Usability test of a MP3 Player

Usability test of a MP3 Player : Results

Below are the results and outcome of statistical analyses on data collected in a usability test of a MP3-player. The experiment ran from Friday 26 January 2007 to Monday 5 February 2007. The introduction text, task instruction, the MP3 player and music file are still available online. A SPSS file (version 13.0) with the data can also be downloaded. The data and the results are presented as follows:

Table 1: Order in which MP3-player and its components were evaluated in the questionnaire
Table 2: Background information participants
Table 3: MP3-player Evaluation
Table 4: Evaluation Play Control
Table 5: Evaluation File Control
Table 6: Evaluation Volume Control
Table 7: Evaluation Information Control
General statistics Participants
Reliability usability scales
Hypothesis tests of cut off point usable versus not usable components
Consistency between ratings and comments

Participants were automatically and randomly assigned to one out of eight versions of evaluation questionnaire. These eight versions were needed to control for order effect of the questions. Table 1 shows the order of the questions of these questions.

Table 1: Order in which MP3-player and its components were evaluated in the questionnaire

Version	MP3-Player	Play Control	File Control	Volume Control	Information Control
0	1	3	4	2	5
1	1	4	3	5	2
2	1	4	5	3	2
3	1	5	4	2	3
4	5	4	1	3	2
5	5	1	4	2	3
6	5	1	2	4	3
7	5	2	1	3	4
Total	24	24	24	24	24

Table 2: Background information participants

ID	Questionnaire version	Gender	Age	Brunel Staff	Brunel Undergraduate student	Brunel Master student	Brunel PhD student	Other
1	5	male	21	no	yes	no	no	no
2	0	male	18	no	yes	no	no	no
3	5	male	22	yes	yes	no	no	no
4	0	female	22	no	yes	no	no	no
5	5	male		no	no	no	yes	no
6	1	male	36	yes	no	no	no	no
7	2	male	25	yes	no	no	no	no
8	0	female	34	yes	no	no	no	no
9	5	female	24	no	no	no	yes	no
10	2	male	59	yes	no	no	no	no
11	6	female	41	yes	no	no	no	no
12	7	male	62	yes	no	no	no	no
13	7	female	22	no	yes	no	no	no
14	4	female	27	yes	no	no	no	no
15	7	female	26	no	no	no	yes	no
16	0	male	32	no	no	no	yes	no
17	3	female	30	no	no	no	yes	no
18	2	male	33	yes	no	no	no	no
19	3	male	35	yes	no	no	no	no
20	5	male	42	yes	no	no	no	no
21	4	male	30	yes	no	no	no	no
22	4	female	33	yes	no	no	no	no
23	0	male	31	yes	no	no	no	no
24	2	female	24	yes	no	no	no	no
25	2	female	39	yes	no	no	no	no
26	3	female	24	no	yes	no	no	no
27	2	male	24	no	no	no	yes	no
28	0	male	29	yes	no	no	no	no
29	1	male		no	no	no	yes	no
30	5	male	32	no	no	no	yes	no
31	1	female	25	yes	no	no	no	no
32	0	male	41	yes	no	no	no	no
33	7	female	34	yes	no	no	no	no
34	2	male	29	no	no	yes	no	no

Table 3: MP3-player Evaluation

ID	Item 1	Item 2	Item 3	Item 4	Item 5	Item 6	Average	Level*	Rationale	Rater 1**	Rater 2**	Rater Total**
1	6	6	6	3	6	3	5	high	The volume control is the worst part. The information form is useless as it's now.	1	1	1
2	6	3	3	6	7	3	4.67	medium	Buttons and interaction are ok, although it is not clear how one would adjust the volume? Any available instructions? Was the bar on the right meant to operate something?	2	2	2
3	5	3	3	6	5	2	4	medium	The volume control issues (mentioned above) brought down the overall rating of this interface. The play controls (although slightly ambiguous) are fine and the application in general performs as expected with all the functions required (although perhaps some additional functions such as loop and random order would have been helpful).	2	2	2
4	1	1	1	1	1	1	1	low	Any navigation is clearly shows on the MP3 Player	3	3	3
5	6	6	7	7	6	7	6.5	high	It has nothing ambiguous.	3	3	3
6	4	3	5	2	4	2	3.33	medium	I could find no volume controller, despite clicking around. I’m a very experienced computer use. I’ve got over 25 years of experience. I have experience of programming. But either I’ve missed something, or the software hasn’t functioned properly.	1	1	1
7	2	1	1	1	1	2	1.33	low	Not sure if you have deliberately tried to develop a hard to use system, when compared with itunes this is terrible, the volume controls are counter intuitive and have no labelling, the feedback for loud seems to be the wrong way around. The fact that you have had to tell the person to hold down shirt for multiple files in an instruction manual is poor. The MP3 player has no metaphoric connection to a normal hi-fi which is disappointing.	1	2	1
8	5	5	6	5	6	5	5.33	high	I found it easy to play but couldn’t find any switches for volume. However I’m sure once indicated it would be easy for me to operate.	2	2	2
9	7	7	5	6	7	7	6.5	high	Apart from the volume control which I couldn’t really see why it had to be so complicated, I believe that it is a very usable mp3 player and if you have a little familiarity with any player before, it doesn’t take more than a couple of minutes to understand how everything works. The lay-out is well designed, too, and one finds everything where he/she expects them to be. Basically, the moment you install the software, you are pretty much able to play your music. To be honest, I would prefer it over windows media player (not over winamp, though).	2	2	2
10	7	7	7	6	6	6	6.5	high	This Player seems easy to use but would be improved by the addition of internationally recognised symbols for Play, Stop Rewind, Fast Forward, Volume up, Volume down etc.	2	2	2
11	5	3	2	3	2	2	2.83	low	Changing the volume was impossible!	1	1	1
12	7	7	7	7	7	7	7	high	I liked this player very much. Seemed to have everything except a working volume control	2	2	2
13	6	5	5	5	6	6	5.5	high	Overall it’s easy to use except the volume control, which is a bit hard to understand.	2	2	2
14	5	3	3	3	5	4	3.83	medium	Simple layout, but not user-friendly. Information control is terribly time-consuming to work with, and volume control unnecessarily complex.	2	2	2
15	5	1	2	1	3	1	2.17	low	The volume controls are the biggest reason for the low scores overall. In general the controls were slow to use and a bit clunky. There was little feedback to know whether I was doing, or selecting the right thing or not.	1	1	1
16	3	2	2	1	3	2	2.17	low	I’m sorry, but this isn’t going to replace Winamp :-) It doesn’t have enough basic functions to be useable on a day-to-day basis, like even displaying the artist and title in the main window. It also has specific failings as discussed below. Visually it’s nice and simple, but it’s also pretty big. In its favour, it does actually seem to play music, which is more than some you download do!	2	2	2
17	6	6	6	5	6	6	5.83	high	it was very easy for me to use the mp3 player from the first time except for the volume control	2	2	2
18	6	6	6	5	6	6	5.83	high		0	0	0
19	2	2	2	4	4	4	3	medium	I could not adjust the volume, and didn’t know where the volume change setting was. The MP3 Information screen was confusing, and inconsistent	1	1	1
20	2	2	2	2	2	2	2	low	I do not persevere with software that I find difficult to use. I do not see it as a challenge. If I am struggling, I usually get my ten year old to explain it to me.	1	1	1
21	5	5	5	5	4	5	4.83	medium	It is in general easy to use, however with a few functions not working properly.	2	2	2
22	4	4	3	2	4	2	3.17	medium	It was awful, looks wise and trying to operate it.	1	1	1
23	5	4	5	2	5	3	4	medium	I could not get the volume control to work, and it was not clear at all as to how I should do this. Finding out information about the artist involved different methods for different pieces of information and some did not work.	1	1	1
24	6	5	2	.	2	3	3.6	medium	Could not use volume control (not sure exactly where this is), and was not easy to do last question; entering information in the ‘MP3 Information’ as would not allow me to change/enter things in.	1	1	1
25	5	3	5	4	5	5	4.5	medium	It was generally straightforward, but I had a few problems. I couldn’t get the volume control to work (but was able to identify the correct area), and found opening tracks tiresome, but could see a work around (either go through the process once opening all files I would want in a session or make it the default MP3 player)	2	2	2
26	7	2	3	1	5	1	3.17	medium	While opening and playing the files was very easy and straightforward, the volume control did not work and was also not clearly labelled.	2	2	2
27	6	6	3	2	5	2	4	medium	Overall it is visually unappealing, inflexible (e.g. storing playlists), the volume and information controls are very frustrating to use, and all of this would make this player redundant when I can have a leading, usable, free software like iTunes.	1	1	1
28	6	6	5	5	6	6	5.67	high	Quite easy to learn / operate – bad volume control.	2	2	2
29	6	3	5	3	5	5	4.5	medium	1. Learning to operate this player is I think easy as it has few controls to operate on. 2. But I was not able to find a volume control to increase volume or decrease it. 3. Overall my interaction with every part of player was clear and I understood what I am doing. 4. This player could be more flexible by the introduction of icons with text on buttons. 5. I still think it is quite easy to use player with if few enhancements take place.	2	2	2
30	5	6	5	3	6	3	4.67	medium	While the application allows me to play the file, the information interface was not appealing, and the volume control/controls were not responsive. Hence, I was not able to fully control the use pf the application.	2	2	2
31	7	7	7	7	7	7	7	high		0	0	0
32	5	6	3	5	3	5	4.5	medium	The basics work – but pretty much nothing else – at least what I could glean from the UI. But all I want is the basics so I’m not bothered about bells and whistles.	2	2	2
33	5	5	5	5	5	5	5	high	I found the use of the MP3 player easy but there were some aspects of usability concerning volume control and information control which I found not be very user friendly	2	2	2
34	6	5	6	5	5	6	5.5	high		0	0	0

* Average rating converted into 3 levels (low: [1,3); medium: [3,5); high: [5,7])
**Raters used the following categories to code the rationale:

missing or not relevant
Mainly negative comments, and/or improvement advice
Both negative and positive comments, and/or improvement advice
Mainly positive or no negative comments and no improvement advice

Table 4: Evaluation Play Control

ID	Item 1	Item 2	Item 3	Item 4	Item 5	Item 6	Average	Level*	Rationale	Rater 1**	Rater 2**	Rater Total**
1	7	5	7	5	7	7	6.33	high	It need a seek bar	1	1	1
2	7	7	7	7	7	7	7	high	Clear and easy to understand. What you get is what you see.	3	3	3
3	6	5	5	6	7	6	5.83	high	With the exception of the ambiguity between “next” and “fast forward” (and also with previous and fast backwards) the play control was easy to use. The next and previous buttons could be better labelled as “Next Track” and “Previous Track” to avoid ambiguity.	2	2	2
4	1	1	1	1	1	1	1	low	It is shows clearly.	3	3	3
5	7	7	7	7	7	7	7	high	It has the control area well designed and easy to understand by using self explained lables	3	3	3
6	7	7	7	7	7	7	7	high	Again, it depends on what I want to do. Within the limits of its own functionality. It was straightforward. Why have a separate pause and play button?	2	2	2
7	6	6	6	6	6	6	6	high	Follows standard conventions such as double click to start	3	3	3
8	7	7	7	7	7	7	7	high		0	0	0
9	7	7	7	7	7	7	7	high	It had the most prominent position on the interface, plus it was the larger button. I think that this goes along with what people expect the play button to be like in a player. So, they don’t really have to think when using it.	3	3	3
10	7	7	7	7	7	7	7	high	The button is large & easy to use.	3	3	3
11	7	7	7	6	7	7	6.83	high		0	0	0
12	7	7	7	7	7	7	7	high	Seemed very straight forward and it worked well	3	3	3
13	6	6	6	6	6	6	6	high	Overall it’s ok but the Fast Forward and Fast Backwards are a bit confusing. Generally they are not displayed as buttons and that’s a bit strange.	2	2	2
14	6	5	6	3	6	3	4.83	medium	Too many buttons. The expectation again, from using other digital and analogue sound applications, is that the “play” button should be combined with the “pause” button. Icons rather than text on the buttons would also be more user-friendly.	1	1	1
15	5	2	5	1	5	2	3.33	medium	The controls were very basic, there was no manipulation of the music file related to a time bar, which proves very useful with other similar applications.	1	2	1
16	6	4	5	4	3	4	4.33	medium	Okay, the basic functions work fine, but fast forward and fast backwards don’t work well at all, IMHO. Normally you assume that holding a FF/RW button down would skip through the track. But these seemed to leap by 10 seconds (which is OK), but the delay was considerable, so it was easy to think that it hadn’t worked and do it too many times.	2	2	2
17	7	7	7	7	7	7	7	high	it was easy to jamp from file to another	3	3	3
18	7	7	7	.	7	7	7	high	I had no difficulties with play control	3	3	3
19	3	3	3	3	3	3	3	medium	Easy enough to use, and quite intuitive, but I am used to the icons usually used to represent these functions, and having to read words on the interface buttons is a slower process.	2	2	2
20	7	7	7	7	7	7	7	high	Basic functions for playing a CD.	3	3	3
21	5	5	5	5	5	5	5	high	In general it works. But when press the next and previous buttons the blue bar on the screen doesn’t seem to move with the selected music.	2	2	2
22	4	4	4	3	4	3	3.67	medium	Nothing special about this control but needs to be more interactive with the files you are playing and presented in the window to see what you are doing it doesn’t look user friendly.	2	2	2
23	6	6	6	6	6	6	6	high	It seemed intuitive and I got it working no problem	3	3	3
24	7	7	7	7	7	7	7	high	Easy to understand and use.	3	3	3
25	7	7	7	7	7	7	7	high	It was clear where to click and it worked	3	3	3
26	7	7	7	7	7	7	7	high	Clearly labelled and very easy to understand and use.	3	3	3
27	7	7	7	4	7	6	6.33	high	It is very self explanatory and conforms to norms found on many physical and software media players	3	3	3
28	7	7	6	7	6	6	6.5	high		0	0	0
29	6	6	6	6	6	6	6	high	The buttons are clear to understand. However the support of icons with text could have earned full points.	2	2	2
30	7	7	7	7	7	7	7	high	Play control is easy to locate and easy to use	3	3	3
31	7	7	7	7	7	7	7	high		0	0	0
32	6	6	7	7	7	7	6.67	high	Easy to use. Liked it a lot.	3	3	3
33	7	7	7	7	7	7	7	high	The play control is straightforward and similar to what I have been used to.	3	3	3
34	6	6	5	6	6	5	5.67	high	I’m more used to symbols for play controls	2	2	2

* Average rating converted into 3 levels (low: [1,3); medium: [3,5); high: [5,7])
**Raters used the following categories to code the rationale:

missing or not relevant
Mainly negative comments, and/or improvement advice
Both negative and positive comments, and/or improvement advice
Mainly positive or no negative comments and no improvement advice

Table 5: Evaluation File Control

ID	Item 1	Item 2	Item 3	Item 4	Item 5	Item 6	Average	Level*	Rationale	Rater 1**	Rater 2**	Rater Total**
1	7	7	7	6	7	6	6.67	high	It needs shortcuts to places on my computer	1	1	1
2	7	7	7	7	7	7	7	high	Nothing new about this, using the file control is the same for other software applications which use the same method	3	3	3
3	7	7	7	7	7	7	7	high	I am used to the standard Windows open dialogue therefore found this control easy to use as it conformed to Windows standards.	3	3	3
4	1	1	1	3	1	3	1.67	low	It have to using open file button, but like other music player, we just select and drag into the play list, then It is working.	3	2	0
5	7	7	7	7	7	7	7	high	It is just the conventional open dialogue used in windows, where most pc users are likely to be expert in this control window	3	3	3
6	7	7	6	4	5	5	5.67	high	Again, there’s some problems with questionnaire design here. All of these questions need a point of reference. For example, relative to iTunes or WinAmp, this is not a flexible application. It depends on what I want the application to do. For example, if my expectations were to create a flexible playlist according to various criteria, then I would answer accordingly.	1	1	1
7	6	6	6	6	6	6	6	high	It is good to see that the tool is using existing systems that are likely to be familiar to the user, although the method is not elegant it is at least familiar. It would be nice to seeadded functionality such as drag and drop	2	2	2
8	5	5	6	6	6	6	5.67	high	I didn’t like having to open two different places to play the music but I suppose this would become easier.	2	2	2
9	7	7	7	7	7	7	7	high	The file control was particularly easy to use. I was able to use it intuitively.	3	3	3
10	7	7	7	7	7	7	7	high	It looks to be a familiar Windows style file interface.	3	3	3
11	7	7	7	7	7	7	7	high		0	0	0
12	4	4	4	3	3	3	3.5	medium	I think you need to have drag and drop enabled to get this to work well. I tried to do this and it didn’t work for me	1	1	1
13	7	7	7	7	7	7	7	high	I have used it before in other applications.	3	3	3
14	7	7	7	6	7	7	6.83	high	Seems a simple enough system, analogous to other file controls in Windows	3	3	3
15	7	7	7	7	7	7	7	high	The file control operates in a similar way to other windows based applications that I am used to. The ability to select multiple files is also very good (some applications don’t allow this)	3	3	3
16	6	6	6	6	6	6	6	high	This was fine, if basic. I know it’s not meant to do this, but drag and drop from Windows Explorer is always nice… And no control of the order of files once in the player. The dialogue did remember where in the directory tree you’d left it, which was a plus.	2	2	2
17	7	7	7	7	7	7	7	high	It was is easy to choose the file/files that I want to play	3	3	3
18	7	7	7	7	7	7	7	high		0	0	0
19	6	6	6	6	6	6	6	high	My experience of this is tainted by my existing use of MS Windows Applications. I would prefer a drag and drop approach, but it is easy to use the method used because it is familiar from other applications. Double clicking on a file in the explorer would also be intuitive, but .mp3 is attached to another app in my OS.	2	2	2
20	3	2	3	2	2	2	2.33	low	MPV players I use automate storing the files.	1	1	1
21	7	7	7	7	7	7	7	high	It is a standard open file box	3	3	3
22	6	6	6	5	2	2	4.5	medium	IT’S THE SAME TYPE OF FILE CONTROL ON OTHER APPLICATIONS	3	3	3
23	6	6	6	4	6	6	5.67	high	Again it seemed intuitive and I managed to use it for all the tasks without a problem.	3	3	3
24	7	7	7	7	7	7	7	high	Use to use, but need to be careful not to right click on mouse, as could open other music player automatically.	2	2	2
25	2	2	3	2	2	2	2.17	low	I found it fiddly – I had to follow the file path from the file button each time I wanted to open a file. I had left the music window open but trying to access the files directly from this window led to the default MP3 player opening them. I guess this would be fixed by making the test application the default association.	1	1	1
26	7	7	7	7	7	7	7	high	Out of all the features, it was the most straightforward.	3	3	3
27	7	7	7	7	7	7	7	high	The file control uses windows standards which I have already invested a great deal of time learning and being familiar with, so using it is intuitive. However, a problem I could foresee in the long term would be organising the music and saving it so you don’t have to open the files every time you use the player – again comparing this with the iTunes experience.	2	2	2
28	6	6	7	6	7	6	6.33	high		0	0	0
29	6	3	6	4	6	5	5	high	Learning to operate these kinds of file controls is easy for me as I am frequent user of windows kind of dialogs. However this file control does not provide me what I want. I want to open the files from desktop and I can not go to the desktop directly by clicking some desktop icon. However my interaction with file control was clear. I know what to do to open a file from a specified location. However this file control was not very flexible specially when accessing files from some commonly used locations. I can be a skilful user for this file control. But easy to use is still questionable as direct access to some locations is my prioroity.	2	2	2
30	7	7	7	7	7	7	7	high		0	0	0
31	7	7	7	7	7	7	7	high		0	0	0
32	2	2	2	1	2	1	1.67	low	Hated this about the player – users shouldn’t have to apply such rudimentary techniques to get the list in. A bit of a shocker. Worst feature of the player by far.	1	1	1
33	7	7	7	7	7	7	7	high	File control is a very common feature of Windows so I am very familiar and confident in using it	3	3	3
34	6	6	6	6	6	6	6	high		0	0	0

* Average rating converted into 3 levels (low: [1,3); medium: [3,5); high: [5,7])
**Raters used the following categories to code the rationale:

missing or not relevant
Mainly negative comments, and/or improvement advice
Both negative and positive comments, and/or improvement advice
Mainly positive or no negative comments and no improvement advice

Table 6: Evaluation Volume Control

ID	Item 1	Item 2	Item 3	Item 4	Item 5	Item 6	Average	Level*	Rationale	Rater 1**	Rater 2**	Rater Total**
1	4	3	7	2	6	2	4	medium	You need to press the big bar each time, and it doesn't matter how many times you press the Up/Down buttons. The Up/Down buttons are not in the expected positions.	1	1	1
2	1	2	2	2	6	5	3	medium	Nothing new about this, using the file control is the same for other software applications which use the same method	0	3	0
3	2	1	1	1	3	1	1.5	low	It took me a good minute to work out how to use the volume control. The need to click the correct button and then confirm with another button meant that what is normally a simple task took a lot longer. I also tried to drag the volume control as it looks like a scrollbar. There were also no labels on the volume buttons and in my opinion the volume bar was upside down (the fuller it was the quieter the volume was).	1	1	1
4	1	1	1	6	6	6	3.5	medium	Description is clearly. But when I use the Volume Controller, I need to press two buttons, then I can doing turn down or turn up.	2	2	2
5	7	7	7	7	7	7	7	high	It is just very good	3	3	3
6	1	1	1	1	1	1	1	low	I couldn’t get the volume to adjust. I tried left-clicks, right-clicks, dragging, scroll-wheeling, clicking in and outside the box, clicking on the diver marker. Nothing worked.	1	1	1
7	1	1	1	1	1	1	1	low	As a designer and human factors expert I found this particularly frustrating that I had to do 2 clicks to increase or decrease the sound, I would like a quicker way of moving it right up or down.	1	1	1
8	4	3	3	3	4	3	3.33	medium	I couldn’t get the volume control to work but with nstructions it ought to be easy.	2	2	2
9	7	2	2	1	7	2	3.5	medium	It took me more that I expected to realise how to the volume control works. First, I couldn’t see any indication that these buttons were the ones that control the volume. Second, it usually takes just one button to increase/decrease the volume in other players that I have used. Third, in order to get to the maximum or minimum you need lots of clicks and that’s a bit impractical, in cases you need to turn off/on the volume quickly.	1	1	1
10	1	1	1	1	1	1	1	low	I was not able to adjust the volume in either direction - I assume that the vertical bar to the right of the screen is the volume control.	1	1	1
11	2	1	1	2	2	1	1.5	low	I had difficulty working out what was the volume control as it wasn’t labelled. I noticed that the bar on the right seemed to move the orange bar up and down but it made no difference to the volume and so I wasn’t sure if changing the volume was a trick task as it didn’t seem to be possible.	1	1	1
12	1	1	1	1	1	1	1	low	I have downloaded this player onto three computers so far and not been able to operate the volume control at all. It looks like it should be easy to use if it worked.	1	1	1
13	2	2	2	2	2	2	2	low	It didn’t work for me at all.	1	1	1
14	2	5	1	1	2	1	2	low	Lack of labels makes it difficult to work out where the volume control actually is. Having to press on two buttons, rather than one, to get the control to work is not user-friendly. Finally, the volume control behaves unlike I expect volume controls to behave from using other sound applications: the expectation is that the control at the top of the column will increase the volume, and the control at the bottom will decrease the volume.	1	1	1
15	1	1	1	1	1	1	1	low	It was hard to work out how to move the volume bar (although the volume indication bar seemed obvious enough). The direction for the loudest and quietest also seemed the wrong way round. Changing the volume was very slow and cumbersome.	2	2	2
16	1	1	1	1	1	1	1	low	Volume control basically didn’t work on my machine (Win XP SP2). Clicking the buttons at top and bottom of what I assume was the volume scale (blue/orange thing) didn’t change anything. Clicking the tall button on the right occasionally moved the orange bar up or down a bit, but very intermittently. So I can’t really evaluate this function.	1	1	1
17	2	2	3	3	3	2	2.5	low	I could not use the volume control from the first time and I could not recognise it first	1	1	1
18	5	5	5	5	6	4	5	high	The volume control appears to require a 2 button pathway to either increasing or decreasing. It would be better if it used a one click pathway or a click-and-drag.	1	1	1
19	1	1	1	1	1	1	1	low	Havingn been given this extra information, I tried to make the volume control work. It is very counter intuitive requiring two clicks to achieve any change in volume, and the sense of volume up/down is reversed in the interface. Down is up! It is not possible to quickly make volume changes.	1	1	1
20	1	1	1	1	1	1	1	low	I couldn’t get it to work.	1	1	1
21	2	2	2	2	2	2	2	low	Can’t get the volume control to work.	1	1	1
22	4	1	1	1	4	1	2	low	It wasn’t obvious that it was even a volume control bar (although previous experience tells you it is) and it didn’t seem to adjust the volume in any discernable way.	1	1	1
23	2	1	1	1	4	1	1.67	low	I couldn’t work it at all – though I think I know how it SHOULD work	1	1	1
24	1	1	5	5	6	6	4	medium	I thought it was this but did not work for me. It would be simply to use if it did work.	2	2	2
25	1	1	1	1	1	1	1	low	I have tried left click, right click, multiple clicks on each button, but nothing seems to happen	1	1	1
26	1	1	1	1	1	1	1	low	As I said previously, it was not clearly labelled and it did not work when I attempted to use it.	1	1	1
27	5	3	2	1	3	1	2.5	low	The buttons are unlabelled, making the initial learning f the control frustrating. Altering the volume takes 2 mouse clicks for each step of volume which seems unnecessary as most media players allow you to slide a bar and change the volume many steps with . one click. Also, the movement of volume seems counterintuitive as to increase the volume you must click the bottom button and the orange area which appears most prominent moves down – this implies something is decreasing to me. I would not use this volume control in real use, I would prefer to use the physical volume wheel on my laptop.	1	1	1
28	3	3	3	1	2	2	2.33	low	Inverse display(loud at bottom), and it is annoying to keep pressing the right button	1	1	1
29	2	2	2	1	5	1	2.17	low	This is the only control I was not able to understand how to operate it. I clicked vaguely here and there as there were no clear instructions on it. However if this control work on some principal I think I can get skilful in using that.	2	2	2
30	1	1	1	1	1	1	1	low	No activity on the volume control/controls. There appear to be three controls and none appear to function as expected. The particular function of each control button is not know as there was no clear markings or label. Even then, this could not be determined by trial and error because the controls don’t seem to work for me.	1	1	1
31	1	1	1	1	1	1	1	low		0	0	0
32	1	1	1	1	1	1	1	low	What volume control? No indication of where it was given and despite pushing every button on the entire panel I Couldn’t get any change in volume! The circled panel above does nothing in my player.	1	1	1
33	3	3	3	3	3	3	3	medium	I found the volume control difficult to manipulate, additionally its function was not obvious so I had to experiment with it.	1	1	1
34	1	1	1	1	1	1	1	low	Volume control didn’t work	1	1	1

* Average rating converted into 3 levels (low: [1,3); medium: [3,5); high: [5,7])
**Raters used the following categories to code the rationale:

missing or not relevant
Mainly negative comments, and/or improvement advice
Both negative and positive comments, and/or improvement advice
Mainly positive or no negative comments and no improvement advice

Table 7: Evaluation Information Control

ID	Item 1	Item 2	Item 3	Item 4	Item 5	Item 6	Average	Level*	Rational	Rater 1**	Rater 2**	Rater Total**
1	5	5	6	3	6	2	4.5	medium	I would prefer to choose the song just once. The selection of the song is different for each piece of information	1	1	1
2	7	7	7	7	6	5	6.5	high	The year option did not seem to work for me when I selected the mp3 title. It would be nice if I could select the mp3 title and all of the relevant information appeared without having to select the mp3 title each time.	1	1	1
3	2	1	2	3	4	2	2.33	low	I would have preferred to open the track and receive all the information in one action, rather than having to lookup each attribute in a separate box.	1	1	1
4	1	7	1	1	4	1	2.5	low	When I using it, there no Year is shown and the Album is wrong.	1	1	1
5	6	6	6	6	6	6	6	high	It has been separated logically and made it easy for the used to understand each part functions	3	3	3
6	7	5	6	5	5	7	5.83	high	I think there’s some problems with the way you’ve worded this questionnaire: is the questionnaire a commentary on my abilities or on the application’s UI?	0	0	0
7	1	1	1	1	1	1	1	low	Again nothing compared to the itunes version	1	1	1
8	4	4	4	4	4	4	4	medium	Some information was not visible, some was. But I wouldn’t really eb interested in this information anyway.	2	2	2
9	6	3	2	2	7	2	3.67	medium	First, for some reason I couldn’t get it to display the year even though I tried a couple of times. Second, I think you cannot view the album info if the song which is in that album is playing at the time. Third, the use of tick boxes, scroll bars and buttons in one window that basically has one functionality (displaying song info) is quite confusing- one suggestion is the use of one method (e.g. double clicking). Fourth, in the “track” part, you can tick more than one boxes (tracks) and it still gives you a number. Last, in my opinion in would be great if, when a song was playing and you clicked on the info button, it displayed all info automatically without requiring you to set up everything yourself.	1	1	1
10	1	1	1	1	1	1	1	low	I could not gain any useful information from this screen. It only showed the title of the track in each of the blue boxes.	1	1	1
11	7	7	7	7	7	7	7	high		0	0	0
12	7	7	7	7	7	7	7	high	Seemed to be excellent	3	3	3
13	6	6	6	6	6	6	6	high	Most of the fields seem to work fine except the Year field, which did not display anything when I selected the song name.	2	2	2
14	6	1	1	1	4	1	2.33	low	I prefer to have all the file information available in one place, rather than having to perform individual tasks to get each bit of information	1	1	1
15	1	1	2	1	2	1	1.33	low	I could not find all the information I wanted. I was not sure whether I was seeing the information about the correct track as there seemed no way of telling which track was highlighted for some of the options. The track had to be selected for each of the options which made things harder.	1	1	1
16	2	2	2	2	2	2	2	low	I didn’t find that this worked well at all. The more logical behaviour would be to have the all information related to the playing track (or maybe the one selected) appear at once. But here you have to click the track multiple times to get all the info. And only Genre seems to work for me – clicking files in the Year combo doesn’t work. The “Track” section is very badly designed – I assume you mean track number. Why can you select multiple tracks? Why do you have to click OK after selecting them to get the track number? And surely these are the track titles appearing under Artist Name?	1	1	1
17	6	6	6	6	7	7	6.33	high	It was very easy to me to use the information control and it was very clear and understandable	3	3	3
18	5	5	5	5	5	5	5	high	Many existing mp3 players do this automatically without the need to click and search. It seems an inefficient method by comparison with market leaders.	1	1	1
19	2	4	3	4	3	3	3.17	medium	Different modes of input were required to retrieve different pieces of information. Too much input was required to receive the information.	1	1	1
20	2	2	2	2	2	2	2	low	I’m a Luddite.	1	1	1
21	4	4	4	4	4	4	4	medium	The interface looks a biting confusing. If I click on most of the buttons there is no response. A instruction is needed for me to understand them	1	1	1
22	6	5	4	4	4	4	4.5	medium	I didn’t seem able to pull up the information the instructions said I could pull up.	1	1	1
23	5	5	5	4	5	4	4.67	medium	There were too many different methods to get the information. I failed to retrieve one piece of info as it didn’t seem to work	1	1	1
24	6	1	5	1	5	4	3.67	medium	I did not have a good time on this, it was going well then I found I could not change things that needed to be changed; entering year, title and album	1	2	2
25	5	5	5	5	5	5	5	high	Everything worked except for the year. I guess I’d like all information to come up in one click or a display button for each component.	2	2	2
26	6	3	3	4	5	5	4.33	medium	It becomes a bit tedious if you have to keep clicking search for it to search for the artist name or track number	1	1	1
27	3	1	1	1	1	1	1.33	low	I am comparing this to the information control in iTunes which is easier to use as all of the track info appears at once, whereas in the above player you must select each individual piece of info – sometimes this can lead to information from multiple tracks appearing in the same window which is very confusing. The Year window had no way I could find of confirming the choice and displaying the year. Very frustrating to use.	1	1	1
28	5	3	2	3	3	3	3.17	medium	Not able to get year. Find album for a track I am not listening to.	1	1	1
29	6	5	6	6	6	5	5.67	high	Over this control was easy to find as I don’t have to find some thing to do. All the information about music files is in front of me.	3	3	3
30	6	6	6	6	6	6	6	high		0	0	0
31	1	1	1	1	1	1	1	low		0	0	0
32	6	6	6	7	6	7	6.33	high	Liked the IC, but its buggy and didn’t give proper info…..so someone didn’t the coding properly. Sorry	2	2	2
33	3	2	2	2	3	2	2.33	low	The information control was not straightforward and not very easy to use.	1	1	1
34	3	3	3	3	3	3	3	medium		0	0	0

* Average rating converted into 3 levels (low: [1,3); medium: [3,5); high: [5,7])
**Raters used the following categories to code the rationale:

missing or not relevant
Mainly negative comments, and/or improvement advice
Both negative and positive comments, and/or improvement advice
Mainly positive or no negative comments and no improvement advice

General Statistics Participants

age
gender
position
- Brunel staff
- Brunel Undergraduate student
- Brunel Master student
- Brunel PhD student
- Other
Questionnaire version

Age

	N	Minimum	Maximum	Mean	Std. Deviation
Age	32	18	62	31.44	9.876
Valid N (listwise)	32

Gender

		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	female	14	41.2	41.2	41.2
	male	20	58.8	58.8	100.0
	Total	34	100.0	100.0

Position

Brunel staff

		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	yes	20	58.8	58.8	58.8
	no	14	41.2	41.2	100.0
	Total	34	100.0	100.0

Brunel Undergraduate student

		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	yes	6	17.6	17.6	17.6
	no	28	82.4	82.4	100.0
	Total	34	100.0	100.0

Brunel Master student

		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	yes	1	2.9	2.9	2.9
	no	33	97.1	97.1	100.0
	Total	34	100.0	100.0

Brunel PhD students

		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	yes	8	23.5	23.5	23.5
	no	26	76.5	76.5	100.0
	Total	34	100.0	100.0

Other

		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	no	34	100.0	100.0	100.0

Questionnaire version

		Frequency	Percent	Valid Percent	Cumulative Percent
Valid	0	7	20.6	20.6	20.6
	1	3	8.8	8.8	29.4
	2	7	20.6	20.6	50.0
	3	3	8.8	8.8	58.8
	4	3	8.8	8.8	67.6
	5	6	17.6	17.6	85.3
	6	1	2.9	2.9	88.2
	7	4	11.8	11.8	100.0
	Total	34	100.0	100.0

Reliability Usability scales

	Cronbach's alpha
MP3-Player (items 1-6)	0.94
Play Control (items 1-6)	0.98
File Control (items 1-6)	0.98
Volume Control (items 1-6)	0.90
Information Control (items 1-6)	0.96

To simplify the analysis and because of the high Cronbach's Alpha the following analysis was done with the average rating on each item of MP3-player or the components (i.e. Play, File, Volume or Information Control).

Usability rating - Hypothesis testing Component rating

	N	Mean	Std. Deviation	Std. Error Mean
Average score MP3 Player	34	4.3657	1.59098	.27285
Average score Play Control	34	6.0392	1.45010	.24869
Average score File Control	34	5.9020	1.68281	.28860
Average score Volume Control	34	2.1324	1.40273	.24057
Average score Information Control	34	3.9559	1.89644	.32524

One-Sample t-Test

Cut off point of 5.29 is based on previous research where 5.29 was the breakeven point of the probability that a rating would come from distribution of a set difficult to use component or from a and ease to use components.

	Test Value = 5.29
	t	df	Sig. (2-tailed)	Mean Difference	95% Confidence Interval of the Difference
	t	df	Sig. (2-tailed)	Mean Difference	Lower	Upper
Average score Play Control	3.013	33	.005	.74922	.2433	1.2552
Average score File Control	2.120	33	.042	.61196	.0248	1.1991
Average score Volume Control	-13.126	33	.000	-3.15765	-3.6471	-2.6682
Average score Information Control	-4.102	33	.000	-1.33412	-1.9958	-.6724

Consistency between ratings and comments

As part of analysing the consistency between the participants' average rating and their comments, the comments were first categorised by two Raters. They used the following categories:

missing or not relevant
Mainly negative comments, and/or improvement advice
Both negative and positive comments, and/or improvement advice
Mainly positive or no negative comments and no improvement advice

The results of this can be found in Table 2, Table 3, Table 4, Table 5, Table 6, and Table 7. Table 8 shows the agreement between the raters expressed by Cohen's Kappa, also referred to as Index of Inter-rater Reliability.

Table 8: Measure of Agreement between Rater 1 and Rater 2 when categorising the comments

Object of evaluation	Cohen's Kappa
MP3-Player Comments	0.95
Play Control Comments	0.95
File Control Comments	0.96
Volume Control Comments	0.93
Information Control Comments	0.95

The results of the two ratings were combined in Rater Total. In the few cases where the to raters disagreed, they discussed the comments and agreed on classification. The next step was to compare the categorisations of the comments with the average rating. To this the average rating was first group in following interval classes :

Low -> average rating between 1-<3,
Medium -> average rating between 3-<5
High -> average rating between 5-7

The following step was to examine the consistency between the comments as classified by the two raters and the average rating when group in the 3 interval classes. As can be seen in the cross tables (Table 9, Table 10, Table 11, Table 12 and Table 13) there were only a nine cases where a participants rated usability high while the comments were mainly negative or and improvement advice, and also a few cases where participants rated the usability low while the comments were mainly positive or none negative comments and no improvement advice. The five participants responsible for these apparent inconsistency were contacted and their reply can be found in Table 14.

Table 9: MP3-player - Evaluation rating versus comments

Table 10: Play Control - Evaluation rating versus comments

Table 11: File Control - Evaluation rating versus comments

Table 12: Volume Control - Evaluation rating versus comments

Table 13: Information Control - Evaluation rating versus comments

Table 14: Additional information on apparent inconsistency between rating and comments

ID	Question	Reply
1	I am in the process of analysing the data of the MP3 usability test and would like to ask you for some further explanation about your rating, specifically for the rating of the MP3 Player, the Play Control and the File Control. On average you have given high rating whereas your comments are mainly negative or suggest improvement to be made. For example you wrote: MP3-Player: “The volume control is the worst part. The information form is useless as it's now.” Play Control: “It need a seek bar” File Control: “It needs shortcuts to places on my computer” Although I am not suggesting that this is wrong, I would be grateful if you can give me some additional information about your rating strategy in these cases. For your convenient, I have also included the questionnaire that you had email to me. I am looking forward receiving your reply, and like to thank you again with helping me with this study. Best wishes,	Maybe I've misunderstood the meaning of the statements. My native language is not English. I would try to explain my "rating strategy". For instance, regarding the Play Control I said "It needs a seek bar" because I really think that it's necessary to facilitate the playing. However that doesn't implies that the current Play Control is difficult to operate. Even thought I think that the Play Control is not flexible enough, I may think that I can get to use it in a pretty good way. With the other parts is the same. Most of them have an "unexpected" behaviour or lack of functionality, but it doesn't implies that I cannot master their use or make them do what I want. As regards to "My interaction with the Play Control would be clear and understandable.", I must say that I was not sure of the meaning of that statement. To sum up. The player does almost all what it should do, but I don't like too much the way it works, though I could get used to it. I hope this helps. :) Kind regards.
2	I am in the process of analysing the data of the MP3 usability test and would like to ask you for some further explanation about your rating, specifically for the rating of Information Control. On average you have given high rating for this control whereas your comments are mainly negative or suggest improvement to be made. You wrote: “The year option did not seem to work for me when I selected the mp3 title. It would be nice if I could select the mp3 title and all of the relevant information appeared without having to select the mp3 title each time.” Although I am not suggesting that this is wrong, I would be grateful if you can give me some additional information about your rating strategy for this control. For your convenient, I have also included the questionnaire that you had email to me. I am looking forward receiving your reply, and like to thank you again with helping me with this study. Best wishes,	Looking back at the specific question, "I would find it easy to get the Information Control to do what I want it to do." i marked "extreamly". As an overall this is correct. I did not want to mark it down on the one problem which i had, and insted noted this down in the comments about the year option not working for me. Aside from this, the rest of the options which i selected were correct; learning to operate was easy for me, interaction was clear and understandable, it was extreamely likely that the info control was flexible to work with. To become skillful and easy to use, i found to be of simular questions. Overall, results where highly rated because usage/learning/interaction where all extreamly true. There were no major problems i identified just an improvement. Although i can see that "I would find it easy to get the Information Control to do what I want it to do." was over rated. My final point, about the info control; i was confused about what the user would use this for; It seems that for each option you could select different artists and albums, would this be relevant though? maybe the user would just want to select the artist and all the rest of the information would appear automatically. Insted of browsing through each of the options and having to "reselect" artist/album again to obtain information. I "nit-pick" on this only because i have used a wide range of music applications e.g. WMP, Winamp, Foobar, etc and from using these have high expectations. I hope this helps you out. Any more questions i will be happy to answer.
4	I am in the process of analysing the data of the MP3 usability test and would like to ask you for some further explanation about your rating, specifically for the rating of MP3-Player and the Play Control. On average you have given low ratings for the MP3-player and the Play Control whereas your comments are mainly positive. For example, for the MP3-player you wrote: “Any navigation is clearly shows on the MP3 Player”, and for the Play Control you wrote: “It is shows clearly.”. Although I am not suggesting that this is wrong, I would be grateful if you can give me some additional information about your rating strategy . For your convenient, I have also included the questionnaire that you had email me. I am looking forward receiving your reply, and like to thank you again with helping me with this study. Best wishes,
6	I am in the process of analysing the data of the MP3 usability test and would like to ask you for some further explanation about your rating, specifically for the rating of the File Control. On average you have given high ratings for the File Control whereas your comments are mainly negative. For example, for the File Control you wrote: “Again, there’s some problems with questionnaire design here. All of these questions need a point of reference. For example, relative to iTunes or WinAmp, this is not a flexible application. It depends on what I want the application to do. For example, if my expectations were to create a flexible playlist according to various criteria, then I would answer accordingly.” Although I am not suggesting that your rational is wrong, I would be grateful if you can give me some additional information about your rating strategy for this control. For your convenience, I have also included the questionnaire that you had email to me. I am looking forward receiving your reply, and like to thank you again with helping me with this study. Best wishes,
18	I am in the process of analysing the data of the MP3 usability test and would like to ask you for some further explanation about your rating, specifically for the rating of the Volume Control and the Information Control. On average you have given high ratings for these controls whereas your comments are mainly negative. For example, for the Volume Control you wrote: “The Volume Control appears to require a 2 button pathway to either increasing or decreasing. It would be better if it used a one click pathway or a click-and-drag.” Likewise for the Information Control you wrote: “Many existing mp3 players do this automatically without the need to click and search. It seems an inefficient method by comparison with market leaders.” Although I am not suggesting that your rational is wrong, I would be grateful if you can give me some additional information about your rating strategy for these controls. For your convenience, I have also included the questionnaire that you had email to me. I am looking forward receiving your reply, and like to thank you again with helping me with this study. Best wishes,	I felt that the controls were easily used and I had little problem operating them, hence the reasonable scores. However I felt that improvements were possible, hence my comments. The difficulties I stated are the reason that the controls in question were not awarded the “extremely” positive responses. I could easily learn to use them but they could be better. I hope this helps.