Computers, creativity and learning

Equal Bite recipe template

For Equal Bite workshop attendees, here is the blank recipe template :  Download Recipe_template_blank v1.

Headings are:

Title

Author:

Flavour:

Serve to:

Number of servings:

Cost/resource:

Background

 

 

 

 

Ingredients

• A …
• A…
• A…

 

Method

1.  
2.  

 

 

 

 

 

 

Notes on ingredients

 

Cook’s Tips

 

Warnings

 

Other flavours/related recipes

 

References

 

Arguments against motherhood and HCI

[edited after a lunch argument with a friend]

Our workshop on Motherhood and HCI at the CHI 2013 conference seems to have angered some feminists. I wouldn't have predicted this, but then I don't devote much time to studying feminist theory. We didn't get any negative comments from men that I'm aware of. Maybe they were fearfully gathered in the corner of the men's toilets, worrying about being marginalised by powerful women.

[EDIT: I just recalled a criticism of the general area of Motherhood and HCI which emerged from a pre CHI event from a man. This memory was provoked at the weekend by a male doctor friend who was criticising a mum in a cafe for cuddling a very new baby while reading her phone. "Look at the perfect example of mother baby bonding", he said in his doctorly way. "The baby thinks her mum is wrapped up in cuddling her, but she's actually ignoring her". The comment from the male HCI academic was similar: "Isn't it a bit sad that all these new mums are using technology instead of spending time with their babies?" The baby in the cafe WAS ASLEEP! Newborns do that. A lot. They also feed for hours at a time. Mummy is meant to sit there enjoying the platonic ideal of motherhood and do nothing else? Mummy is meant to soley devote her cognitice capacity to watching a mini-person sleep? I realise this argument is presented as a bit of a straw man, but I find it too irritating to spell out that yes, of course, parents should attend to their children but there is a balance to be found between the well being of the mother and the needs of the child.]

The workshop was highly productive in terms of mapping out an emerging reseach area and it was also very helpful to hear about the participants' experiences as mothers, daughters, midwives, obsetricians and designers (and also from the two fathers and one baby who joined us).

For now, I want to focus on the criticisms to get the arguments clearer in my head. There were two main areas of contention which I came across.

1. Criticism of the unintended implication that only women who are mothers can perform a nurturing role. Something in the call for papers for the workshop seems to have given this impression although exactly what eludes me.  To be clear: I believe that people who are not biological mothers can and do nurture others. But being a literal minded computer scientist, I did initially baulk at the statement "You don't have to have a vagina to be a mother".  This is a question of semantics. The OED defines the noun "mother" as "a woman in relation to a child or children to whom she has given birth" (definition 1).  This clean cut definition does leave out adoptive mothers or step mothers, who obviously can have a nurturing role. "Mothering" as a verb is potentially more useful here (definition 2),  defined thus "bring up (a child) with care and affection: the art of mothering.  look after (someone) kindly and protectively, sometimes excessively so: she mothered her husband, insisting he should take cod liver oil in the winter". So from this point of view, dads (or anyone else) can mother people, regardless of their vagina-less state. It's maybe also worth noting that the object of the nurture is a child in the first definition, but is more generally a person in the second definition. So, as mentioned at the workshop, the mothering of students could potentially be included, but the mothering of cats would be out.  Maybe the workshop should have been entitled "Mothering and HCI" to encompass this broader notion.  

2. Criticism of the focus on mothers rather than parents, the (again unintended) implication being that it is only women's work. At the ACM Women's Breakfast, Kia Hook took a swipe at the workshop without really explaining what her problem with it is. But in response to a follow up email from me, she wrote:

"What I started from was the position that a feminist position that singles out women as having other traits and abilities than men is a problematic position. Your workshop of course makes sense when it comes to the biological  aspects of becoming a mother (pregnancy and breastfeeding is, for the most part, a female only experience). But monitoring sleep? Monitoring need for food? If we write this off as only being for women - what have we done to fathers? What have we done to the possibility of sharing maternity/paternity leave? The word for this in Swedish is "särartsfeminism" - in english it might be "specificity feminism"? To me, deeply problematic! CHI has had a range of papers on "mother's work", "soccer moms" etc. In essence telling fathers that this is not their work, their duty, their fun, their life. Problematic if we want to share responsibilities, share the work-life-balance, and move forwards with our equality strive. "

It is not the first time I have had this sort of comment from a person from Sweden, that world leader in gender equality . Here parental rather than maternity leave is enshrined in law: both parents may share the leave although in fact only 25% of it is actually taken by men. In the UK, men are entitled to two weeks paternity leave, and women one year of maternity leave. This may change to a more equal arrangement by 2015 but I am not holding my breath. Being a pragmatic computer scientist, I say we should help the users who are actually performing the tasks here and now. And the evidence base is that more women perform nurturing and domestic work than men. An Australian study found that children add (an alarming) thirty five hours of domestic chores a week to a couple’s household [1]. In childless couples in the UK, 62% of domestic chores are done by women; for couples with a child, this rises to 75% [2]. Furthermore, there is evidence that mothers and fathers have different expectations about fathers’ roles in parenting. A study in the US found that mothers have higher expectations of support from the father in housework and emotional support. Men have lower expectations of themselves in terms of emotional support and baby care. [3].

Focus on blood, milk and tears

I can't remember which workshop participant used this evocative phrase, but "blood, milk and tears" covers the challenging biological aspects of being a mother (definition 1 above) rather nicely. Let's not write off the biological aspects so glibly. In the political correctness of including more people with a wider definition of mothering, or telling fathers that nurturing is their joint responsbility, let's not forget the millions of women who mop up their blood, milk and tears every day. Let's design technology which assists in small ways, by helping mothers monitor the changes in their bodies, or find information or share experiences when they need to. Let's consult with mothers about the design of the technology they use, and invent new consultation methodologies to suit this user group if necessary.

Until we reach the wonderful era of gender equality, I say we should focus considerable design effort on women, on mothers and on mothering. If technology can make some of the more mundane tasks just a little easier, or some of the challenging experiences slightly more bearable then it will have been worth it. 

 

References

 [1] Craig, L., & Bittman, M. (2005). The effects of children on adults' time-use: Analysis of the incremental time costs of children in Australia. Social Policy Research Centre. Retrieved on 8/1/13 from http://education.arts.unsw.edu.au/media/File/DP143.pdf

[2] Schober, P. (2013). Gender equality and outsourcing of domestic work, childbearing, and relationship stability among British couples. Journal of Family Issues.34(1). 25-52.

[3] Fox, G. L., Bruce, C., & CombsâOrme, T. (2000). Parenting Expectations and Concerns of Fathers and Mothers of Newborn Infants. Family relations. 49(2). 123-131

 

Does making computer games in class make kids like computing more?

Wouldn't you think that making games in the class would convert kids to the joys of computing? It doesn't really. In fact, it looks like it might put them off. I wanted to have that in the conclusions of this paper, but a reviewer wouldn't let me. :-)

I'm coming round to the idea that kids are very confused about what computing actually is when they start secondary school in the UK, and that as they get an more accurate idea of which it might involve, then their attitudes drop. Not really surprising, as we can't all be computer scientists.

Anyway, read all about it for yourself in this stonkingly interesting article.

Robertson, J. (2013). The influence of a game making project on male and female learners’ attitudes to computing. Computer Science Education 13(1). Download LearnerAttitudeGameMakingProjectAccepted

Little Miss Geek

[cross posted from my CACM blog]

Little Miss Geek is a book – and a wider campaign– by Belinda Parmar to shift women’s role in technology from consumer to creator. The book is mostly aimed at technology business leaders, offering practical solutions to help them make their workplaces better at attracting and maintaining female staff in technical roles. It argues that the technology industry needs to have more female employees in the interests of equality as well as their own bottom line: more women involved in the creation of new technology products will open up exciting new products tailored for an expanding female market.

Here’s the current sorry technology trajectory for girls in the UK, as summarised in the book. Little Miss Potential Geek starts schools. Due to poor quality IT lessons at primary school, she associates computers with boredom, maths or typing.  Little Miss Potential Geek is persistent, however and wants to study it further. When she joins an A-Level computing class at secondary, she finds only 8% of the pupils are girls. Even though she is in the minority, Little Miss Geek (I think we can safely drop "potential" at this point) decides to study computing at university, and graduates with one of the 18% of technology or science degrees which are awarded to women. Initially not daunted by the fact that only 17% of the technology workforce are women, LMG gets a job.  But  danger lies ahead. For 41% of women leave tech companies after 10 years, compared to 17% of men. Will LMG (like 47% of other Little Miss Geeks) feel that she has to act like a man to get ahead? Will she want to have her own Baby Geeks and become one of the 84% of women in technology who thinks that their employers need to do more to encourage women to return after maternity leave? Or as an ambitious professional, will she simply object to the fact that with the tech industry that only 26% of women reach senior management or board level? Or will Little Miss Geek continue to rise within the profession and eventually become Elderly Dame Geek with a MBE for services to the technology industry? As much as we wish her well, we must admit that probability does not favour poor Little Miss Geek. Parmar’s point is that we have to get past just wringing our hands about gender imbalances in IT and take positive action.

What positive actions might companies take? The book discusses a number of workable proposals such as apprenticeship ,sponsorship, bursaries and mentoring for  women; a supportive mum returner scheme; investment in female start-up companies; female heroes programmes; raising the profile of female employees within a brand; and even running coding workshops to encourage talented women from other parts of the company to consider more technical roles.

I applaud the book. The tech industry is indisputably not a female friendly work environment and this needs urgently addressed.  I admire Belinda Parmar’s energy in mounting a campaign to bring the public’s attention to the issue, in a similar way to Jamie Oliver’s campaign to fix school dinners in the UK.

But – and for heaven’s sake don’t let anyone use this paragraph as an excuse to wriggle out of doing the right thing – I have to dispute one point in the book. I can’t help it. I’m an academic first and a member of the sisterhood of technical women second. The book quotes Baron-Cohen’s findings about women having poorer spatial awareness, then says that spatial visualisation skills are directly linked to the ability to interacting with computer software and interfaces. But it’s Ok, the book consoles us, because such skills can be taught. Women engineering students can be taught how to do mental rotations and such. Such minor deficiencies can be overcome before ladies can manage to do all those hard technical things. I found this argument both unpalatable and bewildering, so I followed up one of the (extremely obscure) references which linked spatial awareness to computing skills. For a start, that article didn’t directly discuss gender differences. But the main point is that it does not discuss the impact of weak spatial awareness skills  on programming, or any other technically creative role. It is all about using technology. (In the early ‘90s. When people still used Hypercard.)  Little Miss Geek is a technology creator, not a technology consumer. So her putative lack of spatial awareness will neither hold her back nor need fixed. Unless she particularly wants to pursue a career in computer graphics.

But let’s ignore this odd digression into spatial ability. Let’s focus on the key message: the time is right to inspire girls to become technology creators. The Little Miss Geek book will tell technology business leaders how to do it.

Should a supermodel judge a youth programming competition?

Help me out here- I can't make up my mind about this. Is a super model and actress an appropriate judge of a youth programming competition?  There was an article in yesterday's Guardian about how the supermodel and actress Lily Cole will be judging the Young Rewired State programming competition this year. There is something about this which makes my skin crawl slightly but I think I am being unfair. What do you think?

Here's some more background. Young Rewired State is a fabulous UK programming competition which brings together programmers (under 18 years old) to work with more experienced software developer mentors to create socially useful applications which harness open data sets. There is a a very cute example of an app from last year created by under 10s in a public library which visualised data on happiness in different English counties as asteroids. YRS seems like a great idea to me, and I must remember to encourage my students to sign up as mentors. [Hey, students! Sign up as mentors!] 

Lily Cole has a double first from Cambridge. She appears to be thoughtful, articulate and humble, and holds laudable views about the gift economy. She is founder of a social website called impossible.com so I can see the connection to the aims of the YRS competition. I have no doubt she could be inspiring to the young programmers. 

So why do I feel uneasy about this?  Let's see... her double first is in history of art, not computer science! She has no technical background. Her celebrity status (which in turn is gained from her appearance) seems to have been a factor in recruiting her as a judge. This worries me in the wake of the ridiculous EU Science:It's a Girl Thing debacle from last week. In this video, developed by the European Commission, gorgeous women in high heels prance around shaking test tubes making male scientists peer at them above their spectacles. The video has been widey derided (apparently it's not judge me who vomits when watching it). An interesting New Scientist article in reponse to the video quotes a study which found that girls are demotivated by seeing (manufactured) successs stories of female scientists with overtly feminine characteristics."Worse still, girls who already disliked science were especially demotivated by the feminine scientists. Sekaquaptewa [the researcher] says this was probably because they saw simultaneous success in both domains as totally unattainable." I totally understand this: who has the time?

I don't like the thought of girls being turned off the programming competition because they feel they can't be both geeky and gorgeous. But then, Lily Cole is gorgeous but not geeky. Will girls think they can bypass the geeky bit as long as they are gorgeous? What message does it send to boys if the high profile women are not programmers? Will they care, or will they just like to shake hands with the pretty lady? Am I just a bitter old hag? But mostly, I wonder: why didn't YRS get a female geek as a judge instead?

“Girls can’t program in their heads”: gender and games in the computing classroom

[cross posted from my CACm bog]

When I was an undergraduate computer science student one of my fellow students enraged me by telling me condescendingly that “girls just can’t program C in their heads”. (This is true, of course. People of both genders usually deploy a text editor and compiler rather than computing in their heads). Twenty years later, the underlying problem of gender in computer science persists. We can’t recruit female computer science students, and the industry is desperately short of women (e-skills UK, 2011). Contrary to my student colleague’s perception, this is not because of women’s lack of ability (Singh et al., 2007). Often it’s lack of confidence in technical abilities, coupled with the paradoxical tendency for women to believe that while women in general have equal ability and opportunity to pursue computing careers, they personally don’t happen to be good at it (Beyer, Rynes, Perrault, Hay & Haller, 2003). What can we do to solve this problem? How can we encourage our best and brightest students, whether male or female, to consider careers in computing? This is a question which I have been considering over several years with particular reference to computer game making in the classroom. I don’t have a simple answer, but I do have some empirical observations to add to the debate.

Computer access at home is more equal now In the olden days, girls’ lack of interest in computing at school could in part be attributed to lack of access to hardware (Margolis and Fisher 2002). Girls got elbowed out of the way by their brothers and the boys in their class so they didn’t get to explore and fall in love with the BBC micro or the Apple II. I suspect this is changing, as laptop prices drop, smart phone ownership increases, and social networking becomes more prevalent. In a recent study of 744 secondary school pupils in Scotland , aged between 11 and 14, we found that there was no significant difference in the number of hours boys and girls spent using the computer at home per day (on average between one and two hours). In terms of activities while using the computer, girls were significantly more likely to do homework and talk with friends while boys were significantly more likely to play games. There was no significant difference in the amount of time spent learning more about a topic of interest.

Boys and girls start off high school liking computers and believe they are good at computing Previous studies have found that there is early socialisation of girls away from technology: “Both boys and girls need to be taught that computers are for them” (Varma, 2010; p 314). Our findings suggest that girls do in fact have positive attitudes to computers, although not quite as positive as boys. In our recent study, there was a significant difference between boys’ and girls’ responses to the statement “I like computing”. Essentially girls were more inclined to agree with the statement on a 5 point Likert scale, whereas boys were more likely to strongly agree. Although previous authors have reported low self confidence in computing skills among female students (Beyer et al., 2003), this was not the case here: only 7% of the girls disagreed or strongly disagreed with the statement “I am good at computing”.

 

Kids don’t know what computer science is These figures seem pretty positive. Both boys and girls start off high school liking the technology, and believing that they are good at the subject. When I dug deeper, though, I discovered that these children actually didn’t know much about what the subject involves. This point has been well made in a recent review by the Royal Society (Furber, 2012): there is a good deal of confusion in the school system about whether children are learning ICT, computing or computer science. This is nicely illustrated in the following excerpt from an interview with two 13 year old girls who were asked if they would use computers in their careers.

“Girl 1: To be an optician or a vet, you have to use the computer quite a lot for that.

Girl 2: programming and stuff

Girl 1 : To be an optician you have to program what it is, know what it is, certain parts. Like what’s wrong , how they can help and stuff.

Interviewer: Have you done any programming yet in school?

Girl 1: I don’t know

Girl 2: We did, We did our own program. “My computer of the future” that was a programming one.

Girl 1: We know that programming is like typing and stuff.”

As a computer science lecturer, I put my head down on my desk and wept when I listened to that recording! Something has gone quite badly wrong if learners are unsure if they have learned how to program, and confuse it with typing. The rest of our interview data paints a picture of learners being taught very basic IT skills such as saving files, and using basic applications such as spreadsheets. There was also evidence of good practice in using software to support creative expression, such as making music, videos or animations. There was a general lack of familiarity with the concept of programming, and certainly no mention of theoretical computer science. My guess is that we are going wrong in the early years of secondary school. Starting from a base of confident, keen IT users coming in from primary schools, there is potential to turn at least some of them into confident, curious, creative and rigorous computer scientists by exploring what lies inside their phones and laptops. In my view, we’re not exploiting this potential at the moment because the curriculum is muddled, dull, out of date and inadequately challenging.

Making games in the class isn’t necessarily the panacea we hoped A lot of hope for improving computer science education has been placed on using environments for making games. This seems a reasonable notion: games are part of children’s everyday lives (97% of young people in the US play computer games Lenhart and Kahne (2008)). If children are motivated to spend a good deal of leisure time playing games, perhaps we can interest them in programming and other CS concepts by introducing game making activities to classrooms. Carefully designed software environments which enable game making such as Scratch and Alice have become popular globally, and are increasingly used in Scottish schools. However, there is still a lack of empirical evidence from studies in schools on their impact on motivation or learning. In our study, nineteen schools used our Adventure Author software to teach ~900 learners how to use their own computer games over a period of 9 weeks on average. We studied the impact the project had on learners’ attitudes to computing, and were disappointed to discover that game making may make pupils less inclined to study computing in the future. Furthermore, girls enjoyed the project significantly less than boys. In an interesting twist, it turns out that girls are actually better at making this sort of games than boys are (Robertson 2011), even although girls are generally less experienced gamers. We need more studies of this sort to discover the longer term effects of games programming projects using various tools.

Conclusions

Encouraging more bright learners of both genders to study computer science is a complex educational problem, and we have to tackle it with a range of approaches ( some of which are outlined in the Royal Society report (Furber, 2012)). I believe that we should be realistic in our expectations about the impact from new hardware or software initiatives such as game authoring environments or the much sought after Raspberry Pi. I suspect that improving teacher training in concert with reforming the curriculum will be more effective. But in the end it doesn’t matter what I think, or what any other expert thinks. We need good research evidence on the impact of the interventions we make. And that requires adequate funding.

[ A version of this article will appear in a forthcoming EPSRC/ESRc TEl report edited by Rose Luckin]

References

  Beyer, S., Rynes, K., Perrault, J., Hay, K., & Haller, S. (2003). Gender differences in computer science students. Proceedings of the 34th SIGCSE technical symposium on Computer science education - SIGCSE ’03, 49. New York, New York, USA: ACM Press. doi:10.1145/611892.611930 e-skills UK. (2011). Summary Technology Insights 2011.. London.Retrieved 19/6/12 from http://www.e-skills.com/gamesreport Furber, S. (2012). Shut down or restart ? The way forward for computing in UK schools. Technology. London. Retrieved from http://royalsociety.org/education/policy/computing-in-schools/report/ Lenhart and Kahne, 2008. Teens, video games, and civics (2008) Pew Internet and American Life Project. Retrieved 23/12/11 from http://www.pewinternet.org/Reports/2008/Teens-Video-Games-and-Civics.aspx Margolis, J., & Fisher, A. (2002). Unlocking the clubhouse: Women in computing. Cambridge, MA: MIT Press. Robertson, J. (2012). Making games in the classroom: Benefits and gender concerns. Computers & Education, 59(2), 385-398. Elsevier Ltd. doi:10.1016/j.compedu.2011.12.020 Singh, K., Allen, K. R., Scheckler, R., & Darlington, L. (2007). Women in Computer-Related Majors: A Critical Synthesis of Research and Theory From 1994 to 2005. Review of Educational Research, 77(4), 500-533. doi:10.3102/0034654307309919 Varma, R. (2010). Why so few women enroll in computing? Gender and ethnic differences in students’ perception. Computer Science Education, 20(4), 301-316. doi:10.1080/08993408.2010.527697

I am not "Miss"

New Scientist have an interesting article about why there are so few female scientists.  It's not all "about the babies" as they so tactfully put it:

"More than half of female scientists have experienced gender bias, according to a 2010 survey by the American Association for the Advancement of Science for L'Oréal. Examples include being ignored in meetings, students calling you Mrs. instead of Dr. or Professor, receiving unwanted comments on your appearance, and hearing that you were hired not on merit, but because you're a woman."

Of the crimes above, the one I most frequently encounter is not getting called Dr. I am NOT "Miss". It's wrong on 2 counts. I had a student who persisted in calling me "Miss" (with no surname) for 12 weeks. I think he thought he was at school. Did he call my male colleagues "Sir"? If so, he should have called me "Madam", dammit.

 

[Found this link on Mark Guzdial's blog]

When everything is embarrassing: designing for teenage girls

[Cross posted from CACM blog]

Do you remember what it was like to be 15? I was reminded today of the toe curling embarrassment which teenage girls seem to face on a regular basis. I went with my PhD student, Andrew MacVean, to a user centred design session with six fifteen year-olds. Andrew is designing a location based fitness game for iPhone, and he wanted to observe members of the target user group playing fitness games on the Wii Fit to give him some feedback about effective game mechanics.

I often find user centred/ participatory design work useful just to get a feeling of what it is like to be a user in the target group. Mostly this has involved working with cute little kids (10 year olds are my favourite) who give copious amounts of useful feedback and little gems of wisdom which you can write fondly about in papers. Teenagers are not so straightforward, it would seem. But I still think it is useful to work with them because it gives you a sense of what it really important to them, and it makes you aware of stumbling blocks to using your technology which you may not have anticipated.

It is important to us that teenage girls should enjoy using the fitness game, because in Scotland girls between 12 and 16 are at particular risk from obesity. We would like to encourage them to exercise more in an enjoyable way, which is one half of the battle in fighting weight gain (the other half being caloric intake). However, it is an incredible design challenge because body image is such a sensitive issue among teenage girls. Actually, based on today's session, everything is a sensitive issue if you are a teenage girl. Standing up in front of 5 other friends to use a Wii Fit? Too embarrassing! Everyone would be watching! The horror! You'd think we had asked them to tango naked around the assembly hall. (We didn't. The ethics board wouldn't approve it). We finally managed to convince two girls to play a water skiiing game together. And then suddenly I realised what they had been making a fuss about. The girls were subject to altogether more negative commentary and patronising remarks about their performance in the game than the boys were. In fact, the girls did not comment on the boys' performance at all. I am not sure whether this is because the boys went first so the girls chose not to comment to avoid retaliation when their turn came, or whether the girls did not feel qualified to comment. At any rate, I don't think I would have wanted to play the game with such a critical audience. And these boys were quite sweet and gentle in comparison to their peers, their teacher assured me.

There also seemed to be an issue relating to image management. The girls claimed not to like games, apart from Tetris which they would play on their phones on the bus. They didn't want to be associated with gamers or gaming. Yet, when asked further, they were actually familiar with quite a range of games and admitted to enjoying them. During the session today they spontaneously exclaimed about bits they liked and were disappointed when the bell rang to signal the end of the lesson. So while they enjoyed playing games, they were not willing to admit it.

So designing with and for teenagers is hard. It seems to me that observation is as important as interview in terms of methodology because they may be reluctant to answer honestly in front of peers. Their behaviour may reveal their opinions more honestly, but in a way you could argue that is irrelevant because if they are reluctant to be seen to play with a piece of technology, then they will not use it willingly for long enough to enjoy it, or in this case increase their fitness. This project will be a challenge. But who says research is meant to be easy?

EPSRC, shame on you

 EPSRC have published a report on their ICT research capability as part of planning their research portfolio in the next decade. Part of this is a SWOT analysis for various sub parts of their programme. For computer science, they publish the following graph of demographics of people they funded. Look at the green bits. Yeah, you might have to look real hard because the green bits represent women they funded in computer science. Not very many are there? I don't necessarily blame EPSRC for that without knowing the base rate of funding applications by gender. But I do blame them for not identifying this as a weakness in their SWOT analysis. Complacent, huh?

Stats - summary of what I made myself learn

Since January I have been steadily working on a paper in which I report the results of a likert scale attitude study. It's about how children's attitudes to CS change after taking part in a school game making project. The results or somewhat counter-intuitive, not to say alarming. However, the point of this post is to record what I learned about stats so I can read it later to remind myself. It may also be of use to other people who share my lack of stats knowledge and do similar analysis.

Analysis of Likert Data

It's quite common for HCI or computer science education researchers to use attitude questionnaires to examine people's opinions of new software or teaching interventions. These are often on a likert-type scale of strongly agree to strongly disagree. And the sad truth is that researchers typically use the wrong statistical techniques to analyse them.(Kaptein, Nass, & Markopoulos, 2010) published a paper in CHI last year found that in the previous year's CHI proceedings, 45% of the papers reported on likert type data but only 8% used non-parametric stats to do the analysis. 95% reported on small sample sizes (under 50 people). This is statistically problematic, even if it gets past reviewers! Here's why.

Likert scales give ordinal data. That it, the data is ranked i.e. "strongly agree" is usually better than "agree". However, it's not interval data. You can't say the distances between "strongly agree" and"agree" would be the same as "neutral" and "disagree" for example. People tend to think there is a bigger difference between items at the extremes of the scale than in the middle (There is some evidence that this is the case cited in Kaptein's paper). For ordinal data, one should use non-parametric statistical tests (so 92% of the CHI papers got that wrong!) which do not assume a normal distribution of the data. Furthermore, because of this it makes no sense to report means of likert scale data - you should report the mode (entry which occurs most frequently in the data set).

Which classic non-parametric tests should you use? I strongly recommend the flow chart on p274 of "How to Design and Report Experiments" by Field and Hole. This text book is also pretty good for explaining how to do the tests in SPSS and how to report the results. It also mentions how to calculate effect sizes (see later).

Why is it so common to use parametric tests such as the T-test or ANOVA  instead of non-paramtric counterparts? (Kaptein, Nass, & Markopoulos, 2010) suggest it is because HCI researchers know that non-parametric tests lack power. This means that they are worried that the non-parametric tests will fail to find a test where one exists. They also suggest it is because there aren't handy non-parametric tests which let you do analysis of factorial designs. So what's a researcher to do?

Robust Modern Statistical Methods

It turns out that statisticians have been busy in the last 40 years inventing improved tests which are not vulnerable to various problems classic parametric tests stumble across with real world data and which are also at least as powerful as classic parametric tests (Erceg-Hurn & Mirosevich, 2008). Why this is not mentioned in psychology text books is not clear to me.  It must be quite annoying for statisticians. A catch about modern robust statistical  methods is that you can't use SPSS to do them. You have to start messing around with extra pacakages in R or SAS, which are slightly more frightening than SPSS, which itself is not a model of usability. Erceg-Hurn & Mirosevich (2008) and Kaptein, Nass, & Markopoulos ( 2010) both describe the Anova-type statistics which is powerful and usable in factorial designs and works for non-parametric data. By the way, a lot interval data from behavioral research (such as reaction times) does not have a normal distribution or is heterscedastic (groups have unequal variance), and so shouldn't be analysed with classic parametric tests either. To make matters worse, the tests which people typically use to check the normality or heterscedaticity of data are not reliable when both are present. So basically, you should always run modern robust tests in preference to the classic ones. I have come to the sad conclusion that I am going to have to learn R. However, at least it is free and there is a package called nparLD whi-ch does Anova type statistics. Kaptein's paper gives an example such analysis, which I am currently practising with. I must say he is a very helpful author indeed!

Effect Size

You might think that this is the end of the  statistical jiggery pokery required to publish some seemingly simple results correctly. Uh-uh. It gets more complicated. The APA style guidelines require authors to publish effect size as well as significance results. What's the difference? Significance testing checks to see if differences in the means could have occured by chance alone. Effect size tells you how big the difference was between the groups. Randolph, Julnes, & Lehman( 2008), in what amounts to a giant complaint about the reporting practices of researchers in computer science education, pointed out that the ways stats are reported by CSE folk doesn't contain enough information, and missing effect sizes is one problem. Apparently it isn't just us: Paul Ellis reports that psychologists don't tend to. And they ought to know better. Ellis also comments that there is a viewpoint that not reporting effect size is tantamount to with-holding evidence. Yikes! Robert Coe has a useful article on what effect size is, why it matters, and which measures one can use. Researchers often use Cohen's d or the correlation co-efficient r as a measure of effect size. For Cohen's d, there is even a handy way of saying whether the effect size is small, medium or big. Unfortunately, if you have non-parametric data, effect size reporting seems to get more tricky, and Cohen's way of interpreting the size of effect no longer makes sense (indeed, some people question whether it makes sense at all). Also, it is hard for non-experts to understand. Common Language Effect Sizes, or Probability of Superiority statistics can solve this problem (Grissom, 1994).It's  "the probability that a randomly sampled member of a population given one treatment will have a score (y\) that is higher on the dependent variable than that of a randomly sampled member of a population given another treatment (y2)" Grissom, 1994;314). As an example from Robert Cole, consider a common language effect size of 0.92 in a comparison of heights of males and females.  In other words 'in 92 out of 100 blind dates among young adults, the male will be taller than the female". If you have likert data with an independant design and you want to report an effect size, it's quite easy. SPSS won't do it for you, but you can do it with Excel: PS = U/mn where U is the Mann-Whitney U result, m is the number of people in condition 1, and n is the number of people in condition 2 (Grissom, 1994). If you have a repeated measures design, refer to Grissom and Kim (2006, p115). PSdep = w/n, where n is the number of participants and w refers to "wins" where the score was higher in the second measure compared to the first. Grissom (1994) has a handy table for converting between probability of superiority and Cohen's d, as well as a way of interpreting the size of the effect.

This is my current understanding of the topic, although I have one or two remaining questions. I welcome any corrections from stats geniuses! I hope it is useful but I suspect colleagues will hate me for bringing it up. I hate myself for reading any of this in the first place. It's much easier to do things incorrectly.

References

Erceg-H-urn, D. M., & Mirosevich, V. M. (2008). Modern robust statistical methods: an easy way to maximize the accuracy and power of your research. The American psychologist, 63(7), 591-601. doi: 10.1037/0003-066X.63.7.591.

Grissom, R. J. (1994). Probability of the superior outcome of one treatment over another. Journal of Applied Psychology, 79(2), 314-316. doi: 10.1037/0021-9010.79.2.314.

Kaptein, M. C., Nass, C., & Markopoulos, P. (2010). Powerful and consistent analysis of likert-type ratingscales. Proceedings of the 28th international conference on Human factors in computing systems - CHI ’10 (p. 2391). New York, New York, USA: ACM Press. doi: 10.1145/1753326.1753686.