Computers, creativity and learning

Musings on early years activities for computational thinking

 [I'm thinking about how to develop computational thinking skills from the early years up. Here are some foundation facts about information and process which might be helpful for teachers to identify what is important].

Computational involves an understanding of how to organise information according to its attributes, and processes for systematically manipulating this information to solve real world problems. There are often multiple ways of solving the same problem, some of which are more efficient than others. Computational thinking involves considering trade-offs such as saving time but using more space, or spending time now to save time later.

Here are some foundation facts about information and how to systematically process it. They do apply to the real world, but really come into their own for dealing with large digital data sets. 

 

• You can classify objects according to attributes such as size, colour, weight.
• Objects have more than one attribute e.g. this ball is both red and round.
• A collection of objects can be arranged into groups, depending on their attributes. e.g. a pile of toys for each colour or a heap of heavy and a heap of light objects.
• In order to decide which category an object belongs to, you need a way of deciding equality of attributes. That is, you need to be able to compare objects to decide whether they are the same on a particular attribute (e.g. they are both red),
• Depending on which attribute you use, a collection can be split into different groupings. A red ball could belong to a group of red objects or a group of round objects, or a smaller group which is the intersection of these.
• Objects can be sorted into an order according to a chosen attribute.  For this you need to decide on an ordering operator - a way to decide which is bigger or smaller. For example, a shelf contains a collection of books sorted in alphabetical order.
• There are different ways to sort collections (known as sorting algorithms) and some of these are more efficient than others - that is, some are quicker, some take up less space.  It is useful to consider efficiency for sorting very large collections e.g. all the books in a library.
• There is an efficiency trade-off between search and sort. It can be useful to sort objects within an collection because it makes it quicker to find a particular object next time you look for it. On the other hand, if you seldom need to search for objects in your collection it may just be a waste of time to sort it in advance. The scale of the collection matters too - it might not be worth using a complicated sorting algorithm for a small number of objects, but it may be impractical to sort a very large data set without one.

Whistling Vivaldi book and stereotype threat

As part of  editing our Equal Bite book about gender equality in Higher Education, I've been reading about stereotype threat and how to overcome it. I just finished reading Whistling Vivaldi by Claude Steele. It's an account of several decades of psychological research into academic under-performance of African-American students in US universities. Stereotype threat  affects the performance of people when they belong to a group which is negatively stereotyped at being poor at a particular task. For example, white sports people, French working class undergraduates at language tests (apparently), women in maths tests, men in situations which require empathy, older adults in memory tests and so on. It is possible to evoke stereotype threat in a lab situation even for activities and groups who might not normally experience it. "Whenever we're in a situation where a bad stereotype about one of our identities could be applied to us - such as those about being old, poor, rich or female - we know it. We know what 'people could think'. We know that anything that we do to fit that stereotype could be taken as confirming it, And we know that, for that reason, we could be judged and treated accordingly".(Ch 1.1). 

The consequences of stereotype threat are powerful. It affects highly motivated, highly skilled individuals who are deeply committed to an outcome, causing them to under-perform. Even without encountering direct prejudice or discrimination, an individual can labour under the ghost of stereotype threat to the detriment of their performance and well being. In a situation where a person identifies with a group which has a related negative stereotype , the stereotype will be evoked. For example, imagine a high ability female maths student sitting a difficult exam who has just noticed a cue that reminds her of the stereotype that women are no good at maths.  The student then becomes anxious and the anxiety has associated physiological symptoms which detract from performance (curiously, the participant might not be aware of this anxiety even if their heart rate and blood pressure tell a different story). The student's mind starts to race: considering the stereotype and how she doesn't want to confirm it by failing the test, denying that the stereotype will apply to her, suppressing unwanted thoughts, giving herself a pep-talk and monitoring her performance. The problem is that if this happens in a situation where the person needs all her cognitive resources to solve a challenging problem at the cusp of her abilities, the mind-racing slurps up some of the raw thinking power which is needed to solve the problem. She then self-monitors, find that her distraction has impacted her performance and the cycle goes on during that test, and at the next test and in other learning situations.

There is strong experimental evidence for this from a range of sources, with effect sizes large enough to substantially suppress a student's grades over the years of a degree course. It's not that people under stereotype threat aren't clever, or don't have the pre-requisite learning, are unmotivated or don't work hard. It's that the environment that they learn and are assessed in interacts with their identity in a negative way. It is actually less of a problem for lower ability students, or those who don't care about the outcomes of the test. Unfortunately increasing effort to combat it can be counter productive. For example, a series of studies of black students found that they redoubled their efforts in the face of failure and stereotype threat, but that those efforts did not use particularly effective strategies. For example, the black students in the study were more likely to spend long hours studying in their rooms and getting stuck, whereas white and asian students were more likely to benefit from shared cognitive resources by studying in groups.

Stereotype threat can be triggered by environmental cues, such as a low proportion of other women in the room, a low proportion of women in positions of power, or the environment reflecting the cultural interests of the more dominant group. For me as a female computer science undergraduate, this meant noticing that I was the only woman in the tutorial group, that the lecturers were almost always male and that the labs were covered in posters about Red Dwarf. Such environmental cues do make a difference to behaviour and performance under lab conditions: women taking maths tests under stereotype threat in the company of men performed worse. Women scientists looked more frequently around the room and were better able to recall who was present when they watched a video about a science conference which depicted a low proportion of women; the cue of stereotype threat made them more vigilant to their surroundings [9].

Dealing with stereotype threat day after day is tiring and can lead to vicious cycle of under performing, then possibly drop out. Evidence suggests that it has a longer term negative impact on health outcomes.  It is not surprising that there are low proportions of women opting for advanced scientific and maths departments in universities. Even without encountering direct prejudice, the threat in the air caused by stereotypes is quite seriously damaging.

Is there anything we can do about this? Happily, yes. Several seemingly small educational interventions have recently been shown to be surprisingly effective, even in the long term. In an article in Current Directions in Psychological Science, Walton [1] refers to interventions of this type as "wise psychological interventions". They are based on a specific, precise well-founded psychological theories which have been validated in the lab and then developed to be applicable in a real world setting. The reason that small interventions of this sort can have surprisingly long lasting effects is that they operate recursively. A small adjustment to the way an individual perceives their situation (when applied early enough) can disrupt a self-reinforcing downward spiral, and begin a snowball of positive effects. Some wise psychological interventions are based on the story-editing approach (see Redirect - the New Science of Psychological Change) in which individuals are encouraged to change the narratives they tell themselves in order to address a wide range of social and personal issues. It's based on the assumptions that changing behaviour requires changes in the way individuals interpret themselves in their social worlds, that these interpretations can be fruitfully and wisely redirected, and that these redirections can be self-sustaining, which leads to long term behavioural change.

Here are some examples of successful wise psychological interventions which have important lessons for how we can reduce stereotype threat in our teaching environments.

Belonging

A successful intervention which aimed to help participants edit their stories about themselves focussed on developing a sense of belonging to the student community [2]. If you're a first year black student at college where most of your classmates are white, or a woman student on a engineering course which is male dominated, you may question whether you belong ( belonging uncertainty). This may well provoke stereotype threat and its dampening effect on performance. If you  find classes hard, or have trouble making friends you may attribute these difficulties as being related to your sense that you don't "fit".  In Walton and Cohen's study [2], first year college students were presented with the results of a survey which showed that all first year students, regardless of ethnicity, had initial worries about fitting in, but that over time they settled in, made friends and felt part of the community. This encouraged the students watching the presentation to construct a positive story in which their feelings of belonging uncertainty were attributed to the stage in their academic career rather than their social identity. This gave them hope for the future. Black students who took part in this hour long intervention got on average one third of a letter grade higher next semester than those in the control group. This study has implications for possible successful interventions to assist women studying subjects where they are a minority and which are associated with negative gender stereotypes.

Mind set

Dweck's work on mindset has far reaching consequences for education in general, and has implications for productively intervening to reduce the impact of stereotype threat.  Mindset refers to people's theories about how ability relates to challenge. Those with a "fixed" mindsets believe that ability and intelligence is fixed and can't be expanded: you're either good at maths or you're not. Those with growth mindset believe that it is possible to meet challenges by learning new abilities: you can get better at maths if you practice. If you have a fixed mindset, and are under stereotype threat then you are not in a hopeful situation because it seems as though it is not worthwhile to take on new challenges in case you fail , thus confirming the negative stereotype. If you have a growth mindset, then you have the more optimistic view that you should take on challenges because there is always room for improvement. The stereotype threat is removed because someone with growth mindset doesn't see failure as confirmation of innate intelligence of themselves or their group, but as a step in a journey to improving performance. Luckily, quite short interventions can change people from fixed to growth mindset. Indeed, in  study of women maths students, Good, Rattan and Dweck demonstrated that  "the message that math ability could be acquired protected women from negative stereotypes, allowing them to maintain a high sense of belonging in math and the intention to pursue math in the future" [4;1]. As educators, then, we need to support all our students to learn growth mindsets in order to help them to overcome the difficulties which they will encounter during their studies. Dweck and colleagues write: "We have found that what students need the most is not self-esteem boosting or trait labeling; instead, they need mindsets that represent challenges as things that they can take on and overcome over time with effort, new strategies, learning, help from others, and patience" [5; 312].

Values affirmation

Self-affirmation theory gives us another clue about how to interrupt the downward spiral caused by stereotype threat. This is based on the idea that it's basic human nature to see oneself as good and competent, and that if that perception is threatened we try to repair that self image. Repairing one's self image may involve rationalising and re-explaining events to fit with the view of self-competence. However, it is productive to encourage people to "self-affirm" their wider valued sense of self so that this minor threat to self-image seems smaller and there is less need to rationalise it away. Stereotype threat  causes regular damage to one's self-image of competence, so one possible intervention to reduce it is to give students an opportunity to develop a self-affirming narrative. It turns out this is an astonishingly effective intervention. In an experiment with 7th grade children, researchers asked a random sample of children to write a paragraph for 15 minutes on their three most important values (e.g. family, friends, being good at music) and why they were important to them [7]. There were a few similar follow up writing exercises in later school terms. The affirmation writing exercise improves the grades of all but the strongest performing black students, with those with the poorest initial performance improving the most. It closed the racial achievement gap between black and white students by 40%, and this lasted for two years. More recently, Miyake et al found that a similar writing exercise for female college students studying physics boosted students' modal grades from a C to a B [8].

These results are extremely promising.They suggest that refocusing students' attention on the values that are important to them helps them not to dwell on poor early performances and so frees up cognitive resources which otherwise might get consumed worrying about stereotype threat. This enables a better performance next time around and interrupts the negative snowballing.

Feedback

Much has been made of poor feedback practices in HE in recent years, particularly because in the UK, national student survey results are scathing about feedback on undergraduate degrees. Negative feedback has a detrimental effect for those studying under stereotype threat [6]-  in a study of women tackling a challenging maths test, Mangels and colleagues found that negative feedback was a predictor of disengagement from learning and interference with learning attempts. That is, stereotype threat when combined with negative feedback drags down not only assessment performance but learning itself. Steele describes studies [3] which found that black students interpret critical feedback on their writing in different ways from white students; they are less likely to trust it and less likely to find it motivating. The studies found that it didn't work to try to be neutral in the feedback, or begin the feedback with a general positive comment. What did work for both black and white students was for the marker to explain that she had high standards, and that she believed the students could meet these standards by acting on the following specific advice. This strategy is successful because stereotype threat is reduced. The learner knows that the clearly stated high intellectual standards are for everyone (not just their group) and that the marker believes that they personally are capable of meeting those standards by improving various features of the work. They have hope, and they have a plan for next steps in their learning. As a related note, publishing clear and transparent assessment criteria along with an assignment helps with the first part of this equation. No-one has to second guess what the marker might be "looking for".

 Lastly, but not necessarily related to wise psychological interventions, paying attention to the environment for teaching and learning can reduce stereotype threat.  "If enough cues in a setting can lead members of a group to feel 'identify safe', it might neutralise the impact of other cues in the setting which might otherwise threaten them." (section 8.3). A useful concept here is critical mass: which refers to the point where there are enough members of a minority group in the setting so that individuals no longer uncomfortable or vulnerable to identity threat.  It's hard to quantify what critical mass means in proportional terms. However, Steele reports studies of the gender balance in orchestras in which orchestras with 20% female members experienced problems, in contrast to 40% female member orchestras in which all members reported more satisfying experiences. For me, this converts into a practical teaching suggestion about allocating groups in classes with minority groups. In a maths class with a small number of women for example, if you prioritise critical mass, then you could have some groups with at least 2 out of 5 female members and many groups with no women.This would enable the women to be closer to critical mass, and would work better for them rather than putting one woman in as many groups as possible. (I did a 6 week long group project in which I was the only women as an undergrad and it was not fun). 

Other environmental cues of stereotype threat which we can address in teaching environments is the proportion of women staff members, and the proportion of women in positions of power. This is something which one would hope would be addressed by other initiatives such as Athena Swan; my point here is that women staff members should be highly visible to the students. My point is not, however, that women should bear the responsibility of "fixing" gender balance on our degree programmes. It's everyone's responsibility and so if women staff are asked to teach high profile courses as part of a gender equality commitment, then it should be factored into their workload rather than being an added extra.

Summary

There is a convincing pool of research which indicates that women under-perform in challenging intellectual situations where they encounter stereotype threat. This is not a unique characteristic of women, but a general psychological effect which is possible to trigger in members of any group by negatively comparing their group to another. Fortunately, it is possible to reduce stereotype threat by paying attention to cues that might trigger it in our teaching environments, and by using small but extremely effective wise psychological interventions including promoting a sense of belonging, using values affirmation writing exercises, encouraging growth mindset, and structuring feedback in a way that encourages the student to strive for high standards.

[1] G.M. Walton, The new science of wise psychological interventions, Curr. Dir. Psychol. Sci. 23 (2014) 73–82. doi:10.1177/0963721413512856.

[2]  G.M. Walton, G.L. Cohen, A question of belonging: race, social fit, and achievement., J. Pers. Soc. Psychol. 92 (2007) 82–96. doi:10.1037/0022-3514.92.1.82.

[3]  G.L. Cohen, C.M. Steele, L.D. Ross, The Mentor’s Dilemma: Providing Critical Feedback Across the Racial Divide, Personal. Soc. Psychol. Bull. 25 (1999) 1302–1318. doi:10.1177/0146167299258011.

[4] C. Good, A. Rattan, C.S. Dweck, Why do women opt out? Sense of belonging and women’s representation in mathematics., J. Pers. Soc. Psychol. 102 (2012) 700–717. doi:10.1037/a0026659.

[5] D.S. Yeager, C.S. Dweck, Mindsets that promote resilience: When students believe that personal characteristics can be developed, Educ. Psychol. 47 (2012) 302–314. doi:10.1080/00461520.2012.722805.

[6] J.A. Mangels, C. Good, R.C. Whiteman, B. Maniscalco, C.S. Dweck, Emotion blocks the path to learning under stereotype threat, Soc. Cogn. Affect. Neurosci. 7 (2012) 230–241. doi:10.1093/scan/nsq100.

[7] G.L. Cohen, Reducing the Racial Achievement Gap: A Social-Psychological Intervention, Science (80-. ). 313 (2006) 1307–1310. doi:10.1126/science.1128317.

[8] A. Miyake, L.E. Kost-Smith, N.D. Finkelstein, S.J. Pollock, G.L. Cohen, T.A. Ito, Reducing the Gender Achievement Gap in College Science: A Classroom Study of Values Affirmation, Science (80-. ). 330 (2010) 1234–1237. doi:10.1126/science.1195996.

[9] M.C. Murphy, C.M. Steele, J.J. Gross, Signaling Threat, Psychol. Sci. 18 (2007) 879–885. doi:10.1111/j.1467-9280.2007.01995.x.

 

Seeking computer science students to inspire children about computing

 

When I was a lecturer in the Computer Science department at Heriot-Watt University, one of my favourite courses to teach was Computing in the Classroom. This was a fourth year course where students spent 40 hours helping out in a high school computing class. The students really loved the course, and I enjoyed seeing how much they learned as a result. Now I'm at Edinburgh, in the School of Education, and we're offering a similar course. This time we're looking for Edinburgh based CS/Informatics students to sign up for a course to work with teachers and help out with computing in primary school classrooms. This course isn't credit bearing (so no essays or presentations!) and it is open to students in all years of study. And -let's face it- primary aged kids are soooo much more fun than high school students.

Here's some reasons why it's a good idea:

• Help to inspire and mentor children in the subject you love.
• Show children that computing can be for everyone.
• Explore whether you'd like to be a teacher one day. One of my previous students wrote about the Computing in the Classroom course “The course gave me the opportunity to see what it is like to be a teacher and has convinced me that this is something I would like to do for a career. I understand there are negatives to being a teacher but the joyous feeling you get when someone learns something you taught them can’t be beat”
• Get out of university and meet new people. As one of my students said “Overall the Computing in the Classroom course has been a very positive and rewarding course and it is something that I will remember for the rest of my life. I like the idea of being out in a real-life working environment and doing things out side of university".
• Meet software developers who have also volunteered to help - useful for networking.
• Build your cv. Students from my previous courses told me that they really appreciated the chance to develop their communication and organisation skills, and that employers were impressed by them at interview.

To apply for a place, download this form and email it to me: ComputationalThinkingCourseAdvertCSStudents. You can also see the official advert for teachers here. 

Computational thinking course for Edinburgh Primary Teachers

We're running a course about computational thinking for primary teachers in Edinburgh! I'm excited about this - it's going to be so much fun. For the last year or so I've been yakking on about why we should be focussing on teaching computational thinking (rather than just how to use computers). This is our chance to discuss how best to do this.

In a world over-flowing with technology, the ability to use digital technology is insufficient: education for the digital age needs to be broadened to encompass computational thinking and to enable learners to understand and critique the effects of technology on society and individuals. Children need to learn more than simply how to use software packages, as previously taught in subjects such as Information and Communications Technology. They also need help to learn how to use technology in pursuit of their studies, for example using search tools appropriately to capitalize on the vast sea of information on the Internet. But as educators, we can and should do more. Children need our help in understanding how the technology they encounter in the world around them works and the ways in which it impacts on society. Further, they need a deep conceptual understanding of the information and processes underlying software so that they have the ability to shape technologies of the future.

Computational thinking is an approach to solving problems which has been incredibly useful within computer science. It includes programming (aka coding) but there's more to it than that. It's a way of thinking which enables us to tame huge, complex problems into more easily solvable parts. This is going to be a key skill for children growing up in our digitally driven society. 

During the course, teachers will work with computer science students and industry mentors to develop hands-on games and activities to teach computational thinking without even going near a computer. (You'll be surprised how much of an overlap there is with familiar classroom activities). They'll also learn the basics of programming a simple blocks language like Scratch or Hopscotch. We'll focus on how to make computational thinking inclusive and approachable by all learners. I've been working with Education Scotland and academic colleagues on re-designing the computing part of the Technologies curriculum, so the course will help you get to grips with the upcoming changes.

So, if you're an Edinburgh primary teacher, please apply to join the course! You can get the application form and find out practical information here. This is a pilot year; in future we will consider extending it more widely.

Algorithms to Live By - book review

Ten years ago Jeanette Wing, proposed that computational thinking should be taught to children along with reading, writing and 'rithmetic. She wrote "Just as reading, writing, and arithmetic are fundamental skills every child learns, computational thinking is a skill needed for every citizen to function in today's global society. Computational thinking is an approach to solving problems, building systems, and understanding human behavior that draws on the power and limits of computing. Computational thinking is the use of abstraction to tackle complexity and the use of automation to tackle scale." Then, and since, I have been pretty sceptical of that position. This is partly because her examples of the ways in which computational thinking are useful to everyday life aren't very well developed in her initial paper. In fact, I used to teach  a course entitled "Critical and Computational Thinking" but eventually I dropped the "Computational" bit from the title because I could never find any good meaty material for it. (In retrospect, this was not a surprise as I was in a computer science department at a time and in a sense we were fish in computational thinking water: it was just the way we thought).

In the past few years I have begun to change my mind and in the last weekend I became a convert. I read Algorithms to Live By by Brian Christian and Tom Griffiths and it is superb! It is an account of exactly how computational thinking can be applied to our everyday lives, written by a computer scientist and a cognitive scientist. It is satisfyingly geeky, but doesn't get bogged down in mathematical detail. The premise of the book is that computer scientists grapple with huge, messy, complex problems in the real world, and that the algorithms they have designed to solve those problems can give us insight into the problems that we stumble across routinely everyday. Seemingly mundane problems like knowing when to stop searching for the perfect hire in an applicant pool, or how to schedule tasks in an over crowded day are more complex than we give them credit for. In fact, computer scientists and mathematicians have devoted decades to solving theoretical problems which are structurally identical to those we encounter in our lives. In some cases, they have found optimal solutions which Christian and Griffiths make a good job of relating to the real world examples. In other cases, the problems are so thorny as to be intractable - because the solutions take so long to calculate - or indeed completely insoluble.

The authors make the point that computer science has moved away from reliance on algorithms which exhaustively and painstakingly grind through a series of steps until they find the right answer. Instead developing algorithms for"tackling real-world tasks requires being comfortable with chance, trading off time with accuracy and using approximations" (p5). This means that our best solutions for solving complex problems might not be guaranteed to be correct, or accurate and might give different answers each time. This is quite a different characterisation of algorithms to the watered down version of algorithms which has recently been used in schools where children give step by step instructions to aliens about how to make jam sandwiches. (I will save my jam sandwich rant for another day).

Similarly, the authors note, "people need to make decisions while dealing with uncertainty, time constraints, partial information and a rapidly changing world" (p6). Findings from how computer science has tackled problems of this sort suggest advice like "Don't always consider all your options. Don't necessarily go for the option that seems best every time. Make a mess on occasion. Travel light. Let things wait. Trust your instincts and don't think too long..." (p6). And the best bit about these nuggets of wisdom is that they are not fluffy fortune cookie drivel from a self-help book; they have formal mathematical proofs behind them.  

To be a computer scientist is to make peace with the idea of trade-offs. Do you want the problem solved quickly with an approximate answer, or do you want the exact solution so much that you can wait for ages? How much resource can you throw at the problem to get it solved in a reasonable time? Once you accept that trade-offs are built into many problems, it is liberating because it takes the pressure off you personally. Here's an example:everyone has finite time available to do their jobs, even academics. Every new task which gets added to your academic workload means you have less time to allocate to existing tasks. The more papers you write, the less time you have to review papers for a journal. You cannot do it all. You have to make choices about which tasks to prioritise to optimise your own job satisfaction. And there is simply no point in feeling guilty about the tasks which you didn't have time to do.

Here are some of the topics in the book which I found particularly interesting.

Searching and Sorting

Computer scientists love to sort, and they spend a lot of time analysing the efficiency of  different sorting methods. Some algorithms for sorting are better than others. Sorting goes hand in hand with searching because it is generally easier to locate an item in a sorted collection than ferreting around in a dirty great heap.  "The basic principle is this: the effort expended on sorting materials is just a preemptive strike against the effort it'll take to search through them later." (p72). This is one reason why it makes sense to sort your clean laundry into different drawers - you sort laundry up front when you have time at the weekend so that when you are rushing around in the weekday mornings half-asleep you can lay your hand on your socks quickly. However, Christian and Griffiths advise to "err on the side of messiness... Sorting something you will never search is a complete waste; searching something you never sorted is merely inefficient" (p72). It's another trade-off. It will take you a long time to sort your books shelves into alphabetical order (if you have a large set of books) but you may not often need to search for a book. From this point of view, you'd be better to accept the trade-off that it might take you a while to uncover a book when you do need it than sink a day into sorting your books upfront.  As a more commonly occurring example: this principle says it's a waste of time to sort your email into folders as it arrives.  It is more common to get a message than it is to need to search for one, and furthermore your computer is able to perform a highly efficient search for you should the need arise (unless you happen to use Outlook, in my experience). So a tidy inbox is a sign of wasted time. 

The sort/search trade-off is a long running debate in my house with respect to my son's giant Lego collection. It's easiest to store the Lego by dumping it into a big box (no sort), but when it comes to finding fiddly bits when you're making a model (search) it means endless scrabbling in the box. My husband attempts to sort the pieces into different box according to size and shape, but that's a) hard to maintain and b) time consuming to set up. In theory, setting it up is a high one time cost which should then pay off in the future. Unless, of course, someone drops the Lego boxes all over the stairs so an afternoon of sorting is lost. Oops.

Caching

So apparently there are lots of books devoted to telling people how to organise their possessions which dish out computational contradictory advice. If you are having a spring clean, you need to decide what to keep and then how to arrange those things. Are you going to take the advice of the clutter guru who tells you to group like things together? Or to consider when you bought it? When you last used it? Or indeed, quantify how much joy it is likely to bring you? It turns out that the techniques that computer scientists use to store information are useful here.  There's a trade-off between size and speed in storage devices: you can't have limitless lightning fast storage so computer architecture generally compromises by using a hierarchical combination of small amounts of very fast memory and large amounts of slower storage devices (like hard discs). If you can predict which items of data are likely to be needed next, you can speed things up by storing those items in fast memory where you can get at them quickly. In fact, you can have a series of caches and shuffle data between them depending on when they're likely to be needed next. This is a bit like library books. If you're studying in the library, you take a few of the most relevant books to your table so you can refer to them easily. If you can't find what you need in this cache, you can go and look on the library shelf. In some cases you may have to ask the librarian to fetch the book from stacks storage in the basement, or indeed make a request for it to be brought from an off-site repository. The cache which stores the data which is likely to be used next should be physically closest to where the data will be used. In libraries, the books which will be in demand should be on easy to reach shelves and on computer chips, fast access memory should be situated close to the processor.

So what's the best way to predict what data will be needed in the immediate future?  How do you decide which items get kicked out to free up precious space in the cache? It turns out that the Least Recently Used approach is optimal. The items which you used most recently are likely to be needed again soon, and the ones you haven't used for a bit probably won't needed again for a while. 

Applying this to our spring cleaning, our authors have several pieces of advice based on caching. If space is a a premium, it makes sense to throw away items which you haven't used for a while rather than items you bought a while ago. You might have a very old t-shirt which you wear regularly for jogging which would be a better candidate to keep than a smart suit of the same vintage which you last wore in 2008. A legitimate objection to this is the cost of the item in the first place - why would you throw away an expensive suit? Well, if you have the  space, you might want to move the least recently used items such as the suit to a secondary storage system farther away from your main wardrobe (like a box in the attic). Another lesson is to have storage for items close to their point of use (like wellies at the back door). The authors have no comment to make on how to predict expected joy from one's possessions, but I bet if pressed, they would apply some form of Bayes rule to the prior joy they have brought you. 

One of my favourite results from this part of the book stems from the maths of self-organising lists. A pile of papers on your desk is not chaotic, it's optimal.  Keep your papers in a pile, search the pile from top to bottom. When you pull out an item put it back on top rather than where you found it.That is the best way you can organise the documents to help you find an item quickly in the future. So there.

Scheduling

Processors and people alike need to schedule time to complete multiple tasks, taking into account possible unpredictable interruptions. This can actually be a really hard problem to solve (84% of scheduling problems are intractable according to the authors) so it's no wonder many people find it stressful. What algorithms can help us here? The answer depends on what you're trying to optimise - once again we encounter trade-offs. Suppose you care about reducing the maximum time a customer has to wait. Then your optimal strategy is to start with the soonest due task and work your way forwards to the task due last (the Earliest Due Date approach). If you're concerned about reducing the number of late projects then Moore's Algorithm is your friend. Proceed as for the Earliest Due Date but if it looks as though you are going to start missing deadlines, stop and consider the tasks remaining on your to-do list. Throw out the biggest item (that which would take longest to complete). Then at the very end, you can go back to the items which were eliminated from the to-do list and do them in any order because they're late by that time anyway.

If you're more concerned about how much you get done, then the Shortest Processing Time approach will work. Work out how long each task is likely to take, and then always do the quickest task you can. This lets you race through items on your list to-do quickly, but it does ignore that some tasks are more important than others. You can modify this approach by assigning an importance weight to each item on the list, then divide the importance by how long it'll take you to complete. Then do the tasks in order of importance-by-unit of time. This gives the rule of thumb: only prioritise a task that takes twice as long if its twice as important.  You'd think that prioritising important weighty tasks would be a good bet but you also need to watch out for unfortunate side effects of dependencies between tasks. In short, a seemingly important small task should be given priority if an important task has to wait for it to be completed. (In operating systems design this is called priority inheritance).

What should you do if you get interrupted with a new task while you're working your way through your to-do list? Shortest Processing Time is still a good approach. As each task comes in, evaluate the importance-by-time-unit. If it's higher than the task you are working on, switch to the new task. Otherwise, stick with the current task. This reactive approach is easier to computer than elaborate re-planning and gives pretty good results.

Still, too much multi-tasking caused by interruptions does introduce additional problems. In operating systems, there is a concept of thrashing whereby the processor spends so much time swapping information in and out of memory for each new task that it never gets enough time to actually perform the task before it needs to switch again. In a state of thrashing, the processor is working to capacity but not completing any tasks. Does this sound familiar? You start to write an article, when the phone rings, and you're just writing down the new task the phone call brought up when someone knocks on your door with a new request. Trying to remember where you were with the writing task will take some time, and you can only hope that you can remember it and get some more words on the page before the next interruption. This is why it is a good idea to turn off your email if you're trying to get an important task done - context switching has an overhead.

Another trade-off: you can optimise for responsiveness or you can optimise for throughput. You can get back to student emails quickly or you can get a large number of exams marked, but if you favour one then the other will suffer. The theory would suggest that you should decide what your minimum acceptable level for responsiveness should be (e.g. how quickly do you need to reply to an email?). Once you have decided that, stay on a single task for as long as possible without overstepping the minimal responsiveness level. You shouldn't try to be more responsive than the minimum acceptable level. You could also time box so that you never spend less time on a task than it takes to remember what it is all about. For example, the pomodoro technique suggests focusing on a single task for 25 minutes without interruption. Lastly, applying interrupt coalescing might help. Instead of interrupting a task as a new request comes in, wait for a fixed interval before before checking for new tasks and then handle similar requests in a batch. For example, you could decide to only check  your email once a day and deal with messages then and only then.

 Reflection

I enjoyed reading Algorithms to Live By because it is an in depth justification of why and how computational thinking can be applied to everyday life. The examples in the book refer to the everyday life of adults, probably most for those of us in knowledge economy jobs. The authors weren't primarily setting out to explain computer science concepts for their own sake (although I think the books does this pretty well!). For these reasons, I don't the book can be used directly as a way to teach children about computational thinking. But I do believe that the algorithms and examples in the book can be a starting point for developing classroom activities along with teachers who understand what matters to children, and how best to explain concepts to different age groups. This is what I intend to do this summer.

On the nth day of Christmas my true love sent to me

There was apparently a problem with the delivery of the parcels from my love. Days 6 - 12 got backed  up and arrived altogether. These things happen. Just as well they all arrived by Christmas Eve, right?

On the twelfth day of Christmas, my true love sent to me

Latent class model (because he knows how much I enjoy recovering hidden groups from observed data).

Bias variance trade-off (well, we wouldn't want to over fit our models, would we?)

Aligned rank transforms (because no-one loves you if you analyse non-parametric factorial data with ANOVA).

No missing data (this is really important for all it seems unglamorous. Missing data can seriously mess up your interpretation of results).

Time series values (because it's a good idea to track user experience over time rather than in a one off session).

Power calculations  (because you want to be sure to detect an effect if it is there, and if your love has sent you an effect size from a previous experiment you are able to calculate power without any hassle).

Effect size reporting

No p-values

 Clear stopping rules

 Bayes testing

Anscombe's Quartet

And fair stats communication.

 

Merry Christmas One and All!

 

 

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

On the sixth day of Christmas, my true love sent to me: effect size reporting

On the sixth day of Christmas, my true love sent to me: 

Effect size reporting

No p-values

 Clear stopping rules

 Bayes testing

Anscombe's Quartet

And fair stats communication.

 

Suppose my love likes to quantify his affection. Like the "Guess How Much I Love You" bunny, he gives numerical estimates of his love, perhaps in comparison to another suitor. He uses real world measurements to do so (usually units of distance) , rather than standardised measurements of effect size such as Cohen's d. This is a reasonable decision, as the real word units are easier to interpret (and more poetic). I suppose he might use Cohen's d if he wanted to give me context of related effect sizes in the literature or if he was trying to establish the positive feelings of rabbits to a range of species.

But suppose I was trying to estimate the feelings of my true love for me using a similar strategy to Null Hypotheses Significance Testing (NHST). Then I might adopt an arbitrary threshold and make a binary decision based on that. If p < .05, he loves me. If p > .05 he loves me not. I suspect this might not be the path to happiness. 

This is probably not a very helpful way to introduce the concept of effect sizes. I recommend that you read Koji Yatani's chapter on effect sizes instead for a more cogent discussion of the importance of effect sizes and a guide to how to compute them in R.  Here is a rather clearer explanation than  mine, written by Koji in his characteristically friendly style:

 

"Even if you have a significant result, it is “significant” only in the context of statistics. It does not necessarily mean that the difference you have is meaningful or important in a practical setting. Imagine that you are reviewing a very typical interaction technique paper. It compares two techniques: one is what the authors developed, and the other is a conventional technique. Suppose that the performance time of some tasks was improved with a new interaction technique by 1% with the standard deviation of 0.1%. In this case, their NHST would show a significant difference. So should you automatically accept that paper because it shows significant improvements backed up with well-known statistical testing methods? No – it is probably too early to decide the fate of the paper. 1% improvement may be a difference between 10 and 9.9 sec. Would this 100 msec. be really important? Maybe so for fighter pilots, but we definitely need more contextual information to assess the magnitude of this improvement.

Now you are reviewing another paper. In this paper, the performance time was improved with a new technique by 15%, but the standard deviation was also as large as 15%. The results would not show a significant difference because the standard deviation is too large. So would your conclusion be that the new technique is useless? I would say not. 15% improvement in average (e.g., 10 sec vs. 8.5 sec) could have an impact on user experience. There may have been some factors that caused such variances. For example, some participants might have been able to use experience from other interactive systems (e.g., heavy smartphone users might be able to adapt to new touch interaction more quickly than others). Other researchers might come up with an even better technique which can remove that variance while maintaining faster performance. Thus, the developed technique may be important enough to be shared even though the present paper could not find a significant difference. Assuming the paper is of high quality in other aspects, I think that it should be accepted (though I would ask the authors to explain why the standard deviation was so large and how their technique could be improved to make it smaller). Thus, we should not overrate statistically-significant results and underrate non-statistically-significant results. All results should be interpreted in a larger context."

 

In summary, if you are reporting quantitative results, report either standardised effect sizes or means and standard deviations so that the reader can compute the effect sizes if necessary. And if you are reading a paper, take a look at the descriptive statistics before you get too carried away by a significant result.

 

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials

On the fifth day of Christmas, my true love sent to me: No p-values

On the fifth day of Christmas, my true love sent to me: NO P-VALUES

 Clear stopping rules

 Bayes testing

Anscombe's Quartet

And fair stats communication.

 

You may be wondering about the relationship I have with my true love at this point. My husband (to his credit) has never sent me any of the above. It clearly differs according to tastes in different disciplines, as my friend Phil who is a reformed physicist apparently received the "5 sigma rule" from his true love. All I can say is that his love is extremely stringent in these matters.

 But why avoid sending me p-values? If we follow Pierre's  advice from the first day of Christmas, they are not consistent with fair statistical communication, particularly failing on transparency and clarity. Here is an excerpt from our introductory chapter (written by me and Maurits) on why p-values might not mean what you think they mean.

"The NHST approach, fiercely promoted by Fisher in the 1930’s (Fisher, 1934; and also Pearson, Fisher, & Inman, 1994), has become the gold standard in many disciplines including quantitative evaluations in HCI. However, the approach is rather counter-intuitive; and subsequently many researchers misinterpret the meaning of the p-value. To illustrate this point Oakes (1986) posed a series of true/false questions regarding the interpretation of p-vales to seventy experienced researchers and discovered that only two had a sound understanding of the underlying concept of significance. 

So what does a p-value actually mean? “…the p-value is the probability of obtaining the observed value of a sample statistic (such as t, F, ) or a more extreme value if the data were generated from a null-hypothesis population and sampled according to the intention of the experimenter” (Oakes 1986, p.293). Because p-values are based on the idea that probabilities are long run frequencies, they are properties of a collective of events rather than single event. They do not give the probability of a hypothesis being true or false for this particular experiment, they only provide a description of the long term Type I error rate for a class of hypothetical experiments – most of which the researcher has not conducted.

The erroneous interpretation of the p-value by many researchers brings up the first actual threat to the validity of conclusions that are supported by null-hypothesis testing. Researchers often interpret the p-value to quantify the probability that the null hypothesis is true. Thus, a p-value smaller than .05 indicates to large groups of researchers – be it conscious or unconscious –that the probability that the null hypothesis is true (e.g.   = u₀) is very small.

Under this misinterpretation the p-value would quantify P(H₀|D) - the probability of the null hypothesis (H₀) given the data collected in the experiment. However, the correct interpretation of the p-value is rather different: it quantifies P(D|H₀) – the probability of the data given that H₀ is true. Researchers who state that it is very unlikely that the null hypothesis is true based on a low p-value are attributing an incorrect meaning to the p-value.

The difference between P(H₀|D)  and P(D|H₀)  is not merely a statistical quirk that is unlikely to affect anything in practice. Not at all. It is easy to understand why this misconception is incorrect and hugely important by the following example: consider the probability of being dead after being shot, P(D=true|S=true). Most would estimate this to be very high, say 0.99. However, the mere observation of a dead person does not lead most people to believe that the corpse was shot – after all, there are many possible ways to die which don’t involve shooting. P(S=true|D=true) is estimated to be rather small. Luckily, the relationship between P(D|H₀)  and P(H₀|D)  is well-known and given by Bayes rule (Barnard & Bayes, 1958). Thus, if one wishes to estimate P(H₀|D)  - which we often do – the proper analytical tools are at our disposal." (see my gift from the 4th day of Christmas).

I am personally not arguing for a full-out ban on p-values (although one psychology journal has done so). I am arguing for using them carefully and appropriately as part of a wider statistical argument. I would also like reviewers to accept other statistical approaches rather than favouring NHST practices over others.

 

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials

On the fourth day of Christmas, my true love sent to me: Clear stopping rules

On the fourth day of Christmas, my true love sent to me: Clear stopping rules.

 Bayes testing

Anscombe's Quartet

And fair stats communication.

 

Simmon at al's paper about researcher degrees of freedom (such as lax stopping rules for data collection) is a disturbing read. In one of our own chapters, Maurits and I describe this study before going on to recommend some guidelines for authors and reviewers in HCI to avoid such problems. Here is an excerpt:

"Simmons et al. recently introduced the concept of “researcher degrees of freedom” to describe the series of decisions which researchers must make when designing, running, analysing and reporting experiments. The result of ambiguity when making such decisions is that researchers often (without intention of deceit) perform multiple alternative statistical analyses and then choose to report the form of analysis which found statistically significant results. The authors show through a cleverly constructed simulation study that “It is unacceptably easy to publish “statistically significant” evidence consistent with any hypothesis” (Simmons, Nelson & Simonsohn, 2011, p1). The authors illustrate this by reporting evidence in support of the hypothesis that listening to the song “When I’m 64” reduces chronological age.  The simulation study investigated how the significance of the results changed when different forms of analyses were conducted on a simulated data set randomly drawn from the normal distribution, repeated 15000 times. The different forms of analysis manipulated common researcher degrees of freedom: selecting dependant variables, setting sample size, the use of covariates and reporting only selected subsets of experimental conditions. If the researcher is flexible about when she stops data collection – a common practice although it is not at all advocated in the NHST literature – and gathers more data after doing a significance test, the false positive rate increases by 50%. By using covariates, such as the ubiquitous gender factor, the researcher can introduce a false positive rate of 11%, and by reporting only a subset of the data, a false positive rate of 12.6% is produced. Finally, if our researcher was too flexible in all four of these areas, the false positive rate would be 61%.

Simmons et al (2011) recommend a set of ten simple practices which reduce the problem of false positives. Firstly, they recommend that authors should decide in advance on their criteria for when to stop data collection. Given the prevalence of problems of underpowered tests, it would be sensible to use stopping rules based on power analysis. The second guideline of providing at least 20 observations per cell is directly related to power, and reduces the risk of type II errors.  According to guidelines 3 and 4, authors would be required to report all the variables collected, and all experimental conditions investigated even if they are not reported in the final analysis in the paper. This would enable readers to judge how selective authors are in reporting their results. Guideline 5 recommends that if observations (such as outliers) are removed, they should report both the results with and without their removal to enable the reader to determine what effect it has. To reduce the occurrence of false positives by the introduction of covariates, guidelines 6 recommends that results should be reported with and without covariates. As changes in authorial practices require monitoring by reviewers, Simmons et al. (2011) propose four related guidelines for reviewers, starting with ensuring that authors follow the first 6 recommendations. Following these guidelines is likely to lead to publications with fewer significant and less perfect seeming results; reviewers are therefore encouraged to be more tolerant of imperfections in results (but less tolerant of imperfections in methodology!). Reviewers are also asked to check that the authors have demonstrated that their results don’t depend on arbitrary decisions while conducting the analysis, and that the analytic decisions are consistent across studies reported in the same paper.  Lastly, Simmons et al. (2011) invite reviewers to make themselves unpopular by requiring that authors conduct replications of studies where data collection or analysis methods are not compelling."

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials

On the third day of Christmas, my true love sent to me: Bayes testing

On the third day of Christmas, my true love sent to me: Bayes testing

Anscombe's Quartet

And fair stats communication.

 

Michael Tsikerdekis does a fantastic job of explaining the concepts behind Bayesian inference.  If you're new to Bayes theorem and Bayesian approaches to analysis, you may have an Emperor's New Clothes moment where you wonder if it really can be true that scientists have been deluding themselves by industriously computing and reporting p-values even although they don't actually tell them what they want to know. Joris Mulder builds on these concepts to illustrate how to calculate the strength of evidence for competing hypothesis using Bayesian testing. Liberate yourself from the arbitrary, conservative and somewhat absurd null hypothesis! Directly compare hypotheses which have some plausibility! I am very pleased to have these two chapters in the book, because since I first read about Bayesian inference, I wanted to try on my own data sets but I really needed some accessible examples to help me learn. Michael and Joris provide these for others in the same position.

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials