Computers, creativity and learning

Turing's sunflowers

[cross posted from my CACM blog: http://cacm.acm.org/blogs/blog-cacm/148589-turings-sunflowers/fulltext]

It’s the Turing Centenary Year. Why not plant a sunflower to celebrate? Manchester University, Manchester Science Festival and the Museum of Science and Industry are asking people to grow sunflowers and count the number of spirals in their seed heads in order to test Turing’s  theory that the seeds grow in Fibonacci sequences.

 

Turing was greatly interested in mathematics in nature, studying geometric patterns in flowers from an early age . At the time of his death in 1954 he was working on a problem in phyllotaxis: whether the seed patterns in sunflower heads (and other plants such as pine cones) conform to Fibonnaci sequences. If this were the case, what could it tell us about the growth of plants? Within the centre of a sunflower, there are interlocking spirals in clockwise and anti-clockwise directions. The number of spirals in each direction (referred to as the plant’s parastichy numbers) are usually consecutive Fibonacci numbers. The ratio of one Fibonacci number to the next is an approximation of the aesthetically pleasing golden ratio, which might explain why people like admiring flowers.  You might wonder why sunflowers bother with the maths – why not just grow random numbers of spirals? One theory is that growing them in Fibonnaci sequences allows for more efficient packing of seeds, hence more baby sunflowers, hence greater evolutionary advantage. 

As far as we know, the last empirical work on sunflowers was conducted in 1939, when J C Schoute scrutinised 319 seed heads.  Now the team in Manchester (where Turing did his work on phyllotaxis) have hit on a fantastic idea to run a citizen science project to involve members of the public in helping to finish off one of Turing’s last projects.  It should result in a dataset of thousands of seed heads, which may be deeply exciting to phyllotaxologists everywhere.  Hopefully the family friendly activity of planting a flower will whet the curiosity of young people, and introduce them to the rest of Turing’s work.  These beautiful flowers planted in Turing’s memory will be a fitting tribute to the man to whom Computer Science owes so much.

For more information

Follow @TuringSunflower on twitter, and tweet your sunflower news to #Turing #sunflowers

http://manchestersciencefestival.blogspot.co.uk/

http://www.sciencenews.org/view/generic/id/8479/title/Math_Trek__The_Mathematical_Lives_of_Plants

 

Avoiding the rabbit hole of Bad Science

[cross posted form my CACM blog]

I like to follow scandals in other disciplines: the stem cell biologists’ cloning fraud, physicists and their time traveling particles. The psychologists are having their own scandal just now about extrasensory perception. Can people see into the future? Do experimental participants do better on memory tests of words which they are just about to practice memorising? Is there such a thing as a precognitive response to erotic stimuli? Daryl Bem and his nine experiments say “yes” (Bem, 2011). The other psychologists, Occam’s razor and me say “no”.  Bem exhorts his colleagues to follow the White Queen in Alice in Wonderland and believe six impossible things before breakfast (Bem, 2011). His colleagues prefer to avoid vanishing down the rabbit hole (Ritchie et al., 2012).

As well as the sheer entertainment value, such debates in other fields are worth studying because they can help us study the practices in our own field. Let’s have a look at what the subfields of computer science which use experimental methods (such as my own research in educational technology and HCI) could learn from the great Bem v.s. rationality debate. Are we in danger of falling down the rabbit hole of bad science too?

Don’t torture the data

Wagenmakers et al (2011) make the distinction between exploratory and confirmatory studies. The point of an exploratory study is to develop new theory, so you may feel free to prod and poke your data with whatever statistical tests you most enjoy. As Bem himself (rather inadvisably in my view) wrote previously, “If you see dim traces of interesting patterns, try to reorganize the data to bring them into bolder relief. If there are participants you don’t like, or trials, observers, or interviewers who gave you anomalous results, place them aside temporarily and see if any coherent patterns emerge. Go on a fishing expedition for something–anything–interesting.” (Bem 2000). But if you are trying to run a confirmatory study, the fun stops and you have to be all grown up with your data analysis. No more hiding participants under the carpet. No more running post-hoc sub group analyses until you get some – any!- result. The purpose of a confirmatory study is hypothesis testing; searching for evidence to support a theory which you have perhaps developed through an exploratory study. Here you’re meant to declare in advance what your hypothesis is (given some theoretical justification), the direction and size of effect you expect, a stopping rule for data collection and what tests you will conduct.

In the fields I work in, researchers often develop a novel system and then run a small exploratory user evaluation. That’s perfectly reasonable. What’s not reasonable is presenting the results of exploratory analysis as if they were confirmatory.

Bem’s nudey picture study is a good example of what not to do. He used exploratory data analysis approaches, but then claimed results as if it was a confirmatory study. He not only tested for pre-cognitive responses to erotic pictures, but also positive and negative pictures.  But he only got significant results for the erotic ones, so he pounced on them and breathlessly reported them. Similarly he checked for gender differences without there being pre-existing evidence or theory to suggest that there should be a difference or why. This flaw is close to home. HCI/educational technology people check for gender differences for no apparent reason all the time. I have done it myself in the past, but I swear to reform from this moment forth.

The more statistical tests you perform on your poor tortured data, the more likely it is that you will find a significant effect by chance alone. If you perform 20 tests, and you have an alpha level of .05, you’re going to find something, right? You’re meant to correct the p –value when conducting multiple tests because of this. Bem didn’t. HCI researchers are not perfect in this regard either (Cairns, 2007).

Replicate

The reason the Bem story is in the news again is that another team of psychologists have published a paper about their attempts to replicate Bem’s study (Ritchie et al 2012). Guess what? They didn't replicate his results, so if you were counting on pre-cognitive arousal being real, you will be disappointed. There are five registered replications ongoing, which just goes to show that the psychologists are serious about showing this media friendly nonsense to be rubbish. Interestingly, Ritchie’s replication paper was rejected by the journal which published Bem’s original article on the grounds that they don’t publish replications at all. My first thought was “That’s really silly. Surely journals should be supporting the advancement of science? Of course they should publish replications. What are these psychologists thinking?” Then it occurred to me that replications in HCI or educational technology journals are extremely rare too. In fact, this was the subject of a paper and panel session at CHI 2011 ((Wilson et al, 2011). The authors observed that “As a community we are not encouraged to first replicate and then extend. Instead, we are encouraged to differentiate into novel spaces, design novel interfaces, and run novel experiments that create new insights into human behavior.” If you think of human-computer interaction as an artistic pursuit, replication doesn’t matter. But if you think of it as a scientific or even an engineering enterprise then it does matter and we should be doing it more. If our goal is to design systems which are socially useful (for example in education or healthcare) then we should be sure whether they actually work with their intended user groups. That means we can’t stop at the first exploratory study. It means that we need to go on to run a confirmatory study, open source the system and encourage other teams to replicate. And that in turn has implications for the way our research is funded and the sorts of articles journals will accept.

 

References

Bem, D. J. (2000). Writing an empirical article. In R. J. Sternberg (Ed.), Guide to publishing in psychology journals (pp. 3–16). Cambridge: Cambridge University Press.

Bem DJ (2011) Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. J Pers Soc Psychol 100: 407–425. doi:10.1037/a0021524.

Cairns, P. (2007). HCI... not as it should be: inferential statistics in HCI research. Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI... but not as we know it-Volume 1 (pp. 195–201). British Computer Society. Retrieved from http://portal.acm.org/citation.cfm?id=1531321

 

Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). Journal of personality and social psychology, 100(3), 426-32. doi:10.1037/a0022790

 

Max L. Wilson, Wendy Mackay, Ed Chi, Michael Bernstein, Dan Russell, and Harold Thimbleby. 2011. RepliCHI - CHI should be replicating and validating results more: discuss. In PART 2 ----------- Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems (CHI EA '11). ACM, New York, NY, USA, 463-466. DOI=10.1145/1979482.1979491 http://doi.acm.org/10.1145/1979482.1979491

 Ritchie SJ , Wiseman R , French CC (2012) Failing the Future: Three Unsuccessful Attempts to Replicate Bem's ‘Retroactive Facilitation of Recall’ Effect. PLoS ONE 7(3): e33423. doi:10.1371/journal.pone.0033423

Double dividending

Double dividending is a cunning way of combining your activities to achieve multiple goals. Killing two birds with one stone, you know. It's a good way of being lazy*. My research, teaching and public engagement goals are so intertwined that I sometimes get away with triple dividending. The time I save I wisely spend eating bon-bons.

In the last few weeks I have reached new heights of double dividending in teaching. At one class recently I had three additional students sitting it for three quite different purposes: one because she was going to teach the same material (but simplified) to high school student, another because he wanted to revise and extend the lesson structure for a masterclass in software engineering, and the third because she needed to learn some of the practical skills for her PhD. The student who is working with high school students is going to help me plan a public engagement workshop for kids, with the help of a class member. The school she works with has taken part in various research studies of mine. I also have a group of 5th year students who have made the most fabulous news feed visualiser on the Kinect demoing their work for a UCAS applicant visit day. Another student who is helping on that day is doing it for part of the Computing in the Classroom module which I teach.

Lazy as it may be, you do have to be sharp during the teaching to make sure everyone is learning what they ought from the experience, and you also have to be careful to plan the logistics. But it has a high pay-off, and not just for me. The students learn a lot from each other. They learn about teaching from watching me teach, in a kind of apprenticeship model. So I can indirectly convince more people that they actually do want to learn computer science.

 

*I mean lazy in the same sense as Jim Smith http://www.lazyteacher.co.uk/

What staff can expect from students

I am working on set of guidelines of what lecturers and students can expect from each other, in an attempt to prevent communication problems and general huffiness! What do you think? Are these reasonable expectations? Have I missed anything? If you work at a different uni, can you point me to any similar documents? (If you work at my uni, you'll get consulted about this more officially too!)

What staff can expect from students

• Your commitment to your learning
• Politeness and respect
• A reply to email within 5 working days, particularly for mentors and project supervisors
• Attendance at classes
• Attention, courtesy and participation during classes
• If you can’t make a scheduled meeting with a staff member, please notify them in advance rather than just not turning up
• Your attention to the feedback we give you, and an attempt to improve your work based on that feedback

What you can expect from staff

• Commitment to helping you learn
• Politeness and respect
• A regular office hour slot where you can see them face to face
• Written feedback and a mark for coursework within 20 working days after the hand-in time.
• A reply to general email questions within 5 working days
• A response from your mentor within 2 working days
• If you would like to see your exam script to see where you went wrong, you can get it from the school office. You can make an appointment with the lecturer to get further advice on how to improve your work.

 

Design adventures in a children's hospital: part 1

[cross posted from my CACM blog]

I am very excited about a new project I am starting with the Sick Kids Friends Foundation, a Scottish charity which raises money to support the children’s hospital in Edinburgh. It’s an artistic residency (with Madeline Balaam ) with the aim of working with children at the hospital to produce a cutting edge interactive technology installation. As the hospital is in the process of relocating to a new building, there is also a longer term goal of advising on how play technology can be an integral part of the new design. In both cases, the goal is to support children’s development through play. Play is an essential part of childhood – a right- necessary for learning and social development, and enjoyment. This is particularly important for patients in hospital who may be there for lengthy periods. The Sick Kids hospital makes great efforts to help their patients to play as much as possible even when they are very ill. The question is: how can technology help with this? How can technology be successfully integrated in such a challenging environment with so many design constraints? What’s the best way to make the design process user centred? And how on earth do you do it with a budget of £3K (for the installation, not the new building!)?

There are a number of constraints which make this design extremely challenging:

• Infection control. All toys/books in the play spaces have to be wipe-clean to avoid to spread of infection.
• User characteristics. The users are patients, their parents, nurses and play specialists(trained nursery nurses with specialisms in hospital care). The patients may have restricted mobility due to surgery, may be weak, in pain, or anxious. As only parents are allowed in some wards to visit, the children may be missing their brothers and sisters, grandparents, friends, teachers or classmates.  Parents are likely to be worried and upset.
• Age range. The wards could have patients from day old babies to sixteen year olds all using the same play spaces.
• Time spent in different wards. The length of time in the play spaces varies considerably between wards and patients. In the Accident and Emergency and out-patients space, patients and their parents are likely to be there for around 30 minutes.  A&E gets 39000 patients per year, so this is an incredibly high traffic area. There is an acute receiving ward in which patients spend a shorter time (e.g. 3 days) before going home or being transferred to another ward if necessary. The other wards such as infectious diseases, oncology or neurology are likely to keep their patients for longer (in the surgery ward, we heard of a patient who had been there for 6 weeks, but unfortunately there will be others who are there for longer).On these wards patients can visit the play space supported by the play facilitator or parents as they are able. The play specialist also visits bedsides to take toys and laptops to children who are not able to leave their bedsChildren who have been in the hospital for a long time can get very bored and feel they have played with all the toys already.
• Space. The hospital does not have much free space so the playrooms are small. This will not presumably be so much of an issue in the redesigned buildings
• Internet access.  Internet access is restricted by filters for child protection reasons.
• Potential theft. There is a concern that expensive, mobile equipment such as laptops will get stolen.

Over the next few months, I will be blogging progress on the project. I’d love to have your ideas. If you have any design ideas or tips from similar projects, please leave a comment at the bottom of the page. If you’d like to support the Sick Kids Friends foundation, why not visit http://jackdrawsanything.com/ and get six year old Jack to draw at your command!

Better

I have been rereading Atul Gawadne's book "Better", mostly because I have been thinking about how we can improve teaching in my department since I became a course director. Here's an inspiring quote. Betterment is: "to live a life of responsibility. The question... is not whether one accepts the responsibility. Just by doing this work, one has. The question is, having accepted the responsbility, how does one do such work well?"p9. Gawadne suggests there are three aspects to making things better: diligence ("giving sufficient attention to detail to avoid error and prevail against obstacles"), doing right, and ingenuity (thinking anew). He also has some suggestions for how you can become a positive deviant (i.e. striving to make things better even if the people around you don't). 

1. Ask an unscripted question (when talking to people)

2. Don't complain

3. Count something (i.e. monitor something which interests you numerically)

4. Write something (to reflect on what you have counted)

5. Change

This is me doing number 4 - writing!

I find the quote above striking because even although it was written by a doctor about medical practice, it applies nicely to teaching. I believe that as lecturers, we have signed up to educate and that we can always do better. Even if you get great course feedback and high averages and win teaching awards, there will always be something which you can improve on. Having spent 6 months being a course director and talking to both staff and students about teaching, I am beginning to see that not everyone sees things that way. I think that maybe some staff see the pursuit of betterment only as a criticism, not something which is just built into the job.Maybe this is a "if it ain't broke, don't fix it" attitude. But my point of view is that you have to check whether it is broken regularly, and try to keep it in good working order! It's part of the cycle of reflective practice, planning, teaching, reflecting, then replanning. Trying new teaching methods or content from time to time will make things more interesting for you, and might well make things even better for the students.

In being positively deviant, I am also doing numbers 3. and 5. from Gawadne's list. The counting is to do with the dreaded national student survey results - logging students' attitudes towards their teaching and seeing if in the long term we can increase scores. I am not convinced this is an end in itself, but it is a benchmark for us to use to see whether our changes are actually making things better. Number 5 - change- is hard, it turns out. Universities tend to be conservative beurocracies which work on long time scales. It is hard to achieve anything within one academic year simply because of when meetings at which approval decisions can be taken are scheduled. Staff and students are both busy and busy people tend not to want to make changes which will cost them time. It is hard to coax people out of the habit of complaining and into the more positive frame of mind where they will make constructive suggestions.

But I am trying, and I will keep trying.

The way forward for CS Education in the UK

[cross posted from my CACM blog but the links are screwed up for some reason]

The last week has seen some interesting developments in Computer Science Education in the UK. Or at least, there has been some public hand-wringing about the poor state of computer science education, some sensible proposals for what could be done about it, and some government promises to dramatically overhaul the school ICT curriculum. This could be enormously beneficial for computer science, assuming the momentum is kept up. The hand-wringing is along the lines of “We used to do this so well. Look at Alan Turing” and “Think back to the glory days of the BBC Micro project in the ‘80s”. And then there is a public shaming of the geeks in the nation by the CEO of Google who rapped us on the knuckles, and admonished us for “throwing away” our “great computer heritage”. The Guardian newspaper has embarked on a Digital Literacy Campaign which has a comprehensive selection of articles from kids, politicians, teachers, parents, academics and business people all bemoaning the state of computing education in schools. All this does seem to have had an effect; the Secretary for Education (Michael Gove) announced last week that the current ICT curriculum would be scrapped, and more high quality qualifications in CS developed with freedom for schools to develop their own curricula. Fortunately, the Royal Society published a report on Fridaywhich brings reason, evidence and considered recommendations to the well intentioned but confused discussions. It clarifies the terminology, distinguishing between “Computing”, “ICT”, “Computer Science”, “Information Technology” and “Digital Literacy”. It recommends that school children in the UK should be educated in digital literacy; in the same way as they become fluent in reading and writing text, they should become fluent in using computers confidently. The existing ICT curriculum attempted to do this, albeit in a boring, simplistic and out of date way. A real contribution of the Royal Society report is to argue that children should also have the opportunity to study Computer Science as a rigorous academic discipline and that the curricula and qualifications should be restructured to reflect this. The report recommends that the shortage of specialist Computer Science teachers should be addressed (alarmingly, in England, 66% of teachers of ICT are not considered as qualified by the Department of Education), and that teachers should have greater access to continuing professional development, perhaps offered through industry sponsorship. Resources should also be improved in terms of increased access to software and hardware, and a loosening of network security restrictions which currently hamper access to online materials. Extra-curricular computing activities should also be encouraged. Less cumbersome assessment methods should be developed. So maybe - just maybe- the much lamented golden age of UK computing can return. If we follow these recommendations, our schools will be full of little Alan Turings and Ada Lovelaces all busy in their sand pits exploring the fundamental nature of computation.

Positive interdependence with opal fruits

It's been a while since I blogged about teaching, but for once I find myself with a little time so I shall use it to think about a class I taught today. I am teaching a fourth year/MSC course which requires a lot of group work which the students hate. Last year I had a weekly litany of studenty woes, with people weeping in my office regular as clockwork. People get frustrated about their group members for a variety of reasons, but it is a valuable skill for our graduates to have, so we can't just leave them to stew in their own isolated juices (what a curious metaphor). So how to help the students manage group interactions better?

I was reminded of the work on co-operative learning recently when I heard a teacher give a talk about it. If you plan co-operative learning carefully through structuring tasks in the right way you can promote positive interdependence. That is, the group sinks or swims together. If you make the task difficult or effortful enough the students realise they can't do it by themselves. They can only do it if everyone contributes, and so they have a vested interest in helping other group members succeed. There are various structured exercises around this (and a whole nice set of teaching patterns ). The one I used today was Formulate-Listen-Share-Create. I used it for an interface design task. The students individually formulated their own design ideas, then listened as group members presented them in turn and shared their ideas. Then they created a design as a group based on everyone's input. I insisted on the individual work first to ensure more equal participation in the subsequent discussion. During the discussion I gave each group member 5 opal fruits (sweets/candy for those without a sweet-toothed childhood in the UK). Every time a student took a turn to speak, they paid one of their opal fruits into the centre of the table. Once their 5 opal fruits ran out they were meant to remain silent until all the other group members had used their 5 sweets. At that point they could eat the sweets and talk as much or as little as they wanted.

I think this worked quite well for a group forming phase. It made them conscious of their input to the group but in a fun way (and they could laugh at me for coming up with such a stupid idea which is always good). I found it extremely useful because I could see at a glance how quickly people were exchanging ideas (sweets used up quickly), or if individuals talked too much (sweets used up quickly), or if some people were shy (5 sweets left when everyone else had finished). It's a good visual representation of group process for the instructor. The idea is that the other students encourage the people with sweets left to join in. This worked to some extent, although I was slightly concerned about the pressure it put on very shy people.At any rate, the groups did seem to work well together for the most part even although they had not all worked together before. Watch this space for further tales of(hopefully not pathalogical) group work later in the semester. 

We needn't be malevolent grumps in 2012

[cross posted from my CACM blog]

A few months back, Bertrand Meyer wrote about the nastiness problem in computer science, questioning whether we as reviewers are “malevolent grumps”. Judging by the user comments on the page, this hit a nerve with readers who were the victims of such grumpiness!  Jeanette Wing then followed up on this with some numbers from NSF grant rejections which did indeed indicate that computer scientists are hyper-critical. Much as I enjoy the colourful phrasing, I feel that a field full of malevolent grumps is not something we should simply accept. In fact, even if there are only a few grumps out there, it’s in all our interests to civilise them.

So what can computer scientists do to reduce the nastiness problem when reviewing? Reviewers, authors, programme committee members, conference chairs and journal editors can all do their bit by simply refusing to tolerate discourtesy. Let’s embrace the “no asshole” rule: we no longer ignore bad behaviour. As reviewers, we can aim to be polite (yet stringent) ourselves but also to point out to co-reviewers if we find their impoliteness unacceptable. As authors, we don’t have to accept a rude review and just lie down to lick our wounds. We can (politely!) raise the issue of rudeness with the programme chair or editor so it is less likely to occur in the future. As editors, chairs and programme committee members we can include the issue of courtesy in the reviewing guidelines and be firm about requesting reviewers to moderate their tone if we notice inappropriate remarks.

One of the first steps is to separate intellectual rigour from discourtesy. It is possible to be critical without being rude or dismissive.  We can maintain standards in the field without resorting to ill-natured comments. (Believe it or not, it is also possible to ask genuine questions at a conference without seeking to show off one’s own intellectual chops, but that is another matter). The purpose of reviewing, in my view, is to help an author improve their work, not to crush them under the weight of your own cleverness. It’s not the author’s fault that you had a bad day, or that some other reviewer just rejected your own paper. 

Of course, there are some pockets of good reviewing practice within the field which we can draw on. I am sure there are many, but I have chosen CHI because I have been writing for it recently. The CHI conference is one of the biggest well respected annual human computer interaction conferences. Last year there were 2000 attendees from 38 countries. This year there were 1577 paper submissions with a 23% acceptance rate.  This was the first year I submitted papers to it, and I have been impressed by the quality of the reviews in terms of their fairness, constructiveness and level of detail. They contained greater insight and intellectual oomph than the reviews I had from a high impact journal recently. For one of my CHI submissions, the reviewers did not agree with the paper on some points – it is on a controversial topic- but they still offered suggestions for how to resolve these issues rather than simply rejecting the paper. Was I just lucky in the reviewers I was allocated? Possibly, but the CHI reviewing process has some interesting features built in to maintain review quality*.

1. In the guidelines for reviewers, courtesy is explicitly mentioned: “please be polite to authors. Even if you rate a paper poorly, you can critique it in a positive voice. As part of polite reviewing practice, you should always state what is good about a paper first, followed by your criticisms. If possible, you should offer suggestions for improvement along with your criticism.” 

2. Authors can select both the sub-committee and the contribution type for a paper, which maximises the chance that the paper will end up with reviewers with appropriate expertise , and that the reviewers will use criteria appropriate to the paper when assessing its suitability (e.g. not insisting on empirical evidence for a theoretical contribution).
3.  The reviewing process is thorough and has several opportunities for unfairness or discourtesy to be weeded out. Each paper is blind reviewed by three or more experts, and then an associate chair writes a meta-review to summarise the assessment of the paper, and what action (if any) should be taken to improve it. In this way, individual grumpiness is moderated. A variant of this good practice from other conferences is when reviewers of the same paper can see each other’s reviews (once they have submitted their own), thus introducing peer pressure not to be awful.
4.  Authors have a right to reply by writing a rebuttal of the review. The rebuttal is taken into account along with a revised meta-review (and potentially revised individual reviews) at a two day committee meeting when final accept/reject decisions are made.
5. All submitting authors are surveyed about their opinions of the reviewing process – yet another chance to raise issues about unfairness or discourtesy which have not been addressed in a rebuttal.
6. This point is more about the nature of the conference itself, rather than the reviewing procedures. Because CHI is so interdisciplinary, participants have a wide range of backgrounds from art and design to hard core engineering. They are therefore exposed to – and may in fact seek out- different perspectives which may make them open to different paradigms as reviewers. Could colleagues from the arts and social sciences be having a civilising influence on the grumpy computer scientists?

Ts This is a fairly heavy weight process, but if conference organisers adopted even just one more of the practices from points 1-5, or if journal editors added a courtesy clause to their review instructions the world would be a slightly better place. Make it your New Year's resolution to stop tolerating malevolvent grumps in 2012!

 

*Thanks to Tom Erickson – the person who runs the CHI author survey – for kindly raising some of these points.

 

To my nemesis

Oh reviewer, did you realise that you wrote 13 pages of review without raising any substantive issues? Your enthusiasm is not matched by your insight.