Computers, creativity and learning

On the nth day of Christmas my true love sent to me

There was apparently a problem with the delivery of the parcels from my love. Days 6 - 12 got backed  up and arrived altogether. These things happen. Just as well they all arrived by Christmas Eve, right?

On the twelfth day of Christmas, my true love sent to me

Latent class model (because he knows how much I enjoy recovering hidden groups from observed data).

Bias variance trade-off (well, we wouldn't want to over fit our models, would we?)

Aligned rank transforms (because no-one loves you if you analyse non-parametric factorial data with ANOVA).

No missing data (this is really important for all it seems unglamorous. Missing data can seriously mess up your interpretation of results).

Time series values (because it's a good idea to track user experience over time rather than in a one off session).

Power calculations  (because you want to be sure to detect an effect if it is there, and if your love has sent you an effect size from a previous experiment you are able to calculate power without any hassle).

Effect size reporting

No p-values

 Clear stopping rules

 Bayes testing

Anscombe's Quartet

And fair stats communication.

 

Merry Christmas One and All!

 

 

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

On the sixth day of Christmas, my true love sent to me: effect size reporting

On the sixth day of Christmas, my true love sent to me: 

Effect size reporting

No p-values

 Clear stopping rules

 Bayes testing

Anscombe's Quartet

And fair stats communication.

 

Suppose my love likes to quantify his affection. Like the "Guess How Much I Love You" bunny, he gives numerical estimates of his love, perhaps in comparison to another suitor. He uses real world measurements to do so (usually units of distance) , rather than standardised measurements of effect size such as Cohen's d. This is a reasonable decision, as the real word units are easier to interpret (and more poetic). I suppose he might use Cohen's d if he wanted to give me context of related effect sizes in the literature or if he was trying to establish the positive feelings of rabbits to a range of species.

But suppose I was trying to estimate the feelings of my true love for me using a similar strategy to Null Hypotheses Significance Testing (NHST). Then I might adopt an arbitrary threshold and make a binary decision based on that. If p < .05, he loves me. If p > .05 he loves me not. I suspect this might not be the path to happiness. 

This is probably not a very helpful way to introduce the concept of effect sizes. I recommend that you read Koji Yatani's chapter on effect sizes instead for a more cogent discussion of the importance of effect sizes and a guide to how to compute them in R.  Here is a rather clearer explanation than  mine, written by Koji in his characteristically friendly style:

 

"Even if you have a significant result, it is “significant” only in the context of statistics. It does not necessarily mean that the difference you have is meaningful or important in a practical setting. Imagine that you are reviewing a very typical interaction technique paper. It compares two techniques: one is what the authors developed, and the other is a conventional technique. Suppose that the performance time of some tasks was improved with a new interaction technique by 1% with the standard deviation of 0.1%. In this case, their NHST would show a significant difference. So should you automatically accept that paper because it shows significant improvements backed up with well-known statistical testing methods? No – it is probably too early to decide the fate of the paper. 1% improvement may be a difference between 10 and 9.9 sec. Would this 100 msec. be really important? Maybe so for fighter pilots, but we definitely need more contextual information to assess the magnitude of this improvement.

Now you are reviewing another paper. In this paper, the performance time was improved with a new technique by 15%, but the standard deviation was also as large as 15%. The results would not show a significant difference because the standard deviation is too large. So would your conclusion be that the new technique is useless? I would say not. 15% improvement in average (e.g., 10 sec vs. 8.5 sec) could have an impact on user experience. There may have been some factors that caused such variances. For example, some participants might have been able to use experience from other interactive systems (e.g., heavy smartphone users might be able to adapt to new touch interaction more quickly than others). Other researchers might come up with an even better technique which can remove that variance while maintaining faster performance. Thus, the developed technique may be important enough to be shared even though the present paper could not find a significant difference. Assuming the paper is of high quality in other aspects, I think that it should be accepted (though I would ask the authors to explain why the standard deviation was so large and how their technique could be improved to make it smaller). Thus, we should not overrate statistically-significant results and underrate non-statistically-significant results. All results should be interpreted in a larger context."

 

In summary, if you are reporting quantitative results, report either standardised effect sizes or means and standard deviations so that the reader can compute the effect sizes if necessary. And if you are reading a paper, take a look at the descriptive statistics before you get too carried away by a significant result.

 

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials

On the fifth day of Christmas, my true love sent to me: No p-values

On the fifth day of Christmas, my true love sent to me: NO P-VALUES

 Clear stopping rules

 Bayes testing

Anscombe's Quartet

And fair stats communication.

 

You may be wondering about the relationship I have with my true love at this point. My husband (to his credit) has never sent me any of the above. It clearly differs according to tastes in different disciplines, as my friend Phil who is a reformed physicist apparently received the "5 sigma rule" from his true love. All I can say is that his love is extremely stringent in these matters.

 But why avoid sending me p-values? If we follow Pierre's  advice from the first day of Christmas, they are not consistent with fair statistical communication, particularly failing on transparency and clarity. Here is an excerpt from our introductory chapter (written by me and Maurits) on why p-values might not mean what you think they mean.

"The NHST approach, fiercely promoted by Fisher in the 1930’s (Fisher, 1934; and also Pearson, Fisher, & Inman, 1994), has become the gold standard in many disciplines including quantitative evaluations in HCI. However, the approach is rather counter-intuitive; and subsequently many researchers misinterpret the meaning of the p-value. To illustrate this point Oakes (1986) posed a series of true/false questions regarding the interpretation of p-vales to seventy experienced researchers and discovered that only two had a sound understanding of the underlying concept of significance. 

So what does a p-value actually mean? “…the p-value is the probability of obtaining the observed value of a sample statistic (such as t, F, ) or a more extreme value if the data were generated from a null-hypothesis population and sampled according to the intention of the experimenter” (Oakes 1986, p.293). Because p-values are based on the idea that probabilities are long run frequencies, they are properties of a collective of events rather than single event. They do not give the probability of a hypothesis being true or false for this particular experiment, they only provide a description of the long term Type I error rate for a class of hypothetical experiments – most of which the researcher has not conducted.

The erroneous interpretation of the p-value by many researchers brings up the first actual threat to the validity of conclusions that are supported by null-hypothesis testing. Researchers often interpret the p-value to quantify the probability that the null hypothesis is true. Thus, a p-value smaller than .05 indicates to large groups of researchers – be it conscious or unconscious –that the probability that the null hypothesis is true (e.g.   = u₀) is very small.

Under this misinterpretation the p-value would quantify P(H₀|D) - the probability of the null hypothesis (H₀) given the data collected in the experiment. However, the correct interpretation of the p-value is rather different: it quantifies P(D|H₀) – the probability of the data given that H₀ is true. Researchers who state that it is very unlikely that the null hypothesis is true based on a low p-value are attributing an incorrect meaning to the p-value.

The difference between P(H₀|D)  and P(D|H₀)  is not merely a statistical quirk that is unlikely to affect anything in practice. Not at all. It is easy to understand why this misconception is incorrect and hugely important by the following example: consider the probability of being dead after being shot, P(D=true|S=true). Most would estimate this to be very high, say 0.99. However, the mere observation of a dead person does not lead most people to believe that the corpse was shot – after all, there are many possible ways to die which don’t involve shooting. P(S=true|D=true) is estimated to be rather small. Luckily, the relationship between P(D|H₀)  and P(H₀|D)  is well-known and given by Bayes rule (Barnard & Bayes, 1958). Thus, if one wishes to estimate P(H₀|D)  - which we often do – the proper analytical tools are at our disposal." (see my gift from the 4th day of Christmas).

I am personally not arguing for a full-out ban on p-values (although one psychology journal has done so). I am arguing for using them carefully and appropriately as part of a wider statistical argument. I would also like reviewers to accept other statistical approaches rather than favouring NHST practices over others.

 

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials

On the fourth day of Christmas, my true love sent to me: Clear stopping rules

On the fourth day of Christmas, my true love sent to me: Clear stopping rules.

 Bayes testing

Anscombe's Quartet

And fair stats communication.

 

Simmon at al's paper about researcher degrees of freedom (such as lax stopping rules for data collection) is a disturbing read. In one of our own chapters, Maurits and I describe this study before going on to recommend some guidelines for authors and reviewers in HCI to avoid such problems. Here is an excerpt:

"Simmons et al. recently introduced the concept of “researcher degrees of freedom” to describe the series of decisions which researchers must make when designing, running, analysing and reporting experiments. The result of ambiguity when making such decisions is that researchers often (without intention of deceit) perform multiple alternative statistical analyses and then choose to report the form of analysis which found statistically significant results. The authors show through a cleverly constructed simulation study that “It is unacceptably easy to publish “statistically significant” evidence consistent with any hypothesis” (Simmons, Nelson & Simonsohn, 2011, p1). The authors illustrate this by reporting evidence in support of the hypothesis that listening to the song “When I’m 64” reduces chronological age.  The simulation study investigated how the significance of the results changed when different forms of analyses were conducted on a simulated data set randomly drawn from the normal distribution, repeated 15000 times. The different forms of analysis manipulated common researcher degrees of freedom: selecting dependant variables, setting sample size, the use of covariates and reporting only selected subsets of experimental conditions. If the researcher is flexible about when she stops data collection – a common practice although it is not at all advocated in the NHST literature – and gathers more data after doing a significance test, the false positive rate increases by 50%. By using covariates, such as the ubiquitous gender factor, the researcher can introduce a false positive rate of 11%, and by reporting only a subset of the data, a false positive rate of 12.6% is produced. Finally, if our researcher was too flexible in all four of these areas, the false positive rate would be 61%.

Simmons et al (2011) recommend a set of ten simple practices which reduce the problem of false positives. Firstly, they recommend that authors should decide in advance on their criteria for when to stop data collection. Given the prevalence of problems of underpowered tests, it would be sensible to use stopping rules based on power analysis. The second guideline of providing at least 20 observations per cell is directly related to power, and reduces the risk of type II errors.  According to guidelines 3 and 4, authors would be required to report all the variables collected, and all experimental conditions investigated even if they are not reported in the final analysis in the paper. This would enable readers to judge how selective authors are in reporting their results. Guideline 5 recommends that if observations (such as outliers) are removed, they should report both the results with and without their removal to enable the reader to determine what effect it has. To reduce the occurrence of false positives by the introduction of covariates, guidelines 6 recommends that results should be reported with and without covariates. As changes in authorial practices require monitoring by reviewers, Simmons et al. (2011) propose four related guidelines for reviewers, starting with ensuring that authors follow the first 6 recommendations. Following these guidelines is likely to lead to publications with fewer significant and less perfect seeming results; reviewers are therefore encouraged to be more tolerant of imperfections in results (but less tolerant of imperfections in methodology!). Reviewers are also asked to check that the authors have demonstrated that their results don’t depend on arbitrary decisions while conducting the analysis, and that the analytic decisions are consistent across studies reported in the same paper.  Lastly, Simmons et al. (2011) invite reviewers to make themselves unpopular by requiring that authors conduct replications of studies where data collection or analysis methods are not compelling."

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials

On the third day of Christmas, my true love sent to me: Bayes testing

On the third day of Christmas, my true love sent to me: Bayes testing

Anscombe's Quartet

And fair stats communication.

 

Michael Tsikerdekis does a fantastic job of explaining the concepts behind Bayesian inference.  If you're new to Bayes theorem and Bayesian approaches to analysis, you may have an Emperor's New Clothes moment where you wonder if it really can be true that scientists have been deluding themselves by industriously computing and reporting p-values even although they don't actually tell them what they want to know. Joris Mulder builds on these concepts to illustrate how to calculate the strength of evidence for competing hypothesis using Bayesian testing. Liberate yourself from the arbitrary, conservative and somewhat absurd null hypothesis! Directly compare hypotheses which have some plausibility! I am very pleased to have these two chapters in the book, because since I first read about Bayesian inference, I wanted to try on my own data sets but I really needed some accessible examples to help me learn. Michael and Joris provide these for others in the same position.

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials

On the second day of Christmas, my true love sent to me: Anscombe’s quartet

 

 

On the second day of Christmas, my true love sent to me: Anscombe's Quartet

And fair stats communication.

 

 One of the first steps in data analysis is plot your data to get an intuitive feel for it. Jo Young and Jan Wessnitzer at the Scientific Editing Company wrote us a chapter about data visualisation in R. I love a good graph (just ask my indulgent co-authors), so I was particularly interested to read their description of  Anscombe's quartet. 

"Anscombe’s quartet consists of four datasets with almost identical^[1] descriptive statistics, including mean (to a minimum of 2 decimal places in the case of y) (¯x = 9.0,y¯ = 7.50), sample variance (SD(x) = 11.0,SD(y) = 4.12), correlation between x and y in each case (0.816 to 3 decimal places), and linear regression line in each case (y = 3.00 + 0.500x, to 2 and 3 decimal places respectively). 

However, by graphing Anscombe’s quartet (see below), it becomes clear that a linear regression is probably a reasonable fit for set 1. However, a polynomial regression fit is more appropriate for set 2. By plotting sets 3 and 4, the effects of outliers on descriptive statistics is clearly demonstrated. In both cases, the fitted regression line is ”skewed” by a single outlier. The outliers could be genuine outliers or they could be erroneous data points, e.g., typos during data entry, but they highlight the importance of screening your data visually."

So, next time you get a net full of fresh data, graph it before you get too carried away with your number crunching!

 [1] to at least two decimal places

 

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials

 

On the second day of Christmas, my true love sent to me: Anscombe's Quartet

And fair stats communication.

 

 One of the first steps in data analysis is plot your data to get an intuitive feel for it. Jo Young and Jan Wessnitzer at the Scientific Editing Company wrote us a chapter about data visualisation in R. I love a good graph (just ask my indulgent co-authors), so I was particularly interested to read their description of  Anscombe's quartet. 

"Anscombe’s quartet consists of four datasets with almost identical^[1] descriptive statistics, including mean (to a minimum of 2 decimal places in the case of y) (¯x = 9.0,y¯ = 7.50), sample variance (SD(x) = 11.0,SD(y) = 4.12), correlation between x and y in each case (0.816 to 3 decimal places), and linear regression line in each case (y = 3.00 + 0.500x, to 2 and 3 decimal places respectively). 

However, by graphing Anscombe’s quartet (see below), it becomes clear that a linear regression is probably a reasonable fit for set 1. However, a polynomial regression fit is more appropriate for set 2. By plotting sets 3 and 4, the effects of outliers on descriptive statistics is clearly demonstrated. In both cases, the fitted regression line is ”skewed” by a single outlier. The outliers could be genuine outliers or they could be erroneous data points, e.g., typos during data entry, but they highlight the importance of screening your data visually."

So, next time you get a net full of fresh data, graph it before you get too carried away with your number crunching!

 [1] to at least two decimal places

 

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials

On the first day of Christmas, my true love sent to me: fair stats communication

In celebration of the present giving season, and because I am procrastinating reading final proofs of our new book, I bring you the Twelve Days of Christmas in statistical style!

 Maurits and I started discussing statistical methods in HCI with our paper for CHI 2012 "Rethinking Statistical Analysis Methods for CHI". Several years later, we are in the final stages of editing a book on the topic which draws together contributions from an amazing set of generous and gifted authors from within and beyond the HCI community. I'll post some concepts from the book in a series of short articles, one for each day of the season. (Yes, I know the twelve days start from Christmas and work to 6th Jan but who in their right mind would read a post about statistical methods on Christmas day?) The selection of topics is mostly related to whether I could torture the syllables into the song, so I wouldn't call it a balanced treatment of the book. (There is a bias against the most advanced topics because for some reason there are more syllables in the names of the trickier techniques. Show offs...) 

On the first day of Christmas, my true love sent to me: fair stats communications.

In one of the last chapters in the book, Pierre Dragicevic argues that researchers should aim for fair statistical communication when presenting results:   "While we cannot be fully objective when writing a study report, we can give our readers the freedom to decide whether or not they should trust our interpretations... this is the essence of fair statistical communication. " He describes the principles of fair statistical communications as: 

• Clarity
• Transparency
• Simplicity
• Robustness
• Unconditionality
• Precision

Don't get hung up on whether you have followed the magical Null Hypothesis Significance Testing traditions which have evolved in your field.  Let Pierre  persuade you to consider using an estimation approach, following his comprehensive series of tips to help to make your statistical communication fair.

 

Modern Statistical Methods in Human Computer Interaction (edited by Judy Robertson and Maurits Kaptein) will be published by Springer in early 2016. Here is the table of contents:

 

 

 

 

Modern Statistical Methods for HCI

Preface

1. An introduction to Modern Statistical Methods for HCI.
J. Robertson & M.C. Kaptein

Section 1: Getting Started With Data Analysis.

2. Getting started with [R]: a brief introduction
L. Ippel.

3. Descriptive statistics, Graphs, and Visualization.
J. Young & J. Wessnitzer

4. Handling missing data
T. Baguley & M. Andrews

Section 2: Classical Null Hypothesis Significance Testing done properly

5. Effect sizes and Power in HCI
K. Yatani

6. Using R for repeated and time-series observations
D. Fry & K. Wazny

7. Non-parametric Statistics in Human-Computer Interaction
J.O. Wobbrock and M. Kay

Section 3: Bayesian Inference

8. Bayesian Inference
M. Tsikerdekis

9. Bayesian Testing of Constrained Hypothesis
J. Mulder
 

Section 4: Advanced modeling in HCI

10. Latent Variable Models
A. Beaujean & G. Morgan

11. Using Generalized Linear (Mixed) Models in HCI
M.C. Kaptein

12. Mixture models: Latent profile and latent class analysis
D. Oberski

Section 5: Improving statistical practice in HCI

13. Fair Statistical Communication in HCI
P. Dragicevic

14. Improving statistical practice in HCI
J. Robertson & M.C. Kaptein

Online supplementary materials