Koji Yatani (http://yatani.jp)
This wiki was initially started as my personal note of statistical methods commonly used in HCI research, but I decided to make it public and put more content in it because I think this may be useful for some of you (particularly if you use R). I will also put some codes for R, so you can quickly apply the methods to your data. This wiki does not emphasize mathematical aspects of statistics much, and rather tries to provide some intuitions of them. Thus, if you know maths, you may be unhappy about this wiki, but this is the way this wiki exists.
Keep in mind that I am not an expert of statistics. The contents provided here is basically what I learned from my experience of HCI research and by reading different online/offline materials. I always double-check the content before posting, but it still may be not 100% accurate or even wrong. So, use the contents in this website at your discretion. I own no responsibility on any kind of consequences, such as you have done wrong analysis after reading my wiki or your papers do not get into a conference or a journal, or your adviser doesn't like your analysis.
I also strongly recommend you to get the second opinion on your analysis from other kinds of resource before you really run a test. If you have found any factual errors, please leave your comment. Your comments would be greatly appreciated. You can leave your comments anonymously or with your name. I don't care about the anonymity or non-anonymity (I don't consider you as a coward even if you don't put your name with your comment), but you might be formally acknowledged in this wiki at some point if you leave your name. I will update the content at some point (depending on how serious the problem is and how busy I am…)
In this website, I use R to show some examples of how you can run statistical tests. I assume you can install R in your machine, and you know some basics of how to write a code in R, install packages, etc. You can read the online manual for R. So, please avoid asking how to use R or that kind of stuff in this website. This website is not intended to be a wiki or forum for R.
This wiki (it's not a wiki like Wikipedia because only I will edit the content to avoid spams and confusions, but you can leave comments) initially started as my personal notes of statistics for HCI research, but I decided to make them public because some pieces of information here may be useful to others who are doing HCI research. This wiki doesn't really explain mathematical details of each statistical method. Rather, this wiki's intention is to help you choose which statistical tests to use, how to use them, how to interpret the results, and how to report them. I also talk about some experimental designs as well because they are closely related to how you analyze the data.
Another reason I decided to make this public is there isn't really a good training of statistics for HCI people and isn't a good place to collect the knowledge of statistics for HCI research. People from psychology and bio-related fields usually have strong statistics trainings, so it seems that they have fewer problems than us. But it's not uncommon that HCI researchers have not gone through those trainings. And thus we see papers do different statistical tests in different ways, some of which look kind of questionable. I hope this wiki will provide a better understanding of statistics for HCI research, and help HCI researchers execute statistics more appropriately.
The third reason is that there are some criticisms on Null Hypothesis Significance Tests (NHST) raised by researchers in other fields and they are proposing to address problems that NHST have (see more details in the page about Null Hypothesis Significance Testing. However, such dicussions are not visible yet in the field of HCI, and I thought it is a good time to think about what we should do. For this, I explain two approaches: effect sizes and regression analysis. Effect sizes can complement some information that the results of an NHST cannot provide. Regression analysis can provide a deeper understanding on your data. Furthermore, these two are (more or less) already supported in R. Currently, this wiki is relatively focusing on NHST and effect sizes, but I will be going to put more contents about regression analysis.
There are different kinds of statistical software, such as SPSS and SAS. In this wiki, I use R. Why R, instead of SPSS or SAS? Well, I simply like R. :) It is a great tool and it is open-source and free. And there is a huge collection of packages for various kinds of statistical methods. You can create very cool graphs and run machine learning techniques as well as statistical tests. I am sure that you will be comfortable with using R if you have some programming experience (particularly in Matlab, Mathematica or Python). But it is a little harder to use, and not necessarily be well-documented for HCI research. And there are some differences between SPSS and R (probably SAS and R too). I wanted to explain how to use R appropriately in the context of HCI research.
Here is a report about some comparisons between R and other stats software, which also explains some advantages of using R. R Relative to Statistical Packages: Comment 1 on Technical Report Number 1 (Version 1.0) Strategically using General Purpose Statistics Packages: A Look at Stata, SAS and SPSS. by Patrick Burns
But I also would like to say that you should use the software which you feel the most comfortable with using. For most of us (HCI researchers), we use statistical tests for analysis, rather develop new statistical methods or implement them in particular software. So, if you already know how to use SPSS, for example, you don't have to switch to R. Some of the content in this wiki should be useful for users of other software. But if you don't have a preference on statistical software and you like programming, just try R! You don't have to pay anything, which is one of the best parts of R. :)
If you are an SPSS or JMP user, you should also check out Jake Wobbrock's Practical Statistics for HCI. It is very comprehensive, and lots of examples for SPSS and JMP are available. I always admire Jake's effort on improving statistical analysis in our field.
Statistics is really powerful and in most of the cases, you can find some kinds of statistical tests you can do for your data. Thus, people think about statistics after they have done the experiment, but I would not recommend to do so.
My suggestion is that you should think about what statistical tests you will use when you think about the experimental design. Although there are many kinds of statistical tests, some tests requires you to exercise lots of cautions to perform. Thus, it is better to design an experiment so that you can just do a simple or common statistical test. It is also helpful to make the experiment simple for the analysis. A simple experiment generally needs only simple statistical tests. If the experiment you are going to do doesn't look simple or is not commonly done in HCI research, make sure you can run an appropriate statistical test after the experiment.
Another thing you should be careful about is the type of data. Try to make your dependent variable ratio or interval. This allows you to do a much wider variety of statistical tests than ordinal and nominal data. If you cannot make it ratio or interval, think about making it ordinal. If this is not possible either, you have to give up and have nominal data, but make sure that you can do an appropriate analysis on them, and can test what you want to examine after the experiment. You can see more details about the types of data here.
There is a great webpage to identify what kind of statistical tests you want to use. The following table is made based on that website. Although this table works well for many cases (I think), keep in mind that you need to double-check whether there is any more appropriate statistical test for your data than what this table suggests.
| Interval/Ratio (Normality assumed) | Interval/Ratio (Normality not assumed), Ordinal | Dichotomy (Binomial) | |
|---|---|---|---|
| Compare two unpaired groups | Unpaired t test | Mann-Whitney test | Fisher's test |
| Compare two paired groups | Paired t test | Wilcoxon test | McNemar's test |
| Compare more than two unmatched groups | ANOVA | Kruskal-Wallis test | Chi-square test |
| Compare more than two matched groups | Repeated-measures ANOVA | Friedman test | Cochran's Q test |
| Find relationship between two variables | Pearson correlation | Spearman correlation | Cramer's V |
| Predict a value with one independent variable | Linear/Non-linear regression | Non-parametric regression | Logistic regression |
| Predict a value with multiple independent variables or binomial variables | Multiple linear/non-linear regression | Multiple logistic regression | |
This wiki is now focusing on regression analysis. In most cases, linear regression or multi-level linear regression is the first thing you should try, but here is a simple decision tree (or a decision bullet point?) for regression analysis.
There is a paper which talks about statistics in HCI by Paul Cairns. This is one of the motivations why I made this wiki. HCI... not as it should be: Inferential Statistics in HCI Research.
There is a great paper talking about some dangerous aspects of NHST (Null Hypothesis Significant Test). The Insignificance of Statistical Significance Testing by Johnson, Douglas H. Nevertheless, NHST is still the major statistical method used in the field of HCI, and this wiki aims to promote the understanding of NHST and encourage discussions about how this research community should move forward in statistical analysis practices.
This is another paper talking about how we need to be careful to interpret the results of NHST. The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant by Gelman, A., and Stern, H.
I haven't bought any book for R so far. I found that I can get enough information online in most cases, and the R project also offers a nice intro pdf. So, I don't think you really need to buy any book to use R. You can also take a look at the page for some R tips.
My favorite stats book is SPSS survival manual. This is written for SPSS users, but the book explains the procedure of common statistical methods very nicely, and I found it is also useful for R users (particularly if you are not familiar with statistical methods). The style of the explanation in this wiki is kind of similar to that book (although that book explains way better).
If you are interested in regression analysis and don't know much about it, I recommend the following book. Data Analysis Using Regression and Multilevel/Hierarchical Models by Andrew Gelman and Jennifer Hill. I found that the book really nicely explains regression analysis, and I think it is easy to follow even if you don't have strong math background.
Some very useful links related to statistics and R.
If you have any general comments on this wiki, please feel free to leave them here.