User Tools

Site Tools


Cochran's Q test


Simply speaking, Cochran's Q test is a binomial data version of repeated-measure ANOVA or Friedman test. So, you have multiple binomial data (like “yes” or “no” responses), and you want to see whether the ratio of the responses are different across the groups (e.g., methods, software or devices the participant use). Let's say you have data as follows.

Are you using this software? (o=no, 1=yes)
Software ASoftware BSoftware C
User 1101
User 2001
User 3010
User 4011
User 5001
User 6111
User 7001
User 8011
User 9011
User 10011

Now, you want to compare the responses across the kinds of software with Cochran's Q test.

Effect Size

I haven't figured out how to directly calculate the effect size of Cochran's Q test. So, here, similarly to Kruskal-Wallis test, I calculate the effect size for a post-hoc McNemar's test.

R code example

The code for Cochran's Q test is similar to those for ANOVA. First, you need to prepare the data.

Answer <- c(1,0,1,0,0,1,0,1,0,0,1,1,0,0,1,1,1,1,0,0,1,0,1,1,0,1,1,0,1,1) Participant <- factor(c(1,1,1,2,2,2,3,3,3,4,4,4,5,5,5,6,6,6,7,7,7,8,8,8,9,9,9,10,10,10)) Software <- factor(rep(1:3, 10)) data <- data.frame(Participant, Software, Answer)

Then, include coin package, which is necessary for Cochran's Q test. After that, you just need to specify which the independent or dependent variable is with a similar syntax for ANOVA.

library(coin) symmetry_test(Answer ~ factor(Software) | factor(Participant), data = data, teststat = "quad")

Now, you get the result.

Asymptotic General Independence Test data: Answer by factor(Software) (1, 2, 3) stratified by factor(Participant) chi-squared = 8.2222, df = 2, p-value = 0.01639

Thus, you have a significant effect of Software on the responses.

Post-hoc test with McNemar's test

When you find any significant effect, you need to do a post-hoc test as you do for ANOVA or Friedman test. For Cochran's Q test, you can basically run multiple McNemar McNemar's tests and adjust the p values with Bonferroni or Holm correction. Unfortunately, this requires you to do a little bit of work. First you need to make a 2×2 table for each combination of Software.

(Later, I found that you don't need to make a 2×2 table to do McNemar's test. For more details, please see below.)

Software 2
Software 1yes11
Software 3
Software 1yes20
Software 3
Software 2yes51

And run a McNemar's test for each 2×2 table.

data1 <- matrix(c(1, 1, 5, 3), ncol=2, byrow=T) mcnemar.test(data1) McNemar's Chi-squared test with continuity correction data: data1 McNemar's chi-squared = 1.5, df = 1, p-value = 0.2207 data2 <- matrix(c(2, 0, 7, 1), ncol=2, byrow=T) mcnemar.test(data2) McNemar's Chi-squared test with continuity correction data: data2 McNemar's chi-squared = 5.1429, df = 1, p-value = 0.0233 data3 <- matrix(c(5, 1, 4, 0), ncol=2, byrow=T) mcnemar.test(data3) McNemar's Chi-squared test with continuity correction data: data3 McNemar's chi-squared = 0.8, df = 1, p-value = 0.3711

Then, adjust the p value. Here we use the Bonferroni correction.

p <- c(0.2207, 0.0233, 0.3711) p.adjust(p, method="bonferroni") 0.6621 0.0699 1.0000

Thus, there is a significant difference between Software1 and Software3. McNemar's test in SPSS uses the binomial distribution, which often makes the results different from those you can get in R. For more details, see the code example for McNemar's test. Some versions of SPSS applies Cochran's Q test to each pair combination of the groups. This is also legitimate because Cochran's Q test becomes equivalent to Mcnemar's test without continuity correction when it applies to the data for two groups.

Finally, calculate the effect size for the difference between Software1 and Software3. Again, I am not 100% sure that this is the best way to calculate the effect size for Cochran's Q test, but I just followed the way we do for Kruskal-Wallis test.

sqrt(5.1429 / 20) 0.5070947

Please note that we need to use the total sample size (10 * 2 = 20) to calculate the effect size . Another way to do McNemar's test is to just take the data from the dataframe.

mcnemar.test(data[data$Software==1,]$Answer, data[data$Software==2,]$Answer) McNemar's Chi-squared test with continuity correction data: data[data$Software == 1, ]$Answer and data[data$Software == 2, ]$Answer McNemar's chi-squared = 1.5, df = 1, p-value = 0.2207

Make sure that an element in the same index in the two vectors for the test represents the data taken from the same participants (or the data considering the within-subject factor).

You may have some cases where you see a significant difference with Cochran's Q test, but don't see any difference with the post-hoc comparison. It is probably because Cochran's Q test may be too strong or a post-hoc comparison may be too strict (or the sample size may be too small). But please note that this can happen fairly often.

How to report

You can report the results of Cochran's Q test as follows: With a Cochran's Q test, we found that there exists a significant difference in usage among the three kinds of software we surveyed ((2) = 8.22, p < 0.05). A pairwise comparison using continuity-corrected McNemar's tests with Bonferroni correction revealed that significantly more participants used Software3 than Software1 (p<0.1, = 0.51).


Nik, 2015/10/31 18:09
I don't really sure if your explanation is accurate. Both Cochran and McNemar are "before-and-after" within-subject tests. While your example with Software A, B, C is acceptable ("event" is a "change of software"), example with McNemar is unclear. First, how you came up from software A,B,C to 1,2,3. Second, in case of simple contingency table the chi-squared test would be more appropriate. In your sample we don't have prior information about choice of any of two software.
Mangiafico, 2016/07/02 21:17
I think the critique by Nik is not correct. These tests can be used as before-and-after, but can also be used in other cases where the observations are paired in some way (not independent). If we just think about comparing Software 1 to Software 3, we want to know which is more popular. A chi-square test would ask a very different question. It would ask if there is an _association_ between using Software 1 and Software 3. McNemar is the test we want. It asks if there are many more cases in which a respondent uses Software 3 and not Software 1 than cases where a respondent uses Software 1 and not Software 3. If true, this means Software 3 is more popular than Software 1.
EdwardBleva, 2017/01/20 19:00
<a href=>купить детский квадроцикл</a>
FrankieZen, 2017/01/31 11:38
Тайланд. Любые туры в Паттайе. Такси От тайской тур агентства Русалочка. Все описания туров, цены и фотографии можно посмотреть по ссылке Телефон в Таиланде (WhatsApp, Viber, Line, Telegram): +66–861559142
RobertDut, 2017/02/03 09:24
<a href=>mrwhite</a> - buy dumps with pin online, validdumps.
Bruceweerm, 2017/02/04 09:42
good website
RobertKer, 2017/02/08 23:07
cool site <a href=></a>
Michelaborm, 2017/02/13 19:10
Имеется такая услуга - добровольное медицинское обслуживание (или ДМО).
Она предполагает, что пациент вносит небольшую сумму за то, что посещает врачей в течение года БЕСПЛАТНО.
Однако соцопросы показали, что лишь 3% жителей города знают о ее существовании.
Почему так происходит?
Потому что частным клиникам намного выгодней сдирать с людей деньги за каждое посещение.
А если честный врач попытается посоветовать добровольное медицинское обслуживание клиенту - это сулит ему увольнением.
Информация о ДМО уже вызвала много скандалов, сразу после того как информацию об услуге рассекретил один врач.
Его уволили , после того, как он предложил ДМО своему пациенту.
Самое удивительное, что информация по ДМО есть в открытом доступе, просто натыкались на эту информацию единицы.
Как отстоять свои права?
О правилах предоставления такой услуги и обязанностях частных клиник можно узнать, сделав запрос в Яндексе: "добровольное медицинское обслуживание".
Обязательно обслуживание, а не страхование.

Please enter your comment. You cannot remove your comments by yourself. So double-check before you submit.:
If you can't read the letters on the image, download this .wav file to get them read to you.
hcistats/cochran.txt · Last modified: 2014/08/14 05:23 by Koji Yatani

Page Tools