Power Analysis: What we learned

We had a great time talking through Jenny’s design and thinking about what tests will be the most important in her pilot data. We determined that the comparison between “words” and “part-words” will be the key test for her to decide whether or not each of her two artificial tone “languages” are learnable in her pilot data. Since each subject will respond to both words and part-words, that should be a paired-samples test. The simplest solution is to use the pwr.t.test() function from the pwr package, like this:

pwr.t.test(n=NULL, d=.5, sig.level=.05, power=.8, type="paired", alternative="two.sided")

Effect size for paired t-tests can be a little confusing, with different analysts recommending different denominators for the standardized difference (i.e. cohen’s d). Here’s a little summary on that topic from 611. We learned that the d in this function is the mean difference divided by the standard deviation of the differences (the same d gpower uses for this type of design). To estimate effect size for Jenny’s study, we found two papers with similar-ish designs – an infant study using similar stimuli but a different operationalization of the DV, and an adult study with pretty different stimuli but a similar DV. We pulled out the relevant t-test from each study (a paired t-test comparing participants’ familiarity with “words” to “part-words”), and hacked it into d by dividing by the square root of the number of paired scores. For example…

For t(26) = 3.65, d = 3.65/sqrt(27) = .70

There were a couple experiments with relevant tests reported in each study, and we got a range of d’s from 0.5 to 1.2. Plugging these estimates into pwr.t.test(), we found that Jenny will probably want to run 20-30 participants in each “language” for her pilot study (she wants to be able to test participants from each language separately).  Hooray!


Join us next week (4/7, 3pm Straub 008) for another gorgeous problem! Melissa has a big, complicated dataset from a longitudinal study, and she needs it reformatted from wide to long. “I can do that in one line of code!” you say, “even SPSS can handle that!” you complain, but wait! It’s more fun than that. 🙂 She is missing entire variables at each time point (e.g. measure A is collected at time 1, 2, 3, and 4, but measure B is only collected and 1 and 4), so there are columns “missing” in the wide dataset, causing some otherwise reliable functions (like reshape) to choke. And the dataset has like a hundred variables, so adding the missing columns by hand would be a pain.

Flash the bat signal!!



Post a comment

You may use the following HTML:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>