When should I do post hoc comparisons?

It depends on the motivation for the comparison. If the reason you want to compare two means is NOT based on what you saw in the data you collected (e.g. because your theory suggests they should be different), then that should be an a priori comparison. An example of this would be the contrasts we run in the factorial ANOVA example below – the polynomial trend contrasts are based on the assumption that low, medium, and high dosage (which is a naturally ordered categorical variable) will affect scores in an ordered way. On the other hand, if the reason you want to compare two means is because they look like they might be different (you look at the data, and then decide which comparisons to run based on what looks promising), then you should be doing post hoc tests. Post hoc tests feel a little different to run than a priori contrasts: generally, post hoc tests analyze ALL of the possible comparisons rather than picking out just a few. For example, Tukey’s HSD is a post hoc test. To run it, you correct your significance level (you make it more stringent, to correct for the fact that your Type 1 error rate would otherwise be inflated because you’re doing so many comparisons), and then check every pairwise comparison in the dataset to see which ones reach significance.

Also see this post by Nicole.