Previous studies suggest that researchers tend to prefer one therapy orientation and that this preference biases the way they conduct research, the outcomes of research, and the interpretation of findings. This phenomenon is called the researcher allegiance effect. When researchers conduct studies comparing the efficacy of two treatments, they are more likely to find that their preferred treatment is more effective than the comparison treatment. In one meta-analysis, the researcher allegiance effect accounted for 12% of the difference between treatments, which is substantial. However, there is some controversy about how to understand this effect. On the one hand, it is possible that this simply reflects the tendency of researchers to have a stronger allegiance to a more effective treatment. Still, on the other hand, it could reflect a bias in which researchers consciously or unconsciously stack the deck in favour of their preferred therapy. Zimmerman and Marcus uniquely approached the question in this study to determine what drives this researcher allegiance effect. They found 19 pairs of studies (38 studies in all) in which they identified the researcher's allegiance to one of two different therapies. In each pair of studies, one research team favoured therapy A over therapy B, while in the paired study, another research team favoured therapy B over therapy A. Each paired study treated the same disorder for the same length of time. Zimmerman and Marcus then assessed if the outcomes in the paired studies differed and by how much. This approach allowed them to assess if studies of the same treatments for the same disorder had different outcomes due to different researcher allegiances. On average, for all 19 pairs, the difference was small to medium and statistically significant, ddiff = 0.35 (95% CI = 0.08–0.61; Z = 2.59, p = 0.001). The authors also rated the degree of researcher allegiance (how strongly the researcher identified with a particular therapy orientation). When they looked only at the 7 pairs of studies with the strongest researcher allegiance, the effect was notably larger and statistically significant, ddiff = 0.69 (95% CI = 0.07–0.1.31; Z = 2.19, p = 0.029). In other words, the more a researcher identified with a treatment orientation, the larger the allegiance effect favouring their preferred therapy. Overall, 22 of the 38 studies found no difference between the two treatments that were compared.
Several factors could be the cause of the researcher’s allegiance effect. Researchers may be more skilled at providing (or supervising) the treatment they are allied with, may subtly express greater enthusiasm for their preferred treatment, or may select measures that are more sensitive to the outcomes of their preferred treatment. Researchers may also “spin” their findings to suggest that their preferred treatment is more effective, thus promoting conclusions that are wholly or partly inconsistent with the results. Clinicians must be cautious and skeptical when they read the results of a single study or the interpretation of a study from a secondary clinical source in which the authors claim that one treatment is superior to another. Such studies and their interpretations are inconsistent with the many meta-analyses showing few or no differences between time-limited manualized treatments for most disorders.
Zimmerman, J.A. & Marcus, D.K. (2024). Does researcher allegiance bias outcomes in psychotherapy research? A quasi-experimental secondary analysis. Clinical Psychology and Psychotherapy, 31, e2973. https://doi.org/10.1002/cpp.2974
To learn more about the Psychotherapy Practice Research Network, visit www.pprnet.ca.