I quoted Nickerson in my prior post on confirmation bias. His article is quite good and can be reviewed here. I like this quote in the article attributed to Francis Bacon:
The human understanding when it has once adopted an opinion (either as being the received opinion or as being agreeable to itself) draws all things else to support and agree with it. And though there be a greater number and weight of instances to be found on the other side, yet these it either neglects and despises, or else by some distinction sets aside and rejects; in order that by this great and pernicious predetermination the authority of its former conclusions may remain inviolate.. . . And such is the way of all superstitions, whether in astrology, dreams, omens, divine judgments, or the like; wherein men, having a delight in such vanities, mark the events where they are fulfilled, but where they fail, although this happened much oftener, neglect and pass them by. (p. 36)
Nickerson then outlines several types of confirmation bias:
-Restriction of attention to a favored hypothesis.
-Preferential treatment of evidence supporting existing beliefs.
-Looking only or primarily for positive cases.
-Overweighting positive confirmatory instances.
Recently, several readers asked me about a report on the NARTH website claiming that reorientation therapy reduced suicide attempts. In reviewing the claim, it appears to me to be an example of confirmation bias. However, before I discuss it, I want to assert that I believe confirmation bias is common to humans. For reasons I will lay out in future posts, I believe cognitive activity serves (at least) to simplify complexity, create a sense of predictability to the world, and to justify investments of time and energy – in this case mental time and energy. I am not above it, nor do I believe anyone to be. I do think we can help prevent and/or correct errors by being aware of it.
It is no secret that I think reparative therapists who believe there is only one path to same-sex attraction engage in confirmation bias. Another recent instance from NARTH is the use of a study by Shidlo and Schroeder to make a claim that reorientation therapy reduces suicide risk. President-elect, Julie Hamilton, in her report from the 2008 NARTH conference, wrote:
Regarding the claims that reorientation therapy harms clients, Dr. Whitehead cited studies that found suicide rates decrease after therapy. In fact, he pointed out that Shidlo and Schroeder (2002) sought to prove the adverse effects of therapy by collecting stories of harm; however, instead of finding therapy to be harmful, they found it to be helpful, in that suicide attempts by these clients actually decreased after therapy. For more information on the content and references for Dr. Whitehead’s keynote address, see the NARTH Collected Convention Papers or soon-to-be-released book, What the Research Shows: NARTH’s Response to the APA Claims on Homosexuality.
First, this is misleading because the way it is worded, it sounds as though Shidlo and Schroeder found and reported something they did not intend to find. More relevant to this post, however, is Dr. Hamilton’s reference to an analysis by Dr. Neil Whitehead, bio-chemist with numerous scientific publications including some on sexual orientation. Neil often provides interesting perspectives so I was surprised to see him quoted in this context. When I asked Neil about the claim, he said he reanalyzed the reports of suicide from Shidlo and Schroeder’s paper and stands by it. While I have not seen the reanalysis, I don’t need to in order to know that a relationship between reorientation and suicidality cannot be inferred from an analysis of Shidlo and Schroeder. Even so, Neil stunned me by saying that his analysis did not reach statistical significance but revealed a non-significant trend for reorientation therapy to reduce suicidality among same-sex attracted people. On that basis, he made his claim which was amplified by Dr. Hamilton.
Here is what Shidlo and Schroeder reported about their participants’ suicide attempts.
In examining the data, we distinguished between participants who had a history of being suicidal before conversion therapy and those who did not. Twenty-five participants had a history of suicide attempts before conversion therapy, 23 during conversion therapy, and 11 after conversion therapy. We took the subgroup of participants who reported suicide attempts and looked at suicide attempts pre-intervention, during intervention, and post-intervention to see if there was any suggestive pattern. We found that 11 participants had reported suicide attempts since the end of conversion interventions. Of these, only 3 had attempted prior to conversion therapy. Of the 11 participants, 3 had attempted during conversion therapy.
I am guessing that Neil is taking the 25 and 23 people who reported attempts before and during intervention as being helped by therapy since they apparently (although the numbers may overlap and are not clear) reported no suicide attempts after therapy. The 11 after therapy are perhaps conceded as a minority of clients with an adverse reaction. Since I am not sure, I won’t knock down what might a straw man of my making. However, what seems clear is that whatever effect may have occured, Neil and by extension Dr. Hamilton, assumes it to be a positive benefit from the therapy. However, this seems to me to be a biased attribution with at least one other explanation. Perhaps these people were not suicidal after conversion therapy because they went to a support group for conversion therapy survivors. Perhaps, a fuller examination would find that people are alive today despite the therapy not because of it.
If anything, these reports do not seem favorable to reorientation therapy. Anyone can play with numbers. I could take the 23 plus 11 and come up with a 16.8% (34/202) probability of adverse consequences due to reorientation efforts. However, these reports cannot be the basis for any statements about the general impact of reorientation efforts on suicidality. About all we can say is that some people reported feeling worse due to their reorientation experiences. For at least some same-sex attracted clients, the experience was not benign but was associated with a worsening of their distress. Ordinarily, in absence of prospective studies, professionals should inform their clients of such reports to give clients ability to consent to care. But any general statement of efficacy or probability with regard to suicidality would require a specific study to test that hypothesis.
A study that would permit the statements made by Dr. Hamilton would require a prospective design with follow up and with a control group of people who did not received reorientation therapy but some other appropriate intervention. At the least, a waiting list control group would be required. The prospective nature of the study is crucial to capture not only suicide attempts but any completed suicides which occured during the course of the interventions or thereafter (during the follow up aspect of the study).
Shidlo and Schroeder’s design does not permit any general probablity statement. Just prior to reporting these findings, Shidlo & Schroeder said the numbers should not be viewed as complete or representative of the actual degree of harm:
After participants’ responses to the open-ended question, we followed up with a checklist of symptom areas (self-blame for not trying hard enough to change, self-esteem, depression, difficulties with intimacy, social isolation, loneliness, self-harmful behavior, suicidal thoughts, suicide attempts, feeling paranoid, self-monitoring behavior for “homosexual mannerisms,” and alcohol and substance abuse) and asked them to tell us whether they noticed negative changes in these areas. This symptom checklist was developed in our pilot interviews.
We do not report here on the frequency of responses to these items because of two methodological limitations. First, because we emphasized breadth of inquiry and yet were constrained to keep the interview within a reasonable time limit (approximately 90 min), we used single items for each domain of functioning; this methodological decision came at the expense of sensitivity, reliability, and content and construct validity. Second, participants who felt harmed and unhappy about their therapy experience may have answered affirmatively to a deterioration in a particular area and attributed it to the conversion therapy because of a negative halo-effect or narrative smoothing (Rhodes et al., 1994) rather than having provided an accurate recollection of actual change in that particular area. Thus, instead of using the checklist as a quantitative measure of negative effects, we used these items as qualitative interview-prompts to help respondents explore areas of deterioration. Our results, therefore, focus on the meanings of harm attributed by clients, and the accuracy of these attributions remains to be determined by future process-and-outcome research.
Even though Shidlo and Schroeder have their own confirmation bias issues in this study, here they take a cautious approach. Perhaps, the halo-effect colored the recollections negatively; perhaps some people blocked out suicidal thinking. Without a prospective study with a control group, these numbers tells us nothing reliable about the matter at issue: whether reorientation therapy reduces, enhances, or has no effect on suicidality for the population of people who are inclined to seek it.
Furthermore, as Shidlo and Schroeder note, the actual numbers of attempts of episodes may not be accurate. These were retrospective accounts. It is quite possible that some suicide attempts were not reported to Shidlo and Schroeder.
It seems to me that NARTH’s use of Shidlo and Schroeder illustrates points 2 and 4 above (“Preferential treatment of evidence supporting existing beliefs” and “Overweighting positive confirmatory instances”). In a study where Shidlo and Schroeder set out to confirm a pre-existing view (we believe reorientation is harmful, let’s look primarily for people who have been harmed to test our belief), it is ironic to see Drs. Whitehead and Hamilton engage in the same activity (we do not believe reorientation is harmful, let’s pull these data out of context to confirm the point). I do not mean to imply nefarious motives to Shidlo, Schroeder, Whitehead or Hamilton. Rather, I wonder aloud if both the study and the misuse of it are clear examples of confirmation bias at work.
Bias or not, therapists, ministers and others who advise others about the risks of some kind of reorientation therapy should not provide NARTH’s statement to prospective clients. Instead, these clients can be advised that some people taking these interventions report harm and some report benefit. The best course is to ask the individual counselor or ministry about their specific results. Also, if a person feels worse or becomes depressed, a second opinion or evaluation should be sought.