Confirmation bias, NARTH and the use of research

I quoted Nickerson in my prior post on confirmation bias. His article is quite good and can be reviewed here. I like this quote in the article attributed to Francis Bacon:

The human understanding when it has once adopted an opinion (either as being the received opinion or as being agreeable to itself) draws all things else to support and agree with it. And though there be a greater number and weight of instances to be found on the other side, yet these it either neglects and despises, or else by some distinction sets aside and rejects; in order that by this great and pernicious predetermination the authority of its former conclusions may remain inviolate.. . . And such is the way of all superstitions, whether in astrology, dreams, omens, divine judgments, or the like; wherein men, having a delight in such vanities, mark the events where they are fulfilled, but where they fail, although this happened much oftener, neglect and pass them by. (p. 36)

Nickerson then outlines several types of confirmation bias:

-Restriction of attention to a favored hypothesis.
-Preferential treatment of evidence supporting existing beliefs.
-Looking only or primarily for positive cases.
-Overweighting positive confirmatory instances.

Recently, several readers asked me about a report on the NARTH website claiming that reorientation therapy reduced suicide attempts. In reviewing the claim, it appears to me to be an example of confirmation bias. However, before I discuss it, I want to assert that I believe confirmation bias is common to humans. For reasons I will lay out in future posts, I believe cognitive activity serves (at least) to simplify complexity, create a sense of predictability to the world, and to justify investments of time and energy – in this case mental time and energy. I am not above it, nor do I believe anyone to be. I do think we can help prevent and/or correct errors by being aware of it.
It is no secret that I think reparative therapists who believe there is only one path to same-sex attraction engage in confirmation bias. Another recent instance from NARTH is the use of a study by Shidlo and Schroeder to make a claim that reorientation therapy reduces suicide risk. President-elect, Julie Hamilton, in her report from the 2008 NARTH conference, wrote:

Regarding the claims that reorientation therapy harms clients, Dr. Whitehead cited studies that found suicide rates decrease after therapy. In fact, he pointed out that Shidlo and Schroeder (2002) sought to prove the adverse effects of therapy by collecting stories of harm; however, instead of finding therapy to be harmful, they found it to be helpful, in that suicide attempts by these clients actually decreased after therapy. For more information on the content and references for Dr. Whitehead’s keynote address, see the NARTH Collected Convention Papers or soon-to-be-released book, What the Research Shows: NARTH’s Response to the APA Claims on Homosexuality.

First, this is misleading because the way it is worded, it sounds as though Shidlo and Schroeder found and reported something they did not intend to find. More relevant to this post, however, is Dr. Hamilton’s reference to an analysis by Dr. Neil Whitehead, bio-chemist with numerous scientific publications including some on sexual orientation. Neil often provides interesting perspectives so I was surprised to see him quoted in this context. When I asked Neil about the claim, he said he reanalyzed the reports of suicide from Shidlo and Schroeder’s paper and stands by it. While I have not seen the reanalysis, I don’t need to in order to know that a relationship between reorientation and suicidality cannot be inferred from an analysis of Shidlo and Schroeder. Even so, Neil stunned me by saying that his analysis did not reach statistical significance but revealed a non-significant trend for reorientation therapy to reduce suicidality among same-sex attracted people. On that basis, he made his claim which was amplified by Dr. Hamilton.
Here is what Shidlo and Schroeder reported about their participants’ suicide attempts.

In examining the data, we distinguished between participants who had a history of being suicidal before conversion therapy and those who did not. Twenty-five participants had a history of suicide attempts before conversion therapy, 23 during conversion therapy, and 11 after conversion therapy. We took the subgroup of participants who reported suicide attempts and looked at suicide attempts pre-intervention, during intervention, and post-intervention to see if there was any suggestive pattern. We found that 11 participants had reported suicide attempts since the end of conversion interventions. Of these, only 3 had attempted prior to conversion therapy. Of the 11 participants, 3 had attempted during conversion therapy.

I am guessing that Neil is taking the 25 and 23 people who reported attempts before and during intervention as being helped by therapy since they apparently (although the numbers may overlap and are not clear) reported no suicide attempts after therapy. The 11 after therapy are perhaps conceded as a minority of clients with an adverse reaction. Since I am not sure, I won’t knock down what might a straw man of my making. However, what seems clear is that whatever effect may have occured, Neil and by extension Dr. Hamilton, assumes it to be a positive benefit from the therapy. However, this seems to me to be a biased attribution with at least one other explanation. Perhaps these people were not suicidal after conversion therapy because they went to a support group for conversion therapy survivors. Perhaps, a fuller examination would find that people are alive today despite the therapy not because of it.
If anything, these reports do not seem favorable to reorientation therapy. Anyone can play with numbers. I could take the 23 plus 11 and come up with a 16.8% (34/202) probability of adverse consequences due to reorientation efforts. However, these reports cannot be the basis for any statements about the general impact of reorientation efforts on suicidality. About all we can say is that some people reported feeling worse due to their reorientation experiences. For at least some same-sex attracted clients, the experience was not benign but was associated with a worsening of their distress. Ordinarily, in absence of prospective studies, professionals should inform their clients of such reports to give clients ability to consent to care. But any general statement of efficacy or probability with regard to suicidality would require a specific study to test that hypothesis.
A study that would permit the statements made by Dr. Hamilton would require a prospective design with follow up and with a control group of people who did not received reorientation therapy but some other appropriate intervention. At the least, a waiting list control group would be required. The prospective nature of the study is crucial to capture not only suicide attempts but any completed suicides which occured during the course of the interventions or thereafter (during the follow up aspect of the study).
Shidlo and Schroeder’s design does not permit any general probablity statement. Just prior to reporting these findings, Shidlo & Schroeder said the numbers should not be viewed as complete or representative of the actual degree of harm:

After participants’ responses to the open-ended question, we followed up with a checklist of symptom areas (self-blame for not trying hard enough to change, self-esteem, depression, difficulties with intimacy, social isolation, loneliness, self-harmful behavior, suicidal thoughts, suicide attempts, feeling paranoid, self-monitoring behavior for “homosexual mannerisms,” and alcohol and substance abuse) and asked them to tell us whether they noticed negative changes in these areas. This symptom checklist was developed in our pilot interviews.
We do not report here on the frequency of responses to these items because of two methodological limitations. First, because we emphasized breadth of inquiry and yet were constrained to keep the interview within a reasonable time limit (approximately 90 min), we used single items for each domain of functioning; this methodological decision came at the expense of sensitivity, reliability, and content and construct validity. Second, participants who felt harmed and unhappy about their therapy experience may have answered affirmatively to a deterioration in a particular area and attributed it to the conversion therapy because of a negative halo-effect or narrative smoothing (Rhodes et al., 1994) rather than having provided an accurate recollection of actual change in that particular area. Thus, instead of using the checklist as a quantitative measure of negative effects, we used these items as qualitative interview-prompts to help respondents explore areas of deterioration. Our results, therefore, focus on the meanings of harm attributed by clients, and the accuracy of these attributions remains to be determined by future process-and-outcome research.

Even though Shidlo and Schroeder have their own confirmation bias issues in this study, here they take a cautious approach. Perhaps, the halo-effect colored the recollections negatively; perhaps some people blocked out suicidal thinking. Without a prospective study with a control group, these numbers tells us nothing reliable about the matter at issue: whether reorientation therapy reduces, enhances, or has no effect on suicidality for the population of people who are inclined to seek it.
Furthermore, as Shidlo and Schroeder note, the actual numbers of attempts of episodes may not be accurate. These were retrospective accounts. It is quite possible that some suicide attempts were not reported to Shidlo and Schroeder.
It seems to me that NARTH’s use of Shidlo and Schroeder illustrates points 2 and 4 above (“Preferential treatment of evidence supporting existing beliefs” and “Overweighting positive confirmatory instances”). In a study where Shidlo and Schroeder set out to confirm a pre-existing view (we believe reorientation is harmful, let’s look primarily for people who have been harmed to test our belief), it is ironic to see Drs. Whitehead and Hamilton engage in the same activity (we do not believe reorientation is harmful, let’s pull these data out of context to confirm the point). I do not mean to imply nefarious motives to Shidlo, Schroeder, Whitehead or Hamilton. Rather, I wonder aloud if both the study and the misuse of it are clear examples of confirmation bias at work.
Bias or not, therapists, ministers and others who advise others about the risks of some kind of reorientation therapy should not provide NARTH’s statement to prospective clients. Instead, these clients can be advised that some people taking these interventions report harm and some report benefit. The best course is to ask the individual counselor or ministry about their specific results. Also, if a person feels worse or becomes depressed, a second opinion or evaluation should be sought.

Misconceptions in sexual identity ministry

I also blog on Crosswalk.com and publish articles there occasionally. A recent one regarding misconceptions in sexual identity ministry has not been getting great reviews over there, which is I suppose to be expected given the content. Here is one section which should surprise no regular readere here:

One – All gay people are attracted to the same sex because they did not bond with their parents or were sexually abused.
Untold pain and confusion has been caused to parents and their same-sex attracted children by well-meaning writers and counselors who promote this unsupported view of homosexual origins. The usual evangelical narrative is that persons attracted to the same sex did not get sufficient bonding or love from the same-sex parent and seek these experiences in the present via sexual relationships from members of the same sex. For males, the concept of an over involved, smothering mother is often thrown in as an additional family factor. In addition, claims have been made that most if not all same-sex attracted people have been sexually abused.
The truth is that research on causal factors in sexual orientation is still in the early stages. However, we do know from experience that there are many same-sex attracted people who had loving homes and were not sexually abused or otherwise mistreated as children. On the other hand, some say they believe their sexual desires were shaped in some way by unhappy growing up experiences. What we cannot identify with any certainty is why any given individual experiences same-sex sexual attractions. Recent research on twins suggests that pre-natal factors are associated with same-sex attraction, as are individual environmental experiences which vary among homosexuals. The best we can say at present is that different pre- and post-natal factors may operate differently in different people. For now, not only is it unnecessary to pigeon hole people, it can be harmful and intensely discouraging for parents and children alike to pursue therapy for non-existent problems of bonding or parenting. Where abuse or bonding problems exist, they should be addressed but successfully dealing with issues from the past will not of necessity lead to sexual reorientation.

I then take on change within the context of evangelical doctrine. The comments let me know how large the gaps still are.
UPDATE: The Christian Post also published this article. The comments section is again quite lively.

Dutch government funds sexual identity ministry

This news report from the Netherlands Information Services, an English language Dutch new service, is intriguing.

Govt Subsidises ‘Homosexuality Healers’
THE HAGUE, 22/11/08 – Emancipation Minister Ronald Plasterk subsidises two Christian organisations that see homosexuality as a curable complaint.
Plasterk recently announced he would give subsidies to Christian gay organisations with the goal of making homosexuality discussible in these circles. But according to De Pers newspaper, these organisations work with therapists that are supposed to ‘cure’ homosexuals of their proclivity. They also only accept homosexuality when it is not actually practised.
Refo Anders is receiving 84,000 euros. “Homosexual feelings, okay. But we reject practising,” confirmed chairman Johan Quist in De Pers. Dealing with homosexual feelings means either celibate living or becoming heterosexual, is also the view of Johan van de Sluis, board member of the Onze Weg (Our Way) foundation, which is receiving 50,000 euros.
Gay interests organisation COC chairman Wouter Neerings is astonished. “Minister Plasterk is normally a man of sound judgement. It looks like a political compromise with ChristenUnie.”
In a reaction, Plasterk said: “These organisations have access to circles where there would otherwise be no homo-emancipation at all. Of course I do not support the aims completely, but they have taken important hurdles. For example, the recognition that there are homos in Christian circles.”

Plasterk is an acclaimed molecular biologist, writer and of late politician, who serves as the Minister of Education, Culture and Science in the Netherlands. As such, he is responsible for “emancipation” or setting policies supportive of equal rights for women and gays. He rejects Christianity, yet apparently believes there is value in making “homosexuality discussible in these circles.”
I have some question about the characterization of the recipients as “homosexuality healers.” Refo Anders, one of the organizations receiving the money, has expressed reservations regarding change therapy. I cannot vouch for Google translation but here is a link to a page where it appears that Anders believes his group has been mischaracterized by the Dutch press. An excerpt:

The Press newspaper in an article called Refo Anders as a gay “healer”
Despite that Refo Anders in his vision document wrote that they skeptical about change therapy The Press considers it better to know and places it among the gay Refo Otherwise healers.
Otherwise Refo Foundation regrets this “low to the ground” way of journalism companies, particularly since the journalist (Koen Verhelst) a nuanced story that he has heard total passes even after he was recalled on factual inaccuracies in his story, even the inappropriate use of the name of Refo Otherwise urged not to him by corrections that after 37 minutes of us got on his concept press release (which he himself for a few days needed to write).

As I read through the materials, the approach sounds more like Wendy Gritter’s New Directions, than a change ministry.
UPDATE: Johan Quist, of Refo Anders wrote to explain a little further. Although his English is much better than my Dutch, it may take a couple of reads to catch the meaning.

We do not like to change the human with therapy, but otherwise we believe that God has the power to change people, it’s a work inside the heart of a human from God himself. Also its really important that people find the way back to the warm relation with God and from this warm relation God will show him the way to live; some people have to live his total life with homo-feelings And other people God will change, it’s the free way of God !

This is not a homosexuality healing or reparative therapy organization. Rather, this is a Christian ministry with a reformed theology which supports non-gay-affirming people who strive to live by their traditional reformed faith. I like this approach and think it has much to recommend it over the reparative based ministry often practiced in the U.S.

Obama may delay repeal of "don't ask, don't tell"

The Washington Times is reporting what seems like a smart move: Barack Obama will probably deal with the most serious problems facing the nation first and postpone action on potentially divisive actions, like the repeal of “don’t ask, don’t tell.” My guess is that Obama will delay the push until after the 2010 mid-term elections.
Bob Knight is quoted as suggesting that Americans don’t have this policy on their radar now but will be quick to react negatively if the policy is debated nationally. I think he is probably on target. While attitudes toward homosexuals continue to become more positive, I think questions about practical matters of living arrangements, morale and recruitment will be raised by opponents.

Ex-gay history: Charles Socarides and Richard Cohen on CNN

Richard Cohen has added some content to his YouTube account with a new infomercial and a clip of an interview on CNN with Charles Socarides and Cohen from what appears to be the early to mid 90s (probably 1994 given this article). Socarides mentions NARTH so it would be after the founding of that organization in 1992. The sound does not match the speakers but this is an interesting clip nonetheless which demonstrates that NARTH’s premise in the early days was to return homosexuality to a status of mental disorder.