University of Cambridge Home Physics Dept Home TCM Group Home

Scientists' unethical use of media for propaganda purposes

This web page is intended to draw the attention of scientists, the media, and the public to a problem that, while being very familiar to some, is probably unknown to the majority of visitors to this web page.  Propagandising of the kind described in the following bypasses the normal carefully considered processes of science, and may well create a distorted impression in the mind of the unsuspecting reader or viewer.

The following account of a not untypical case concludes with recommendations for the media.

1. A TV demonstration is fixed?

Many viewers of a recent Discovery Channel programme, previewed in a Guardian article, where the claims of Natasha Demkina, a 17-year-old Russian girl who says she is able to "look deep inside people's bodies, watch their organs at work and spot when things are going wrong", were investigated, ended up with a strong impression that the main test in the investigation had been deliberately set up with a view to ensuring that she would fail it.  The test involved her being given a set of seven cards, with a medical condition indicated on each.  Medical subjects with these seven conditions (one of which was 'no condition'), each bearing an identifying number, stood in a row and Natasha had to mark each card with the number of the person whom she thought had the condition indicated on the card.  Despite the difficulties associated with the rigorous and unfamiliar conditions imposed by the experimenters, she identified four of the seven correctly.  A fairly straightforward calculation shows that the odds of getting 4 hits or more out of 7 by chance are more than 50 to 1 against.  Surely a case for celebrating Natasha's success?

Well, no, actually.  The experimental protocol, to which Natasha and her agent had been asked to agree, rather curiously states:

"If Natasha correctly matches fewer than 5 target medical conditions, then the Test Proctor will declare that results are more consistent with chance guessing and does not support any belief in her claimed abilities."

Accordingly, it was announced that Natasha had 'failed the test'.  In the article about the programme in the Guardian, Richard Wiseman, one of the investigators, emphasised this conclusion, declaring "a failure is a failure".

(added November 11th., 2004) The investigators' own account is now available on the web: observe that the 50 to 1 statistic does not feature anywhere in it.  The fact that "everyone [had] agreed to the written protocols" (including the above italicised condition) is given as sufficient justification for asserting "[the] test, as preliminary as it was, will likely close the chapter in this case".  I think not: real science does not work on a basis of getting someone to sign their agreement to a long list of conditions, then later coming back saying "this is what you signed; the challenge goes to us!".

2. Motivations I

On the face of it, it looks as if there was some kind of plot to discredit the teenage claimed psychic by setting up the conditions to make it likely that they could pass her off as a failure.  Might the investigators have simply made a computational error when they decided to place the cut off point at 5 hits?  Apparently not: Wiseman, when questioned about this, appeared to know about the 50 to 1 statistic, but would not accept that the cutoff point had been set wrongly. What then did he have to say about the fact that Natasha achieved four hits?  He admits that the result is "interesting", but is then quoted as saying: "At best, she's done this a lot and she has a real expertise at being able to look at people and make reasonably accurate diagnoses."  Perhaps realising that this statement might not quite convince, he goes on to suggest that perhaps Natasha cheated, a handy way to evade the issue.

A second suspect argument for Natasha having no special abilities (except for her 'expertise at being able to look at people and make reasonably accurate diagnoses') involved discrepancies in a preliminary test, but given the limited extent to which these discrepancies were followed up it would be hard to conclude just from these that Natasha was not actually picking up some problem that the subjects had not written down on their card (having viewed the actual programme, I now consider the argument based on these discrepancies to have been grossly overblown).

In the words of Sven-Goran Eriksson, 'this is a nonsense'.  A statistically very significant result was obtained in the quantitative part of the investigation, but the experimenters concealed the fact with their talk of 'failing the test'.  The investigation claims to be science, but fails almost every test of good scientific practice.  Does this matter for a TV programme?  I suggest that if a program pretends to be science then it does matter, and its failure to conform to scientific norms is a serious matter.

3. Motivations II

If this line of enquiry leaves us uncertain, perhaps we should look elsewhere to understand why the investigation took the form that it did.  According to the Guardian article, the experiment was designed by scientists working for the Committee for the Scientific Investigation of Claims of the Paranormal (CSICOP), an organisation of professional sceptics.  In addition, the leader of the investigation, Andrew Skolnick, is director of the organisation CSMMH, the so-called Commission for Scientific Medicine and Mental Health.  This organisation is not an official body as one might think from its title Commission, but rather a collection of people interested in "the scientific examination of unproven alternative medicine and mental health therapies", Their mission statement at makes instructive reading, with  references to 'aberrant remedies' and 'dubious research'.  The situation now perhaps becomes a little clearer.  It would be consistent with the aims of members of such organisations for the experiment to be designed to maximise the chances of being able to discredit Natasha.  Setting the threshold to 5, rather than the value that would make more sense, namely 4, would help achieve this, as would the scenario whereby it would be announced that Natasha had 'failed the test'.  Perhaps we need look no further to see why the threshold of 5 rather than 4 was chosen?

4 Wiseman and the media

Wiseman's investigations are amazingly well covered by the media, which surely cannot come about except by design.  Other investigations follow a similar pattern to the one described.  Take the following report of one of his mega-demonstrations. Here is an example:

No meeting of minds at telepathy trial
By David Derbyshire
(Daily Telegraph, 14 Dec. 2000)

THE world's biggest psychic experiment last week failed to come up with a shred of evidence for telepathy.

But then, the true psychics would have known that anyway.

Over the course of 10 experiments, several hundred people failed to project a set of images to volunteers in a sealed room several hundred feet away. The volunteers should have got between two and three images right just by random chance. In fact they scored one out of 10.

Derbyshire's jocular comment 'true psychics would have known that anyway' is truer than he might have realised, since many professional parapsychologists did predict in advance that the test would 'fail', just from their knowledge of the test conditions.

We see the same features as in our earlier example: a highly publicised test is carried out in a way that is unlikely to succeed, and the fact that it does not succeed is trumpeted loudly as if it were a highly significant result: a show dressed up to look like science.

5. Comments on the TV programme

I suspect that most viewers would have ended up being deeply sceptical as to the way the 'test' was done, and wondered about the investigators' motives.  A comment made by Wiseman himself in the closing stages of the programme would seem to sum up the investigators' mindset precisely:

"... it doesn't matter what [the experiment achieves] in terms of testing.  It's not about the results; this is about belief".

It seems to me that the only correct conclusion to have come to in regard to the test is that it was inconclusive; no viewer of any intelligence would have felt that the fact that Natasha did not achieve 5 hits ('the score that the experimenters had set'), under the conditions obtaining at the test, meant much at all.

The three investigators saw fit to ignore the inconvenient statistics of the outcome, and to talk instead as if the outcome of the test had refuted Natasha's claims, which it clearly did not.  Doing so was not only unscientific but, under the circumstances, unethical.

6. Afterthoughts

(added November 28th., 2004) In correspondence, Andrew Skolnick has expressed in strong words the view that the CSICOP/CSMMH experiment was good science. while the critique on this web page was unscientific, assertions that I found hard to take seriously.  There is, however, one imaginable perspective from which the sceptics' way of doing things could appear rational. Let us examine it.

The scenario concerned is one where (a) one did not believe Natasha's claims could possibly be true; (b) taking them to be true could have undesirable consequences.  If one thought that, then disproving the claims would take a high priority.  Suppose also (c) that there is a possibility that it might appear as the outcome of an investigation that Natasha had the claimed skill, when actually she did not (a situation that the sceptics assert is the actuality), which could lead to the undesirable consequences presumed in (b).  In that case, it could appear rational to design the experiment to minimise the probability of a false positive outcome as in (c).  This line of argument could have been the rationale underlying the choice of 5 out of 7 as the cutoff point.

That line of thinking is problematic because, in minimising the probability of a false positive in this way, the experimenters significantly enhanced the probability of the alternative, equally undesirable, false negative, i.e. declaring that Natasha appears to have no abilities when in fact she has some.  All that getting less than 5 successes really proved, under conditions that were very unfavourable to the exercise of her skills (including, it would appear, one of the subjects appearing to be laughing at her), was that she could not get perfect results under arbitrary conditions (and even then, there have been disputes as to whether her 'failures' were really failures; for example, proper medical records have not been supplied to confirm that the patients did not have the conditions that Natasha believed they had).

The very fact that there was no null hypothesis, other than the scientifically suspect one that she could get a score of at least 5 under the unfamiliar conditions of the experiment, makes the 'test' dubious from the scientific point of view.  And the experiment provided no justification at all for Prof. Hyman to say as he did on the programme, "my hope for Natasha is that she will grow up ... and give up this aspect of her life ... I don't think it is good in the long run for any of us to be living an illusion".

Was the experiment CSICOP propaganda?  Yes.  Was the TV demonstration 'fixed'?  In the sense discussed in this section, again yes.

(added Dec. 13th., 2004)  Further analyses, by Julio Siqueira, based in part on correspondence with the investigators, are now available:

7. Conclusions

Should one be suspicious of CSICOP's procedures?  Let the reader decide.

The initial decision to consider a score of 4 hits out of 7 as 'failure' when the probability of getting such a score is less than two per cent is hardly in step with normal scientific practice.  Having adopted it, and used it to declare Natasha a failure ("she had the claim, we tested it, she didn't pass the test"), the investigators moved on to what seems to have been an automatic presumption of deception or self-deception ("people believe that she can do it ... how come smart people can get to believe things that aren't so?").

Manipulation of concepts such as 'failure', and the abuse of statistics, are commonplace in the world of propaganda.  Is that what is happening here, or honest science?

The CSICOP organisation is not infrequently taken to have an authority that it does not deserve.  Such organisations are in reality pressure groups, taking every chance they can get to press their beliefs in the media, often in ways that have been characterised as misleading.  Representatives of the media need to be on their guard against this kind of thing.  Some recommendations directed toward this end follow:

7. Links

Some sites and links giving critiques of these kinds of activities:

and here are links to the home pages of the two organisations discussed, CSICOP and CSMMH.

The above report was produced by Brian Josephson