Critique of conventional research methodologyJohn Heron

Published in Complementary Medical Research, Vol 1 No 1, 1986, pp 12-22

This critique is concerned with the clinical trial which in its basic form compares the effects on a group of patients receiving a certain treatment with those on a matched group receiving no treatment. It is argued that unless this comparison can be systematically made in ways that control for possible sources of error and bias we can never know whether a treatment actually does any real good. Once we know that a treatment is better than no treatment then we can conduct trials to compare it directly with some other new treatment to find out whether the latter is better still.

I wish to raise serious doubts about this approach as it is currently conceived and practised, but before doing so I want to stress that, while making the negative case is a necessary labour in clearing the ground, the more important task is that of laying new foundations and of developing an alternative methodology.

CLINICAL TRIAL METHODOLOGY (CTM): TECHNICAL CRITIQUE

The main technical limitation of CTM is that it assumes the homogeneity of the research populations which it studies. It is looking for similarities of response to a treatment but it does this in a way which obscures differences of response. The assumption of homogeneity is enshrined in the use of averages and the statistics based on their comparison. The researcher is comparing the mean effect for the treatment group with the mean effect of the no-treatment or comparison group.

For the individuals in the trial a comparison of means may hide much more than it reveals. It is possible for there to be a significant difference in means between two treatment groups, so it may be said that one treatment is better than another. Yet there are sub-groups of patients in both groups who are actually worse after the treatments (Canter, 1984); or the mean effect of a treatment may show no significant difference, concealing the fact that the treatment has benefited some and harmed others in roughly equal proportions (Anthony, 1985).

In general CTM tends to ‘obscure, rather than illuminate, interactive effects between treatments and personal characteristics’ (Weinstein, 1974). The data from CTM do not throw any light on the fact that patients respond differently, and do not help with the everyday practical question: What is the treatment of choice for this individual and idiosyncratic patient? There is a curious mismatch between research method and clinical reality: the research method assumes that patients are the same and obscures their differences, while clinical reality repeatedly reveals the patient differences which defeat this assumption. The result of this mismatch is that so-called scientific medicine inescapably generates a significant iatrogenic effect. Medical treatment which is based on the findings of conventional inferential statistics is bound sooner or later to harm some patients in ways which the research culture can neither predict nor understand.

To understand the relation between individual differences and the effects of a treatment is important, morally, theoretically and above all in clinical practice. One approach is to establish a range of physiological and psychological characteristics which are reliable markers of individual differences in response to treatment, and can be used in helping to select the appropriate treatment for the individual patient. Such markers may include peripheral blood leukocyte count, factors which define immune responsiveness, blood group or HLA type (Anthony, 1985), and further work is needed.

Another approach is to develop a research method which does not bury individual differences under the descriptions of aggregate analysis such as averages and measures of variation, but retains the data of each individual in the final results ‘in order to develop a more detailed understanding of what makes them similar to or different from the overall trend identified’ (Canter, 1984a). Facet theory (Canter, 1984b) offers scope for an alternative set of research strategies, but its application to therapeutic outcome studies is still in the early stages.

Another major technical limitation of CTM is that it regards treatment causes and effects in terms of separate univariate analysis. It aims to separate the single treatment variable out from all other extraneous variablesof a psychological or physical kind, and consider its causal impact in termsof changes in level of one or more relatively separate effect variables. This linear univariate causal view — this single variable causing changes in this or that single variable — gives a unidimensional and hence a limited and misleading view of the multidimensional reality within which practitioners and their patients live, work, and participate in therapy. To seethe world as a self-contained mechanical realm compounded of sets of point-to-point linear cause-effect sequences is out of date in theoretical physics (Capra, 1982); and it is odd that it should still have any claim in the scientific basis of therapeutics.

The point is that any single treatment variable such as a drug, a pill, a needle, or a manipulation is, in the real world, set in a whole context of use compounded of many interacting variables. We need to know not simply the relatively independent effect of the physical treatment at a physical level, but how the physical treatment interacts with a whole system of other variables, including psychosocial ones, to produce the therapeutic effect. Variables come in concatenated webs, in dynamic systems with mutually interacting components. General systems theory (Laszlo, 1972) is a better guide here than Cartesian-Newtonian mechanics. We are looking for system-to-system sequences and how one whole system can metamorphose into another.

The univariate analysis of CTM leads to the paradox of the extraneous variable. In conventional research you have to control for and design out the influence of an extraneous variable so that you can obtain the exclusive effect of the particular experimental variable in which you are interested. Hence the placebo effect is an extraneous variable which you have to control out if you want to see whether a drug does have some effect in its own right. Here there is a curious paradox. If a variable has no real influence on the treatment, then you can discount it; you do not need to control for it and you do not bother to call it extraneous. It is only when it does really influence the treatment that you have to control it out. Such a variable may be extraneous to our understanding of the independent effect of the experimental treatment variable, it is certainly not extraneous to our understanding of what is going on during treatment.

The paradox of CTM lies in wanting to discount the influence of a variable precisely because it is influential. The placebo effect, the effect of mental expectation on physiological functioning, is powerful. The univariate research method of clinical trials, which must discount it as extraneous, leads to a therapeutic culture which also discounts it, and this is crazy. What we need is a research method which devises models of and looks at patterns of interaction among variables, and thinks in terms of the interactive effects of mental expectation and physical treatment, so that the therapeutic culture is systemic, interactive and holistic in its thinking and practice.

Facet theory (Canter, 1984b) has done pioneer work in devising a variety of different geometrical models for depicting the pattern of relationships between a number of different variables. These geometric models are not just illustrative but are directly tested in and modified by the data. This multivariate approach requires that the researcher should formulate a probable pattern among the variables before considering changes in individuals. It also allows that during treatment this pattern itself may change as well as the levels of any particular variable.

When we talk about the pattern of variables in treatment, we do not want to rush in with existing models of multivariate statistical analysis and their assumptions about the logical status of the variables involved. We need first of all to think comprehensively about the types of patterned systems which might be involved and understand the range of possible relationships which might be exhibited in such systems. It is clear that research method influences therapeutic practice. If we develop the intellectual tools for research in terms of significant patterns of variables, then the therapy based on such research will probably look different from current conventional or complementary therapy.

A third technical limitation of CTM is that it is exclusively researcher centred. The selection and categorization of variables is entirely determined by the medical researchers. This relates to the distinction between disease and illness. Disease concerns observable pathological change in the body and is what the researcher can reach. Illness is the patient’s experience of being diseased and this is subjective to the patient only. Let us assume three things:

1. Disease and illness are to some degree functionally independent of each other — a patient can be ill without being diseased and vice versa.
2. Disease and illness are in other respects functionally interdependent — how you are diseased can influence your experience of being ill, and how you feel ill can influence the state of your disease.
3. How you feel ill is always and in every instance a choice in the important sense that, provided you are conscious and not brain-damaged, you can choose to alter the way in which you feel ill; you can alter your attitude, and apply one of a range of different internal, mental actions to the content of your subjective experience, and you can change your beliefs about various aspects of the objective disease which you have (Heron, 1985).

In other words, if I can choose to change the way in which I feel ill, then the way I felt ill in the first place was a choice which I had not noticed I had made. There is a basic intentionality about illness which simply goes unnoticed in our culture. Even when it may be true to say that I catch a disease (in the same sense that that disease happens to me or accidentally overcomes me), it is also true that I choose how I will experience the subjective illness which is its correlate, perhaps even whether I will experience it.

The statement that CTM is exclusively researcher-centred implies that it is entirely concerned with the researchers’ selection and categorization of objective disease variables. It studies the fortunes of disease only, and this solely in terms of aggregate analysis and of means and measures of variation. It does not study the whole subjective-objective reality which is illness-disease. It ignores a fundamental sleeping variable, the intentionality of illness, and a systematic study of what would be involved in waking that variable up and putting it to work. If you take assumptions (2) and (3) — that how you are ill can influence your disease, and that you can choose how you are ill — then for any disease and any patient, ceteris paribus, there is an unexplored potential for intentional self-healing. There is also an interesting possibility, which is implicit in assumption ( I), that a person can turn illness into wellness even though the disease is unchanged.

If research is to become more patient-centred then it will need to educate the patient to select and categorize illness variables and work consciously and intentionally with them. Once this active subjectivity of the patient becomes part of the domain of medical research, we are moving radically away from CTM towards forms of action research and cooperative inquiry in which the absolute distinction between the roles of researcher and patient breaks down. We also start to move away from generalizations based on inferential statistics to exhortations based on exemplary and illustrative case histories.

To summarize, the use of CTM assumes the homogeneity of research populations and distorts reality by obscuring the consequences of their heterogeneity; thinks of treatment and its outcomes as essentially separate univariate matters and so can give no account of a multivariate reality in which patterns of interacting variables are at work; focuses exclusively on objective disease measures, thus misrepresenting the subjective-objective illness-disease reality and suppressing a hidden variable — the intentionality of illness — of unknown power.

LOGICAL CRITIQUE

I wish to consider issues dealing with the logic of valid claims because they penetrate deeper into the heart of the critique than the important technical points which I have made. The conventional view is that the claim that a treatment actually does some good is only valid if a properly designed comparison has been made between a treatment group and an untreated matched control group. This view presupposes the restricting technical parameters of homogeneity, univariate analysis, and observable disease measures; the claim to validity itself rests on assumptions whose validity can be called into question because they so misrepresent the reality which they purport to reveal.

Suppose we take a wider and more representative set of parameters and think of individual differences among patients studied as well as the general trend; of the treatment and its effects in terms of a whole pattern of factors or facets; of studying the experiential illness component together with the observable disease component whilst enlisting the patients’ active subjectivity, their internal agency. Then the question we ask is not: Does this particular physical treatment have a general tendency to reduce or cure observable disease independently of the operation of other factors such as the placebo effect and unaided recovery? but: What pattern of factors in the total treatment including the patients’ intentionality enables the patients involved to transform illness into wellness and results in the reduction or cure of observable disease, and what pattern of factors inhibits such effects?

What is important to grasp is that in the second question there is no such thing as an extraneous (but influential) variable and therefore no such thing as a control group, whose sole function is to provide data to discount the influence of an extraneous variable. What the question seeks is the interacting effect of all the identifiable major influential variables including patient self-help. It may be said that there is always the possibility of one basic extraneous variable — unaided spontaneous reduction of or recovery from observable disease — and unless there is control for this an observer may be deluded into supposing that the pattern of factors in the total treatment is having some effect.

There are two basic answers. First, the factor of spontaneous reduction of observable disease should always be included within the total pattern of factors being studied in the inquiry, so the control for it is internal to the inquiry through discrimination within the total pattern of factors in relation to the data of the inquiry. What we are concerned with here is contextual validity (Diesing, 1972).

Second, the use of a control group will simply throw one back into a comparison of means which will entirely misrepresent the aim of the inquiry group. In order to obtain an effective pattern of factors in the inquiry group you are not just striking a mean through the data but comparing and contrasting both similarities and differences among the individuals. You are not concerned with statistical inference based on a comparison of means, but rather with a discriminating judgment borne out of a dialectical interplay between the data and the conceptualized pattern of factors, and out of the various perspectives on this interplay provided by individual differences. A comparison of mean scores on observable disease measures may show no significant difference between the inquiry and the control group, yet there may also be wide differences within the inquiry group itself which, taken with all the other factors involved, contextually validate claims about a therapeutic effect.

This is a key point and likely to be a stumbling block for conventional researchers until it is fully grasped. If you want to study the interacting pattern of all the influential therapeutic factors you cannot split the factors up between an experimental and a control group. All the factors must be studied together in the inquiry group and control is exercised internally and contextually. Different studies using this model may also be compared and contrasted. They are controls for each other, both conceptually and empirically, in the contextual sense, but not in the conventional sense of a comparison of means. What this whole approach calls for is much closer conceptual attention to possible systems of therapeutic factors, and to the different types of structuring which may occur within them, before any empirical inquiry begins.

The concept of active subjectivity, of intentional self-healing, raises some interesting points in the logic of valid claims. The criteria for knowing that you have cured yourself of your disease by your own internal agency are not the same as the criteria for my knowledge that some physical treatment which I have administered has cured you. The logic of valid claims in each of these different situations is also different and it is important that they are not confused with each other. Furthermore, the criteria for my knowledge that you have cured yourself by your intentional self-healing are not the same as the criteria for my knowing that some physical treatment which I have administered has cured you.

MORAL CRITIQUE

CTM often uses the random allocation of subjects to the experimental treatment group and the control group, since this assists the assumption of homogeneity when making a comparison of means between the two groups. Such randomization is morally problematic. It is axiomatic that respect for patient autonomy is an absolute moral principle. Every patient has a right to exercise fully informed self-determination in the selection of available treatments and this right is only to be waived when the patient is an emergency and in extremis. This entails a duty on researchers to seek the informed consent of patients before entering them for a trial, and their consent to randomization (Heron, 1984).

Even when the patient is properly informed about the treatments being tested and the reason for the trial, where their consent to be entered and to be randomized is sought, I do not think that this resolves the moral problem. It is at least morally questionable to ask people to consent to participate in a research system that: (1) ignores individual differences and is therefore bound to produce results which harm some; (2) often treats the subjects’ mentality and intentionality as extraneous variables whose influence is to be discounted; (3) totally disregards the subjects’ right to participate in decisions about a research design which purports to generate knowledge about them. This third point affirms an important extension of traditional and contemporary doctrines of human rights and merits further development, particularly because it comes as a moral surprise to conventional researchers that the point could even be made.

I will repeat an argument which I have already stated (Heron, 1981): Doctrines of rights form a basic moral principle of respect for personal autonomy which requires that we give impartial consideration to the needs and interests of each person, and provide just conditions for the fulfilment of each person’s well-being. Statements of human rights have spelt out some of the basic conditions required for such fulfilment: the right to freedom of speech and expression, to freedom of association and contract, and to political membership of the community to participate in the framing and working of political institutions (Benn and Peters, 1958).

The last named right is a special case of an all-pervasive right of persons to participate in decision-making which affects the fulfilment of their needs and interests. While acknowledging this right in the restricted political sense, in local and national government, our society has been slow to acknowledge its relevance to industry and commerce, to organizational structures, to education, and to research. In all these areas the obligation is clear: we need to learn how to make consultative and cooperative decisions with people who will be affected by those decisions, rather than make decisions over them unilaterally.

Concepts of rights have developed over centuries. But it is still odd to talk of the political or participative rights of experimental subjects in clinical trials. Nevertheless, I hold that individuals as autonomous beings have a moral right to participate in decisions which claim to generate knowledge about them, and that researchers have a correlative duty to include their subjects in the decision-making which generates, manages and draws conclusions from the research.

Such a right does many things.

1. It honours the fulfilment of the subjects’ need for autonomously acquired knowledge about themselves.
2. It honours the requirement of truth that the subjects’ perspectives on their situation are a necessary corrective and complement to anyone else’s account.
3. It protects them from becoming unwitting accessories to knowledge-claims which may be false and inappropriately or harmfully applied to others.
4. It protects them from being excluded from the formation of knowledge which purports to be about them, and so from being managed and manipulated, both in the acquisition and in the application of the knowledge, in ways they do not understand and so cannot assent to or dissent from.

Such a view is far ahead of current practice. It entails on medical researchers an obligation to initiate patients into the whole rationale of the research and to invite patients to become autonomous inquirers alongside them. Medical research has an important educational commitment to provide conditions in which patients can enhance their capacity for self-determination in acquiring knowledge about their condition. This involves a radical revision of method — from doing research on persons to doing it with persons, and this is not possible within the current assumptions and parameters of CTM. Once medical research becomes participatory research with patients as autonomous persons, then the practice of research overlaps with the practice of education, therapy, and political rights.

PSYCHOLOGICAL CRITIQUE

There is an important point to be made in the field of the psychology of knowledge. Fear is the classic human concomitant of ignorance. When the domain of ignorance is directly to do with the human being and the human condition — as in medicine and psychology — then the fear is particularly acute and problematic. Positively, the anxiety of uncertainty, of not knowing, may be one of the spurs to inquiry and the proper acquisition of knowledge which dispels ignorance. Negatively, especially when such anxiety is not fully identified and acknowledged, it may have an overdetermined effect, and through unawareness distort the direction in which knowledge is sought and the method by which it is sought (Devereaux, 1967; Reason and Rowan, 1981; Heron, 1982; Heron and Reason, 1985).

The result is a curiously alienating mismatch between the knowledge supposedly acquired and the domain of human experience to which it is supposed to refer. Indeed, the findings and the method of arriving at them may be more to do with reducing the unacknowledged anxieties of the researchers, than with illuminating the field of inquiry. As Peter Reason points out, this psychological distortion is a significant threat to the validity of any research directly to do with individuals and steps need to be taken within the research procedure to counter it. Certainly, the degree of control over people that is so much part of the CTM makes it to some degree psychodynamically suspect.

Another major point of a psychological kind relates to what we now know about the voluntary control of internal states, the ability of individuals through acts of will to alter physiological states which were hitherto thought to be entirely under control of the autonomic nervous system (Green and Green, 1977). Although you could do a conventional randomized trial to compare the effects of conscious self-regulation with the effects of doing nothing, it would generate some interesting anomalies.

First, if you seek fully informed consent, as you certainly should, then you have to ask your subjects to agree to the possibility of being randomly allocated to a control group in which they are not to practice conscious self-regulation. Apart from the dubious morality of this request, you have possibly contaminated any placebo effect in the control group by alerting them to what they are not supposed to do. Consciously not practising conscious self-regulation may have a contradictory effect. If you get positive outcomes in the control group you can never be quite sure that this is simply due to a placebo effect. In other words, there is only the thinnest of lines between the unintended effect of mental expectation on physiological states and the intended effect of mental endeavour on them.

Second, the capacity for voluntary control of internal states can scarcely be regarded as any old random independent variable. Those who practise it should be self-selected, exercising free and informed choice on the basis of belief and commitment. Voluntary control is central to personal autonomy and the exercise of personal autonomy is not something you could or should explore through randomized allocation. You need a different type of cooperative and consultative inquiry which both respects and helps to create the conditions under which personal autonomy flourishes. It is no good trying to inquire into something by a method which is inimical to its manifestation.

Third, aggregate analysis and a comparison of means are not needed to demonstrate that there is an effect from voluntary control. One individual, one case, is sufficient to demonstrate that there is an effect. Thereafter, we need to know the whole context and pattern of factors which have enabled that person to practise voluntary control and will enable others to do the same. Once again we come back to a systems view, a contextual view, of inquiry.

One final psychological critique of the CTM states that ‘in its pursuit of the independent physical effect of physical treatments’ it creates a research culture which simply ignores the relevance to treatment of other factors: prevailing belief-systems, social norms, role expectations, various aspects of practitioner-patient relationship, voluntary control, catharsis, psycho-dynamic mechanisms, the use of touch, psi factors and spiritual interventions. This is why a systems approach is so important because we can start to build conceptual models to portray how these various factors dynamically interweave in therapeutically effective situations.

PHILOSOPHICAL CRITIQUE

CTM is really embedded in, and is an expression of, a classical Cartesian-Newtonian view of the world and anybody in it as a self-contained mechanical entity in which linear cause-effect sequences can be discovered by experimental control of variables in the light of certain mathematical assumptions about their status. This old world-view is under assault from two primary directions: developments in theoretical physics (Bohm, 1980; Capra, 1982) and in consciousness research (Valle and Eckartsberg, 1981; Wilber, 1981). What is replacing it is a holistic, ecological or systems view of the world (Laszlo, 1972; Bohm, 1980) which entails a totally different concept of what it is to do research, which I have described elsewhere (Heron, 1985), and which form a central part of Peter Reason’s paper at this conference.

We need to think about putative models of effective treatment systems before any empirical inquiry (which would be a cooperative form of action research), and to think of them in terms of whole multidimensional systems which have sub-systems at different levels (physical, energetic, psycho-social, psi, transpersonal). We need to think about relations between the parts or factors of the systems — both within and between levels — in terms of several different sorts: functional independence, one way causation, mutual interaction and asymmetrical interaction. We need also to think of different types of causation (unifactorial, multifactorial; energetic, formative, intentional). Once we have started to generate clear, coherent and comprehensive models of the patterns of factors, then we can take them to the data to clarify, correct and amend them.

REFERENCES

Anthony, H. M. (1985), personal correspondence.

Bahm, A. J., (1985), ‘Five Types of Systems Philosophy’, International Journal of General Systems, vol. 6: 4, 233-7.

Benn, S. I., and Peters, R. S. (1958), Social Principles and the Democratic State, London: Allen & Unwin.

Bohm, D., (1980), Wholeness and the Implicate Order, London: Routledge & Kegan Paul.

Canter, D. V. (1984a), personal correspondence.

Canter, D. V., ed. (1984b), Facet Theory, New York: Springer-Verlag. Capra, F. (1982), The Turning Point, London: Wildwood House.

Devereaux, G. (1967), From Anxiety to Method in the Behavioural Sciences, The Hague: Mouton.

Diesing, P. (1972), Patterns of Discovery in the Social Sciences, London: Routledge & Kegan Paul.

Green, E., and Green, A. (1977), Beyond Biofeedback, New York: Dell.

Heron, J. (1981), ‘Philosophical Basis for a New Research Paradigm’, in Reason, P. and Rowan, J. Human Inquiry: A Sourcebook of New Paradigm Research, Chichester: Wiley.

Heron, J. (1982), Empirical Validity in Experiential Research, Human Potential Research Project, University of Surrey.

Heron, J. (1984), Draft Policy Statement on the Use of the Randomised Trial, London: Research Council for Complementary Medicine.

Heron, J. (1985), A Systems Approach to Medical Research, London: Research Council for Complementary Medicine.

Heron, J., and Reason, P. (1985), ‘Research with People: The Paradigm of Cooperative Experiential Inquiry’, Working Paper. Centre for the Study of Organisational Change and Development, University of Bath.

Laszlo, E. (1972), Introduction to Systems Philosophy, New York: Harper. Reason, P., and Rowan, J. (1981), Human Inquiry: A Sourcebook of New Paradigm Research, Chichester, Wiley.

Valle, S., and von Eckartsberg, R., eds. (1981), The Metaphors of Consciousness, New York: Plenum Press.

Weinstein (1974), ‘Allocation of Subjects in Medical Experiments’ New England Journal of Medicine, 291, 1278-85.

Wilber, K. (1981), ‘Reflections on the New Age Paradigm, interview with Ken Wilber’ Revision, Spring, 53-74.