EUROPEAN PRESSURE ULCER ADVISORY PANEL

RELIABILITY AND VALIDITY OF EVIDENCE

Adapted from an editorial, first published in the Journal of Tissue Viability 1999; 9(1): 3-4

An essential element of implementing Evidence Based Practice (EBP) is a definition of what evidence is and how different types of evidence compare to one another. To determine the weight of evidence for the effectiveness of a particular health care intervention it is not enough to count the quotations supporting it and where a practice is supported by one type of evidence but not another, how do you decide what to believe? A not unusual example of this type of dilemma is where use of a product is supported by the testimony of many ‘delighted users’ while controlled trials suggest it is useless. To resolve the dilemma we need to assess the quality (reliability) and relevance (validity) of the competing types of evidence. To determine how reliable evidence is, we need to consider the accuracy of the data and any likely biases in the method of its collection. However, however reliable the evidence is, it needs to be relevant or valid to be of any use. Validity is determined by assessing whether the information answers the question we are asking. It is, perhaps, regrettable that a lot of effort has gone into determining and improving the reliability of techniques whose validity has never been ascertained. The importance of asking the right question cannot be overemphasized. In pressure sore prevention the question is usually ‘does an intervention reduce pressure sores and/or by how much’. Other considerations might be the effects of the intervention on patient comfort or activity. In wound care, the questions will relate to how quickly wounds heal, patient comfort, pain etc.

Occasionally intermediate outcomes can be used as a substitute for primary outcomes. However, there needs to be separate evidence of the direct relationship between the two outcomes. For example, it has been mathematically and experimentally shown that two balls of equal size but of differing mass fall at the same rate in a gravitational field. Therefore, the velocity of falling by a ball of a particular mass can be extrapolated to any other ball of identical size. However, it cannot be assumed that pressure relieving device A which produces lower interface pressures than device B is more effective at preventing pressure sores, as that relationship has never been established.

To make sense of these conflicting priorities, the technique of formally grading evidence and establishing a hierarchy was devised, analogous to that of historians. Within healthcare, clinical and research evidence has been graded according to the strength of the evidence. In wound care examples include the NPUAP¹, EPUAP², RCN³ and others. Historians quickly learned that such grading systems have to be used flexibly to be helpful.

Firstly, the fact that a piece of evidence belongs to a highly rated category does not guarantee it’s reliability. Any likely biases must be taken into account. In terms of accuracy not all evidence in a given grade is equal; rather, there is a sliding scale of reliability and indeed a degree of overlap between the grades. Secondly the ordering of the grading has to change, depending on the question being asked. Thirdly, any grading system and theories based upon it have to be reviewed as new techniques and new types of evidence are developed. The laws governing the ranking of sources are not immutable; the evidence has to be considered on its own merits rather than solely on its position in the rankings and it cannot be rejected out of hand just because it does not belong to a favoured category. Has the world of EBP taken on board the experiences of historians and in particular the need for flexibility? Both yes and no, but, sadly, mainly no. There is agreement that the quality of studies varies and depends on factors such as sample size, randomisation, comparability of experimental and control groups etc. However, rather than assessing each piece on its merits and then allotting it an appropriately sized place in the structure of the theory that is being developed, modern reviews of healthcare literature often assess evidence according to a rigid set of criteria. The evidence is then ‘accepted’ or ‘rejected’. Once accepted, the piece becomes a unit of evidence, artificially equivalent to any other accepted unit of evidence. There is even less acceptance for the concept of overlap of value of categories. Usually, only the members of one category (often the randomised controlled trial, RCT) are considered at all. This is unfortunate, since the strongest theories are often built from a triangulation of sources from different categories. Furthermore, techniques from lower categories in the hierarchy can be used to check the reliability of information found in higher ones. In an area such as wound care where there is a plethora of data, but few RCTs, it may be wasteful and counter-productive in the long run to completely ignore much of the data for the sake of an ill-conceived academic purity.

There seems to be a rigid dogma that the RCT is the only source of reliable answer, whatever the question. This view is maintained by the grading systems of all the organisations quoted earlier. In reality, the best methodology for any given question is one which has the fewest threats to validity and reliability built into it. The RCT is undoubtedly the best method for assessing the efficacy (achieving the desired effect under ideal conditions) of health care technologies or drugs. However, in wound care one is often assessing the effectiveness of policies or products in the real world. The best research method under these circumstances is the balanced incomplete block design or a cross-over study, while the RCT is considered second-grade⁴. This is because the biases caused by inter-subject contamination and Hawthorne effects that occur in RCTs, outweigh the effect of learning biases of the other types of methodology. To maintain that one hierarchy suits all circumstances is nonsensical and a different hierarchy has to be constructed for each type of question. Another anomaly of the current systems is the higher rating given to laboratory studies regardless of their validity. If one wants to know how effective a pressure relieving device is, anecdotal personal evidence is a valid source although of minimal reliability. Readings of interface pressure are not valid as the direct connection between interface pressure and efficacy has never been established. In conclusion, the process of grading evidence according to its weight is potentially enormously useful to help sort through the plethora of data that is available on wound care. However, if it is thoughtlessly and rigidly applied, it is equally likely to become a straight-jacket which will strangle further research and development. Indeed this baleful influence can even now be seen. Some manufacturers are claiming that since reviews show that there are few high quality RCTs there is little to choose between the evidence for various products and, as the carrying out of classical double-blind RCT’s is not possible, we should revert to provision of equipment by clinician’s whim. It is also sometimes suggested that this is somehow more ethical than carrying out the best research that can be done under the circumstances. Some clinicians are swallowing this hook, line and sinker.

K. S. Gebhardt

Clinical Nurse Specialist
Pressure Sore Prevention
St George’s Healthcare NHS Trust
London
UK

References

1. Bergstrom N, Allman RM, Alvarez OM, Bennett MA, Carlson CE, Frantz RA et al. Clinical practice guideline, number 15: Treatment of pressure ulcers. Rockville: Agency for Health Care Policy & Research, 1994: 18.
2. European Pressure Ulcer Advisory Panel. Pressure Ulcer Prevention Guidelines. EPUAP Review 1998; 1(1): 7–8.
3. Royal College of Nursing and Midwiferey. The management of patients with venous leg ulcers. London: the College, 1998.
4. Davies J, Freemantle N, Grimshaw J, Hurwitz B, Long A, Russell I, et al. Implementing clinical practice guidelines. Effective Health Care 1994; 1(8).