Key Learning Points

• Clinical policies / practice guidelines are "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances"; they "represent an attempt to distill a large body of medical knowledge into a convenient, readily usable format". (Institute of Medicine)
  
  
• Clinical policies help to differentiate "expert opinion" from research based knowledge.
  
  
• The ACEP Clinical Policies Committee was started in 1987; there are currently 15 clinical policies in existence. Average cost of development is $10,000 vs up to $2 million used by other organizations.
  
  
• www.guidelines.gov is an excellent source to identify clinical policies / guidelines that are in existence.
  
  
• Developing evidence based clinical policies requires focused questions with defined outcome measures. The process is encumbered by the heterogeneity of study populations and variance in outcome measures used.
  
  
• Dissemination and acceptance of clinical policies has proven difficult for a variety of reasons which include difficulties with changing established practice patterns.
  
  
• Early guidelines were primarily consensus based and thus influenced by bias.
  
  
• Evidence based guidelines use well defined methodologies that evaluate the quality of a study and provide recommendations that are graded according to the strength of the evidence available. Different societies use different grading systems and thus it is important for the reader to understand how the recommendations were developed.
  

TOP

Critical Appraisals of Clinical Policies

Introduction

Over the past ten years there has been a proliferation of clinical policies (also referred to as practice guidelines or practice parameters). It is estimated that there are close to three thousand policies in existence, written by a variety of organizations, fulfilling a number of different goals and agendas. 1 To facilitate guideline utilization, a National Guideline Clearinghouse has been established which can be accessed on the internet at http://www.guideline.gov. Readers of clinical policies are left with the task of trying to determine the value of the document and how to appropriately utilize its recommendations. This presentation is intended to provide background on clinical policy development and a systematic approach on how to assess a policy's merit.

History of Clinical Policies

There has been exponential growth in the development of clinical policies, although the concept itself is not new. The medical literature has been used for years in the form of texts and journal articles to guide physicians in the care of their patients. The first effort in creating policies was accomplished by the American Academy of Pediatrics in 1938. 2 Little additional work was done until the American College of Obstetrics and Gynecology began producing problem-specific advisories in 1959. This was followed by work by the American College of Physicians in 1980. In 1986, the American Society of Anesthesiologists published a twelve point practice guideline that not only improved patient care, but also resulted in a decrease in their medical malpractice premiums. 3, 4 This gave the area of guideline development significant visibility, causing great interest at the federal level, resulting in the establishment of the Agency for Health Care Policy and Research (AHCPR). The AHCPR was responsible for efforts at a national level in the development of guidelines, until its funding was cut by Congress in 1996. Currently, there are more than 60 specialty societies or physician interest groups involved in the development of clinical policies.

Efforts in emergency medicine clinical policy development was begun by the American College of Emergency Physicians in 1987. This effort was launched with a series of meetings with one of the pioneers in guideline development, DM Eddy. During those meetings, the terminology "clinical policy" was chosen, and the decision was made to concentrate on symptom or complaint-based policies due to the undifferentiated nature of emergency patients' complaints. Topics for the clinical policies were chosen from those complaints with high frequency, high risk, and / or high cost. The clinical policy on chest pain was the first in a series of such complaint-based policies developed and published by the ACEP Clinical Policies Committee in 1990. 5 In 1998, ACEP switched from clinical policies that took a general approach to undifferentiated disease, to evidence based policies that focused on answering specific, clinically relevant questions. All of ACEP's clinical policies are on a three year review cycle.

Definition and Assumptions

Clinical policies are documents that are developed to assist clinical decision making. 6 They are intended to assimilate the large volume of knowledge available from both the literature and from expert opinion into strategies helpful in diagnostic, management, and resource utilization decision making. Ultimately clinical policies have been used as tools in quality assurance programs, in creating health care policy, in directing research agendas, and in medicolegal determinations.

The full impact on health care delivery and outcomes for many clinical policies is not clear. Interestingly, clinical policy development has been driven by several assumptions that have yet to be validated. 6 These include the assumption that scientific evidence exists to support the recommendations; that mechanisms are available to fund the process; that policies will be distributed, read, and applied by the intended audience; that implementation will improve health care and that utilization will decrease health care costs. There are studies that show that distribution and acceptance of guidelines are problematic, 1, 7, 8, 9 and that implementation in some cases may actually increase cost of health care. 10 Distribution, implementation, and impact on health care are exciting areas for future study, and may be facilitated by combining clinical policies with computer technologies. 11

Clinical Policy Development

The methodologies used to develop clinical policies can be divided into two general categories: Those that are consensus driven, and those that are evidence based. A number of policies have used both methodologies. Evaluating policies involves understanding the rationale for why a policy was developed and how the final recommendations were derived. Under the most ideal of circumstances, a policy is developed to help readers comprehend and apply the large amount of literature available on a given subject and to provide sound recommendations based on the best available information. Unfortunately, at times clinical policies are developed to promote special interest agendas or to give a forum for an opinion or point of view that is not necessarily supported by scientific evidence. Consequently, it is important to fully understand the developmental process used in order to comprehend how to apply (or not apply) the policy's recommendations. 12

Consensus clinical policies: Consensus policies may be formulated either by an informal or a formal process. In informal consensus policy development, there is usually a group of experts who assemble, discuss the issues at hand, and draw their conclusions based on those discussions. This process may or may not involve some degree of literature review. This approach to clinical policy development has been described as "global subjective judgment" and is influenced by the bias that enters the decision making process. 13 ACEP's first chest pain policy is an example of this type of policy. 5 Though informal consensus can be reached by authoritative sources, it should be viewed with skepticism.

In formal consensus policy development, there is again a group of experts assembled but in this case there is an actual process in which the appropriate literature is reviewed and discussed. However, the final recommendations are ultimately determined by the panel's opinion or interpretation of the evidence. This process is limited by its lack of defined analytic procedures or clear criteria on factors used in creating recommendations. Therefore, formal consensus documents must also be viewed with caution since the experts' opinions and biases may have overridden the scientific evidence. An example of a formal consensus document is ACEP's 1997 "Clinical policy for the initial approach to patients presenting with a chief complaint of seizure who are not in status epilepticus". 14

Evidence based clinical policies: Evidence based clinical policies are emerging as the preferred method for policy development. 15, 16, 17, 18 In this method, appropriate literature is reviewed by a panel experienced in reading the literature, and each piece of evidence is graded according to set criteria. Recommendations are then made based on the strength of evidence that is available. Tables 1, 2, and 3 provide the criteria used by the ACEP Clinical Policies Committee in grading strength of evidence and generating recommendations. Examples of evidence based clinical policies are ACEP's Clinical Policy on Sedation and Analgesia, 19 and the American Academy of Neurologic Surgeon's Practice Guideline on Management of Severe Head Injury 20.

The absence of directed research may make the creation of a "standard" problematic. In these cases, expert opinion is often the only evidence available; an example of this is the use of oxygen in patients with suspected myocardial infarction. This is the circumstance that confronted ACEP's Clinical Policies Committee in its complaint based policies.

In some cases, resource availability may limit the implementation of a recommendation made in a clinical policy. However, in such cases, if there is strong strength of evidence to support the action, and the studies findings are externally valid, then the clinical policy's benefit to the health care system would be in effecting change in the system. An example might be forcing the immediate availability of a CT scanner in a hospital that has agreed with EMS to receive suspected stroke patients. Finally, there are situations where there is clear evidence to support an action but the issue at hand may not have value or relevance to society; in these cases the action will be driven by the societal value placed on an outcome. An example of this situation would be the intubation of the hypoxic patient who is end-stage with a terminal process.

Once a clinical policy is developed, its recommendations can be constructed into an implementation tool such as a "clinical pathway" or an "annotated algorithm" which take into account the resources available. 21 However, it is critical that these flow charts allow for practice variability as determined by the strength of evidence discovered in the development process.

Critically Assessing a Clinical Policy

Why was the topic chosen and are the objectives and goals clearly stated: The first step in evaluating the value of a clinical policy is assessing why the topic was chosen, e.g., to improve patient care, to lower cost, to support an organization's practice preference. Evaluating the funding source for potential conflict of interest can identify factors that may have influenced or biased the policy's recommendations. In the case of the clinical policies developed by ACEP, the topics were selected based on their high frequency, high cost, or high risk. Though the motivation was correct, the complaint based topics had limited research to support an evidence based approach to policy development. Other organizations have avoided this difficulty by choosing focused topics such as neuroimaging in first time seizures, 22 or indications for hyperventilation in severe head trauma 20. All funding for ACEP's clinical policies comes from within the organization; no outside funding, e.g., unrestricted educational grants from pharmaceutical companies, is used.

What are the credentials of the policy's authors and do they have the expertise to competently assess the subject: Clinical policy development is a complex process that requires both clinical expertise and proficiency in the scientific process of evaluating the quality of research. Panel members should include both clinicians who understand the practice environment and scientists who can assist in properly evaluating the strength of evidence available in the literature. It is important to be aware of the writers' point of reference. Clearly an organization with a financial or power interest in a specific protocol may be biased towards certain conclusions. The creation of joint (multi-specialty) policy development groups including all affected specialties helps to minimize this bias. Jointly developed policies may promote open discussions and strengthen the utilization of evidence-based methodologies to support recommendations. Joint clinical policies are costly and require significant organization; as a compromise, many groups send out draft policies for multidisciplinary / multi-specialty comment. Documentation of multi-specialty involvement in the creation of a clinical policy enhances the policy's credibility.

What methodology was used to generate the policies recommendations: The clinical policy should clearly define how it went about collecting the scientific evidence it used in generating its recommendations. This process usually takes the form of a reference data base search with explosion of terms as needed. The clinical policy must then describe the mechanism used in evaluating the literature reviewed and assigning it a strength of evidence. Lastly, a description of how recommendations were generated is key including the emphasis placed on consensus opinion when clear scientific evidence was not available. In general, consensus opinion alone is not sufficient to scientifically define a standard of care.

Format: The utility of a clinical policy in clinical practice is largely dependant on its format, which determines the readability, accessibility, and retrievability of information in the policy. Certain topics are amenable to decision rule making while other topics can only be addressed by case-based reasoning. Such topics may allow the development of evidence statements but may not be transformed into decision rules or algorithms. Full text formats allow for ample justification but may be problematic when looking for a specific item if they are not extensively indexed. Algorithm formats are generally easy to follow but do not deal with complex decision making. They are compatible with use as decision support tools and to be effective, algorithm formats should show their evidence justification. No one format meets all needs.23

Was the clinical policy field tested or reviewed in clinical practice: A key component to a clinical policy is its relevance to clinical practice. Field testing and/or careful review by practicing physicians of a clinical policy during its development process can assist in evaluating its relevance. The endpoint in a management recommendation is multi-factorial. Ideally, a policy should take into consideration outcomes that are important to the patient, such as quality of life. 12

When was the policy written, and is there a mechanism to keep it current: Clinical practice is constantly changing as new evidence becomes available. Clinical policies can take years to develop and can become outdated quickly. An example is the changing recommendations related to stroke management. Consequently, date of publication is key to check before accepting a recommendation of a clinical policy.

Clinical Policies and Malpractice Litigation

The development of clinical policies has generated concern that they provide fuel for malpractice litigation. In actuality, evidence-based guidelines generate very few "standards" and generally allow for practice to be tailored to specific patient presentations. In some cases, guidelines may serve to defuse litigation by demonstrating the absence of a scientific basis of some action that an "expert witness" might attest to as "standard of care". Unfortunately, at times a "standard of care" is still defined by how the care is provided in the community, and not by how it is defined by the best available scientific evidence. 24

The role of clinical policies in defining medical practice is evolving. The State of Maine studied the impact of a cervical spine clinical policy, in addition to 20 other policies, on malpractice litigation. 25, 26 The state legislature allowed an affirmative defense for physicians agreeing in advance to follow specified practice guidelines. No cases of malpractice arose during the four year study period but the project demonstrated the interest of states in using clinical policies to define health care. Other states, including Vermont, Florida, and Minnesota, have initiated programs exploring the role of clinical policies in arbitrating malpractice claims. 25

In a report to the Physician Payment Review Commission, Hyams and colleagues performed a three component assessment of clinical policy utilization in malpractice litigation. 27 In component one, a computerized legal search was performed for any U.S. court's use of practice guidelines and standards in medical malpractice litigation from January 1, 1980 to September 30, 1993. Thirty-two cases were identified where guidelines were used to demonstrate that a practitioner departed or adhered to the required standard of care. There were 23 cases where guidelines were used successfully: In 17 cases the guidelines were used by the plaintiff as evidence that the practitioner departed from the standard of care; in 6 cases the guidelines were used by the defense. There were 9 cases where guidelines were used unsuccessfully, 6 by plaintiffs and 3 by the defense. The majority of the cases referenced the American College of Obstetrics and Gynecology's policies; there were no emergency medicine cases cited.

In component two of Hyams and colleagues report, a random sample of 259 claims from two insurance carriers were pooled; 17 of the claims (6.6%) involved the use of practice guidelines, 7 of which involved obstetrics, and 4 involved family practice. In component three, 980 attorneys were surveyed. There was a 60% response rate which showed that approximately 75% of attorneys were aware of practice guidelines, while 36% reported that they had at least one case per year in which guidelines had played an important role. Twenty-five per cent of attorneys reported that a guideline had influenced their decision to settle a case, and 25% reported that a guideline had led them not to take a case.

Clinical policy development has been driven by a need to summarize the medical literature in a critical and constructive way. Recommendations made in clinical policies should allow for practice flexibility that is tailored to an individual patient's situation. Strict "standards of care" are rarely established and the format of grading recommendations based on strength of evidence serves to reinforce the fact that the practice of medicine is an art that must take into consideration a number of variables. From a medicolegal point of view, clinical policies should be a framework; deviations from recommendations are allowed as long as there is appropriate documentation demonstrating the necessity to modify care to meet a particular patient's needs.

Recommendations for Research

Within the realm of health care, the successes and failures of current practice guidelines go largely unevaluated. Evidence-based medicine has brought to light the reality that the diagnosis, treatment and management of patients is frequently based on limited scientific information. Recommendations made in clinical policies should ideally be validated following the development process. To test the utility and efficacy of a clinical policy, outcomes must be operationally defined; that is, study endpoints must be clearly indicated along with plans for their measure. 28

Policy analysts need to concur about the type and volume of data to be collected for policy testing. Additionally, the instruments selected for measuring outcomes must have demonstrated validity and reliability. For example, a policy developed for the purpose of increasing patient compliance with ED discharge instruction should include a protocol which defines what is meant by "compliance", and which specifies how it will be measured. In this scenario, a standardized, validated measure of patient compliance would be the ideal. Furthermore, a research study, which includes the design elements of randomization and blinding, would strengthen the testing approach.

Clinical policy development has set the stage for an explosion of research ideas. The policies themselves provide the impetus for studies measuring their impact, offering an infinite number of hypotheses for testing. It is essential that clinical policies be coupled with strategies to measure their success in meeting predetermined goals. Future research is needed to identify valid, reliable measures of clinical endpoints and outcomes. Last but not least, the dissemination of a clinical policy must be systematically evaluated. Survey research of health care providers is imperative to determine the extent to which intended audiences know about, and adhere to, a clinical policy.

Conclusion

Clinical policies have been developed to facilitate the provision of efficient, comprehensive care. When properly developed they are valuable tools to facilitate resource utilization and management decision making. Clinical policies should rely on sound scientific evidence for generating "standards" or "rules", and default to expert consensus only when necessary. Some organizations will not create "standards" that are based solely on expert consensus. It is critical in assessing a clinical policy that the methodology used in generating its recommendations be carefully examined. In general, clinical policies should have few standards and provide sufficient flexibility for the clinician to individualize care. There is a need for research on the optimal use of clinical policies, particularly in the area of impact on outcome and implications on patient preferences.

TOP

1. US Congress, Office of Technology Assessment. Identifying health technologies that work: Searching for evidence. Washington, D.C.: US Government Printing Office; September 1994: 177-198. OTA-H-608.

2. General Accounting Office. Practice guidelines: The experience of medical specialty societies.
U.S. Government Printing Office. GAO/PEMD-91-11, February 1991.

3. Eichorn J. Prevention of intraoperative anesthesia accidents and related severe injury through safety monitoring. Anesthesiology 1989; 70:572-577.

4. Holzer J. The advent of clinical standards for professional liability. QRB 1990; 16:71-79.

5. American College of Emergency Physicians. Clinical policy for management of adult patients presenting with a chief complaint of chest pain, with no history of trauma. Dallas: American College of Emergency Physicians, 1990.

6. Field M, Lorh K (eds). Guidelines for Clinical Practice: From Development to Use. Committee on Clinical Practice Guidelines, Division of Health Care Services of the Institute of Medicine. Washington, DC: National Academy Press, 1994.

7. Wigder H, Arai D, Narasimhan K, Cohan S. ACEP chest pain policy: Emergency physician awareness. Ann Emerg Med 1996; 27:606-609.

8. Lomas J, Anderson G, Domnick-Pierre E, et al. Do practice guidelines guide practice? The effect of a consensus statement on the practice of physicians. N Engl J Med 1989;321:1306-1311.

9. Kosecoff J, Kanouse D, Rogers W, et al. Effects of the National Institutes of Health consensus development program on physician practice. JAMA 1987; 258:2708-2713.

10. Ellrodt A, Conner L, Riedinger M, Weingarten S. Measuring and improving physician compliance with clinical practice guidelines. Ann Int Med 1995;122:277-282.

11. Schriger D, Baraff L, Rogers W, Cretin S. Implementation of clinical guidelines using a computer charting system. JAMA 1997; 278:1585-1590.

12. Hayward R, Wilson M, Tunis S, et al. Users' Guide to the medical literature VIII. How to use clinical practice guidelines. JAMA 1995; 274-570-574.

13. Eddy D. Practice Policies: Where Do They Come From? JAMA 1990;1269-1275.

14. ACEP Clinical Policies Committee. Clinical policy for the initial approach to patients presenting with a chief complaint of seizure who are not in status epilepticus. Ann Emerg Med 1997; 29:706-724.

15. American Medical Association. Attributes to Guide the Development of Practice Parameters. Chicago, IL: American Medical Association; 1990.

16. Woolf S. Practice Guidelines, a New Reality in Medicine: II. Methods of Developing Guidelines. Arch Int Med 1992; 152:946-952.

17. Institute of Medicine. Guidelines for clinical practice: From Development to Use. Washington D.C: National Academy Press; 1992.

18. Gray J, Haynes R, Sackett D, et al. Transferring evidence from research into practice: Developing evidence-based clinical policy. Evid Based Med 1997; 2:36-38.

19. American College of Emergency Physicians. Clinical policy for procedural sedation and analgesia in the emergency department. Ann Emerg Med 1998;31:663-667.

20. Brain Trauma Foundation. Guidelines for the management of severe head injury. Brain Trauma Foundation, New York, 1995.

21. Hadorn D, McCormick K, Diokno A. An annotated algorithm approach to clinical guideline development. JAMA 1992; 267:3311-3314.

22. American College of Emergency Physicians, American Academy of Neurology, American Association of Neurological Surgeons, American Society of Neuroradiology. Practice parameter: Neuroimaging in the emergency patient presenting with seizure (summary statement). Ann Emerg Med 1996;27:114-118.

23. Adern R, Hodgson L, Hamer D. Flow diagram or prose: does the format of practice guidelines matter? Euro J Emerg Med 1998;5:241-244.

24. Garnick D, Hendricks A, Brennan T. Can practice guidelines reduce the number and cost of malpractice claims. JAMA 1991 266:2856-2860.

25. Hong D, Liang B. The scope of clinical practice guidelines. Hosp Physcian 1996;59:46-53.

26. Smith G. A case study in progress: Practice guidelines and the affirmative defense in Maine. J Qual Improv 1993; 19:355-362.

27. Hyams A, Brandenburg J, Lipsitz S, Brennan T. Report to physician payment review commission practice guidelines and malpractice litigation. Physician Review Commission Grant No. 92-G04, January 25, 1994.

28. Cairns C, Garrison H, Hedges J, et al. Development of New Methods to Assess the Outcomes of Emergency Care. Ann Emerg Med 1998; 31:166-171.

29. Candadian Task Force on the Periodic Health Examination. The periodic health examination. Can Med Assoc J 1979; 121:1193-1254.

30. Agency for Health Care Policy and Research. Acute low back problems in adults: Assessment and treatment. AHCPR Publication No. 95-0643, US Department of Health and Human Services, 1994.