This presentation addresses the questions:

Over the past ten years there has been a proliferation of clinical policies (also referred to as practice guidelines, practice parameters).  It is estimated that there are close to two thousand policies in existence, written by a variety of organizations, fulfilling a number of different goals and agendas. ¹  Readers of clinical policies are left with the task of trying to determine the value of the document and how to appropriately utilize its recommendations.  This report is intended to provide background on clinical policy development and a systematic approach on how to assess the policy’s merit.

 

History of Clinical Policies:  The last decade has seen exponential growth in the development of clinical policies, although the concept itself is not new.  The medical literature has been used for years in the form of texts and journal articles to guide physicians in the care of their patients.  The first effort in creating policies was accomplished by the American Academy of Pediatrics in 1938. ²  Little additional work was done until the American College of Obstetrics and Gynecology began producing problem-specific advisories in 1959.  This was followed by work by the American College of Physicians in 1980.   In 1986, the American Society of Anesthesiologists published a twelve-point practice guideline that not only improved patient care, but also resulted in a decrease in their medical malpractice premiums. ^{3, 4}  This gave the area of guideline development significant visibility, causing great interest at the federal level, resulting in the establishment of the Agency for Health Care Policy and Research (AHCPR).  The AHCPR was responsible for efforts at a National level in the development of guidelines, until its funding was cut by Congress in 1996.  Currently, there are more than 60 specialty societies or physician interest groups involved in the development of clinical policies.

 

Efforts in emergency medicine clinical policy development were begun by the American College of Emergency Physicians in 1987.  This effort was launched with a series of meetings with one of the pioneers in guideline development, DM Eddy.  During those meetings, the terminology “clinical policy” was chosen, and the decision was made to concentrate on symptom or complaint-based policies due to the undifferentiated nature of emergency patients’ complaints.  Topics for the clinical policies were chosen from those complaints with high frequency, high risk, and / or high cost.  The clinical policy on chest pain was the first in a series of such complaint-based policies developed and published by the ACEP Clinical Policies Committee. ⁵ Twelve clinical policies have been published to date; plus one clinical policy developed jointly by ACEP and other organizations, see Table 1.  ACEP clinical policies are on a three-year review cycle.

 

Definitions and assumptions:  Clinical policies are documents that are developed to assist clinical decision-making. ⁶  They are intended to facilitate combining the large volume of knowledge available from both the literature and from expert opinion into strategies helpful in diagnostic, management, and resource utilization decision-making.  Ultimately clinical policies have been used as tools in quality assurance programs, in creating health care policy, in directing research agendas, and in medicolegal determinations.

 

Despite the large number of clinical policies, their impact on health care delivery and outcomes is not clear.  Interestingly, clinical policy development has been driven by several assumptions that have yet to be validated. ⁶  These include the assumption that scientific evidence exists to support the recommendations; that mechanisms are available to fund the process; that policies will be distributed, read, and applied by the intended audience; that implementation will improve health care; that utilization will decrease health care costs.  There are studies that show that distribution and acceptance of guidelines are  problematic, ^{1, 7, 8, 9} and that implementation in some cases may actually increase cost of health care. ¹⁰  Distribution, implementation, and impact on health care are exciting areas for future study, and may be facilitated by combining clinical policies with computer technologies. ¹¹

 

How are clinical policies developed?

The methodologies used to develop clinical policies can be divided into two general categories:  Those that are consensus driven, and those that are evidence based.  A  number of policies have used both methodologies.  Evaluating policies involves understanding the rationale for why a policy was developed and how the final recommendations were derived.  Under the most ideal of circumstances, a policy is developed to help readers comprehend and apply the large amount of literature available on a given subject and to provide sound recommendations based on the best available information.  Unfortunately, at times clinical policies are developed to promote special interest agenda or to give a forum for an opinion or point of view that is not necessarily supported by scientific evidence.  Consequently, it is important to fully understand the developmental process used in order to comprehend how to apply (or not apply) the policy’s recommendations. ¹²

 

Consensus clinical policies: Consensus policies are formulated either by an informal process or by a formal process.  In informal consensus policy development, there is usually a group of experts who assemble, discuss the issues at hand, and draw their conclusions based on those discussions.  This process has been described as “global subjective judgment” and is influenced by the bias that enters the decision making  process. ¹³  Though informal consensus can be reached by authoritative sources, it should be viewed with skepticism.

 

In formal consensus policy development, there is again a group of experts assembled but in this case there is an actual process in which the appropriate literature is reviewed and discussed.  However, the final recommendations are ultimately determined by the panel’s opinion or interpretation of the evidence.  This process is limited by its lack of defined analytic procedures or clear criteria on factors used in creating recommendations.  Therefore, formal consensus documents must also be viewed with caution since the experts’ opinions and biases may have overridden the scientific evidence.  An example of a formal consensus document is ACEP’s “Clinical policy for the initial approach to patients presenting with a chief complaint of seizure who are not in status epilepticus”. ¹⁴

 

Evidence based clinical policies:  Evidence based clinical policies are emerging as the preferred method for policy development. ^{15, 16, 17, 18}  In this method, appropriate literature is reviewed by a panel experienced in reading the literature, and each piece of evidence is graded according to set criteria.  Recommendations are then made based on the strength of evidence that is available. Table 2 provides the criteria presently used by ACEP in grading strength of evidence and generating recommendations.  An example of an evidence based clinical policy is ACEP’s Clinical Policy on Sedation and Analgesia, ¹⁹  or the American Academy of Neurologic Surgeon’s practice guideline on management of severe head injury ²⁰.

 

One of the factors limiting the development of evidence-based policies is the lack of directed research on the subject to support a clear recommendation, such as the use of oxygen in acute myocardial infarction.  The lack of appropriate evidence makes the creation of a  “standard” problematic.  In these cases, expert opinion is often the only evidence available.  This is the circumstance that confronted ACEP’s Clinical Policies Committee in its first nine policies, which were complaint based. In those policies, a “rule” was defined as “an action reflecting principles of good practice in most situations”.  A “guideline” was defined as “an action that should be considered but may or may not be performed, depending on the patient, the circumstances, or other factors”.

 

In some cases, resource availability may limit the implementation of a recommendation made in a clinical policy.   However, in such cases, if  there is strong strength of evidence to support the action, and the studies findings are externally valid, then the clinical policy’s benefit to the health care system would be effecting change in the system.  An example might be forcing the immediate availability of a CT scanner in a hospital that has agreed with EMS to receive suspected stroke patients.  Finally, there are situations where there is clear evidence to support an action but the issue at hand may not have value or relevance to society; in these cases the action will be driven by the societal value placed on an outcome. 

 

An emerging type of clinical policy is the “explicit guideline” in which not only is strength of evidence used to generate recommendations but the actual benefits, harms, and costs of potential interventions are assessed. ¹⁶   These “explicit estimates” are presented in a tabulated form and provide a format for assessing the desirability of a particular outcome as well as taking into account patient preferences.  This process is not only complex but it also is expensive and remains to be tested as a worthwhile approach to guideline development.

 

Once a clinical policy is developed, its recommendations can be constructed into an implementation tool such as a “clinical pathway” or an “annotated algorithm” which take into account the resources available. ²¹  However, it is critical that these flow charts allow for practice variability as determined by the strength of evidence discovered in the development process.

 

How does one critically assess clinical policies?

Why was the topic chosen:  The first step in evaluating the value of a clinical policy is assessing why the topic was chosen.    The policy should have stated objectives and goals.  In the case of the clinical policies developed by ACEP, the topics were selected from the list of most common risk management issues in the practice of emergency medicine.  Consequently, topics such as chest pain, blunt abdominal trauma, seizures, and headache were chosen.  Though the motivation was correct, the difficulty in this approach was choosing topics that did not lend themselves to a comprehensive, evidence-based approach.  Other organizations have avoided this pitfall by choosing focused topics such as neuroimaging in first time seizures, ²² or indications for hyperventilation in severe head trauma ²⁰.  Additionally, the goals of the policy should be clearly articulated since the subsequent evaluation of the policy can only be fairly accomplished if linked to the goals.

 

What are the credentials of the policy’s authors and do they have the expertise to competently assess the subject:  Clinical policy development is a complex process that requires both clinical expertise and proficiency in the scientific process of evaluating the quality of research.  Panel members should include both clinicians who understand the practice environment and scientists who can assist in properly evaluating the strength of evidence available in the literature.  It is important to be aware of the writers’ point of reference.  Clearly an organization with a financial or power interest in a specific protocol may be biased towards certain conclusions.  The creation of joint (multispecialty) policy development groups including all affected specialties helps to minimize this bias.  Jointly developed policies may promote open discussions and strengthen the utilization of evidence-based methodologies to support recommendations.  Joint clinical policies are costly and require significant organization; as a compromise, many groups send out draft policies for multidisciplinary / multispecialty comment.  Documentation of multispecialty involvement in the creation of a clinical policy enhances the policy’s credibility.

 

What methodology was used to generate the policies recommendations: The clinical policy should clearly define how it went about collecting the scientific evidence it used in generating its recommendations.  This process usually takes the form of a reference data base search with explosion of terms as needed.  The clinical policy must then describe the mechanism used in evaluating the literature reviewed and assigning it strength of evidence.  Lastly, a description of how recommendations were generated is key including the emphasis placed on consensus opinion when clear scientific evidence was not available.  In general, consensus opinion alone is not sufficient to define a “standard of care”.

Was the clinical policy field tested or reviewed in clinical practice:  A key component to a clinical policy is its relevance to clinical practice.  The endpoint in a management recommendation is multifactorial.  Ideally, a policy should  take into consideration outcomes that are important to the patient, such as quality of life. ¹²

 

Format:  Whether a clinical policy can ultimately be useful in clinical practice or in  administration depends on its format.  The format will largely determine the readability, accessibility, and retrievability of information.  Algorithm formats are generally easy to follow but rarely can show the evidence justification or deal with complex decision-making.  Full text formats allow for ample justification but are problematic when looking for a specific item.  No one format meets all needs.²³

 

When was the policy written, and is there a mechanism to keep it current:  Clinical practice is constantly changing as new evidence becomes available.  Clinical policies can take years to develop and can become dated quickly.  An example is the changing recommendations related to stroke management.  Consequently, date of publication is key to check before accepting a recommendation in a clinical policy.

 

What are the medical legal implications of clinical policies?

The development of clinical policies has generated concern that they provide fuel for malpractice litigation.  In actuality, evidence-based guidelines generate very few “standards” and generally allow for practice to be tailored to specific patient presentations.  In some cases, guidelines may serve to defuse litigation by demonstrating the absence of a scientific basis of some action that an “expert witness” might attest to as “standard of care”.  Unfortunately, at times a “standard of care” is still defined by how the care is provided in the local or regional community, and not by how it is defined by national practice, ²⁴   however, this is becoming less frequently the case.

 

The role of clinical policies in defining medical practice is evolving.  The State of Maine studied the impact of a cervical spine clinical policy, in addition to 20 other policies, on malpractice litigation. ^{25, 26}  The state legislature allowed an affirmative defense for physicians agreeing in advance to follow specified practice guidelines.  No cases of malpractice arose during the four-year study period but the project demonstrated the interest of States in using clinical policies to define health care.  Other states, including Vermont, Florida, and Minnesota, have initiated programs exploring the role of clinical policies in arbitrating malpractice claims. ²⁵

In a report to the Physician Payment Review Commission, Hyams and colleagues performed a three-component assessment of clinical policy utilization in malpractice litigation. ²⁷  In component one, a computerized legal search was performed for any U.S. court’s use of practice guidelines and standards in medical malpractice litigation from January 1, 1980 to September 30, 1993.  Thirty-two cases were identified where guidelines were used to demonstrate that a practitioner departed or adhered to the required standard of care.  There were 23 cases where guidelines were used successfully: In 17 cases the guidelines were used by the plaintiff  as evidence that the practitioner departed from the standard of care; in 6 cases the guidelines were used by the defense.  There were 9 cases where guidelines were used unsuccessfully, 6 by plaintiffs and 3 by the defense.  The majority of the cases referenced the American College of Obstetrics and Gynecology’s policies; there were no emergency medicine cases cited.  

 

In component two of Hyams and colleagues report,  a random sample of 259 claims from two insurance carriers were pooled; 17 of the claims (6.6%) involved the use of practice guidelines, 7 of which involved obstetrics, and 4 involved family practice.  In component three, 980 attorneys were surveyed.  There was a 60% response rate that showed approximately 75% of attorneys were aware of practice guidelines, while 36% reported that they had at least one case per year in which guidelines had played an important role.  Twenty-five per cent of attorneys reported that a guideline had influenced their decision to settle a case, and 25% reported that a guideline had led them not to take a case. 

 

Clinical policy development has been driven by a need to summarize the medical literature in a critical and constructive way.  Most recommendations made in clinical policies allow for practice flexibility that is tailored to an individual patient’s situation.  Strict “standards of care” are rarely established and the format of grading recommendations based on strength of evidence serves to reinforce the fact that the practice of medicine is an art that must take into consideration a number of variables.  From a medicolegal point of view, clinical policies are a framework; deviations from recommendations are allowed as long as there is appropriate documentation demonstrating the necessity to modify care to meet a particular patient’s needs.  

 

Recommendations for research

Within the realm of health care, the successes and failures of current practice guidelines go largely unevaluated.  Evidence-based medicine has brought to light that practice relating to the diagnosis, treatment and management of patients is frequently based on limited scientific information.   Recommendations that are not based on sufficient strength of evidence should undergo formal testing following policy development.   To test the utility and efficacy of a clinical policy, outcomes must be operationally defined; that is, study endpoints must be clearly indicated along with plans for their measure. ²⁸

 

Policy analysts need to concur about the type and volume of data to be collected for policy testing.   Additionally, the instruments selected for measuring outcomes must have demonstrated validity and reliability.   For example, a policy developed for the purpose of increasing patient compliance with ED discharge instruction should include a protocol which defines what is meant by “compliance”, and which specifies how it will be measured.  In this scenario, a standardized, validated measure of patient compliance would be the ideal.  Furthermore, a research study, which includes the design elements of randomization and blinding, would strengthen the testing approach.

 

Clinical policy development has set the stage for an explosion of research ideas.   The policies  themselves provide the impetus for studies measuring their impact,  offering an infinite number of hypotheses for testing.   It is essential that clinical policies are coupled with strategies to measure their success in meeting predetermined goals.   Future research is needed to identify valid, reliable measures of clinical endpoints.  Other important areas for study include patient acceptability of a policy, satisfaction with care, resource utilization and fiscal outcome.  Last but not least, the dissemination of policy must be systematically evaluated.  Survey research of health care providers is imperative to determine the extent to which intended audiences know about, and adhere to, a clinical policy.

 

Conclusions

Clinical policies have been developed to facilitate the provision of efficient, comprehensive care.  When properly developed they are valuable tools to facilitate resource utilization and management decision making.  Most experts recommend the use of scientific evidence for generating  “standards”, and rely on expert consensus only when necessary.  Some organizations will not create “standards” that are based solely on expert consensus.  It is critical in assessing a clinical policy that the methodology used in generating its recommendations be examined. Many experts believe that clinical policies should have few standards and should incorporate significant flexibility for the clinician to determine the applicability of recommendations for individual patients.  There is a need for research on the optimal use of clinical policies, particularly in the area of impact on  outcome and implications on patient preferences.

 

 

TABLE 1: CLINICAL POLICIES DEVELOPED BY ACEP

 

Clinical policy for the initial approach to children under the age of 2 years presenting with fever. Ann Emerg Med 1993;22:628-637

 

Clinical policy for the initial approach to patients presenting with a chief complaint of nontraumatic acute abdominal pain. Ann Emerg Med 1994;23:906-922

 

Clinical policy for the initial approach to patients presenting with penetrating extremity trauma. Ann Emerg Med 1994;23:1147-1156.

 

Clinical policy for the initial approach to adults presenting with a chief complaint of chest pain, with no history of trauma. Ann Emerg Med 1995;25:274-299. (Revision of 1990 policy)

 

Clinical policy for the initial approach to patients presenting with acute toxic ingestion or dermal or inhalation exposure. Ann Emerg Med 1995;25:570-585.

 

Clinical policy for the initial approach to adolescents and adults presenting to the emergency department with a chief complaint of headache. Ann Emerg Med 1996;27:821-844.

 

Clinical policy for the initial approach to patients presenting with a chief complaint of vaginal bleeding. Ann Emerg Med 1997;29:435-458.

 

Clinical policy for the initial approach to patient presenting with a chief complaint of seizure who are not in status epilepticus. Ann Emerg Med 1997;29;706-724.  (Revision of 1993 policy)

 

Clinical policy for the initial approach to patients presenting with acute blunt trauma. Ann Emerg Med 1998;31:422-454. (Revision of 1993 policy)

 

Clinical policy: Procedural sedation and analgesia in the Ed. Ann Emerg Med 1998;31:663-677.

 

Clinical policy for the initial approach to patients presenting with altered mental status.  Ann Emerg Med, 1999

 

Clinical policy for the initial approach to patients presenting with syncope. Ann Emerg Med 2000

TABLE 2: CRITERIA USED BY ACEP TO GRADE EVIDENCE AND GENERATE STRENGTH OF RECOMMENDATIONS

 

Strength of evidence I: Unbiased* interventional studies including clinical trials¹, observational studies² including prospective cohort studies, aggregate studies including prospective cohort studies, aggregate studies including meta-analyses of randomized clinical trials only.

 

Strength of evidence II: Unbiased* observational studies including retrospective cohort studies, case-control studies, cross-sectional studies; aggregate studies including other metaanalyses.

 

Strength of evidence III: Unbiased* observational reports including case series, case reports; consensual studies including published panel consensus by acknowledged groups of experts.

 

 

Recommendation A: Generally accepted principles for patient management that reflect a high degree of clinical certainty.  Standards generally based on “strength of evidence A” or overwhelming evidence from “strength of evidence B” studies that directly address the question at hand or from decision analysis that directly address all issues.**

 

Recommendation B: Recommendations for patient management that may identify a particular strategy or range of mange strategies that reflect moderate clinical certainty.  Guidelines are generally based on “strength of evidence B” studies that directly addresses the issue, decision analysis that directly addressed the issue, or strong consensus of “strength or evidence C” literature.

 

Recommendation C: Other strategies for patient management for which the clinical utility is uncertain.  Options are based on inconclusive or conflicting evidence.

 

 

 

*          Bias can limit the value of a study; consequently, a study can be downgraded to a lower level of evidence, or, if severely biased, eliminated completely from consideration.

 

**         A standard can not be established by consensus from experts regardless of their credentials unless there is the appropriate, quality research available.

 

¹                Randomized, double-blind, placebo controlled studies allow for the least amount of bias and highest degree of validity.  This experimental design is used primarily to test therapeutic effectiveness. 

 

2              Observational designs are used to study issues such  that do not lend themselves to experimental methodologies.  The prospective cohort is the prototypic analytical observational study and when designed to minimize bias can provide a high level of evidence.

 

TOP

1.   US Congress, Office of Technology Assessment. Identifying health technologies that work: Searching for evidence. Washington, D.C.: US Government Printing Office; September 1994: 177-198. OTA-H-608.

 

2.   General Accounting Office. Practice guidelines: The experience of medical specialty societies. U.S. Government Printing Office. GAO/PEMD-91-11, February 1991.

 

3.   Eichorn J. Prevention of intraoperative anesthesia accidents and related severe injury through safety monitoring.  Anesthesiology 1989; 70:572-577.

 

4.   Holzer J. The advent of clinical standards for professional liability. QRB 1990; 16:71-79.

 

5.   American College of Emergency Physicians. Clinical policy for management of adult patients presenting with a chief complaint of chest pain, with no history of trauma. Dallas: American College of Emergency Physicians, 1990.

 

6.   Field M, Lorh K (eds). Guidelines for Clinical Practice: From Development to Use. Committee on Clinical Practice Guidelines, Division of Health Care Services of the Institute of Medicine. Washington, DC: National Academy Press, 1994.

 

7.   Wigder H, Arai D, Narasimhan K, Cohan S. ACEP chest pain policy: Emergency physician awareness. Ann Emerg Med 1996; 27:606-609.

 

8.   Lomas J, Anderson G, Domnick-Pierre E, et al. Do practice guidelines guide practice? The effect of a consensus statement on the practice of physicians. N Engl J Med 1989;321:1306-1311.

 

9.         Kosecoff J, Kanouse D, Rogers W, et al. Effects of the National Institutes of Health consensus development program on physician practice. JAMA 1987; 258:2708-2713.

 

10.  Ellrodt A, Conner L, Riedinger M, Weingarten S. Measuring and improving physician compliance with clinical practice guidelines. Ann Int Med 1995;122:277-282.

 

11.  Schriger D, Baraff L, Rogers W, Cretin S. Implementation of clinical guidelines using a computer charting system. JAMA 1997; 278:1585-1590.

 

12.  Hayward R, Wilson M, Tunis S, et al. Users’ Guide to the medical literature VIII. How to use clinical practice guidelines. JAMA 1995; 274-570-574.

 

13.  Eddy D. Practice Policies: Where Do They Come From? JAMA 1990;1269-1275.

 

14.  ACEP Clinical Policies Committee. Clinical policy for the initial approach to patients presenting with a chief complaint of seizure who are not in status epilepticus. Ann Emerg Med 1997; 29:706-724.

 

15.  American Medical Association. Attributes to Guide the Development of Practice Parameters. Chicago, IL: American Medical Association; 1990.

 

16.  Woolf S. Practice Guidelines, a New Reality in Medicine: II. Methods of Developing Guidelines. Arch Int Med 1992; 152:946-952.

 

17.        Institute of Medicine. Guidelines for clinical practice: From Development to Use. Washington D.C: National Academy Press; 1992.

 

18.        Gray J, Haynes R, Sackett D, et al. Transferring evidence from research into practice: Developing evidence-based clinical policy. Evid Based Med 1997; 2:36-38.

 

19.            American College of Emergency Physicians. Clinical policy for procedural sedation and analgesia in the emergency department.  Ann Emerg Med 1998;31:663-667.

 

20.        Brain Trauma Foundation. Guidelines for the management of severe head injury. Brain Trauma Foundation, New York, 1995.

 

21.        Hadorn D, McCormick K, Diokno A. An annotated algorithm approach to clinical guideline development. JAMA 1992; 267:3311-3314.

 

22.            American College of Emergency Physicians, American Academy of Neurology, American Association of Neurological Surgeons, American Society of Neuroradiology. Practice parameter: Neuroimaging in the emergency patient presenting with seizure (summary statement). Ann Emerg Med 1996;27:114-118.

 

23.        Hadern R, Hodgson L, Hamer D. Flow diagram or prose: does the format of practice guidelines matter?  Euro J Emerg Med 1998;5:241-244.

 

24.        Garnick D, Hendricks A, Brennan T. Can practice guidelines reduce the number and cost of malpractice claims. JAMA 266:2856-2860.

 

25.        Hong D, Liang B. The scope of clinical practice guidelines. Hosp Physcian 1996;59:46-53.

 

26.        Smith G. A case study in progress: Practice guidelines and the affirmative defense in Maine. J Qual Improv 1993; 19:355-362.

 

27.        Hyams A, Brandenburg J, Lipsitz S, Brennan T. Report to physician payment review commission practice guidelines and malpractice litigation. Physician Review Commission Grant No. 92-G04, January 25, 1994.

 

28.        Cairns C, Garrison H, Hedges J, et al. Development of New Methods to Assess the Outcomes of Emergency Care. Ann Emerg Med 1998; 31:166-171.

 TOP