Fundamentals of Clinical Decision Support Systems

1. Lecture #11 INFO 590: Fundamentals of Clinical Care for Health Informaticians Clinical Decision Support Systems kharrazi@iupui.edu http://www.info590.com

2. Introduction The Forms of Medical Knowledge Learning by Humans and Machines Decision Support Models Acceptance of Decision Support Systems Lecture in a Nutshell

3. Introduction • A large part of all activities in health care deals with decision making regarding which examinations and tests need to be done or on the basis of earlier examinations which further tests need to be ordered. • In short, at many points during patient care, data collection is accompanied by decisions. • Decisions can be categorized as decisions fordiagnosis or treatment. For both types we need complete and accurate medical knowledge.

4. The Forms of Medical Knowledge • Two types of knowledge are involved in decision support: Scientific and Experimental. • Scientific or formal knowledge: This type of knowledge is backed by the medical literature, books, or articles. This type of knowledge deals with cognition or deduction. This form of knowledge will be constructed when the principles of biological processes and relationships between pathophysiological conditions and disease symptoms are knows. • Experimental knowledge: This type of knowledge is related to recognition or induction that is, a clinician has seen certain symptoms before and recognizes the underlying disease. • Both of these types of knowledge will be used when clinicians ‘reason’ about the signs and symptoms of a specific patient.

5. The Forms of Medical Knowledge cont. • Computers may be required because: (1) People sometimes make errors or mistakes even in routine cases (2) Clinicians cannot keep up with the ever increasing medical knowledge (3) It is sometimes more efficient to automate decision making (such as lab tests or ECG results) (4) Health care organizations may mandate certain clinical practices both to improve the quality of care and to lower the cost of care. • The question is whether computers are able to make the same decisions as experienced clinicians using the same patient data and the same knowledge. The key problem here is that in fact we don’t know how people store and use their knowledge. • Computers may not be able to decide properly because the patient data are sometimesincomplete or a specific patient may benew and unique.

6. Learning by Humans and Machines • Decision making knowledge is involved in cognition and recognition. • Learning is done during education and training and the level of cognition is tested during examinations. • Both stages can be observed during the training of medical students; therefore both stages should also be observed in computers. • The most central issue in all learning processes is the selection of features or symptomsby which we recognize events or diseases.

7. Learning by Humans and Machines cont. • Features • Symptoms versus Diseases: In general the disease  symptoms relationship is the result of careful observations during scientific studies and clinical experience. The symptoms  disease is the result of practicing clinicians. Same relationships should be implemented in CDS systems. • Variability: Patient symptoms generally show a large variability between patients who have the same disease. The symptoms also show variability in the same patient when the disease progresses. Variability in symptoms between patients who possess the same disease. The variability can be expressed in, for example, means and variances.

8. Learning by Humans and Machines cont. Symptoms or measurements, in short: features, determined for the same disease category may be correlated and can also be expressed statistically. In this diagram, observations for features f1 and f2 have been plotted.

9. Learning by Humans and Machines cont. Feature Selection: Decision models can assist in feature selection, which is the principal part of training computers for decision support, that is, to determine which signs, symptoms, or measurements are the most important (statistically significant) for discriminating between a healthy or a pathological condition. Features versus Decision Model: When ‘training’ a computer decision model, base on empirical classification, the features of a set of patients with different diseases are entered into the computer (training set or learning set). Two types of learning can be conducted: supervised and unsupervised learning.

10. Learning by Humans and Machines cont. Supervised Learning: In supervised learning the researcher tells the computer the disease or health status of each patient in the training set. The computer is then asked to order the features according to discriminatory power between disease A and disease B. Unsupervised Learning: In unsupervised learning the computer is also given the training set of features. However the truth (information on what disease belongs to which patient) is not known. The computer is then used to discover by what clusters of feature sets the difference disease groups can best be characterized. This process is called clustering.

11. Learning by Humans and Machines cont. Unsupervised Learning (Clustering)

12. Decision Support Models • Decision support models in health care can be: quantitative and qualitative • Quantitative Models: is often based on well-defined statistical methods and makes use of training sets of patient data. Prior probabilities for the occurrence of diseases are generally incorporated into the statistical models. • Qualitative Models: uses features that are generally proposed by experts and that are based on clinical studies. In this category the decision support methods use symbolic reasoning methods such as logical deduction (Boolean logic – If Then) • Some DSS systems may incorporate both models Bayesian Network (Quantitative and Qualitative).

13. Decision Support Models cont. Decision-support models in health care can be grouped into different categories. The main categories are the quantitative (statistical) and the qualitative (heuristic) decision-support models, which can also be further split into subcategories.

14. Decision Support Models cont. • Quantitative Decision Support Systems • One Feature  Single Decision Threshold: Threshold will be defined to categorize the health status to disease or healthy. In principle we are free to choose the decision threshold for each population in the overlapping distributions. Whatever decision threshold we choose the resulting decision would not be error free. Four possible decisions are possible: • TP True Positive: The percentage of people who had the disease and they have been correctly recognized as patients. • TN True Negative: The percentage of people who did not have the disease and they have been correctly not recognized as patients. • FP False Positive: The percentage of people who did not have the disease and they have been mistakenly recognized as patients. • FN False Negative: The percentage of people who did have the disease and they have been mistakenly not recognized as patients.

15. Decision Support Models cont. Relationships between True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN).

16. Decision Support Models cont. Normal distribution + Histogram

17. Decision Support Models cont. Normal Population True Negative False Positive Normal distribution + Normal Population Example: WBC more than a certain amount means…

18. Decision Support Models cont. Disease Population False Negative True Positive Normal distribution + Disease Population Example: WBC more than a certain amount means…

19. Decision Support Models cont. Normal Population Normal distribution: Normal versus Disease

20. Decision Support Models cont. Normal Population Disease Population True Negative True Positive Ideal test Example: WBC more than a certain amount is 100% diagnostic of the disease

21. Decision Support Models cont. Normal Population Disease Population True Negative True Positive False Negative False Positive Real test Example: WBC more than a certain amount is somehow diagnostic of the disease

22. Decision Support Models cont. • All decision models deal with these four situations (TP, TN, FP, FN). The task of a decision model is tominimize the percentages of errors (FP and FN), given the features and the prior probabilities of the occurrences of the disease. • If we want to minimize the total amount of incorrect decisions, irrespective of the costs involved in making erroneous decisions, it can be seen that the choice of the decision threshold is different for each population. The effect of the choice of the decision threshold can graphically be seen in so-called receiver operating characteristic (ROC) curve. • ROC Curve: It teaches us that FP and FN cannot be minimized independently and the ideal point (FP, FN) = (0, 0) cannot be reached. • The less the distribution overlap, the better the ROC approaches the ideal point of (0, 0); the more the distribution overlap, the more ROC approaches the diagonal line that runs from the point (100, 0) to (0, 100)  it is better to have more distinctive and discriminatory features.

23. Decision Support Models cont. Ideal shift Sensitivity 1-Specificity ROC Curves of the left diagrams

24. Decision Support Models cont. • Performance of Decision Models: Often ROC is not practical to create; therefore the most usual way to express performance is by a 2x2 matrix. The following variables can be calculated:Sensitivity = a/(a+b) Specificity = d/(c+d)Positive Predictive = a/(a+c) Negative Predictive = d/(b+d)Prevalence of a disease = (a+b)/(a+b+c+d)Total Performance = T(1) = (a+d)/(a+b+c+d)Total Performance = T(2) = (TP+TN)/2Total Performance = T(3) = (TP+TN) - 1

25. Decision Support Models cont. Sensitivity = TP/(TP+FN) Specificity = TN/(FP+TN) Sensitivity and Specificity

26. Decision Support Models cont. Sensitivity = TP/(TP+FN) Specificity = TN/(FP+TN) Sensitivity and Specificity

27. Decision Support Models cont. Sensitivity = TP/(TP+FN) Specificity = TN/(FP+TN) Sensitivity and Specificity

28. Decision Support Models cont. • Cost and Risks: An ROC curve is determined by the distribution of the features and not by the choice of the decision threshold. The better the feature discriminate, the better the method’s performance. • The cost and risk has an effect on determining the threshold.For instance during population screening finding too many patients with a FP test result (a decision threshold which is too low) would possibly result in costs of follow-up examinations that are too high, while too many FN results would make the screening meaningless.In patient monitoring generating too many false alarms (too many FPs) would lead to less attention from nurses in correct alarms.In a critical decision making processes, such as diagnosis of appendicitis, a high false negative is not acceptable.

29. Decision Support Models cont. FP: Type I error (p-value) FN: Type II error Sensitivity and Specificity details

30. Decision Support Models cont. Sensitivity and Specificity calculations for bowel cancer

31. Decision Support Models cont. • Bayes’ Rule: The effect of prior or conditional probabilities on the probability that a disease is diagnosed by the decision model is expresses by Bayes in 1763. • The Rule of Bayes allows us to compute the posterior probability p(D|S) of a disease D, given its probability p(D) and knowledge about the conditional probability p(S|D) that symptoms occur in a given disease. • The rule is derived for the case of either having a disease (D) or being health or normal (N): • p(D|S) = ( p(D|S) / (p(S|D)*p(D) + p(S|N)*p(N)) ) * p(D)

32. Decision Support Models cont. • Multiple Features: So far we have used examples with one feature or symptoms for the diagnosis of two diseases. Similar to the one-dimensional decision problem we can follow the same strategy for two features which will be characterized as a 2-D classification space (non-supervised diagram). • If we deal with N features the formula becomes a hyperplane in an N-dimensional (N-D) feature space. Even with multiple features we still can calculate the factions FP and FN and have a ROC curve by moving around the threshold line.

33. Decision Support Models cont. • Multiple Disease: When we deal with more than two diseases (classes) the decision models become more complex and neither a single pair of performance measures (FP,FN) nor a single ROC curve is sufficient. • Sometimes a way out is to reduce all decisions to two-class decision problems. The outcome of multiple disease testing can be expressed in a KxL matrix where K is the number of diseases and L the number of ‘true’ diseases that are present. The following table shows the discrimination of cardiac diseases based on ECG: More than two classes

34. Decision Support Models cont. • Qualitative Decision Support Systems • Qualitative methods, which may be inspired by human reasoning, typically are less formal and are not based on mathematical basis. • Symbolic methods may be composed of elementary decision units that in principle do not differ from the two-class, single-feature statistical decision support. These elementary decisions test whether some symptom is present or a measurement is larger than some threshold value. • Rule-based decision support methods the result of a Boolean (logical) test (E) is based on a feature (x) and its threshold (L). E will be either TRUE or FALSE: E = “x > L” • Minnesota code (ECG expert system) is a mixture of a truth table and a flow chart to diagnose the underlying disease for each ECG: E = “Q/R amplitude ratio > 1/3 AND Q >0.03 in V6” • The most common format is: IF E THEN action1 ELSE action2:(1) use micro-decisions simultaneously (decision/truth table)(2) use micro-decisions sequentially (flowchart or decision tree)(3) user micro-decisions is situation action rules (rule-based or qualitative reasoning)

35. Decision Support Models cont. • Decision Tables: It takes all logical expressions into account at once: D = E {E1, E2, …, Ek} In principle a diagnostic decision-support method should assess all the lines (rules) in the truth table before coming to a definite decision. Rules can be combined for a diagnosis. The different combinations of elementary expressions can also be represented graphically by Venn diagrams (Karnaugh diagrams). Truth tables can easily incorporate decisions on multiple diseases. An example of a decision table for n diseases and m possible findings. For a disease, a finding can be positive, negative, or immaterial, symbolized by 1, 2, and –, respectively (e.g. 1 = male, 2 = female, - = both sexes)

36. Decision Support Models cont. Logical Expressions E Used as Elements in the Truth Table for Arrhythmia diagnosis

37. Decision Support Models cont. Truth Table for Arrhythmia Diagnostic Statements D in which the Logical Expressions Ei of the last table are Used as Elements for Arrhythmia Diagnosis. T stands for TRUE, F stands for FALSE and d stands for "don't care."

38. Decision Support Models cont. • Flow Charts: More knowledge is contained in flow charts than truth tables because the rules are assessed sequentially. The outputs are binary (TRUE or FALSE). A flowchart can be considered a decision tree that is upside-down, with the root at the top and the branches and leaves stretching downward. The location where micro-decisions are made are called the nodes of the tree. • The advantage of flowchart is that it is very efficiently processes and not all combinations of input data must be assessed as in truth tables. The disadvantage of flowchart is the rigidity of the paths. Once we are on a wrong path in the tree, no return is possible unless parallel trees are used. • Although flowcharts are considered qualitative approaches, some mathematical algorithms can be used to create them from a training set such as NPPA (nonparametric partitioning algorithm). These trees are usually large and should be pruned which introduces FP and FN cases.

39. Decision Support Models cont. Example of a flowchart in the form of a computer program, based on similar elementary decisions and logical expressions as used in the truth tables.

40. Decision Support Models cont. • Rule Based Reasoning/Systems (RBR/RBS): Updating truth tables and flowcharts is a complicated task when new knowledge is available. In RBR attempts have been made to unbundle the knowledge contained in the micro-decisions from the structure of the decision model. • Inference Mechanism: is the reasoning we need that is a procedure adaptive enough to operate on different patient databases and with different knowledge bases. Different strategies exist for inference, such as forward and backward reasoning. Each time a rule is fired either another rule will be executed or more information will be asked from the user until the result becomes available. The inference engine cycles via a match-fire procedure

41. Decision Support Models cont. Representation of the elements involved in heuristic reasoning. For inferencing, different strategies can be used: forward reasoning and backward reasoning.

42. Decision Support Models cont. • Semantic Networks (frames): In frame representations, knowledge is constructed as a set of concepts in which each concept has a number of attributes that may take on particular values. An inference mechanism known as inheritance allows the decision support system to derive conclusions about certain frames that are related to each other in a hierarchical manner. • Knowledge-Based (Expert) Systems: Characteristic of the expert system is the separation of the case data, the domain knowledge and the inferencing mechanism. Generally an expert system has a knowledge acquisition program that will be used to build an maintain the knowledge base. Expert systems may also contain an explanation module to help justify their recommendations for their users. Sample patient frames

43. Acceptance of Decision Support Systems • The common criteria to assume DSS systems useful are: • support of data acquisition (medical imaging), • data reduction (real time data in intensive care units), and • data validation (lab results and pharmacy drug interactions). • The most important task for decision support systems is enhancement of data reliability. DSS are useful when they: offer guidance to inexperienced users in making complex decisions, give support to experienced users specially in none routine cases and finally integrating critiquing into the CPR system. • Computers may be misused in clinical practice: • Nonqualified user such as a nurse uses the program • Non-trained qualified user uses the system • Program does not integrate well in the existing systems and receives wrong input • Poor judgment of the users.

44. Acceptance of Decision Support Systems cont. • Legal and Ethical aspects: • Negligence theory (malpractice) holds that service providers must uphold the standards of the community for quality and reliability. • Strict product liability (injuries and compensation) applies when the purchaser is harmed as a result of a defect in that product. • Since the patient and not the care provider is the individual who physically suffers from errors in the clinical process while the purchasers are the care providers, it could be argued that strict product liability would not apply to DSS systems.

45. Acceptance of Decision Support Systems cont.

46. Introduction The Forms of Medical Knowledge Learning by Humans and Machines Decision Support Models Acceptance of Decision Support Systems Summary