SPEAKER 0 Okay, That sounds great. SPEAKER 1 If folks are ready, then we'll go ahead and get started. It's my distinct pleasure to introduce Rob Lobato, who's speaking tonight on quality improvement at individual clinician level. Rob is Division chief of the Multi Specialty Anesthesiology and section head of the Thoracic Anesthesiology program at Nebraska Medicine. And his research interests are in cardiac anesthesiology and echocardiography. As you know, Rob often introduces the speakers because he's also in charge of our Grand Round series here at the center. So thank you for for presenting tonight, Rob. And I'll turn it over to you. SPEAKER 2 Well, thank you very much. Good evening, everybody. So as Dr. Wendell mentioned, I'm a clinical anesthesiologist. That's my full time gig. And I dabble in some of the data database stuff that I think is of interest or might be of interest to CHC. I guess I'll start by just prefacing this with this is not going to be exactly the meat and potatoes of what CHC does on a routine basis. I'm going to start with some fairly large broad topics outside the scope of CHC and then really hopefully work my way in in concentric circles to something that I think is perhaps of interest to some of the audience here. I'm focusing mostly on quality things, very clinically oriented, and some of the data sources that I've been using to achieve that. So here's where I'm going to start. Nearly every national ranking is at an institutional or a clinical program level. You look here at the US News 2022, 2023, hospital honor roll, and these are all institutions. So why, you might ask yourself, do I care about individual level clinician performance? I mean, that's nowhere on there. Does it say anything about clinician X or clinician Y? Well, this whole interest of mine started a couple of years ago when I was having a conversation with a colleague and in a really throwaway way, he said, Well, don't worry about it. Anesthesia has passed, fail. And looked at him kind of funny and said, Well, what do you mean by that? And he said, Well, either the patient lives or the patient dies. It's pretty straightforward stuff. SPEAKER 0 And I kind of scratched my head and. SPEAKER 2 Walked away and thought to myself, I hope I really do more than that. So I'm going to start my first concentric circle. It's just a little bit about anesthesia and anesthesiology. So as some of you may know, anesthesia is a relatively recent medical invention. It really didn't come into practice until the 1860s, and even then it was super dangerous. This is an image from the Civil War where. This gentleman right here in the middle is administering some sort of inhalational anesthetic, perhaps while being evaluated for, I assume, an amputation of some sort. And this looks a little more staged than an actual clinical photo, but I think it's reflective of probably what anesthesia was around that time. We've all watched movies that sort of say, you know, you take a big slug of whiskey and then out comes the saw, off goes the leg. I think it was maybe slightly better than that, but really only slightly. Fast forward 100 years to something around the time of the Vietnam War and anesthesia is now starting to look like a medical discipline. There's an anesthetist here in the foreground administering an anesthetic, and we can assume a trauma surgery is taking place. Even though a hundred years had passed, the mortality for emergency surgery and emergency anesthesia was still fairly high. 1 in 100. 1 in 1000. I mean, that's that's still relatively high. This is what an hour looks like today. And it looks absolutely nothing like even what was there 50 years ago. So here in the foreground, this is an anesthesiologist or two anesthesiologists here working all of this stuff in the background. These are the monitors that we use to transducer arterial pressure, PA pressure, central venous pressure. This is our anesthesia machine with three different types of volatile anesthetics. Lots of information about the ventilatory mode that's being used, plenty of syringes around here, infusions that are running. And then this is an echocardiography machine that's being used to guide some of the hemodynamic management during the surgery. This is very, very different from what they were doing 50 years ago, or especially 150 years ago when you look at the Civil War images. In fact, to most people, this looks a lot more like this. It looks like a cockpit. It's a ton of instruments. It's a lot of information, very much a high tech endeavor. And I think that for a lot of us in anesthesiology, commercial aviation has been a guidepost of the way that we improve quality and improve safety. It's now to the point where commercial aviation I mean, people step on planes every day and rarely do they think of the plane falling out of the sky. I mean, if so, it's a brief thing. It's not a real concern. And everybody has heard the statistic that it's safer to fly somewhere than to drive, which is almost certainly true. Well, at the same has proven out with anesthesia. Really. This is a little new snippet from the Canadian Broadcast channel saying that the risk of dying shortly after surgery has. Dropped 90%. So it's now one tenth of what it was prior to 1970. That's pretty good, really. Our crude mortality rate is about three and 10,000 for routine ish things to perhaps as high as 30in 10,000 or more emergency things. And this, of course. SPEAKER 3 Is is irresistible. SPEAKER 2 This includes several different patient factors and surgical factors and all kinds of other things. But really, these are pretty good odds. I mean, if someone were going into something and said there's a 0.03% chance of death, most people would say that's that's worth gambling. Our complication rate, however, is still fairly high. And there's a number of things that I bucket into complication rate, many of which I'll talk about here in a little bit. But if we were only concerned with crude mortality in improving anesthesia, we'd have a hard time making a whole lot of difference. Moving. Moving this in order. Another order of magnitude would probably take another 50 years. When looking at the things that we consider minor complications, patients often feel very differently about when we call them minor complications. That's really the ground for a lot of the quality improvement work within anesthesiology, and that's where I'm primarily going to focus today. So this, I thought, was a really interesting study that was done by by a former colleague of mine at Stanford. And this really speaks to my gut feeling that pass fail is not good enough. He took 101 patients and showed them a list of common non fatal anesthesia outcomes or complications. He then asked them if they could allocate $100 to avoid the listed outcomes. How would they divide that $100? Kind of a neat way to ask a question, right? What he's really trying to get to is what matters to the patient and how much does it matter? SPEAKER 3 This is the results. SPEAKER 2 But this is the results from his study and. I think it's easiest if we look here at the relative value portion and he does a nice job of telling you what these numbers actually mean. So when you look at the thing that patients most want to avoid with that $100 vomiting, they give a relative value of 18.05. And the author describes this as this means, for example, patients assigned $18.05 of the $100 who avoid vomiting. And then there's a list of other things and their relative weights of importance. This is really, I think, pretty impactful because it helps me know what the patients are concerned about. It gives me a sense of of what really matters to them. And so it provides some stakeholder perspectives, the patient perspective, which is primarily concerned with comfort immediately after the procedure. But there are also some health care stakeholder perspectives. Things like time into hospital discharge or AC discharge. Every one of those minor complications, every one of those clinical, non-fatal clinical outcomes delays the amount of time that a patient is going to be in the recovery area or is going to be at the ASC and in some cases delays the amount of time they're going to be in the hospital. So from a health care system point of view, anything we can do to minimize those non-fatal outcomes is probably going to result in better financial health of the health care system. And then we want what we really want to be concerned with is the overall return to baseline quality of life for patients. This is more of a public health perspective so that patients come in, they have a procedure, they get better afterwards and they go back to living their normal lives or at least similar life to what they were before they came in from surgery. SPEAKER 3 So if we know what's important to stakeholders, what's the problem? Well, not all medical specialties have the. SPEAKER 2 Same ability to assess. SPEAKER 3 Clinician performance. SPEAKER 2 If you look at medical subspecialties like internal medicine, a lot of the primary care specialties cardiology, for example, the clinician see the same patient several times over, sometimes very, very long periods of time and really get to know that individual patient's response to medications, the natural course of their disease longitudinally, they have a chance to see what works, what doesn't work. And it's there's often a lot of feedback involved. I'll contrast this with other medical specialties like anesthesiology, perhaps like diagnostic radiology, perhaps like pathology, where. SPEAKER 3 These are relatively brief encounters. SPEAKER 2 And it's uncommon for me to see the same patient more than once in any reasonable period of time. Every once in a while someone will come back after 5 or 6 years and Oh, hey, I remember you. SPEAKER 3 Good, you know, Nice to see you. But it's. SPEAKER 2 Not a longitudinal encounter the same way a lot of other medical specialties. SPEAKER 3 Have. So for those who don't know much about anesthesiology, and I'm specifically speaking to O.R. anesthesia. SPEAKER 2 Which is a little different than some of the other anesthesia subspecialties. Most of the time when I'm working with patients in the OR, my patient. SPEAKER 3 Encounter. SPEAKER 2 Begins in the preoperative holding area. It starts on the day of surgery. The patient, as is illustrated in this stock photo, is probably already gowned and ready to go back to the operating room. And I'm coming in and meeting him or her for the first time after having reviewed most of their medical history and their laboratories and stuff like that. We then go to the operating room where I care for the patient sometimes for several hours, and then at the end of the procedure I take the patient to recovery or intensive care, do a sign out. And that's really the end of my encounter. SPEAKER 3 The one big problem with that, though, from my point of view, is I usually don't know. SPEAKER 2 Much about how the patient does. SPEAKER 3 After signing up. Could be the patient does great. Could be the patient really has a hard time and. SPEAKER 2 Experiences several of. SPEAKER 3 Those. SPEAKER 2 Those common non fatal outcomes. I don't usually get to know about that. If something really bad happens, I probably do find out about it, but it's not a very linear feedback system. SPEAKER 3 And I think we've all seen. SPEAKER 2 Schematics kind of like this, which talk about deliberate practice and moving here from the 12:00 position from novice performance to competent to proficient to mastery. SPEAKER 3 You can see how often feedback. SPEAKER 2 Plays a role within each of. SPEAKER 3 These increases in. SPEAKER 2 Knowledge or expertise. SPEAKER 3 The real challenge is where does this feedback come from? So when I sit back as a clinical anesthesiologist and someone who manages clinical anesthesiologists, I think to myself, how can we improve our practice without feedback? And that can be pretty hard to come by. Well, one proposed answer is self assessment. Is this a solution? Could this be a solution? Not usually. SPEAKER 2 I think we've all seen anecdotes like this. The media tends to enjoy news stories such as this, where this is an example of a survey that was done in two thirds of American drivers rate themselves excellent or very good drivers. Those same folks rate only about 22% of people their age as excellent or very good. So it's one of these Lake Wobegon things where every kid is above average, right? Everybody thinks of themselves as better than their peers at whatever task. Scientifically, this has been. SPEAKER 3 Termed the. SPEAKER 2 Dunning-Kruger effect, and this. SPEAKER 3 Is a cognitive bias. SPEAKER 2 In which people believe they're smarter and more capable than they actually are. New learners are especially prone to overrate their. SPEAKER 3 Performance because they just don't have the. SPEAKER 2 Experience to really see good and bad. SPEAKER 3 Performance. SPEAKER 2 Ironically, the lowest ability individuals are more likely to dramatically overestimate their expertise. You may have seen this sort of diagram of the Dunning-Kruger effect where the knowledge in the field people start out with zero knowledge. SPEAKER 3 And relatively quickly their confidence increases. SPEAKER 2 Even though they haven't really learned a whole lot. It's not until we start to learn more about what it is that we do that we start to say, Wow, there's more to this than I thought. And much, much later in our careers, we start to realize, Oh. This is now making sense and it's actually pretty complicated. SPEAKER 3 So in the face of poor group performance. The Dunning-Kruger response from. SPEAKER 2 From folks is often, well. SPEAKER 3 I'm fine, but everyone else needs to change. And I think we're starting to see how. SPEAKER 2 That could play a. SPEAKER 3 Role in large scale quality improvement projects. I'm going to take another concentric circle here and talk about. SPEAKER 2 Quality improvement kind of as a discipline because again, it's it's. SPEAKER 3 Very much. SPEAKER 2 Related to what she. SPEAKER 3 Does, but it's not something that people. SPEAKER 2 Run across every day, at least as. SPEAKER 3 A discipline. SPEAKER 2 Zooming out all the way. The goal of QE. SPEAKER 3 Is to refine a process, and for the most part, this takes place in three steps. Step one we try and reduce variation in how a process is. SPEAKER 2 Performed by. SPEAKER 3 Establishing standards. SPEAKER 2 Mostly using evidence. SPEAKER 3 Based guidelines. We then try to deploy a standardized. SPEAKER 2 Practice into a. SPEAKER 3 Test environment and see how well that works to to provoke change. SPEAKER 2 If we're happy, we try and. SPEAKER 3 Expand adoption and sustain improvement over time. And this is true for. SPEAKER 2 Almost every quality improvement project. One of the challenges, though, is how do you know that a. SPEAKER 3 Program is working? And how do you communicate. SPEAKER 2 Your progress to stakeholders? There's a tool that exists in. SPEAKER 3 Engineering that was then. SPEAKER 2 Translated into business that is now started to appear in quality improvement literature called a run chart. So I'd like to just introduce the concept of run charts as a potential answer to both of these questions. SPEAKER 3 Now run charts are a graphical method for visualizing process variation and identifying. SPEAKER 2 Trends or changes in. SPEAKER 3 Performance. This is very low tech and I. SPEAKER 2 Realize my audience is informative. Programmers, artificial intelligence experts. This is kind of the opposite of that. This is a very low tech way of looking at things, but it's it can be pretty powerful. SPEAKER 3 And the. SPEAKER 2 Intuitive appeal of it really makes. SPEAKER 3 It effective for communicating to stakeholders what the problem is and how. SPEAKER 2 Progress at improving that has. SPEAKER 3 Gone. So how do we construct a run chart? SPEAKER 2 This is really as simple as it looks. SPEAKER 3 You find an outcome of interest. SPEAKER 2 You find a time interval, you plot it, and you calculate immediate. Right. Super simple. Well, how can run charts really help us? One of the things that run charts can do is talk about. SPEAKER 3 Variation in existing progress. Run charts can. SPEAKER 2 Also help us. SPEAKER 3 Detect changes in performance over time. Now, within this context. SPEAKER 2 What I mean by variation. SPEAKER 3 Is the difference between an ideal. SPEAKER 2 And an observed process. SPEAKER 3 So in other words, how consistent is a process? Graphically, we. SPEAKER 2 Assess this by looking. SPEAKER 3 At the dispersion around a central tendency, and in this case, that. SPEAKER 2 Central tendency is the median. SPEAKER 3 So if you look. SPEAKER 2 On the left side of the figure, you can see the original. SPEAKER 3 Process and there's some variation between intervals. Things go up, things go down. SPEAKER 2 It's willy nilly kind of all over the place. SPEAKER 3 On the right side, you can see an improved process that has reduced variation, even though it hasn't. SPEAKER 2 Changed the median at. SPEAKER 3 All. SPEAKER 2 We now have a process. SPEAKER 3 That produces an output that's much more consistent. We're much more likely to get out of the process what we expect to get out of the process. So even though that's not a. SPEAKER 2 Change in a median, it's. SPEAKER 3 An improvement in quality because. SPEAKER 2 It's more predictable. SPEAKER 3 And more consistent. So variation can really result from any portion of a process. And this is where. SPEAKER 2 Quality improvement becomes more systematic, is trying to dissect all these various processes and figure out where are the sources of. SPEAKER 3 Variation, which of those are controllable, which of those are uncontrollable, and how do we do this at an individual. SPEAKER 2 Level? SPEAKER 3 The other thing that I think. SPEAKER 2 Run charts can be helpful for is looking at a change. SPEAKER 3 In performance. Now they're called run. SPEAKER 2 Charts because there are things called runs. SPEAKER 3 And that's a predefined number of consecutive events above or below the original median that constitutes a run. When a run is identified, a new median is calculated. SPEAKER 2 Graphically, it looks like the illustration here and you can see there's been a change, a consistent change in our process and a new median has been calculated. SPEAKER 3 Now, how does one define a run? Well, as I mentioned, it's a pre-specified number of consecutive events above or below the baseline. Really, this is going to be specific to each individual program, and it's based on the concept of meaningful change. SPEAKER 2 So it depends. SPEAKER 3 What is it you're measuring, how much of an improvement matters and what is the interval that you're using between measurements? SPEAKER 2 One could imagine that if I were measuring measuring something on a daily interval, I might need several days in a row before I thought that something really meaningful had happened. In contrast, if I'm measuring something quarterly or annually, it may not take very many. SPEAKER 3 Consecutive. SPEAKER 2 Intervals for me to think, Yeah, there's probably been a change and I should I should calculate a new median to see where we are. SPEAKER 3 The nice thing about this is that this becomes algorithmic. We. If we pre specify a number of consecutive events, we then have a way of looking at this without. SPEAKER 2 Having to scratch our heads each time and wonder is. SPEAKER 3 This is this something we want to do or. SPEAKER 2 Is this something we don't. SPEAKER 3 Want to do? SPEAKER 2 We have experts weighing and telling us this amount of change for this period of time. SPEAKER 3 Is meaningful to us. So we'll try making another concentric circle here. What I would like to do is start looking at anesthesia outcomes within our institution. Luckily for me. SPEAKER 2 There's something called the Epic. SPEAKER 3 Anesthesia Registry. This is a portion of epic. SPEAKER 2 This lives in Kabuto, which is one of the databases within Epic inside our Enterprise data warehouse. And this is available after. After some training to people who are interested in doing this. And I wrote a ton of code in our code to start digging into the anesthesia registry. SPEAKER 3 To look at individual outcomes. The nice thing about this is the first few things that we're going to look. SPEAKER 2 At are. SPEAKER 3 Really well described, at least within the epic. SPEAKER 2 Literature of how they've come up with these. So most of them are. SPEAKER 3 Consistent between. SPEAKER 2 Institutions. They're outcomes that several places are looking at. Some of them are even governmental measurements. And I'll get. SPEAKER 3 Into more specifics about this here in just a minute. The things that our group thought were really important to start looking at. SPEAKER 2 At least at the outset of. An individualized performance reporting system is. SPEAKER 3 Postoperative nausea and vomiting, which I'm going to. SPEAKER 2 Just call because it's kind of a mouthful. SPEAKER 3 Post-operative temperature and neuromuscular blockade reversal, which. SPEAKER 2 Is a fancy way of saying. SPEAKER 3 Residual weakness after anesthesia. SPEAKER 2 Now, if we go back to the table that the doctor Macario published, you can see that nausea and vomiting are numbers. SPEAKER 3 One and four. Residual weakness. SPEAKER 2 Here is, I don't know, number six and shivering, which would be temperature is one of. SPEAKER 3 The others here on the list. The other things, of course, are. SPEAKER 2 Very important pain we try and address every time. And I think it's we. SPEAKER 3 Try and do our. SPEAKER 2 Best every time we take care of a patient. SPEAKER 3 With regards to pain. SPEAKER 2 Gagging on the endotracheal tube, that's. SPEAKER 3 A. SPEAKER 2 Pretty uncommon occurrence and I think is mostly a fear that people have based on watching medical dramas. It's not something that happens very often for patients in. SPEAKER 3 The O.R.. SPEAKER 2 So we picked the things that we thought were probably highest. SPEAKER 3 Importance. SPEAKER 2 To patients as our. SPEAKER 3 Stakeholders. Specifically, I'm going. SPEAKER 2 To start with PO and V as a first target. SPEAKER 3 Now, why envy? SPEAKER 2 It's pretty common. This happens to ten, 15, 20% of patients, depending on how they're treated and what their risk factors are. It's pretty important to patients, as we saw from the Macario. SPEAKER 3 Study, the number one and the number four most important. SPEAKER 2 Things. And I think, in fact, about $30 of that imaginary $100 allocation was to nausea and vomiting. So pretty impactful. SPEAKER 3 It is associated with. SPEAKER 2 Increased health care costs, mostly as a result of increased. SPEAKER 3 Stay within. SPEAKER 2 The post-operative recovery area. Sometimes it even results in inpatient admission for intractable nausea and vomiting after an anesthetic. SPEAKER 3 Which may have not. SPEAKER 2 Been planned in an ambulatory. SPEAKER 3 Environment. It also has well recognized risk factors. SPEAKER 2 Things like patient sex, previous history of nausea and vomiting, smoking, whether or not opioids are. SPEAKER 3 Going to be used. SPEAKER 2 Some really pretty predictable things. And most importantly, there are evidence based society guidelines. SPEAKER 3 For preventative treatment. So all in all, we have a lot of the ingredients that we would want in order to start a quality improvement project on postoperative nausea and vomiting. Even better. Is a CMS MIPS measure right now? SPEAKER 2 That was the alphabet soup. So did he code that all. SPEAKER 3 CMS is the center for Medicare and Medicaid Services and MIPS stands for the merit based Incentive payment system, which is. SPEAKER 2 Part of the quality. SPEAKER 3 Payment program. So this is an effort from the Center for Medicare and Medicaid Services to try to financially incentivize institutions to pay more attention to postoperative nausea and vomiting. SPEAKER 2 Centers. SPEAKER 3 That don't. SPEAKER 2 Do well on this measure, receive a. SPEAKER 3 Small deduction in their reimbursement. So it's more of a penalty than an incentive. But the important part for what we're. SPEAKER 2 Talking about is that they have a very full. SPEAKER 3 Description, technical description of what it is. SPEAKER 2 That the that they're going to measure what goes into the numerator, what goes into the denominator. This whole spec is several pages long and includes things that we would exclude patients for. It includes which medications count. SPEAKER 3 Which don't count. SPEAKER 2 So really a pretty nice specification for an outcome. The fact that it's federally mandated or not federally mandated, federally defined is really a better way to say. It means that. SPEAKER 3 Each center that participates using this measure is. SPEAKER 2 Going to define it the. SPEAKER 3 Exact same way. So it's benchmark level between institutions. SPEAKER 2 And can really be used as a common yardstick. SPEAKER 3 Between. SPEAKER 2 Anesthesia programs in different locations. SPEAKER 3 So in. About two years ago, around April of 2021, we decided. SPEAKER 2 We wanted to take on the MIPS for 30 the. SPEAKER 3 Pond as. SPEAKER 2 A quality improvement. SPEAKER 3 Project. And the first thing we did was pull some historical data. This is the previous nine months of pon v success rate and that is. SPEAKER 2 Defined as. SPEAKER 3 According to guidelines. How many what proportion of patients received the preventative treatment that they should have given the guidelines that exist? And our success rate was about 91%. Not bad, but there's plenty of room for improvement. When I look at tables. SPEAKER 2 Like this that have an interval measurement and a numeric value, it makes me want to put them in a run. SPEAKER 3 Chart. And so this is what the run chart. SPEAKER 2 Looks like for our historical. SPEAKER 3 Data. And you can see that. SPEAKER 2 Here from July through April of. SPEAKER 3 2021, our. SPEAKER 2 Performance had a fair amount of. SPEAKER 3 Variation, but the median. SPEAKER 2 For that sample was right about. SPEAKER 3 91%. Well, the first thing that most quality improvement programs do is start with education. So our group put together a department wide education where we spent a. SPEAKER 2 Lot of time talking about the evidence based guidelines for. SPEAKER 3 Prevention and treatment, which medications we would use, what the risk factors were. We also stressed the importance of pins to patient satisfaction and comfort. SPEAKER 2 That's really the most motivating thing that we can present. SPEAKER 3 To clinicians is how it matters to patients. SPEAKER 2 So we presented those and some. SPEAKER 3 Strategies for treatment and prevention. And got on our way. Well, three months later. SPEAKER 2 We came back and looked at our data and this is what we saw here. Shaded in gray is our historical baseline data and boom, boom, boom. We had three months above the previous. SPEAKER 3 Baseline. SPEAKER 2 Which we were really pretty excited about. SPEAKER 3 And if that were the whole story. SPEAKER 2 It would really be a fairly. SPEAKER 3 Unexciting. SPEAKER 4 Story. SPEAKER 2 Because three months after that, we looked again and our performance crashed right back. SPEAKER 3 Down to baseline. SPEAKER 2 For people who are familiar. SPEAKER 3 With quality improvement projects, this is a very common occurrence. In fact. SPEAKER 2 It's been dubbed the Sustainability. SPEAKER 3 Challenge and some of the literature. This is one particular. SPEAKER 2 Publication that talks about the challenge involved, and it reports that within the national health system in. SPEAKER 3 The United Kingdom, 33% of quality. SPEAKER 2 Improvement. SPEAKER 3 Projects are not sustained one year. SPEAKER 2 After. SPEAKER 3 Evaluation. The next sentence, I think, is even more important Some improvement. SPEAKER 2 Leaders believe that the. SPEAKER 3 Challenge is not starting improvement work, but rather continuing the work after. SPEAKER 2 The initial enthusiasm has. SPEAKER 3 Dissipated. SPEAKER 2 That's pretty clearly what happened here. We had a brand new program. People got fired up. They changed their practice, and then they kind of forgot about it or went back to doing what they were doing and just lost excitement. SPEAKER 3 This happens so often and it's. SPEAKER 2 One of the biggest challenges. SPEAKER 3 In quality improvement. So let me try and bring these things together. SPEAKER 2 I've talked a little bit about anesthesiology. I've talked a little bit about common outcomes and how they matter to patients. I've talked about the importance of feedback in improving performance. SPEAKER 3 Over time. SPEAKER 2 And some possible data. SPEAKER 3 Sources for doing that. So this is really where we. SPEAKER 2 Decided to try and. SPEAKER 3 Use individual level, individual level performance data for our clinicians. We started issuing quarterly reports of individual clinician performance using the. SPEAKER 2 Epic Anesthesia. SPEAKER 3 Registry data. We also went to a more. SPEAKER 2 Frequent review. SPEAKER 3 Of department level performance at our monthly quality program. SPEAKER 2 I spent a lot of time. I'm primarily an art programmer for my data analysis, so I spent a lot of time generating. SPEAKER 3 A parameterized reporting system. SPEAKER 2 That would go out and query kaboodle, pull those outcomes, match them up with. SPEAKER 3 The patient records. SPEAKER 2 Match those with the anesthesiologist and. SPEAKER 3 The anesthetist anesthetist. SPEAKER 2 That was doing the doing the anesthetic that was administering the anesthetic, and then find. SPEAKER 3 Those outcomes and report them back quarterly in a way that hopefully was impactful. This is a high level report. This is meant for. SPEAKER 2 Our our leaders in the department to. SPEAKER 3 Look at the entire group at one time. So this is clinician level. This is a leadership version, and it would list each patient I'm sorry, each clinician. This is a report, each clinician and each clinicians performance relative to our standard. I'm going to zoom in a little bit. SPEAKER 2 Because it's I realize it's a bit of an eye chart. SPEAKER 3 Especially if you're looking at this on a smaller screen. So each nurse anesthetist here, this reports his or her performance for the quarter of interest. SPEAKER 2 You can see that our aspirational goal, our stretch goal for. SPEAKER 3 The department was. SPEAKER 2 Greater than 97%. SPEAKER 3 And that about half of our clinicians already meet or exceed that goal. That's fantastic. It also lists the clinicians who are not meeting that goal, which is an opportunity for improvement. So in addition to having a parametrized report for departmental leadership. I also generated reports for each individual clinician. So this is what an individual clinician report would look like. It's blinded to every. SPEAKER 2 Other clinician. SPEAKER 3 That's included in the sample. In this case, this is again, the nurse anesthetist. SPEAKER 2 So you someone would not see any of their colleagues names here, but would simply see his or her own name, which I've blacked out. It shouldn't really have a black bar. It would have. SPEAKER 3 The individual clinicians own name would highlight his or her performance and then say what exactly it was and how many cases they had performed. So in this case, this clinician had a 90% success rate by appropriately treating 18 out of the 20 high risk. SPEAKER 2 Patients that he or she had cared for during. SPEAKER 3 That quarter. And each clinician got one of these. SPEAKER 2 This is what it looks like in a little bit zoomed in, although on my screen now, it's pretty pixely Hopefully it's a little. SPEAKER 3 More readable for you, but. SPEAKER 2 Highlights. SPEAKER 3 What that individual clinician's performance. SPEAKER 2 Was, where it was relative to the group. SPEAKER 3 Standard and relative to their peers. And I think this is really three important things that I wanted to convey. I wanted. SPEAKER 2 To have a comparative. SPEAKER 3 Sense, a comparative anchor for people to see how they're doing. SPEAKER 2 To really help combat the Dunning-Kruger effect of I'm fine, everybody else needs. SPEAKER 3 To change. SPEAKER 2 I cherry picked this particular report because this clinician is in the bottom third bottom quarter. SPEAKER 3 And I am. SPEAKER 2 It's pretty impactful when someone sees themselves compared to their colleagues and ends up being below average or maybe. SPEAKER 3 Even in the bottom. SPEAKER 2 Quarter of the bottom third. I think that really can be jarring enough to provoke change. SPEAKER 3 The other thing I wanted to do was use a departmental stretch goal here to reinforce the standard with the goal really of reducing variation. SPEAKER 2 So over time. SPEAKER 3 One would hope more and more of these folks start to line up above the line until in a perfect. SPEAKER 2 World, all the clinicians are meeting are exceeding the standard and the. SPEAKER 3 Variation is practically none. I think together these things form a powerful motivator for growth for clinicians and. SPEAKER 2 This is something that that I have at times had access to in my previous practices. SPEAKER 3 And I think that. SPEAKER 2 It helped me improve. SPEAKER 3 My practice over time. SPEAKER 4 So in. SPEAKER 3 The year plus since we started doing the individual performance report, this has been our departmental monthly performance. SPEAKER 2 On the NB. SPEAKER 3 Success metric. You can see our. SPEAKER 2 Historical data. SPEAKER 3 Here right around 91%, the initial three months where. SPEAKER 2 We did well and then dropped back down. SPEAKER 3 To our baseline performance by reminding people more frequently at monthly meetings, by generating individual performance reports. We've not only helped increase our departmental performance, but more importantly, sustain it over time. This is really the tough part of quality improvement, is getting this to stick. And now we're. SPEAKER 2 18 months. SPEAKER 3 14. SPEAKER 2 Months into this. SPEAKER 3 Individual reporting and it's. SPEAKER 2 Really seems to have been helpful, seems to have made. SPEAKER 3 A difference in our departmental report. So. As I mentioned, this is not exactly. SPEAKER 2 What. SPEAKER 3 The Center for Intelligent. SPEAKER 2 Health Care is about. SPEAKER 4 But. SPEAKER 3 It involves a number of the. SPEAKER 2 Things that I. SPEAKER 3 Think. SPEAKER 2 Are impactful and really are meaningful to clinicians and. SPEAKER 3 To patients. I'll happily take any questions or feedback you might have. SPEAKER 1 Thank you, Rob. Actually, I think especially in view of Chris's presentation last month, this follows directly on it. So one quick question is about data source. Do you use structured data and follow up? So there is someone who is filling out a form afterwards that says here is the outcomes. SPEAKER 4 Well. SPEAKER 3 I took a little bit of that. I started with the low hanging. SPEAKER 2 Fruit, to be quite honest, because Epic. SPEAKER 0 Has. SPEAKER 2 Nice a nice registry that reports the outcome as assessed by the Post operative. SPEAKER 3 Care nurse. SPEAKER 2 And the risk factors and. SPEAKER 3 The medications. Most of the heavy lifting of doing that had been done for. SPEAKER 2 Me by Epic, which is really tremendously helpful. There's no way that I as as a single person working on this, could have made anywhere near that progress in the amount of time if it weren't something that were already pretty well done. SPEAKER 3 So post. SPEAKER 2 Operative knowledge and vomiting is is. SPEAKER 3 Reported on almost every patient who experiences it in. SPEAKER 2 The recovery area. The bedside nurse puts it as part of his or her assessment almost every time. And then Epic was able to use groupers to find out whether the medications were correctly administered and various risk factor tools to figure out what the patient, the re anesthetic patient risk was to help match a lot of these things up. SPEAKER 4 Thank you. SPEAKER 5 Hey, Rob. I enjoyed that. In your run charts, it appeared that and maybe you were just this were these were examples and not not real statistical process control charts that your center line changed with different inputs. One time it was three. One time it was five. What are you using to to move your your. What number of of. Observations above or below, consistently above or below the previous center line are you using to move the the median line? SPEAKER 3 Yeah, Thank you for the question. So you're right, this is not the full statistical process. SPEAKER 2 Control output of this. I find that a lot of times when I'm. SPEAKER 3 Trying to present to an audience. SPEAKER 0 Not. SPEAKER 3 Necessarily y'all as the audience, but more of my clinicians is the audience. SPEAKER 2 Statistical process control can get fairly complicated with the upper and lower bounds and talking about an out of control process. That is not something that typically. SPEAKER 3 People run into. So in this case, to directly answer your question. SPEAKER 2 Our quality group. SPEAKER 3 Decided that three consecutive months. SPEAKER 2 Were enough to trigger a change in a recalculation of the baseline, a. SPEAKER 3 Rebasing. So if you look here, we have. SPEAKER 2 One, two, three. Those are highlighted in red and there's a new. SPEAKER 3 Baseline one, two, three. SPEAKER 2 And unfortunately, that baseline went down. SPEAKER 3 We did not have three successive. SPEAKER 2 Points above or below the baseline until. SPEAKER 3 Several months. SPEAKER 2 Later. SPEAKER 3 So the three that are. SPEAKER 2 Counting are the ones that. SPEAKER 3 Show up in red. The DS three one, two, three knew Rebase bounced around. SPEAKER 2 One, two, three. New rebase. One, two, three, new rebase. The dotted lines are. SPEAKER 3 Continuations of the previous. SPEAKER 2 Median where. SPEAKER 3 The. SPEAKER 0 The. SPEAKER 2 Rebasing rules have not yet been met. Does that answer your question? SPEAKER 5 Yeah, it does. It's just that there are rules of when means should move, when your center line should move. And they're not based upon consensus of a small group of people. SPEAKER 4 You know, I've. This is really funny. SPEAKER 3 I and a couple. SPEAKER 2 People at Harvard and I. SPEAKER 3 Went back. SPEAKER 2 And forth about whether. SPEAKER 3 Or not. Whether or not that made sense within a clinical environment. And I think you're right, there's disagreement. That is an open question. SPEAKER 0 From my point of view. SPEAKER 2 I think it matters the time interval. I think it matters what the clinical meaningfulness of the outcome is. I think it matters what's. SPEAKER 3 Impactful to the group. SPEAKER 0 I agree with you. SPEAKER 2 I have heard of people putting together more. SPEAKER 3 Structured. SPEAKER 2 Rules based on. The number of time points. I think there has not been broad agreement, or at least not that I'm aware of, that all processes must have the same number of. SPEAKER 3 Intervals for a given time point, although I could certainly be wrong about that. Like I said, this is kind of my. SPEAKER 0 Part time gig. SPEAKER 5 Thanks. It's fun. SPEAKER 6 Um, hi, Dr. Lovato. This is Natsuki, and I'm a statistician, so I pretend to know something about these things. And I think Deming was the, uh, at the end, after World War two was the one who went to Japan and improved vastly the quality control there. And he was a statistician. And what I see here is what I have seen when these date traders do on the stock market. It's called technical analysis. It's quite different from what Warren Buffett does say 3 or 5 years projections. But in my view, how many? To go back to Dr. Maloney's question about how many, uh, time points, it's time series analysis, essentially, um, how many time points one needs to change the mean value or the median, rather the median value. I think it very much depends. It's just a tool. So we're using a tool for our purposes. And you have decided to use it for three consecutive, right? So it is it's just a tool to to my mind, there's no absolute there's no absolute, um, numbers of how many points. Of course there are statistical method saying, oh, this is a change point analysis and we're going to statistically detect in a statistically significant way these change points. That's also possible. But if you decide three consecutive points, that's fine too, because that serves your purposes. SPEAKER 5 Thank you. SPEAKER 3 Oh, thank you. That's. That's really. SPEAKER 0 Fantastic. I love it when I can get a free consultation. I'm always looking for opinions from people smarter than me. SPEAKER 2 And if I can get one for free, that's even better. SPEAKER 6 Smarter than you? I don't think so. I'm sorry. I apologize for my ignorance. SPEAKER 7 Hey, Rob, this is Jimmy Chang. Excellent presentation. Appreciate you bringing this forward and to clubs. Gleaves A most recent comment about Deming. This is a very nice demonstration, I believe, of process change that hopefully will continue to have a long lasting impact. Um, as you know, the Center for Intelligent Health Care has a number of different cause and including good data that is high quality, good discrete data available at the point of care and captured at the point of care. But a second core of that is that there has to be good design for capturing that information. So it was curious to me that you had, I think it was two dot outliers that were really far off of performance metrics. If you go back a slide, think a slide or two, you'll see it. Um, yeah. There you go. Any one of those there? There's two outliers there. And the question for me is that bad data is that bad design to capture those pieces of information. Have you actually dug into those? Because the question arises, arises for me at least, what is it about the those two providers that has created the dots that are so far off of the of the performance of everybody else? SPEAKER 2 Yeah. Thank you. That's. That's a really astute question. SPEAKER 0 These sort of outliers are often. SPEAKER 3 Clinicians that. SPEAKER 0 Don't practice very often, and. SPEAKER 2 It's a small numbers. SPEAKER 3 Problem. SPEAKER 2 The denominator is just very small. SPEAKER 0 So when this. SPEAKER 2 Is a quarter, a quarterly report. SPEAKER 3 If a clinician is only practicing 5 or. SPEAKER 2 6 days and only has a small handful of patients. SPEAKER 3 Missing, one not meeting the standard on. SPEAKER 2 One of those 5 or 6, can all of a sudden throw them into 80% or 60% without without showing that they're not good clinicians? I think this. SPEAKER 0 All when. SPEAKER 3 I explain this to the. SPEAKER 2 Clinicians receiving this, I really want to put this in the context of these are not a grade for any individual. SPEAKER 3 Clinician. This is. SPEAKER 2 How you did for the. SPEAKER 3 Number of. SPEAKER 2 Patients that you had over one quarter. SPEAKER 3 And that what we're. SPEAKER 0 Really looking for. SPEAKER 3 In assessing a clinician's performance over time is how they do over several quarters, over tens of patients, maybe even hundreds of. SPEAKER 2 Patients. SPEAKER 3 For those. SPEAKER 2 Folks that just are part time and happen to have 1 or 2 cases where they just didn't meet the metric, it's hard. SPEAKER 3 To say too much about. SPEAKER 2 That. SPEAKER 3 Usually these are small numbers problems. SPEAKER 2 I will say, however, we have some folks. SPEAKER 3 Who quarter after quarter. SPEAKER 0 Appeared in this. SPEAKER 3 Bottom, you know. SPEAKER 2 Lower three lower five. SPEAKER 0 Position. SPEAKER 3 And that's. SPEAKER 2 A good signal for us that we probably need to take the opportunity to maybe refresh them on. SPEAKER 3 Which or. SPEAKER 2 Which of the therapeutics that we want to be using, which are the high risk factors that we really want to be looking for. If someone appears at the very bottom and looks like an outlier for one quarter and they just weren't. SPEAKER 3 In the or very much, I think we take that with a grain of salt. SPEAKER 6 Thank you. May I ask a statistical question here? So basically, this criterion is yes. No. Did they meet or they didn't meet? Is that right? SPEAKER 4 That is correct. Yep. SPEAKER 6 So I would suggest, based on my experience as a statistician, that I would trust more like 100 observations before I judge this. It's like a smoker, non smoker type thing. It is. It requires a larger sample size. It's not a continuous variable with a continuous variable. We may be happy with 25 or 30 sample size, but with this type of dichotomous, quite preferable. I mean really meaningful is 100. SPEAKER 2 Yeah. Thank you. I think that's that's great feedback. Now as we're getting to the point where we have several. SPEAKER 3 Months. SPEAKER 2 Of data and our process seems to be settling down, we very well may be to the point where we can find 100 patients. SPEAKER 3 Per clinician. SPEAKER 2 And really assess their performance. And I think you're right. I think it's perhaps time we start thinking about that in in a more statistical, larger sample way than just kind of a convenient sample that happens quarterly. SPEAKER 1 So I have one more comment. And it goes back to your this this this, quote unquote, is not appropriate for the center, but it's square on because, again, our charter is really about making clinicians more efficient and more effective. And what you really demonstrated and I think really has to be argued that if you want to see real improvement, you have to get to the level of the individual to to effect change, because otherwise it's too diffuse, too amorphous to make a difference. And I think that is a very important observation. My second observation is one of the things I enjoy about the center is the diversity of the participants. So it is the challenge to say, I'm going to give a presentation and I'm going to make a statistical statement and wait a minute, there's a PhD, biostatistician who's going to look at this because that's again, how we get better. That's the hybrid vigor. And as we look at this and we're going to see that in a lot of our AI discussions. It just you need that diversity to come up with the best answers. SPEAKER 0 Well, thank you. And I completely agree with you. I. SPEAKER 2 I think medicine has gotten too complicated for any one individual single expertise. We're always better as a team. SPEAKER 1 Any final questions? If not, thank you very much, Dr. Lovato. We'll see everyone next month. SPEAKER 0 Thank you, everybody. Thank you. SPEAKER 4 Thank you.