Union, I'd say.
Live meetings 10%.
Low.
I think we can go another minute or two and then we'll get
started.
That sounds great.
Can you guys see me on video? Because I can't see myself, OK?
So I won't like brush my teeth, comb my hair, things like that.
Rob, if you feel comfortable, why don't we go ahead and get
started?
Absolutely. That sounds great. Good afternoon, everybody. I'm
Robert Lovato. I'm a clinical psychologist at the University
of Nebraska Medical Center and one of the folks who's helping
coordinate the grand rounds for the Center for Intelligent
Healthcare series. I'd like to introduce Doctor Kathleen Blake
will be tonight's speaker. Doctor Blake earned her medical
degree from Pritzker School of Medicine and the University of
Chicago and her MPH from Johns Hopkins Bloomberg School of
Public Health. She has had a distinguished career in health
information technology.
Policy and is currently vice President for performance
improvement at the American Medical Association, where she
overseas quality initiatives and the development of electronic
clinical quality measures. She also serves as a director the
executive director of the Physician Consortium on
Performance Improvement and has executive responsibility for the
National Quality Registry network.
Doctor Blake will be speaking today on safe and effective
artificial intelligence. She's asked that we place questions
into the chat, and that those questions will be read at the
end.
Welcome, doctor Blake. Thank you.
Thank you so much, and I'm delighted to be here. I do have
to admit that I have moved into a role as senior adviser at the
American Medical Association and let me share this secret, which
is that it then means that I get to work on the topics that I am
passionate about, and one of them is artificial intelligence.
The title that I chose for today's grand rounds is very
intentional, safe and effective. For those of you familiar with
the regulatory process, is the really the mandate for the Food
and Drug Administration when it approves for marketing any
product that is going to be used on patients in the United
States. And this includes medical devices that oftentimes
will have artificial intelligence.
Either as a standalone or as a.
Integral part of it.
So the approach will take is we'll talk about concept to
practice and my goal really is to be able to share a
perspective that I hope will result in as listed here safe
and effective AI. I have no disclosures. I won't be
discussing non approved uses. And as we are required with
anything that involves the AMA, these remarks are my own
although you'll see a lot of AMA policy and similarly.
And served until the end of May of this year for on the national
evaluation system for Health pet health policy nest and was on
the governing and executive committees. This is a public
private partnership to address medical device issues next.
And so just a brief history about the American Medical
Association, because I understand this is a mixed
audience.
Some may be members, but others may really not be familiar with
it at all.
It's an, shall we say, seasoned organization. It was established
in 1847, which means this year is its 175th year of operations.
The mission is to advance the art and science of medicine and
the betterment of public health.
The membership is in the range of about 255 hundred 1000, or
255,000 US physicians, students, residents, and fellows. Policy
is made by a House of delegates that represents many subsidiary
or allied organizations, the states, Territories, National
Medical Specialties societies, professional interest
associations, federal services.
So think Air Force, Navy, etcetera and AMA member
sections. Next slide.
And so the AMA really began out of a need for a code of medical
ethics and this was really a living, breathing document. It
was recently updated and is the code that is used across the
United States to guide our conduct, including as it relates
to AI next.
And so this is a picture of the building that we used to work
in. And you may say what on God's earth is all of that like
laboratory equipment doing there? Well, in the 1940s, turns
out that the AMA was filling a gap, you might say, which was to
analyze drugs and other products that were on the market because
there wasn't a robust structure like we have now in the FDA.
Next slide.
This is an example of a medical device from the 1940s, a Wilhelm
Reich said that an ill person could cure themselves by just
sitting in this quote shooter box and they would get an
infusion of orgone energy that would relieve symptoms from
every malady from cancer to impotence. So the AMA stepped in
the gap and has worked very closely and been very supportive
of the work.
Done by the FDA as it has assumed this responsibility.
Next slide.
And so let's Fast forward from the 1940s to 2018, we really saw
the handwriting on the wall, which was that artificial
intelligence was a growing endeavor, not just in the US,
but worldwide. And the AMA passed policy you see here the
House of Delegates.
Excuse me and says that it would help set priorities for
healthcare in AI that it would not stand by passively. In other
words, that it would work to integrate the perspective of
practicing physicians that would start with the development
process, design, validation and implementation.
That it would provoke and really actively encourage the
development of very thoughtfully designed, high quality,
clinically validated healthcare AI.
We realized the profession had a need for education so that it
would both understand the promise but also the limitations
of AI. And we committed ourselves to helping to advance
and provide that education and the legal implications, which I
will spend more time than perhaps you expected about
healthcare AI on issues such as liability. This is a very hot
topic right now, intellectual property and also to advocate
for appropriate.
Professional and governmental oversight.
Next slide.
But just as important as I've finished describing what the AMA
does, it's important to know what we don't do. We do not
develop AI products. We are not a developer. We are not a
regulator. We don't evaluate or validate AI products. We don't
endorse the AI tools and products that increasingly we
see in the marketplace and we don't sell them. And so I think
that this gives us some very important separation and
neutrality from what you might call the more.
Commercial interests associated with AI. Next slide.
So our vision and this is an eye chart, but really just focus on
the middle and I think the audience will have the slides to
be able to look at some of the fine print, but you'll recognize
the quadruple aim and we've really adapted it to the
development of AI who said AI should enhance the patients
experience of their care. We don't want it to degrade.
Second, it should improve population health and ideally
reduce cost, but at a minimum.
Keep costs stable. And also the fourth thing would be to improve
the work life of healthcare providers, meaning it should be
a benefit. A balm so to speak, not a burden. Next slide.
And so on the basis of some research and extensive
informational interviewing that we did across multiple
stakeholders, we realized that there were three key issues
where we could weigh in and one was on evidence. What is the
evidence that was used to develop the AI product and how
generalizable or how narrow is it, what is ethical use of AI
and how is the ethical charge really the responsibility of
physicians?
Upheld or unfortunately sometimes diminished by the
introduction of AI and then increasingly I think especially
as we've gone through the pandemic, focusing also on
whether AI advances our common goal of HealthEquity. Next
slide.
So the new imperatives and we'll go to the next.
I think this won't surprise anyone, but the chart bears just
where the graph bears a quick look, which is that we have an
aging population and that there will be an imperative to be able
to care for people thoughtfully.
In the context of the quadruple aim, but that we're anticipating
that it will be challenging to do so unless we think now about
how we might use AI's next slide.
And the population as it ages, I wouldn't say inevitably but very
commonly develops an increasing number of chronic conditions
that could very much, very easily outstrip supply and
demand. And this was most visible, I think to us during
the pandemic. But even as we come out of the pandemic, I
think we all realize there will be probably future pandemics,
but also just the normal challenges.
An aging population, which is why we've got to get AI right
the first time around. Next slide.
Ah, the aim about reducing cost.
This data was updated in 2020. I'd be the first person to say
I'm not sure what things will cost and as we come out of the
public health emergency because that has so distorted the cost
and availability of care. But $3.8 trillion is a lot of money.
It's a lot of money per person. I think if anyone thought about
having to pay that themselves, they would agree that's a lot of
money and it is a large percentage of our gross domestic
product next.
So another eye chart. But anybody who talks about a I just
can't resist putting some kind of figure or chart or picture
like this. These are all the different originators and use
cases in the healthcare data ecosystem. So this is the
building blocks. This is the source of data. Sometimes good,
sometimes not so good, sometimes we don't know the quality, but
lots and lots of data.
And lots of usage of it. Next slide.
And that data then is fed into a very complex ecosystem. I put
here digital health because there is so much.
Connect connectivity between AI and digital health products that
at the center are affecting doctors and the patients they
care for. Next slide.
And there's a clinical imperative and this is pre COVID
data and I'll update it a bit for you. Burnout in the
physician population, which is what we measure, we're standing
at about 4849% pre pandemic. Unfortunately, our most recent
data published about two months ago, it's up in the low 60s
percent. So this is an unsustainable.
Situation for clinicians, and I say clinicians purposefully
because I'm really talking about physicians, nurses, anybody who
is in the healthcare system. And if you measure it in that broad
swath of people in in our environment, you'll find that it
is higher than the general population of people who work
and that it is higher now than ever before.
The other thing that we've learned from some research we
did with Dartmouth Hitchcock health system was that
physicians are spending a lot of time with their electronic
health records. Now, some people might say that's great because
they're putting in the data that we can then use to develop
algorithms that might ultimately save time. But what I can tell
you is this is one of the major causes of burnout.
Doctors did not go to medical school to take care of a
computer. They actually went to medical school to take care of
real people. And there's a lot of interest in what you might
call the financial investment value of AI and healthcare
generally, with a lot of private equity firms cropping up that
are doing healthcare deals not just for vertical integration
but also for information.
You might say consolidation. Next slide.
Some of you may have seen this graph, and I call this the like,
why AI and what are we expecting? And this was
developed by Gartner and colleagues. It's a registered
trademark, and this is their most recent version for 2021.
There's a trigger. Someone says we could do this. Then there's
inflated expectations, sometimes justified.
Because it's a narrow population then crash, there's a through of
disillusionment, a slope of enlightenment, and then there's
kind of a plateau of productivity and it's at that
plateau level that we really see is this a.
An approach, a tool that enhances the.
Fulfillment, you might say, of the quadruple aim or does not
next slide.
So some definitions. Let's go to the next.
And this group again, which I realize is a mixed group.
I think would not be surprised by some of these definitions of
artificial intelligence. Machine learning, which is when AI
systems acquire their own knowledge by extracting patterns
from raw data, and then deep learning which is many, many
layers of data that build out complex concepts. The further
you go from machine learning to deep learning, the more you
might say black box and AI product becomes.
Next slide.
And so following through, I put this in and it may seem sort of
simple to put it in a talk for this group or maybe a bit naive.
But I think it's important as we move later to the regulatory
issues to realize that this is probably really, really where
the rubber meets the road because this tells us how well
can the user, the end user.
Understand or relate to the output that comes from the AI.
They're probably able to do pretty well when it's rules
based. These are clinically accepted rules of qualified
physicians will understand them. They're used to using clinical
decision support, guidelines, labels, published literature,
etcetera. So rules based seems safer than some of the other
models. So data based AI is not being explicitly programmed.
This is where you're starting to find the patterns.
With machine learning and deep learning, regression patterns,
things like that, and within the database AI products, we're
always going to be interested in whether it's a locked model.
Is it developed?
Disseminated. Used.
But stays consistent throughout that time? Or is it continuously
learning?
And think about the medical legal case which is that you run
a particular patients data into a system that is unlocked or
continuously learning. So the answer you get today may be
different than the answer that you get six months or a year
from now. And it's important that as a community we
understand how fluid those answers might be. Next slide.
AI.
Is trained on data, it's tuned on data and it's tested and all
of that can vary widely across the products. These are things
that are still being worked out. And I think if there's one
message that I would like to convey, it's that the reason
it's important to pay attention to regulation, to pay attention
to what I think will probably be some legislation in the next
couple of years, is because.
These are the rules that we will live by and they are the
standards to which we will be expected to adhere. So will
there be a training set with over 100,000 images or a
training set with, just as is shown here, 216 images?
What about the tuning set? And what about the test set? So this
is the kind of information that probably we would want to know
or have access to, whether it's in the label, the package insert
something so that we're able to make our own determination of
whether it's ready for use in our setting or not. Next slide.
So the AMA prefers, but oftentimes is not able to make
this distinction because the acronym AI does really stick.
But we.
Would propose that we conceptualize this at least in
the clinical arithmetic or arena as a augmented intelligence. So
we really focus on the way that it can assist physicians and
other clinicians help health systems and enhance and really
have a synergistic effect with human intelligence rather than
replacing it. Next slide.
Four questions let's.
OK, these are the key questions that we ask ourselves about
every product, about every policy, about every regulation.
First of all, does it work? Does the tool actually work? And if
it works and I purchase it for my practice, my health system,
my insurance company, whatever I might be, will I get paid or
will I somehow be able to balance the cost? Or am I just
adding to the cost of providing care? Will I get sued and.
The answer is perhaps you will get sued and I'll show a
particular issue that we're grappling with right now. And
will it work? Not just generally, but will it work in
my practice? Next slide.
So any AI product that you look at, you want to have some sense
of where it fits in this kind of table, the risk categorization
table that was developed by the international medical Device
Regulators Forum for software as a medical device and essentially
the vigilance and the pre release, pre marketing or
authorization review depends very much on the risk.
Level. For example, if it's a critical healthcare situation
and the product is going to be used to treat or diagnose, it's
a class 4 risk level, the highest level of risk. Whereas
if it's informing you and it's a non serious condition, you're
all the way at the level. One risk where vigilance may not be
as as necessary. Next slide.
So does it work, the clinical validation and there are several
different types of validation. Is there a valid association
between the output that you get from the product you've just
used and the patients individual clinical situation?
Does it work from the analytical standpoint? Does it correctly
process the data, provided the data is clean to begin with and
generate accurate, reliable, precise outputs and then
clinical validation? Does it make sense for the patient in
front of me, the people I care for? So how might you answer
these questions? Well, look and see where the results. Did the
authors? Did the developers publish their results in the
peer review?
Literature. Is the data accessible for others to
examine?
Is the population in which it was developed and tested
described the so-called table one of a peer reviewed journal?
Did they consider bias and if so, how did they evaluate the
potential for bias? Is it explainable? Are the outputs
reliable and how does it perform? To what degree is there
a return on the investment that has been made in using the AI?
Does it improve sensitivity and does it improve specificity?
Next slide.
So explain ability whenever you type it. Certainly Microsoft
Word goes crazy and doesn't know what you're talking about. But
here's what the National Institutes of Standards and
Technology is talking about.
Explain ability for them says. Or is that a system delivers
accompanying evidence or reasons for all of the outputs. It's
meaningful.
The explanations themselves are understandable to an individual
user. I call this the plain English clause.
The explanation is accurate. It actually explains it in a way
that is accurate. It's telling you what it really does and is
and it acknowledges the limits that surround anything in
healthcare or in information, the knowledge limits. I call
this the use or don't use kind of boundary. Next slide.
Does it work? And if so, what was the route to market
authorization? This gives me a chance to just briefly say that
most AI products that are out there in common use are not
evaluated or authorized by the FDA. They actually and companies
are quite open in saying they don't want to have to go through
what is a much more rigorous process. What does that mean?
Your patient might be looking at the results of a screening tool
for, let's say, cancer of the skin, and it will say this is
not intended for clinical use. Talk to your doctor. So all of a
sudden your poor doctor is getting, you know, 100 people a
month who've used the tool but really want the clinical, shall
we say, Good Housekeeping seal of approval.
The number of these products is astounding.
Even though the FDA doesn't regulate most of them, just
between 2020 and 2022 we've gone from 161 authorized products to
521.
So one of the fears that the FDA concerns, I should say that FDA
has, is this number is only going to grow and in a world of
limited resources, how on Earth will they keep up with the
regulatory burden?
And it has several choices. Pre market approval is the most
rigorous, typically requires controlled trials and a lot of
human data. 510K authorization means.
We're telling the FDA and we hope it will agree that our
product is so similar to an already approved or authorized
product that it can fall under, shall we say, that same approval
or authorization. And de Novo is the truly novel products for
which there isn't any, shall we say comparator.
I asked two questions. Next, what's in the label? Do we need
a new label? I think you might have a sense already. My view is
we are going to need a new label. But getting a new label,
which people have been talking about for years, is probably
going to require legislation and regulation. But what would be in
that new label? Well, indications for use. So we'd
want to know and this is a little different.
Then the label for other medical products.
How large was that population? As we alluded to earlier that
was used? What were their characteristics, particularly
gender, race, ethnicity, age? We may see more and more use of
social determinants of health information.
We will have a different kind of shall we say AI black box
warning. So we're used to seeing that with pharmaceuticals. But
the anti use cases might be in a black box that say this has not
been tested in children, it cannot be used in children or
this has only been tested in individuals from let's pick on
somebody, Minneapolis, MN and cannot be used in individuals in
Albuquerque, NM.
New Mexico's my home state.
The classification of risk. So this is a shortcut to saying how
concerned, how vigilant should I be, who developed it. I will
admit my bias is that if I see that a strong, clinically active
institution and developers are responsible for a product, I am
much more confident about it and about how they went about
developing it then if it is outside of the clinical
environment.
And so the context, it really does matter. And was it bias
proofed? So let's say my population is, as it is in New
Mexico, very multiracial, multiethnic. So we are one of
the states in America that there's no ethnic majority. And
so a product in New Mexico has to include individuals who come
from the original native peoples of the United States.
Individuals who are of Hispanic or Latinx origin, as well as
individuals who are Anglo.
Our population of Blacks, African Americans is
exceptionally small.
So we would not be the place to develop AI because our data just
will not reflect that.
Population of people. What that means then, and I think it
applies also in Nebraska as well as anywhere else, is that this
is a team sport. The data can start, perhaps, or the
development can start in one place, but it sure as heck needs
to have other places well characterized that are able to.
Prove or disprove its validity or improve the product and then
the methods as we've talked about before. Next slide.
There we go.
Did I just lose?
My screen.
My screen just came to.
Something, yeah, let's keep going.
That would be a quiz at the end of this. Not really. Only
kidding. OK, here we are. So let's assume the AI is going to
change.
It's different than a chemical molecule, which is inert and
will not change. It's different than a medical device that may
have incremental improvements and AI will change in different
ways. It might be how data is collected, features extracted,
how it's trained, etcetera.
And so FDA will be interested in what you might call some pre
specifications of the product.
If it is going to change, how will it do so and will there be
a predetermined agreed upon algorithm change protocol? And
how closely did the developers adhere to what are called the
good Machine learning practices?
When is an amendment needed, meaning an amendment to the
authorization of the AI?
And this is hard probably if I, you know, add a slightly
different population, I may not need an amendment to those, the
label as we've talked about it. But if I add a very different
population or I'm using very different tools or my results
are different, then I probably will need an amendment because
then the users will know to whom it applies.
What does the post market surveillance look like? That if
we have a product that's changing all the time, we'd like
to have, but we don't yet have a strategy for really effective
postmarket surveillance. And how will safety signals be
communicated to and from developers, the FDA, users and
patients, FDA's convened patient groups that have said if
something is changing that will affect my health, I want you to
tell me.
Because then I know how I should respond, what do I need to do?
And then what's needed for all of this change management, you
might say, is methods and standards and there are a whole
variety of groups that are some, only some of which are listed
here that are working on this next slide.
Will I get paid?
So presumably if you develop an AI product within the walls of
your institution, that development is sort of paid from
normal operating expenses. But let's say you're a smaller
facility or a clinical practice and you actually need to
purchase a license or purchase a product.
And in order to purchase it, you need to know what's its value,
how was it valued, what will be the offsets and to what degree
will it reduce the cognitive effort of the human being
delivering the care, interacting with the patient or increase the
cognitive effort depending on how much of it is machine
driven.
And then you get into some technical details with respect
to whether it gets categorized as a practice expense. So a
practice expense might be what you pay for rent, for example,
or what you pay for health IT. But then you have to decide, is
it direct or indirect? These are the things that quite frankly,
I'm glad other people help figure it out because my brain
would just, I think, explode. And is it a professional
service? The more I interact with it, the more the machine
needs.
Me, so to speak, to be effective the closer it is to a
professional service. But if it is substituting for professional
service, payers will tell you they don't wish to pay the same
amount for the professional service delivered by the
clinician.
Next slide.
I.
I'm just going to say that the way that payers pay for things
is by is based on how they are described.
So.
There are different descriptions that are used by different
entities. FDA describes the product based on what it has
been told by and what's been shared with it by the developer.
Things that we've already talked about.
CPT is code descriptors. They have to be discrete. They have
to be differentiable, meaning that they are different enough
from some already approved service so that it justifies a
new code and for example, retinal imaging with an
automatic or automated, excuse me, point of care diagnostic
report, one of the first products approved by FDA in the
United States.
Was able to get a unique descriptor, but description is
not enough because then the CMS coverage and payment group looks
then at some of the questions related to how much work is done
by the machine, what are the circumstances of use and what
are the business models. Next slide.
So AMA, which overseas the CPT terminology, has issued an
appendix. The citation is here, and it was updated about a year
ago, not quite a year ago, for artificial intelligence
taxonomy. So the CPT editorial panel reviews submissions.
Of anything, any service that is provided in healthcare.
And has to fit it into, shall we say, a bucket.
Is it assistive the AI product detects something but the
clinician is required to interpret and report? Or is it
augmentative the machine actually analyzes or quantifies
the data?
And there will be still clinician engagement or is it
fully autonomous? So this helps figure out what lane of review.
Is going to happen when a code is applied next slide.
So this is the area that's changed very, very recently, in
fact, with a use case from the federal government that I'll go
into in some detail. But the question is, will I be sued?
And if I'm sued? So let's just assume you'll be sued. But how
is liability apportioned?
In the US legal system, you can hold people and corporations
responsible or liable. You cannot hold a machine liable.
Most developers are seeking to maximize protection of their
intellectual property while at the same time minimizing their
liability. These are conflicting goals.
The Supreme Court decisions with respect to FDA authorized
medical devices actually protect the manufacturers from
liability, it says. That's the Good Housekeeping seal of
approval, so to speak, so the developer is not responsible so
long as the product went through that FDA process and functioned
as it was meant to function.
The developers have been concerned that they may have
more and more trouble patenting their software.
And what does that mean? A patent applicant has to provide
a disclosure?
So as a scientist of quote ordinary skill, I'm not really
sure what that means. Could make and use the invention?
And the developer has to provide a written description about the
structure of the invention they are claiming. In other words,
there has to be a recipe that I, the ordinary skill scientist,
can use and rely upon.
People who've invested a lot of time and money into developing
something are not very keen on that.
Because they'd like to realize some return on their investment.
And so there's increasing use of trade secrecy laws, but that
comes at the expense of transparency about the device or
the tool and also greater potential liability. In other
words, we're going from a clear box to.
A black box and everything in between and the implications for
patients and clinicians are profound, particularly with
respect to trust. Next slide.
OK, I put these in, but this is, shall we say for the record and
I'll skim through these fairly quickly, but it is a hot topic.
Just this summer, the US Department of Health and Human
Services Office of Civil Rights proposed some new rules.
That would implement provisions in the Affordable Care Act.
That relate to non discrimination health programs
and activities. A specific section in that proposed rule
relates to the use of clinical algorithms and decision making.
There are some very reasonable, shall we say, preamble, type
statements, covered entities, so practices, health systems,
etcetera, must not discriminate.
Against any individual on the basis of race, color, national
origin, et cetera, through the use of clinical algorithms in
its decision making.
Sounds OK.
It says this is office of Civil Rights. It's critical to address
the issue explicitly. Given recent research that shows the
prevalence of clinical algorithms that can result in
discrimination, this is the optimum.
Algorithm that looked at expenditures for healthcare to
decide which individuals would need additional services and it
turned out to discriminate against individuals who were
black because historically and even currently they do not use
health services at the same rate as non blacks.
Here's where it gets tough.
The proposed rule puts covered entities, that's all of us, on
notice that they cannot use discriminatory.
Clinical algorithms and may need to make reasonable modifications
to their use of the algorithms.
And it says the intent of the proposal is not to prohibit or
hinder the use of clinical algorithms, but rather to make
clear that discrimination that occurs through their use is
prohibited. So this is.
A very concerning set of statements because it says we're
not telling you that you can't use these products, but if the
result of using the product is that there is discrimination.
You, the user, will be liable next slide.
Because this is the proposed assignment.
And it says here's how you should consider the tools.
And the fact that a covered entity did not design the
algorithm, does not have knowledge about how it works,
does not alleviate their responsibility to ensure that
they not take actions that result in discrimination.
So.
Now that probably I've shocked people, I will tell you that
this has elicited a hue and cry.
From many, many quarters.
And it's been interesting to watch because our feeling is
quite strong. We think it is premature to put something like
this in place and we also think that it will inhibit.
What is a very important innovation opportunity for
healthcare? Next slide.
So be careful what you wish for. This is a citation by the Office
of Civil Rights saying we propose that providers use the
quote AMA framework.
To achieve all these following things, this is a framework.
But it is not a cookbook, shall we say, but it's a framework
that I and others at the AMA developed saying this would be
the ideal set of circumstances. These are the sets of questions
you should ask of a developer. The problem with OCR's putting
this here is they acted in a way as if all of this already
existed and it does not. Next slide.
So we said a couple of things and this is a huge rule. It
applies to lots of things, not just to AI. But my talk here
today is about clinical algorithms. So we support the
intent which is non discrimination. We support many
of the ways in which that intent would be realized, but we urge
that it not be adopted.
We say let's get all the relevant people at the table.
Let's pause so that we really understand the implications and
in the meantime, that it is reasonable that the developer of
the tool bear the responsibility.
For and therefore the liability if errors result because they
are the most knowledgeable party and we disagree and I put it in
bold, it's in bold in our letter that we absolutely disagree that
holding an individual physician liable.
Will achieve the goal of reducing discrimination. We
think that is not ready for even reasonable people to consider.
Next slide.
So.
I'm going to go through these quickly because I promised I
would leave 10 minutes or so for questions and answers, but this
framework is very specific if you're trying to figure out will
this work in my practice and who's responsible at these
various stages? Planning and development, followed by
implementation and monitoring. And what you'll see here is at
almost every single stage.
More than one party is responsible.
This is a team sport. This is not me pointing a finger at you
or you pointing a finger at me. It's actually developers,
deployers and physicians working together. Next slide.
So this is the example I mentioned earlier of diabetic
retinopathy showing that and this is an really an automatic.
School that is able to evaluate the retinas of individuals with
diabetes and has very high sensitivity for diabetic
retinopathy. Those who are put in that bucket are then
evaluated by a clinician. But think of somebody in a rural
part of New Mexico, Nebraska. They can have a pretty high
level of confidence that something serious will not be
missed. Next slide.
This is another one. It has to do with the use of EKG's and
really looking very carefully at the EKG to be able to diagnose
long QT interval, something that is a potentially life
threatening condition.
That sometimes is very subtle or perhaps not even visible on the
ECG. Just as importantly, it can also tell you who's normal and
who does not need to be worried on this one was developed by the
Mayo Clinic. Next slide.
And going to what some are now calling the quintuple aim, we
have to be careful that algorithms do not codify or
reinforce racism or health inequities. And this was an
algorithm that was developed by Optum to look at that cost of
care as a proxy.
That would predict eligibility for disease management.
Individuals who are black were unfairly excluded from
eligibility because they did not use health services at the same
rate as whites. So people who are equally sick, if you were
white, you got the services. If you were black, you did not.
Next slide.
And this is a slide that we really rely upon often, which
is.
Looking at equality, everybody stands on the same box,
everybody gets the same thing, but it's not getting you to the
same place. Equity people get different resources. They're all
able to see in this graphic the game. And so that is more
equitable. Reality is this unfortunate situation, which is
this chasm between the haves and the have nots, and liberation is
that fence is removed.
So this is what we would like to move towards, away from reality
to equity and liberation. Next slide.
And this is this discussion about bias in algorithms.
It is largely unconscious. I don't think that people wake up
every day and say I'm going to try and be biased in the care I
deliver, but what we need is an environment.
In which we explicitly acknowledge that risk and then
take measures to counteract it.
Next slide.
And so the implications and how they fit into the place, one of
the places I started how do these algorithms and complex
models become trusted?
They are not trusted to start with. How important is it to see
inside and explain a black box? But if you're going to explain
it to me, I need it in plain English.
How do we ensure that new solutions do not discriminate,
make inequities worse, or deny care?
What are the potential concerns of platform dominance? We think
there really needs to be a robust multiplayer environment.
And how do we keep up with the rapid pace of technology? Next
slide.
So Eric Topol, who I love to quote, he always says it very
concisely, the state of AI hype. That graph we saw before has far
exceeded the state of the science, especially when it
pertains to validation and readiness for implementation and
I would say assignment, particularly of liability. Next
slide.
And beyond the algorithm, it really is all about patience. So
let's please always look at that. Does it make that
clinician patient experience better and the trust stronger?
Next slide.
And finally, acknowledgements to my colleagues at the AMA past
and present who have worked so incredibly hard and creatively
to develop the positions, the approach perspective that I've
been able to share with you today. So with that I will close
and Robert, I'll turn it back over to you.
Thank you. That was really a head spinning whirlwind of of
issues and ethics. Wow, there was really a lot in there to
unpack. I think I'll open it up to the audience and see if
anybody has any questions that they would like to ask. If so,
please either put them in the chat or feel free to hop in and
ask directly.
I guess I'll ask a question. Thank you, Kathy. This was
spectacular. One of the things and I appreciate the
thoughtfulness of this and I'm kind of curious with some of the
AI scientists.
Is that there's Merton's law of unintended consequences. And as
I looked at some of the rules that it really seemed to fall
into that that that algorithm can't use any of the following
criteria to make a judgment. But we do that all the time so that
we treat a 25 year old and a 95 year old differently because the
risks and benefits are different. So I applaud the AMA.
For for thinking about that, because I agree, we're a long
way from exploring the unintended consequences.
But as you look at this, what what are the next steps? How can
we be useful in this?
Discussion.
So first of all, by developing the program that you've
developed, you've already created an ecosystem, I would
say, where people of goodwill we might call it and of shared
purpose are able to honestly openly discuss these types of
issues. And I think those discussions should then lead to
a an approach similar to what we've developed, which is an
approach that.
Is pragmatic.
That is based on real world experience and that acknowledges
with some humility I would say.
That this is a work in progress and I think that would be very,
very powerful. And as somebody who comes from, as many of you
do in so-called flyover country in America, I think we sometimes
suffer from.
Where we are different, we're different in ways that are
really good. We have different populations of people. We're
more reliant, I would say, on each other. And so there is, I
can absolutely tell you, enormous interest in having a
group like yours come and say.
This might work in.
Name whatever big city. But here's what we have found in our
particular environment, particular is still being quite
large. You've got a large rural state.
And also.
Being willing to partner with those other types of entities to
say just as they will bring value, we will bring value and
that will increase.
The.
Reliability.
Of, and potentially the use of an AI tool.
So I think there's tremendous opportunity to take what you've
built already.
And share it in that kind of way.
Thank you.
Kathy, this is Jimmy Chang. That was an incredible Tour de force.
Hello, hello. Thank you. It's nice to see you. It was
absolutely wonderful your your treatment of this topic. Really
do appreciate the depth that you went through in terms of the
legal components. You are correct. We, I don't think we
were expecting that, but extremely helpful.
You've touched on the issues of.
Narrowness of applicability of AI solutions and by extension
their brittle nature, that is that you change parameters and
now all of a sudden what you would you would assume would be
working is no longer working because of the fairly brittle
nature. But if you turn that around, and I'm asking now for.
Your perspective in terms of looking at the crystal ball,
where have you seen or where do you anticipate AI and its
derivatives will likely be the most successful? Where is it
most likely to be a, you know, what domains would it be most
likely to be a successful adjunct to? Clinical decision
making, diagnosis, therapeutic decision making, whatever you
can imagine but.
If you could provide a little bit of guidance as to how we
can.
Focus our efforts in ways that are likely to result in a
positive outcome.
So Jimmy, first of all, thank you. And.
I realize we're at time, but I'm not going to evade the question,
and I'm happy to stay for as long as people want to ask
questions. So my approach is pretty pragmatic. Don't waste
your time on things that you're already doing pretty well. So if
your health system or your organization or your specialty
is already doing a good job of, you know, diagnosing XYZ.
Don't. Don't go there.
The second is you do have to go where the data is, and that
means most likely you need to go someplace or to a condition or a
population where there is a lot of data. Either a lot of people
are well understood or are in the system already, or if it's a
less common condition like long QT syndrome that you have a lot
of data on those individuals, so.
With the Mayo algorithm you have a lot of EKG's, so that's sort
of the will I ever be able to get the data I need to arrive at
some algorithms and then the third part.
Has to do with being.
A bit fearless about failure.
OK.
And you can be fearless about failure if you have backup
systems in place. But a beautiful example recently that
came out of Hopkins was a new sepsis algorithm. They said,
we're not going to diagnose sepsis. OK, sepsis is hard to
diagnose and sometimes it, you know, mimics other things.
But what we're going to do is develop an alert system that and
we will do it with clinicians and we will do it at 5 different
facilities and we will test it.
And have the clinicians tell us, was this useful meaning?
Yes, it properly identified someone with sepsis or.
The system said this person might have sepsis. I looked at
it and I was confident, based on everything else I know, that
they did not. In other words, there was.
A separation.
Upon use of the algorithm into groups at higher or lower risk,
so I think that is important. And then.
I can't say it often enough. I think transparency is actually
the developer's friend, and I know that oftentimes they don't
believe that or they think that's naive, but they will get
to a good product quicker than they will if they are not
transparent and epic with their very.
Harshly criticized sepsis algorithm has actually now made
it so that health systems can be much more transparent, can adopt
or adapt the algorithm to their environment, can test it on
their patients and I think that that.
That was a hard learned lesson, but I'm married to a computer
scientist and we used to laugh because I would come home from
work and say oh boy, you know, near Miss. Today we got this
person through whatever it was, the heart attack, the cardiac
arrest, etcetera. And my husband, the computer scientist
would say.
Boy, I got to break more systems today. It was great. So
somewhere between the fear of breakage, that fragility, that
rigidity and the, I used to call it mayhem that some computer
scientists seem to just relish, I think is where we need to be.
Hopefully that gives you a sense of where I would approach it.
Any more questions?
Kathy, what I will say is this actually is very confirmational
for what we're seeing at Nebraska in terms of
transparency and the importance of.
Trust by clinicians in moving forward. So it's a combination
of good data and explain ability that I think is going to, it
won't.
Overcome the power of Google or Amazon and.
8,000,000 records, but I think it can get us in a way where
clinicians will accept it.
And maybe just one last closing comment. I have learned so much
over the years from Brent James who was at Intermountain
Healthcare. And one of the things that he would frequently
make reference to was that in a complex environment such as
healthcare, we don't need everybody to know all the same
things and we don't need everybody to have the same
perspective. And in fact that because people used to ask him,
well, how do you get people at Intermountain?
On board with everything you're doing and he said, really what I
need is I need them to be on board with the part that they
will be doing.
And so I think that there is a sense of relief when you know
that you don't have to do it all because other people are right
there with you doing their part.
Far better than you would ever be able to do it, so I really
appreciate the chance to.
Give share our thoughts and I hope enthusiasm about this
opportunity.