As we discuss editing of essays, electronic applications, and other adaptations to the admissions process it is clear that we need to spend more time and effort on evaluating candidates for medical school. The current paperwork and then a single interview or two is just not enough. The students that most abuse the system will, as usual, cause all to suffer consequences. Efficiencies will only make it easier for them. Medical school deans and faculty, often the most dedicated to keeping their lives from being more complicated, must ante up, or medicine will continue to suffer loss of prestige, loss of trust, and continued misdirection of scarce health care dollars.
The area of evaluation of medical students is becoming even more important, considering that the graduates of medical schools are becoming, in themselves, a major cause of death. Medical errors are gaining ground on the major causes of mortality and morbidity in this country (about 600 die per month or so). Physician incompetence may be at an all time high despite the increased surveillance and increase medical training of the past decade. At the last AMA conference on Physician Competence back in the mid-1980's, basically the consensus was that it was impossible to deal with the issue of incompetence. Those who abused the system just learned how to play the game better rather than reforming.
I find it interesting that we do not take evaluation more seriously regarding people that will be held to such high standards in their various medical careers. We seem to fail to understand the very complexity and multidimensional nature of evaluation in medical education. Even though physicians are not the autonomous entities of yesteryear, they still have the potential for causing great good, or great harm.
A profession that increasingly stresses prevention of death and disability must examine itself and the process of selecting those who will be agents of prevention. In the past we have been unable to proceed for lack of information and techniques. Information is improving and qualitative evaluation has come a long way. We know that in medical school, it is easier to prepare basic science exams, less easy to do internal medicine exams, and nearly impossible to do family medicine exams. There is too much variety in the experiences of students on rotations and there is a greater breadth of material. It is much harder to teach "to the boards" when the subject is more diverse.
Why is it such a stretch to think that medical student applicants are even more complex to evaluate?
In response to the challenge of testing more broad subjects such as family medicine, medical schools are moving to Objective Structured Clinical Exams and essays. These are much better for evaluation than the written tests as noted by one of the family medicine predoctoral exam preparers today on the Family-L list serve (Dr. Jacques). Others on the list serve noted that the MCAT essay may have increased importance. See reflections on med school predictive measures at Best vs Brightest: What is the Best Fit For the State (or Nation)
How can an OSCE stand up the the daily, almost hourly measurements and adjustments of a rural preceptor over 9 months as in RPAP? Why a Preceptorship Is Better Perhaps the reason for OSCE is the lack of actual patient contact in medical education, worsening with each liability and discontinuity factor.
Those of us that are trying to help medical education return to more preceptorship-based experiences have faced the challenge of more demanding evaluation methods. On the usual medical school board tests, preceptorship students do not look much different than mainstream students. When you look more closely and examine 23 measures, beyond the clinical and into behavioral and procedural areas, the students doing preceptorships were equal or superior in all measures (Verby, Minnesota Rural Physician Associate Program - RPAP). This was not selection bias. These students also were at or below the median in the standard measures prior to the preceptorship. This is an example of how more global measures are needed to evaluate students and programs.
Don Madison at UNC Chapel Hill did a nice study noting that admissions committees using only the standard information, could evaluate the service orientation and experiences of medical students. With this extra effort admissions faculty could predict the student's choice of a primary care career. If the selection committee had even more information, they could do even more. Service Characteristics
There are several options. We could
1. Increase the number of interviews and add more structure to the interviews with a high suspicion index for problem areas. At Baylor we had 4 interviews and then if there were questions, student had a 5th interview with one of the behavioral faculty.
2. Set up team-building week long experiences (expanded ropes course stuff, brief project, reports). Of course even the bad apples take a while to sour.
3. Use small group facilitators not only to teach and advise students over the first two years of medical school, but also to evaluate them. Buy out the students that did not measure up.
4. Expand use of college advisors and others in high school and college who, I suspect, often know much more about the applicants when compared to admissions committee members.
5. Require a structured and approved service experience.
We can try all of the first four in a typical American way, after we have exhausted all other possibilities (as Churchill noted). With the proper studies, we may even be able to learn enough to figure out which approach works for various types of students. I do not think we have the time to wait, nor can we afford any longer not to act in a demonstrative way. I would prefer to take the more detailed approach. I would much rather have medical students do a structured and approved service experience in the college years as a prerequisite over any of the other measures.
The challenge would be in the structured and approved part. First for the pro part before getting to the more obvious con. A large number of students have done experiences that would likely qualify them. Others have significant work histories or could obtain them with little extra effort and some potential reward. There is the potential that we could get even more predictability in several areas of evaluation with such a structured prerequisite. These include service orientation and potential for serving the underserved. Also we could explore the candidate's ability and willingness to communicate, their ability to work with weaknesses and know their limitations, etc. They also would demonstrate the ability to do complex problem-solving. Interactions would also give information about the integrity and work ethic of the candidate. Now that the AMA notes that it is important for physicians to deal equitably with all people and populations, the service experience would allow observations in the candidate's ability to work with a variety of individuals from different socioeconomic segments of society.
I also have high hopes that enhanced observations would help to identify potential problem physicians. In the past 20 years in medicine and medical education, I have seen two approaches that actually helped deal with the area of incompetence. One was in Tennessee where for a time a senior physician was granted broad and flexible powers to deal with incompetent or problem physicians. I doubt that many have his wisdom or experience and the due process issues are enormous. The main problem of this approach is that patients (often many patients) have had to suffer before discovery and remedy. The second approach offers more potential. Jack Verby, the director of RPAP, noted that longer term experiences did help them identify problem students as well as problem physicians (impaired, criminal) during the 9 month RPAP preceptorship in rural Minnesota in the M-3 year. A more limited 1 month experience might make this difficult, but those candidates who were suspect could be asked to do longer experiences.
There are the obvious drawbacks to more extensive evaluations such as performance on projects. The evaluation needs to be objective. Local service or community-based projects where the student (or family) knew the evaluator/supervisor would be suspect. There needs to be some structure to the experiences so that there is consistency for selections.
There are also risks. Medical school projects would be subject to potential abuse, but no more so than the current graduate student environment. The cost would be a concern, even if only for supervision and paperwork and liability. Also if all medical schools did not use this method of assessment, the method itself would be a major impediment for the ones that did.
There are some strong arguments for increased evaluations:
Think about our actions for the past decades: Why do we force those who are often most devoted to the ideals that we hold most dear in medicine, to delay admission, go overseas, or take extra expense and even years of remediation when we allow facilitated or early admission to those who are intellectually qualified, but otherwise unknown.
Think about the increased security demands of the 911 era: Why should we allow students unlimited access to be able to wreck havoc through knowledge gone amuck (bioterrorism), thru unbridled greed (billing excess, fraud, or abuse), or pleasing patients or society regardless of ethics (euthanasia, cloning, etc.). We should probably choose our medical students (and certain graduate students) with at least the some of the precautions and security clearances of military academies.
Would pre-professional service requirements delay admission, possibly but some are just not ready for medicine at 21 or 22 years of age. Would this make admission more costly or difficult for those who could least afford it, possibly, depending on how the experience is structured. Would this make service experiences more distasteful in the eyes of the students, possibly, but also it might turn some students on to service experiences, depending on the experience and the evaluator, supervisor, potential mentor.
Remember that for certain patients, as a doctor all I have to do
To increase the probability of someone dying
1. is make an excuse and leave the room early, instead of engaging in some difficult interactions.
2. is to forget to check on a drug interaction
3. is to ignore a phone call
4. is to assume that I knew what this new drug would do to my patient
5. is to fail to call the authorities (even though they will ignore my words) in a patient at risk
6. is to rely on someone that I know is less dedicated or competent to interact with my patient
All I have to do to make someone's life a bit better
1. is to listen
2. is to recognize someone's discomfort in a personal area
3. is to make an extra phone call or otherwise ask someone else for help
Intellectual prowess will not help me in these areas. These areas will help me - service ethic, versatility, self-evaluation, and communication skills. Most important is perhaps my past interactions with patients where I was actively engaged in understanding them and their needs, learning about their problems, learning what I could and could not do, and following through. Applicants will not be able to do much with patients, but they can work with people and projects. These interactions can tell much about the candidate and what kind of physician they might become.
Robert Bowman
Character, Color, Admissions, and Physicians
Testing Fails to Predict Performance or Future Location
Personal Growth and Medical Education
Side Effects of Selecting for Family Medicine
Medicine, Education, and Social Status