CSS Menu - Horizontal
Home
Geriatric Minifellowship
LTC Dr. Magnuson Dementia Series
NEBGEC Conferences
NEBGEC Program Evaluation
Online Educational Modules
Related Educational Opportunities

Evaluation Framework for Nebraska Geriatric Education Center

The mission of the Nebraska Geriatric Education Center (NEBGEC) is to improve health care and health care outcomes of Nebraska’s vulnerable elderly population who reside in rural areas and nursing homes through enhanced training of health professionals and faculty in geriatrics and geron­tology. We will accomplish this mission through five main objectives:

Objective 1.  Increase the effectiveness of geriatrics health care through training for physicians, physician faculty, other clinical preceptors and health professionals in rural and underserved urban areas using an 80-hour interdisciplinary mini-fellowship with more than 105 completers by year 3.

Objective 2.  Improve the interest of students from medicine, pharmacy, nursing, physician assistants, physical therapy, and social work in geriatric health careers through exposure to career opportunities, positive experiences with older people and interdisciplinary training. 

Objective 3.   Improve the health outcomes of older patients in rural areas by providing rural practitioners, preceptors and trainees with geriatric interdisciplinary team training emphasizing evaluation of complex patients, especially those with dementia, mental health concerns and failing functional abilities.

Objective 4.  Improve the quality of care and patient outcomes for residents of nursing homes through interdisciplinary training of nursing home providers.

Objective 5. Improve the diversity and cultural competence of the health professions workforce.  
We will use three frameworks to guide our evaluation activities. These frameworks are Donabedian’s original framework to
evaluate (infer the quality of) an intervention by assessing structure, process, and outcomes[1]; the Logic Model, which refers to
program resources, activities, and outcomes[2]; and an augmented version of Kirkpatrick’s training criteria taxonomies.[3] The
outcome of interest in an educational intervention is the societal impact of the change in the behavior of those educated.[4]
However, as Donabedian explained, it is difficult to attribute a societal impact to the outcomes of a training intervention when there
are multiple intervening factors affecting the outcome, which evolves over a period of time. Consequently, we must assess
intermediate outcomes within the structure and process of educational interventions and infer the effect on society based on
scientific evidence. In decreasing order of distance from societal impact, these intermediate outcomes can be organized according
to an expanded version of Kirkpatrick’s training criteria.

 

Training Criteria Definition
Reactions to Training Learner assessments of satisfaction with the content and mode of training
     Affective reactions Reactions as affect, learners are considered “customers”
     Utility judgments Reactions as utility judgments to determine learners’ perceptions of the training as useful for future job performance
   
Learning Knowledge assessment
     Immediate knowledge Immediate post-training assessment of knowledge (e.g., with a multiple choice test that reflects training objectives)
     Knowledge retention Post-training assessment of knowledge at a specified period of time after training such as 3 – 6 months post-training
     Behavior/skill demonstration Demonstration of behavior change or new skills within the training as in a simulation
   
Behavior /Transfer Retention and transfer of new behavior or new skills to the workplace including changing the process of health care
      
Results Demonstration of organizational or societal impact due to the new behaviors and skills in the workplace. This impact may be change in an individual’s health status due to a change in the process of care and change in population health due to institutional changes in structures and processes of care.                                                                                                                                

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

In additions to reactions to training, we will collect data regarding the number of learners in an activity and their characteristics. 

Based on the above framework, we will create standardized evaluation templates to assess training criteria at appropriate levels for each objective. Items that assess affective reactions and utility judgments will follow the same format for each activity and will build on existing templates used in UNMC Continuing Education. Assessments of immediate knowledge retention at three months post-training will be customized to each objective and educational session. We will assess program completers' transfer of behavior to the work place and attitudes toward functioning in interdisciplinary teams qualitatively and quantitatively in objectives 1, 3, and 4.    

To evaluate Objective 1 (interdisciplinary mini-fellowship), we will conduct qualitative interviews with up to four mini-fellowship completers from each discipline (medicine, nursing, social work, and physical therapy). The interviews will elicit rich information from completers within the first three levels of the training criteria—reactions, learning, and behavior/transfer.    To evaluate Objective 3 (rural geriatric interdisciplinary team training), we will assist the providers at a Critical Access Hospital in rural Nebraska to develop a Geriatric Assessment Clinic (GAC) in Year One of the grant.  

 

To evaluate the effectiveness of the clinic, we will develop a Geriatric Chart Audit tool to assess the extent to which training results in the use of evidence-based processes of geriatric assessment within the Pender GAC. This evaluation will determine the extent of behavior change and transfer of new skills from training to the work environment of the newly developed GAC. In addition, we will evaluate the sustainability of this rural GAC by recruiting one additional Nebraska Critical Access Hospital in Year Two of the grant to implement a GAC. 

 

To evaluate Objective 4 (conducting interdisciplinary training of nursing home personnel), we will conduct an assessment of quarterly Minimum Data Set (MDS) data in a time-series design to control for the effect of background changes in structure and process not attributable to our intervention. This assessment will determine the extent of the impact of the training on the organizational effectiveness of the nursing homes as measured by the MDS.  

[1] Donabedian, A. (1966). Evaluating the quality of medical care. The Millbank Memorial Fund Quarterly, 44, 166-206.
[2] McLaughlin, J.A. & Jordan, G.B. (1999). Logic models: a tool for telling your program’s performance story.   Evaluation and Program Planning,  22, 65-72.
[3] Alliger, G.M., Tannenbaum, S.I., Bennett, W. et al. (1997). Personnel Psychology, 50, 2, 341-358.
[4] Hutchinson, L. (1999). Evaluating and researching the effectiveness of educational interventions. BMJ, 318, 1267-1269.