Expanding Mental And Dental
Health Services
within School-Based Health Centers:
Operations and Evaluation Challenges
Summary
The Caring for Kids initiative funded by The Robert Wood Johnson Foundation
is supporting the development of several models for expanding mental
and dental health services within school-based health centers. A primary
objective of the grant program is to describe the process by which these
models are put in place and assess their strengths and weaknesses. This
edited transcript describes the operations and evaluation challenges
that the grantees have experienced or anticipate and summarizes remarks
by school-based health center research and evaluation expert, Dr. Linda
Juszczak.
Grantees developing mental health initiatives report that their greatest
operational challenges include: securing and sustaining support for
organizational change, blending mental health disciplines, and defining
the relationship between the health center's mental health providers
and the school's academic program. Program evaluation issues include
defining who is a mental health provider and what is a
mental health encounter and determining if a program evaluation can
measure whether the new service improves students' mental health status.
Summary tables are found on pages 10 and 16.
Grantees developing dental health initiatives report their major challenges
as: integrating children's dental issues into the medical staff's physical
assessments, overcoming community and parent disinterest in the importance
of dental health, and connecting dental services with disease prevention/health
promotion initiatives in nutrition, tobacco use, and injury prevention.
Evaluation questions to be addressed include: Who do we serve? Do we
improve access? Are we improving community or parent understanding that
sealants and dental care generally are important?
Introduction
In June 2002, the 15 grantees of The Robert Wood Johnson Foundation
Caring for Kids (CFK) Initiative met in Denver, Colorado to discuss
the challenges and opportunities they are confronting in launching new
mental health and dental health programs within school-based health
centers.
While some of the challenges were similar for both groups, how to collaborate
with colleagues from different disciplines, for example, others were
unique. While mental health programs wrestle with definitional issues
in the design of data-gathering systems, dental programs struggle to
decide whether they need two consent forms or one to provide preventive
and restorative services. Dental programs also felt uniquely challenged
in engaging parents in caring for their children's dental needs.
This discussion reflects grantees' experiences from their first six
months of participation in Caring for Kids.
Project Evaluation: An Overview
Linda Juszczak, DNSc, MPH, CPNP, Director of the National Assembly's
Center for Evaluation and Quality at Montefiore Medical Center, led
off discussions of research, evaluation, and quality improvement and
with an overview of basic issues. The following comments summarize Dr.
Juszczak's remarks on program evaluation.
The goal of most evaluations is to provide "useful feedback" to a variety
of audiences including sponsors, donors, client groups, administrators,
staff and other relevant constituencies. Since their earliest days,
school-based health centers have attempted to document their effectiveness.
Project evaluation efforts by the Caring for Kids grantees will be particularly
important because relatively few published evaluation studies have focused
on mental health services organized by school-based health centers and
even fewer have evaluated the impacts of dental programs organized by
the centers.
To assist in planning an evaluation of CFK-funded projects, Dr. Juszczak
emphasized evaluation basics to be kept in mind:
- The difference between what we might want to measure and what
we can measure. In most evaluation work associated with school-based
health centers, evaluation thinking has run something as follows:
What
we want to measure:
- Wellness
- Holistic health
- Lives saved
- Life success
- Cost efficacy |
What
we can measure:
- Enrollment/visits
- Who uses services, how much they use and if they use more or less
than other sites
- Bad behaviors
- Success in advancing a grade
- Effects of short-term interventions on small groups of students |
To get the evaluation job done, we sometimes have to settle for the
do-able as opposed to the most fundamental questions that we would like
to wrestle with.
- The importance of defining project goals that are to be evaluated.
There are multiple possibilities: improving access to certain services;
achieving certain outcomes; or demonstrating an improvement in quality
of care. Be clear about the questions you will answer in your evaluation.
- After the evaluation question or questions are selected, the
study design, including the data elements required and the methods
to be used to collect the data, can be developed.
- Other important tasks include: assessing the resources available
to the project - money, time, and expertise, and establishing a timeline,
and determining how the study's findings will be used to inform both
the school-based health center's practice and the policies that impact
its programs.
Data sources commonly include: management information systems, standardized
data collection instruments, surveys, interviews, and focus groups.
Examples of evaluation indicators often associated with school-based
health centers are the following:
Percent of students screened for: general health, reproductive health,
nutrition problems, violence and injury risk, pregnancy and STI risk
factors, sexual abuse, mental health or behavioral health problems,
school performance, family dysfunction, and dental health.
Percent of students receiving; general physical exams; dental exams/sealants,
restorations; immunizations; STI testing; annual pap smears and pelvic
exams, and mental health services.
There are many challenges associated with the definitional work that
needs to be done before evaluation studies can be put in place. What
constitutes a service? Who is a mental health provider? What are the
definitions for assessment, evaluation or treatment? Are you going to
track all services or just look at a particular type - individual, group,
family? Do you want to look at service coordination? If you are looking
at outcomes, how do you approach the dose/response issue, ie. how much
of what for how long?
There may be specific mental health outcomes you want to examine. These
could include:
- High parent satisfaction
- Students with behavioral issues can demonstrate self-care skills
- Patient perception that well-being has improved
- Increased compliance rates as measured by follow-up visits completed,
prescriptions filled, therapy attended, referrals completed, etc.
Dr. Juszczak concluded with two main recommendations for developing
successful evaluation studies. First, to be successful, staff must buy-into
the notion that the school-based program is accountable to its patients.
their families, funders, and stakeholders for progress on stated program
goals. The evaluation can serve as a vehicle for reporting back to these
critical audiences. Second, successful evaluations involve staff in
their development as well as implementation. Staff buy-in, essential
to data collection on which evaluation depends, requires not only staff
support but staff insights and contribution to evaluation design.
While evaluation research conducted within school-based health centers
has particular advantages -- access to a convenient population, a well-defined
intervention, the availability of related data from the school system
and perhaps other sources -- there are substantial challenges as well.
These include those listed below.
|
Challenges
in Conducting SBHC Evaluations
Implementation
issues
- Difficulties
in attaining pre-intervention data
- Tremendous
variability in the SBHC model
- Political/administrative
hurdles to implementing evaluation
- Researcher/evaluator
bias
- Written
parental consent
Research
design challenges
- Small sample
sizes
- Difficulty
getting appropriate comparison groups
- Self-selection
in enrollment in SBHC and use of SBHC
- Prevalence
of problems being studied, eg. at-risk of asthma vs. at-risk
of suicide
- Shifting
denominators due to transfers, dropouts, "stop"outs
- Shifting
numerators due to fluctuating daily attendance
Juszczak,
June 2002
|
Other challenges may impede the evaluation. While health centers generally
try to identify if students are receiving services in the community,
not all centers have good data on the role of community-based providers.
Similarly while staff will likely be aware of major developments within
the school and community that may confound intervention effects, this
is an area of uncertainty. School-based health center staffers may also
find the data demands of the evaluation too burdensome for full compliance.
Time commitments -- especially those caused by additional data collection
-- and technology issues may also impede the evaluation.
Obviously, these are reasons why evaluation plans are not always
converted to evaluation reality. But the hard truth is that funders,
whether public or private, have become increasingly sophisticated in
their understanding of research and what constitutes an acceptable level
of evaluation research. We really have no choice. We will need to find
ways to evaluate what we do with credibility and timeliness.
Next: Mental Health Challenges
