METHODS

Presentations and case studies were used to construct a detailed description and analysis for each agency's evaluation activities and outcomes. The case study method proved to be a useful analytic tool for modeling some essential principles and practices of evaluation effectiveness. Each of the five case studies presented was organized into three major sections:

  1. The Background section describes the problem; who was at risk; what health risk communication strategy was used to address the risks; and how the success of the health risk communication effort was defined and measured;
  2. The Evaluation Approach section includes what types of evaluation methods were used, how they were applied, and cost estimates in time and dollars; and
  3. The Lessons Learned section examines lessons learned from the evaluation effort and how those lessons were incorporated into subsequent communication activities.

The evaluation attributes and practices for each PHS agency that submitted a case study are listed in Table 1.

TABLE 1.

EVALUATION MATRIX FOR FEDERAL HEALTH RISK COMMUNICATION ACTIVITIES

Agency Public Health Issue Type of Evaluation Evaluation Strategy
NSC Childhood lead poisoning Formative
Process
Outcome 		Evaluation approach was modeled after the National Academy of Sciences (NAS) recommendations for CDC's AIDS Education Project:
  1. Formative ­ concept testing of messages and copy testing of public service announcements (PSAs).
  2. Process ­ assess the precise time of PSA airplay, the program currently scheduled, the media market, and the station call letters and affiliation. Timing of air play and frequency of calls are closely linked.
  3. Outcome ­ behavior change from advertisements that prompted people to call the 800 number in order to receive information.
NCI Mammography screening guidelines Formative Focus groups were used to assess the likely impact of a change in the NCI mammography screening guidelines on physician decision-making with regard to mammogram referrals. Focus group participants reacted to sample print materials designed to support the introduction of the revised guidelines.
FDA Risk of heart valve
fracture in patients 		Formative Manufacturer evaluation methods:
  1. Pretesting of a letter by cardiologists, risk communication experts, individuals with heart problems, and those having a Shiley heart valve.
  2. A telephone­administered questionnaire directed to a representative sample of patients and physicians.

FDA evaluation method: A telephone or in­person administered questionnaire directed to a sample of patients and physicians.

ATSDR Community health assessment workshops Formative
  1. 3­page survey distributed to workshop participants;
  2. Informal assessment through one-on-one and small group interactions; and
  3. Community recommendations to promote understanding, ongoing dialogue, and information sharing.
EPA Farm workers' exposure
to pesticides 		Formative Evaluation of education materials designed to teach farm workers about pesticide safety:
  1. Hands-on training for EPA staff in conducting focus group discussions, pretesting, and material revision.
  2. Three focus group discussions with farm workers on pesticide safety.
  3. 52 individual interviews with farm workers.

NSC ­ National Safety Council
NCI ­ National Cancer Institute
EPA ­ Environmental Protection Agency
FDA ­ Food and Drug Administration
ATSDR ­ Agency for Toxic Substances and Disease Registry

Return to Table of Contents

Return to Committee Reports Page