Evaluation Reporting Template for Interactive Health Communication Applications,

Version 1.0, Science Panel on Interactive Communication and Health

 
This is an evaluation reporting template for developers and evaluators of interactive health communication (IHC) applications to help them report evaluation results to those who are considering purchasing or using their applications. Because the template is designed to be used for all types of applications and evaluations, some items may not apply to a particular application or evaluation. Complete only those items that apply. This and subsequent versions of the template and other resources on evaluation of IHC are available at: URL: www.scipich.org

Sections:

I.    Description of Application

II.   Formative and Process Evaluation

III.  Outcome Evaluation

IV.  Background of Evaluators


 
I. Description of Application
1.  Title of product/application:
2.  Name(s) of developer(s):
3.  Relevant qualifications of developer(s):
4.  Contact(s) for additional information:
5.  Funding sources for development of the application (e.g., commercial company, government, foundation/nonprofit organization, individual):
6.  Category of application (e.g., clinical decision support, individual behavior change, peer support, risk assessment):
7.  Specific goal(s)/objective(s) of the application (What is the application intended to do? List all, as applicable):
8.  Intended target audience(s) for the application (e.g., age group, gender, educational level, types of organizations and settings, disease groups, cultural/ethnic/population groups):
9.  Available in languages other than English? Yes  No (specify):
10.  Technological/resource requirements of the application (e.g., hardware, Internet, on-site support available):
11.  Describe how confidentiality or anonymity of users is protected:
12. Indicate who will potentially be able to get information about users:
 
 
II. Formative and Process Evaluation*
1.  Indicate the processes and information source(s) used to ensure the validity of the content (e.g., peer-reviewed scientific literature, in-house "experts," recognized outside "experts," consensus panel of independent "experts," updating and review processes and timing):
2.  Are the specific original sources of information cited within the application? Yes  No
3.  Describe the methods of instruction and/or communication used (e.g., drill and practice; modeling; simulations; reading generic online documents, interactive presentations of tailored information, specifying methods used):
4.  Describe the media formats used (e.g., text, voice/sound, still graphics, animation/video, color):
5.  For each applicable evaluation question below indicate (i) the characteristics of the sample(s) used and how they were selected, (ii) the method(s) of assessment (e.g., specific measures used), and (iii) the evaluation results:
6.  If text or voice is used, how was the reading level or understandability tested?
7.  What is the extent of expected use of the application (e.g., average length and range of time, number of repeat uses)?
8.  How long will it take to train a beginning user to use the application proficiently?
9.  Describe how the application was beta tested and debugged (e.g., by what users, in what settings):
 
 
III. Outcome Evaluation?
1.  For each applicable evaluation question below, indicate (i) the type of evaluation design (I-III),? (ii) the characteristics of the sample(s) used and how they were selected, (iii) the method(s) of assessment (e.g., specific measures used), and 
(iv) the evaluation results:
2.  How much do users like the application?
3.  How helpful/useful do users find the application?
4.  Do users increase their knowledge?
5.  Do users change their beliefs or attitudes (e.g., self-efficacy, perceived importance, intentions to change behavior, satisfaction)?
6.  Do users change their behaviors (e.g., risk factor behaviors, interpersonal interactions, compliance, utilization of resources)?
7.  Are there changes in morbidity or mortality (e.g., symptoms, missed days of school/work, physiologic indicators)?
8.  Are there effects on costs/resource utilization (e.g., cost-effectiveness analysis)?
9.  Do organizations or systems change (e.g., resource utilization, effects on "culture")?
 
 
IV. Background of Evaluators
1.  Names and contact information for evaluator(s):
2.  Do any of the evaluators have a financial interest in the sale/dissemination of the application? Yes  No 
 (specify):
3.  Funding sources for the evaluation(s) of the application (e.g., developer's funds, other commercial company, government, foundation/nonprofit organization):
4.  Is a copy of the evaluation report(s) available for review on request? Yes  No (how to obtain):
 
Comments and suggestions regarding the content, scope, utility, and practicality of this template should be directed to: SciPICH, Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services, 200 Independence Ave., SW, Washington DC 20201 or e-mail comments to: scipich@health.org

* Formative evaluation is used to assess the nature of the problem and the needs of the target audience with a focus on informing and improving program design before implementation. This is conducted prior to or during early application development, and commonly consists of literature reviews and reviews of existing applications and interviews or focus groups of "experts" or members of the target audience. Process evaluation is used to monitor the administrative, organizational, or other operational characteristics of an intervention. This helps developers successfully translate the design into a functional application and is performed during application development. This commonly includes testing the application for functionality and also may be known as alpha and beta testing.

? Outcome evaluation is used to examine an intervention's ability to achieve its intended results under ideal conditions (i.e., efficacy) or under real world circumstances (i.e., effectiveness), and also its ability to produce benefits in relation to its costs (i.e., efficiency or cost-effectiveness). This helps developers learn whether the application is successful at achieving its goals and objectives, and is performed after the implementation of the application.

? Design types are grouped according to level of quality of evidence as classified by the U.S. Preventive Services Task Force and the Canadian Task Force on the Periodic Health Exam. (U.S. Preventive Services Task Force. Guide to clinical preventive services. 2nd ed. Washington, DC: US Department of Health and Human Services, 1996.)
I. Randomized controlled trials. Experiments in which potential users are randomly assigned to use the application or to a control group. Randomization promotes comparability between groups. These designs can be (a) double-blinded-neither the participants nor the evaluators know which participants are in the intervention group or the control group, (b) single-blinded-the participants are not aware which experimental group they are in, or (c) non-blinded-both the participants and the evaluators are aware of who is in the intervention group and who is in the control group. Greater blinding lessens the chance of bias.
II-1. Non-randomized controlled trials. Experiments comparing users and non-users (or "controls") who are not randomly assigned to these groups. For this type of design, specify how the participants were recruited, selected, and assigned to the groups and how the groups compare - similarities and differences between users and non-users prior to the evaluation.
II-2. Cohort study/observational study. An evaluation of users with no comparison or control group.
II-3. Multiple time series. Observations of participants as they go through periods of use and non-use of the application.
III. Descriptive studies, case reports, testimonials, "expert" committee opinions.

Source: Wired for Health and Well-Being.

Return to December 1999 Table of Contents

Return to Table of Contents

Return to Committee Reports