h.jabbari
17th February 2012, 08:15 PM
GAO
United States General Accounting Office
Report to Congressional Committees
February 2001
FOOD ASSISTANCE Performance Measures
for Assessing Three WIC Services
GAO-01-339
Page 1 GAO-01-339 Measures for Three WIC Services
United States General Accounting Office
Washington, D.C. 20548
Letter
February 28, 2001
The Honorable Richard G. Lugar
Chairman
The Honorable Tom Harkin
Ranking Member
Committee on Agriculture, Nutrition, and Forestry
United States Senate
The Honorable John A. Boehner
Chairman
The Honorable George Miller
Ranking Minority Member
Committee on Education and the Workforce
House of Representatives
The Special Supplemental Nutrition Program for Women, Infants and
Children (WIC) is a federally funded $4.1 billion-a-year nutrition assistance
program administered by the U.S. Department of Agriculture’s Food and
Nutrition Service (FNS). During fiscal year 2000, this program provided $3
billion for supplemental foods and $1.1 billion for nutrition services and
administration to assist lower-income pregnant, breastfeeding, and
postpartum women, infants, and children up to age 5 who are at nutritional
risk. FNS provided annual cash grants to support program operations at 88
state-level agencies.1 These 88 agencies administered the program through
more than 1,800 local WIC agencies.
To help the Congress better understand the costs of delivering nutrition
services and administering WIC, the William F. Goodling Child Nutrition
Reauthorization Act of 1998 (P.L. 105-336) directed GAO to assess various
aspects of WIC nutrition services and administration. This report, the third
in a series responding to this request, examines the performance measures
that FNS uses to assess the nutrition education, breastfeeding promotion
1State agencies are located in the 50 states, the District of Columbia, American Samoa, the
Commonwealth of Puerto Rico, Guam, the U.S. Virgin Islands, and 33 Indian tribal
organizations. These agencies typically are public or private nonprofit health or human
services agencies; they can be an Indian Health Service Unit, a tribe, or an intertribal
council.
Page 2 GAO-01-339 Measures for Three WIC Services
and support, and health referral services provided to WIC program
participants.2
We based our evaluation on the performance measurement framework
contained in the Government Performance and Results Act of 1993 (Results
Act), which, among other things, encourages agencies to measure program
performance by determining the extent to which intended program
outcomes have been achieved. Program outcomes are the results of
delivering a program’s products and services, while program outputs are
the products and services delivered. An example of the distinction between
the two concepts can be seen in the case of a job-training program. In such
a program, an output could be the number or percentage of program
participants who completed the training. A program outcome, on the other
hand, could be the number or percentage of program participants
employed 1 year after the training. Outcome measures on which objective
data can be collected at reasonable cost can be difficult to develop,
particularly for programs that are intended to influence the behavior of
individuals. While emphasizing the use of outcome measures, the Results
Act recognized that the output measures traditionally used by agencies for
measuring performance remain critical to program management.
Consistent with the Results Act’s approach to performance measurement,
this report discusses FNS’ use of both outcome and output performance
measures. Specifically, it provides information on how FNS measures (1)
the outcomes of three services provided by the WIC program—nutrition
education, breastfeeding promotion and support, and health referral
services; and (2) program outputs for these three service areas.
Results in Brief FNS has an outcome-based measure for one of the three nutrition
services—breastfeeding promotion and support. However, the measure,
breastfeeding initiation rate, examines only one of several important
aspects of the service’s possible impact on WIC participants. Other key
aspects, for which FNS has not established outcome measures, include the
length of time that WIC mothers breastfeed their infants and
breastfeeding’s contribution to an infant’s overall nutritional needs. Several
2The first report in the series was Food Assistance: Financial Information on WIC Nutrition
Services and Administrative Costs (GAO/RCED-00-66, Mar. 6, 2000). The second report was
Food Assistance: Activities and Use of Nonprogram Resources at Six WIC Agencies
(GAO/RCED-00-202, Sept. 29, 2000).
Page 3 GAO-01-339 Measures for Three WIC Services
obstacles have hindered FNS’ efforts to develop and implement outcomebased
measures for nutrition education and health referral services for the
WIC program. These include difficulties in identifying measures that would
allow the agency to appropriately link a particular service’s activity to a
desired outcome and resource constraints affecting FNS’ ability to collect
data needed to implement a proposed measure.
FNS employs a large number of program output measures to gauge
performance of the three WIC services. Generally, these measures are used
to examine the types and quantities of services that state and local agencies
provide and whether the agencies are in compliance with grant expenditure
and other program requirements. FNS has been hindered in measuring
program compliance because of weaknesses in the management evaluation
process used to collect data on these measures. Among other problems,
inconsistencies in procedures used to conduct the evaluations have
resulted in problems in interpreting evaluation results as well as in
comparing and analyzing results across evaluations. FNS is considering
recommendations its special task force made in a January 2001 report to
strengthen its processes for management evaluations.
While we believe that the weaknesses in FNS management evaluations
should be addressed expeditiously, we are not making specific
recommendations at this time because FNS is actively considering
corrective actions. However, we will continue to monitor the agency’s
actions on these matters and, if warranted, make recommendations in the
future. In commenting on a draft of this report, FNS officials generally
agreed with the information presented, but cautioned that the report
discussed performance measures only as they pertained to three nutrition
service components of the WIC program and not the program in its entirety.
They believed that when viewed from this broader context, the program
has resulted in a number of desirable health outcomes.
Background As part of its administration of the WIC program, FNS makes grants to the
88 state agencies that in turn provide program benefits—supplemental
foods and a variety of nutrition services—to participants through more
than 1,800 local WIC agencies. The state agencies develop guidelines
intended to ensure that local agencies effectively deliver WIC benefits to
eligible participants, and monitor local agencies’ compliance with these
guidelines. Local agencies serve participants directly or through one or
more service delivery sites or clinics located within their service areas.
Staff at local WIC agencies and clinics approve applicants for participation;
Page 4 GAO-01-339 Measures for Three WIC Services
provide food benefits, typically in the form of vouchers that can be
exchanged for WIC-approved supplemental foods at specified grocers;
provide nutrition education; and make health referrals to eligible
individuals.
For fiscal year 2000, grants for this program totaled $4.1 billion—$3 billion
for supplemental foods and $1.1 billion for nutrition services and program
administration costs. Grants for nutrition services and administration
(NSA) are typically used to support activities in four broad categories:
participant services, including health care referrals; nutrition education;
breastfeeding promotion and support; and program administration, such as
planning and budgeting. This report focuses on three specific services—
nutrition education, breastfeeding promotion and support, and health
referrals. Table 1 describes some of the specific activities that local WIC
agencies provide as part of these services.
Table 1: Three Nutrition Services by Activities
FNS Has an Outcome-
Based Measure for One
of Its Three Nutrition
Services
FNS established the breastfeeding initiation rate as an outcome-based
measure for the WIC program’s breastfeeding promotion and support
activities, but has no outcome measures for its nutrition education or
Type of service Description
Nutrition education • Providing individual and/or group nutrition education sessions.
• Preparing or obtaining nutrition education materials, such as brochures and videotapes, and interpreting or
translating materials to facilitate nutrition education of non-English-speaking participants.
• Providing or receiving training regarding nutrition education promotion, and evaluating and monitoring
nutrition education.
• Providing information on the benefits of the supplemental foods provided by WIC and how to exchange WIC
food vouchers for food products at WIC-approved grocers.
Breastfeeding promotion
and support
• Providing individual counseling sessions at WIC clinics or in hospitals and by telephone to promote and
support breastfeeding, and maintaining a clinic environment that encourages breastfeeding.
• Consulting with medical providers regarding breastfeeding issues.
• Providing or receiving training regarding breastfeeding promotion and monitoring and evaluating
breastfeeding promotion activities.
Health referrals • Providing each participant with information on available health care providers.
• Referring participants to a health practitioner, when appropriate, such as for immunizations, and to social
services such as the Medicaid program.
Page 5 GAO-01-339 Measures for Three WIC Services
health referral services.3 While this single measure allows FNS to examine
one aspect of the impact of its services on WIC clients, it does not measure
ther important aspects of this service’s impact—such as the length of time
that WIC mothers breastfeed their infants and the percentage of daily
nutrition an infant obtains from breastfeeding. FNS has attempted to
develop outcome-based measures for the WIC program’s nutrition
education and health referral services but has not yet been successful in
implementing these measures. FNS has identified a measure for nutrition
education but has not implemented it because of resource constraints. For
health referrals, FNS has been unable to identify a measure that, among
other things, would permit it to appropriately link the service’s activities
with a desired outcome.
Breastfeeding Initiation
Rate Measures Outcomes of
Breastfeeding Promotion
and Support Activities
FNS established the breastfeeding initiation rate as its outcome-based
measure for breastfeeding promotion and support activities as part of its
fiscal year 2000 performance plan.4 The measure represents the percentage
of WIC infants between 7 and 11 months of age who, at some time, have
been breastfed. FNS first reported the results of this measure in May 2000
as part of its 1998 WIC Participant and Program Characteristics Study
(Characteristics Study), a biennial report on the characteristics of WIC
participants and program activities. The overall rate of 42 percent for 1998
will serve as a baseline against which FNS will gauge future years’
progress.5 FNS has collected data for fiscal year 2000, which it is in the
process of analyzing and expects to be available in the spring of 2002.
Although FNS established an outcome measure for breastfeeding initiation,
this measure does not assess the full range of potential impacts of
breastfeeding promotion and support activities on WIC participants. The
breastfeeding initiation rate includes only WIC infants who are breastfed at
3While FNS has no outcome measure specific to the WIC program’s nutrition education
service, it has an outcome-focused diet quality measure to monitor progress in improving
the diet quality for demographic groups generally eligible for participation in FNS’
programs. According to agency officials, data from this measure will be used in the future to
target the Agency’s nutrition education efforts through programs such as WIC.
4Performance plans, developed pursuant to the requirements of the Results Act, assess
progress toward the goals and objectives of an agency’s strategic plan. FNS’ performance
plan reflects the agency’s priorities for the fiscal year covered by the plan.
5This percentage is based on data from 63 of 88 state agencies, which covered about 80
percent of all WIC infants.
Page 6 GAO-01-339 Measures for Three WIC Services
some time. It does not address other aspects of breastfeeding, such as
duration or the percentage of daily nutrition that an infant obtains through
breastfeeding. FNS attempted to collect data on breastfeeding duration as
part of its 1998 Characteristics Study. However, only 40 of the 88 state
agencies were able to provide this information for at least 75 percent of
WIC infants in their states. According to FNS, these state agencies were
unable to provide the data because their automated information systems
did not contain complete data on each participant. FNS anticipates the
number of state agencies reporting sufficient data to increase as states
become accustomed to reporting these data. FNS has not attempted to
measure the percentage of daily nutrition an infant obtains through
breastfeeding. While acknowledging the importance of this measure, FNS
has not yet identified an appropriate means of implementing it.
FNS’ Efforts to Implement
Outcome-Based Measures
for Nutrition Education and
Health Referrals Have Been
Unsuccessful
FNS established outcome-based measures for both the nutrition education
and health referral services in its fiscal year 2000 performance plan.
However, it was unable to implement these measures and did not include
outcome-based measures for the two services in its 2001 performance plan.
FNS has identified an outcome measure for WIC nutrition education
services, but has not implemented it primarily because of resource
constraints. FNS intended to evaluate the impact of its nutrition education
services by assessing the nutritional quality of meals consumed by a sample
of WIC participants. FNS planned to implement this measure by extracting
data from a national dietary survey of the U.S. population. This periodic
survey, the Continuing Survey of Food Intakes of Individuals, includes WIC
participants. However, FNS determined that the survey did not contain a
large enough sample of WIC participants to establish national estimates.
FNS lacked the resources needed to develop another sample with a large
enough number of WIC participants to establish national estimates. FNS
officials said that the agency is exploring ways to overcome the sample size
problem for future implementation of this measure.
In addition, FNS has not established a measure for WIC health referral
activities that appropriately links the service’s activities with a desired
outcome. Initially, FNS intended to use the percentage of WIC child
participants who were immunized as the outcome measure for its health
referral services. To implement this measure, FNS planned to use data
gathered by the federal Centers for Disease Control and Prevention (CDC)
from its National Immunization Survey. In its 1999 Survey, the CDC began
reporting immunization data for children participating or not participating
Page 7 GAO-01-339 Measures for Three WIC Services
in WIC. However, with its 2001 performance plan, FNS discontinued this
measure. Agency officials gave two reasons for this decision. First, they
noted that FNS is not responsible for ensuring that children participating in
WIC are actually immunized, but rather those children needing
immunizations are referred to doctors as appropriate. Second, they pointed
out that immunization data provided by CDC on WIC participants did not
permit FNS to measure the extent to which WIC health referrals actually
contributed to the immunization of WIC child participants.
FNS officials noted that the problems they faced in finding a measure that
appropriately linked WIC health referral activities to a desired outcome are
not unique to the health referral services. They pointed out that it is
difficult to measure the impact of any of the individual services WIC offers
because, among other things, other variables also influence the behavior
these services are intended to promote. For example, there are several
other state and local programs that, like WIC, are aimed at improving
health through nutrition education. Separating the effects of these efforts
from those of the WIC program is difficult at best.
FNS Relies on a Variety
of Output Measures to
Monitor Program
Performance
FNS employs a large number of program output measures to gauge
performance of the WIC nutritional services it offers. Generally, these
measures are used to examine the types and quantities of services the state
agencies provide and whether the agencies are in compliance with grant
expenditure and other program requirements. (A detailed list of output
measures can be found in app. I.) To monitor these measures, FNS depends
in large part on data provided by state and local agencies through routine
financial reports, its biennial Characteristics Study, and FNS’ management
evaluations of state and local agency operations. Of these monitoring
methods, FNS relies heavily on its management evaluations to measure
state and local WIC agency compliance with program requirements.
However, weaknesses in the procedures for conducting evaluations of WIC
agencies reduce its value as a performance oversight tool. FNS recently
completed an examination of its practices for conducting management
evaluations for all FNS programs, including WIC, and made several
recommendations for improvements.
FNS uses several financial reports submitted by state agencies to measure
the extent to which state and local agencies meet expenditure
requirements. The principal measure is whether states are adhering to a
legislatively established requirement for spending on nutrition education
activities and breastfeeding promotion and support activities. Combined
Page 8 GAO-01-339 Measures for Three WIC Services
spending for these two activities must equal at least one-sixth of the NSA
expenditures plus a target amount for breastfeeding promotion and
support established by FNS at the beginning of each fiscal year. The
Addendum to the WIC Program Annual Closeout Report, prepared by each
state, serves as the primary source of data on the state’s compliance with
this expenditure requirement. For fiscal year 1999, the most recent year for
which complete data were available, FNS reported that two states and four
Indian tribal organizations did not meet this minimum spending
requirement. In addition, the WIC Program Annual Closeout Report each
state prepares provides data on the final status of the state’s grant and costs
for the report year.
To measure the nature and amount of program services WIC agencies
provide, FNS primarily uses data obtained from state and local agencies
and reported in FNS’ Characteristics Study.6 As noted earlier, the
Characteristics Study provides information on a variety of program
activities. Table 2 lists some of the key program activities in the three areas
of service included in the Characteristics Study as well as information on
the percentage of local WIC agencies offering the activity.
Table 2: Selected Program Activities Used by Local Agencies
6FNS also utilizes special reports to measure the nature and amount of program services
provided to participants. For example, FNS uses such reports to evaluate the success of
state adoption of FNS’ national breastfeeding promotion campaign, Loving Support.
Program activitya
Percentage of local
agenciesb
Nutrition education
Nutrition education sessions for pregnant participants provided
during
Certification
Food benefit issuance
Dedicated nutrition education sessions
Other health care appointments
96.8
65.8
44.7
20.2
Nutrition education offered in a language other than English
Spanish
French
Vietnamese
Laotian
Various Native American languages
Hmong
50.0
5.7
5.3
4.6
3.9
3.8
Page 9 GAO-01-339 Measures for Three WIC Services
aA total of 460 local WIC agencies responded to the survey. Responses were weighted to reflect the
universe of 2,203 local agencies, the estimated universe of 8,932 service delivery sites, and the
universe of 8,042,758 enrollees.
bFigures may not sum to 100 percent because respondents could provide multiple responses.
cResponses for these services are by percentage of WIC service delivery sites providing the service.
According to FNS officials, survey data from local agencies, of the sort seen
in table 2, may no longer be collected for the Characteristics Study for 2002
and beyond because research funds are unavailable for that purpose. These
officials explained that the Agriculture, Rural Development, Food and Drug
Administration, and Related Agencies Appropriations Act of 1998 (P.L. 105-
86) effectively transferred primary responsibility for research on food
assistance and nutrition programs from FNS to the Department’s Economic
Research Service, which has not subsequently funded the collection of the
local WIC agency data. FNS officials believe that these data are important
Nutrition education topics especially for pregnant women
Importance of folic acid
Using WIC foods for a healthy diet
Strategies to prevent or manage overweight
81.0
82.9
47.6
Methods or materials always used in providing nutrition education
Counseling/discussion
Food models
Videos, films, slides
Testing participant’s knowledge
90.7
25.5
7.4
4.1
Average length of group nutrition education sessions for non highrisk
participants, in minutes
Less than 10
10-19
20-29
30-39
40-49
50 and longer
18.1
28.8
21.2
5.1
2.1
3.1
Breastfeeding promotion and support
Group education sessions devoted solely to breastfeeding
Breastfeeding support groups
Provision of breast pumps
Individual counseling
Home/hospital visit
Peer counseling
97.4
80.7
43.6
31.4
30.9
16.7
Health referralsc
Type of referral or service provided
Well-baby care and immunizations, referral for all participants
Well-baby care and immunizations, referral as needed
Immunizations only, referral for all participants
Immunizations only, referral as needed
22.6
60.6
10.6
48.1
(Continued From Previous Page)
Program activitya
Percentage of local
agenciesb
Page 10 GAO-01-339 Measures for Three WIC Services
for monitoring the nature and level of services being delivered by local
agencies. To address this and other research concerns, FNS has attempted
to regain direct responsibility over research funding for food assistance
and nutrition programs.
To measure overall compliance with program requirements, including
those related to nutrition education, breastfeeding promotion and support,
and referral services, FNS regional offices conduct management
evaluations at each of the 88 state WIC agencies and at a sample of local
agencies. In general, each of the state agencies and selected local agencies
are reviewed every 3 years. According to FNS, management evaluations are
the primary means of measuring state and local WIC agency compliance
with program requirements. As part of the management evaluations, staff in
FNS’ regional offices conduct interviews, review planning documents,
observe interactions, and examine participants’ files. Listed below are
some of the types of nutrition education, breastfeeding promotion and
support, and health referral service program requirements that can be
assessed during management evaluations:
• Nutrition education
• State agency WIC plan includes nutrition education goals and action
strategies,
• Local agencies offer each participant a minimum of two nutrition
education contacts during each 6-month certification period and
documents the provision of service or the participant’s refusal or
inability to participate,
• State agency has developed or identified nutrition education
resources and materials for use by local agencies, and
• Nutrition education material provided is easily understood by
participants.
• Breastfeeding promotion and support
• State and local agencies have a designated breastfeeding promotion
and support coordinator who provides technical assistance to local
agencies,
• State agencies have developed standards for local agencies for use in
providing breastfeeding promotion and support, and
• Local agency staff has received training on promotion and
management of breastfeeding promotion and support.
• Health referrals
• Local agency staff refers participants, as appropriate, to private or
public health care services.
Page 11 GAO-01-339 Measures for Three WIC Services
FNS has not made full use of management evaluations to measure program
performance and provide needed oversight. In large part this is due to
inconsistencies in the procedures used to conduct the evaluations, which
have caused problems in interpreting evaluation results as well as in
comparing and analyzing results across evaluations. FNS recently
examined its practices for conducting management evaluations for all FNS
programs, including WIC. Results of this review were highlighted in a
January 2001 report to the Administrator. The report noted that (1)
management evaluations at state and local levels were not being conducted
uniformly across FNS regional offices; (2) standard vocabulary did not
exist for identification of certain concepts; (3) standard definitions were
needed for such key terms as deficiency (a judgment that the agency being
reviewed is not in compliance with program requirements and must take
corrective action); (4) additional staff time needed to be allocated to
testing and training prior to implementation; and (5) a learning, feedback,
and modification period would be needed to support consistent data
collection and the development of reporting methods. The task force
recommended a number of actions aimed at obtaining more consistent
information that, if implemented, could provide FNS with better data for
identifying national trends, quantifying deficiencies, developing policies,
and measuring mission effectiveness. Agency officials told us that a
response from the Administrator is expected by April 2001.
Agency Comments and
Our Evaluation
We provided a draft of this report to USDA’s Food and Nutrition Service for
review and comment. We met with FNS officials, including the Director,
Office of Analysis and Nutrition Evaluation; the Director, Supplemental
Food Program Division; and officials in the Grants Management Division.
These officials generally agreed with the information presented in this
report. However, they cautioned that the report addressed performance
measures as they pertained to three nutrition service components of the
WIC program and not the program in its entirety, which includes
supplemental foods and other aspects of nutrition services. FNS officials
believed that research on the effects of the WIC program as a whole
indicated that it has contributed to a number of desirable outcomes
including improved diets and infant feeding practices, reduced incidence of
anemia among children, reduced incidence of infant mortality and low
birthweight infants, and reduced postpartum medical expenses. Also, these
officials provided us with a number of technical comments that we
incorporated into the report, as appropriate.
Page 12 GAO-01-339 Measures for Three WIC Services
Scope and
Methodology
In developing information for this report, we spoke with and obtained data
from officials at FNS’ headquarters in Alexandria, Virginia, and the seven
FNS regional offices—Northeast in Boston, Massachusetts; Mid-Atlantic in
Robbinsville, New Jersey; Southeast in Atlanta, Georgia; Midwest in
Chicago, Illinois; Mountain Plains in Denver, Colorado; Southwest in
Dallas, Texas; and, Western in San Francisco, California. We reviewed
legislation, policies, procedures, and practices; reviewed reports issued by
GAO and others related to WIC; and interviewed various officials
responsible for administering the WIC program. We also reviewed FNS’
strategic and annual performance plans and other documents developed as
part of implementing the Results Act. We conducted our work from
September 2000 through January 2001 in accordance with generally
accepted government auditing standards.
We are sending copies of this report to the appropriate congressional
committees; interested members of the Congress; the Honorable Ann M.
Veneman, the Secretary of Agriculture; and other interested parties. We will
also make copies available to others upon request.
If you or your staff have any questions about this report, please contact me
at (202) 512-7215. Key contributors to this report are listed in appendix II.
Robert E. Robertson
Director, Education, Workforce
and Income Security
Page 13 GAO-01-339 Measures for Three WIC Services
Page 14 GAO-01-339 Measures for Three WIC Services
Appendix I
Appendixes Output Measures for Monitoring Program
Requirements and Activities for the Three
Services AppendxiI
Performance measure Description of requirement or activity Source of data
Nutrition education
Amount of Nutrition Services and
Administration (NSA) funds spent for
nutrition education by state agency
Requirement that state agencies spend an
amount not less than the sum of one-sixth the
total amount of its NSA expenditures on nutrition
education and breastfeeding promotion and
support plus a breastfeeding promotion and
support target amount established by FNS at the
beginning of each fiscal year
Financial report
Extent to which participant actually
receives nutrition education
Requirement that each participant will be provided
with at least two nutrition education contacts
during each 6-month certification period
Management evaluation review of
participants’ files
Extent to which the nutrition education is
in individual or group settings
Requirement that nutrition education contact will
be provided to each participant in individual or
group settings
Management evaluation observations of
staff-participant interactions, and review
of participants’ files and documents
Number of state agencies with a nutrition
coordinator
Requirement that each state WIC agency will
designate a nutrition coordinator responsible for
nutrition education
Special reports to the agency and
management evaluation review of
documents
Number of state agencies that have
developed nutrition goals and action plans
Requirement that state agency will include in its
state plan a nutrition plan with education goals
and action plans, including a description of
education methods
Management evaluation review of state
agency’s State Plan of Operations
Extent to which nutrition education is free,
easy for participant to understand, and
bears a practical relationship to
participant’s needs
Requirement that the participant will be provided
with free nutrition education that is easily
understood and bears a practical relationship to
participant’s needs situation and cultural
preferences
Management evaluation observations of
staff-participant interactions, and review
of participants’ files and documents
Number of local agencies offering nutrition
education in foreign languages
Requirement that, in order to meet the needs of
non-English-speaking participants, WIC agencies
offer nutrition education in languages other than
English, such as Spanish, Laotian, and French
Management evaluation observations
and review of documents; survey of local
agencies through WIC Participant and
Program Characteristics Study
Extent to which nutrition education
emphasizes the relationship between
proper nutrition and good health
Requirement that state agency will provide
nutrition education that emphasizes the
relationship between proper nutrition and good
health
Management evaluation observations of
staff-participant interactions, and review
of participants’ files and documents
Extent to which nutrition education
reinforces positive changes in food habits
that result in participant’s improved
nutritional status
Requirement that state agencies will assist
nutritionally at-risk individuals in achieving positive
changes in food habits, resulting in improved
nutritional status and in prevention of nutritionrelated
problems
Management evaluation observations of
staff-participant interactions, and review
of participants’ files and documents
Extent to which state agency provides the
local agency with nutrition education
resources and materials
Requirement that each state agency will identify
or develop nutrition education resources and
educational materials for use in local agencies
Management evaluation review of state
agency’s State Plan of Operations and
other state documents; review of
resources at local agency
Appendix I
Output Measures for Monitoring Program
Requirements and Activities for the Three
Services
Page 15 GAO-01-339 Measures for Three WIC Services
Number of visits made to FNS’ nutrition
education Web site
Measurement of local agency program activity to
provide nutrition education strategies to WIC staff
and increase staff nutrition education knowledge
through FNS-implemented Web site, WIC Works
Special report
Percent of WIC staff who are Registered
Dieticians (or RD-eligible) or Registered
Dietary Technicians
Measurement of FNS’ recruitment and retention
efforts to increase the number of registered
dieticians, which is critical to the delivery of quality
nutrition services, including nutrition education
Special report (biennial survey by public
health and nutrition program directors)
Breastfeeding promotion and support
Amount of nutrition services and
administration funds spent on
breastfeeding promotion and support
Requirement that state agencies must spend a
target amount of nutrition services and
administration funds on breastfeeding promotion
and support
Financial report
Number of state agencies with a
breastfeeding coordinator
Requirement that the state agency will designate
a breastfeeding promotion and support
coordinator to provide support and technical
assistance to local agencies and to develop state
agency activities
Special report; management evaluation
of state agency’s State Plan of
Operations and other documents
Number of local agencies that have a
breastfeeding coordinator
Requirement that every local WIC agency will
designate a breastfeeding coordinator responsible
for providing promotion and assistance
Special report; management evaluation
of local agency documents
Number of states that implement all or
part of FNS’ National Breastfeeding
Promotion Campaign
Measurement of the success of FNS’ campaign—
Loving Support—designed to promote
breastfeeding in the WIC program
Special report
Extent to which state agency provides the
local agency with breastfeeding education
resources and materials
Requirement that state agency will develop
breastfeeding education resources and materials
and provide technical assistance to local agencies
Management evaluation review of state
agency’s State Plan of Operations and
other documents
Number of state agencies that have
developed standards for local agency use
in providing breastfeeding support
Requirement that the state agency develop
standards for use by local agencies for
breastfeeding support for prenatal and
postpartum women
Management evaluation review of state
agency’s State Plan of Operations and
other documents
Number of local agencies offering
education sessions devoted to
breastfeeding
Measurement of program activity to encourage
pregnant participants to begin and continue
breastfeeding their infants
Survey of local agencies through WIC
Participant and Program Characteristics
Study
Number of local agencies that have staff
that make home or hospital visits to
mothers of newborns
Measurement of local agency activity to
encourage and support breastfeeding by mothers
of newborns through home or hospital visits
Survey of local agencies through WIC
Participant and Program Characteristics
Study
Number of local agencies offering peer
counseling for breastfeeding
Measurement of local agency activity to
encourage and support breastfeeding through
peer support and discussion groups
Survey of local agencies through WIC
Participant and Program Characteristics
Study
Number of local agencies that have
received training on promotion and
management of breastfeeding promotion
and support
Requirement that the state WIC agency develop
and provide to local agencies training on
promotion and management of breastfeeding
promotion and support
Management evaluation review of state
agency’s State Plan of Operations and
other documents
Performance measure Description of requirement or activity Source of data
Appendix I
Output Measures for Monitoring Program
Requirements and Activities for the Three
Services
Page 16 GAO-01-339 Measures for Three WIC Services
Health referrals
Number of participants provided with
information on health care providers
Requirement that WIC staff provide each
participant with information on available health
care providers
Management evaluation review of
participant file data and other documents
Number of participants referred to a
health practitioner when appropriate
Requirement that WIC staff refer participants to a
health practitioner when appropriate, such as
when a pregnant participant has not yet received
prenatal care or a child participant has not
received age-appropriate immunizations
Management evaluation review of
participant file data and other documents
Number of WIC agencies with on-site
pediatric care
Measurement of WIC agencies that are collocated
with pediatric care facilities to assist in meeting, in
part, the requirement that WIC staff help
participants obtain and use preventive health care
services and local agencies serve as a link
between participants and health-care providers
Survey of local agencies through the
WIC Participant and Program
Characteristics Study; management
evaluation of local agency operations
Number of WIC agencies offering wellbaby
care and immunizations
Measurement of WIC agencies that offer wellbaby
care services to assist in meeting, in part,
the requirement that WIC staff assist participants
in obtaining and using preventive health-care
services
Survey of local agencies through the
WIC Participant and Program
Characteristics Study; management
evaluation of local agency operations
Performance measure Description of requirement or activity Source of data
Page 17 GAO-01-339 Measures for Three WIC Services
Appendix II
GAO Contact and Staff Acknowledgments Appendix II
GAO Contact Robert E. Robertson, (202) 512-7215
Staff
Acknowledgments
In addition to the contact named above, Abiud Amaro, Clifford Diehl, Judy
Hoovler, Angela Miles, and Ron Wood made key contributions to this
report.
(150291) Letter
Ordering Information The first copy of each GAO report is free. Additional copies of
reports are $2 each. A check or money order should be made out to
the Superintendent of Documents. VISA and MasterCard credit
cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013
Orders by visiting:
Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC
Orders by phone:
(202) 512-6000
fax: (202) 512-6061
TDD (202) 512-2537
Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.
Orders by Internet:
For information on how to access GAO reports on the Internet,
send an e-mail message with “info” in the body to:
info@www.gao.gov
or visit GAO’s World Wide Web home page at:
http://www.gao.gov (http://www.gao.gov/)
To Report Fraud,
Waste, or Abuse in
Federal Programs
Contact one:
• Web site: http://www.gao.gov/fraudnet/fraudnet.htm
• e-mail: fraudnet@gao.gov
• 1-800-424-5454 (automated answering system)
United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Correction Requested
Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00
United States General Accounting Office
Report to Congressional Committees
February 2001
FOOD ASSISTANCE Performance Measures
for Assessing Three WIC Services
GAO-01-339
Page 1 GAO-01-339 Measures for Three WIC Services
United States General Accounting Office
Washington, D.C. 20548
Letter
February 28, 2001
The Honorable Richard G. Lugar
Chairman
The Honorable Tom Harkin
Ranking Member
Committee on Agriculture, Nutrition, and Forestry
United States Senate
The Honorable John A. Boehner
Chairman
The Honorable George Miller
Ranking Minority Member
Committee on Education and the Workforce
House of Representatives
The Special Supplemental Nutrition Program for Women, Infants and
Children (WIC) is a federally funded $4.1 billion-a-year nutrition assistance
program administered by the U.S. Department of Agriculture’s Food and
Nutrition Service (FNS). During fiscal year 2000, this program provided $3
billion for supplemental foods and $1.1 billion for nutrition services and
administration to assist lower-income pregnant, breastfeeding, and
postpartum women, infants, and children up to age 5 who are at nutritional
risk. FNS provided annual cash grants to support program operations at 88
state-level agencies.1 These 88 agencies administered the program through
more than 1,800 local WIC agencies.
To help the Congress better understand the costs of delivering nutrition
services and administering WIC, the William F. Goodling Child Nutrition
Reauthorization Act of 1998 (P.L. 105-336) directed GAO to assess various
aspects of WIC nutrition services and administration. This report, the third
in a series responding to this request, examines the performance measures
that FNS uses to assess the nutrition education, breastfeeding promotion
1State agencies are located in the 50 states, the District of Columbia, American Samoa, the
Commonwealth of Puerto Rico, Guam, the U.S. Virgin Islands, and 33 Indian tribal
organizations. These agencies typically are public or private nonprofit health or human
services agencies; they can be an Indian Health Service Unit, a tribe, or an intertribal
council.
Page 2 GAO-01-339 Measures for Three WIC Services
and support, and health referral services provided to WIC program
participants.2
We based our evaluation on the performance measurement framework
contained in the Government Performance and Results Act of 1993 (Results
Act), which, among other things, encourages agencies to measure program
performance by determining the extent to which intended program
outcomes have been achieved. Program outcomes are the results of
delivering a program’s products and services, while program outputs are
the products and services delivered. An example of the distinction between
the two concepts can be seen in the case of a job-training program. In such
a program, an output could be the number or percentage of program
participants who completed the training. A program outcome, on the other
hand, could be the number or percentage of program participants
employed 1 year after the training. Outcome measures on which objective
data can be collected at reasonable cost can be difficult to develop,
particularly for programs that are intended to influence the behavior of
individuals. While emphasizing the use of outcome measures, the Results
Act recognized that the output measures traditionally used by agencies for
measuring performance remain critical to program management.
Consistent with the Results Act’s approach to performance measurement,
this report discusses FNS’ use of both outcome and output performance
measures. Specifically, it provides information on how FNS measures (1)
the outcomes of three services provided by the WIC program—nutrition
education, breastfeeding promotion and support, and health referral
services; and (2) program outputs for these three service areas.
Results in Brief FNS has an outcome-based measure for one of the three nutrition
services—breastfeeding promotion and support. However, the measure,
breastfeeding initiation rate, examines only one of several important
aspects of the service’s possible impact on WIC participants. Other key
aspects, for which FNS has not established outcome measures, include the
length of time that WIC mothers breastfeed their infants and
breastfeeding’s contribution to an infant’s overall nutritional needs. Several
2The first report in the series was Food Assistance: Financial Information on WIC Nutrition
Services and Administrative Costs (GAO/RCED-00-66, Mar. 6, 2000). The second report was
Food Assistance: Activities and Use of Nonprogram Resources at Six WIC Agencies
(GAO/RCED-00-202, Sept. 29, 2000).
Page 3 GAO-01-339 Measures for Three WIC Services
obstacles have hindered FNS’ efforts to develop and implement outcomebased
measures for nutrition education and health referral services for the
WIC program. These include difficulties in identifying measures that would
allow the agency to appropriately link a particular service’s activity to a
desired outcome and resource constraints affecting FNS’ ability to collect
data needed to implement a proposed measure.
FNS employs a large number of program output measures to gauge
performance of the three WIC services. Generally, these measures are used
to examine the types and quantities of services that state and local agencies
provide and whether the agencies are in compliance with grant expenditure
and other program requirements. FNS has been hindered in measuring
program compliance because of weaknesses in the management evaluation
process used to collect data on these measures. Among other problems,
inconsistencies in procedures used to conduct the evaluations have
resulted in problems in interpreting evaluation results as well as in
comparing and analyzing results across evaluations. FNS is considering
recommendations its special task force made in a January 2001 report to
strengthen its processes for management evaluations.
While we believe that the weaknesses in FNS management evaluations
should be addressed expeditiously, we are not making specific
recommendations at this time because FNS is actively considering
corrective actions. However, we will continue to monitor the agency’s
actions on these matters and, if warranted, make recommendations in the
future. In commenting on a draft of this report, FNS officials generally
agreed with the information presented, but cautioned that the report
discussed performance measures only as they pertained to three nutrition
service components of the WIC program and not the program in its entirety.
They believed that when viewed from this broader context, the program
has resulted in a number of desirable health outcomes.
Background As part of its administration of the WIC program, FNS makes grants to the
88 state agencies that in turn provide program benefits—supplemental
foods and a variety of nutrition services—to participants through more
than 1,800 local WIC agencies. The state agencies develop guidelines
intended to ensure that local agencies effectively deliver WIC benefits to
eligible participants, and monitor local agencies’ compliance with these
guidelines. Local agencies serve participants directly or through one or
more service delivery sites or clinics located within their service areas.
Staff at local WIC agencies and clinics approve applicants for participation;
Page 4 GAO-01-339 Measures for Three WIC Services
provide food benefits, typically in the form of vouchers that can be
exchanged for WIC-approved supplemental foods at specified grocers;
provide nutrition education; and make health referrals to eligible
individuals.
For fiscal year 2000, grants for this program totaled $4.1 billion—$3 billion
for supplemental foods and $1.1 billion for nutrition services and program
administration costs. Grants for nutrition services and administration
(NSA) are typically used to support activities in four broad categories:
participant services, including health care referrals; nutrition education;
breastfeeding promotion and support; and program administration, such as
planning and budgeting. This report focuses on three specific services—
nutrition education, breastfeeding promotion and support, and health
referrals. Table 1 describes some of the specific activities that local WIC
agencies provide as part of these services.
Table 1: Three Nutrition Services by Activities
FNS Has an Outcome-
Based Measure for One
of Its Three Nutrition
Services
FNS established the breastfeeding initiation rate as an outcome-based
measure for the WIC program’s breastfeeding promotion and support
activities, but has no outcome measures for its nutrition education or
Type of service Description
Nutrition education • Providing individual and/or group nutrition education sessions.
• Preparing or obtaining nutrition education materials, such as brochures and videotapes, and interpreting or
translating materials to facilitate nutrition education of non-English-speaking participants.
• Providing or receiving training regarding nutrition education promotion, and evaluating and monitoring
nutrition education.
• Providing information on the benefits of the supplemental foods provided by WIC and how to exchange WIC
food vouchers for food products at WIC-approved grocers.
Breastfeeding promotion
and support
• Providing individual counseling sessions at WIC clinics or in hospitals and by telephone to promote and
support breastfeeding, and maintaining a clinic environment that encourages breastfeeding.
• Consulting with medical providers regarding breastfeeding issues.
• Providing or receiving training regarding breastfeeding promotion and monitoring and evaluating
breastfeeding promotion activities.
Health referrals • Providing each participant with information on available health care providers.
• Referring participants to a health practitioner, when appropriate, such as for immunizations, and to social
services such as the Medicaid program.
Page 5 GAO-01-339 Measures for Three WIC Services
health referral services.3 While this single measure allows FNS to examine
one aspect of the impact of its services on WIC clients, it does not measure
ther important aspects of this service’s impact—such as the length of time
that WIC mothers breastfeed their infants and the percentage of daily
nutrition an infant obtains from breastfeeding. FNS has attempted to
develop outcome-based measures for the WIC program’s nutrition
education and health referral services but has not yet been successful in
implementing these measures. FNS has identified a measure for nutrition
education but has not implemented it because of resource constraints. For
health referrals, FNS has been unable to identify a measure that, among
other things, would permit it to appropriately link the service’s activities
with a desired outcome.
Breastfeeding Initiation
Rate Measures Outcomes of
Breastfeeding Promotion
and Support Activities
FNS established the breastfeeding initiation rate as its outcome-based
measure for breastfeeding promotion and support activities as part of its
fiscal year 2000 performance plan.4 The measure represents the percentage
of WIC infants between 7 and 11 months of age who, at some time, have
been breastfed. FNS first reported the results of this measure in May 2000
as part of its 1998 WIC Participant and Program Characteristics Study
(Characteristics Study), a biennial report on the characteristics of WIC
participants and program activities. The overall rate of 42 percent for 1998
will serve as a baseline against which FNS will gauge future years’
progress.5 FNS has collected data for fiscal year 2000, which it is in the
process of analyzing and expects to be available in the spring of 2002.
Although FNS established an outcome measure for breastfeeding initiation,
this measure does not assess the full range of potential impacts of
breastfeeding promotion and support activities on WIC participants. The
breastfeeding initiation rate includes only WIC infants who are breastfed at
3While FNS has no outcome measure specific to the WIC program’s nutrition education
service, it has an outcome-focused diet quality measure to monitor progress in improving
the diet quality for demographic groups generally eligible for participation in FNS’
programs. According to agency officials, data from this measure will be used in the future to
target the Agency’s nutrition education efforts through programs such as WIC.
4Performance plans, developed pursuant to the requirements of the Results Act, assess
progress toward the goals and objectives of an agency’s strategic plan. FNS’ performance
plan reflects the agency’s priorities for the fiscal year covered by the plan.
5This percentage is based on data from 63 of 88 state agencies, which covered about 80
percent of all WIC infants.
Page 6 GAO-01-339 Measures for Three WIC Services
some time. It does not address other aspects of breastfeeding, such as
duration or the percentage of daily nutrition that an infant obtains through
breastfeeding. FNS attempted to collect data on breastfeeding duration as
part of its 1998 Characteristics Study. However, only 40 of the 88 state
agencies were able to provide this information for at least 75 percent of
WIC infants in their states. According to FNS, these state agencies were
unable to provide the data because their automated information systems
did not contain complete data on each participant. FNS anticipates the
number of state agencies reporting sufficient data to increase as states
become accustomed to reporting these data. FNS has not attempted to
measure the percentage of daily nutrition an infant obtains through
breastfeeding. While acknowledging the importance of this measure, FNS
has not yet identified an appropriate means of implementing it.
FNS’ Efforts to Implement
Outcome-Based Measures
for Nutrition Education and
Health Referrals Have Been
Unsuccessful
FNS established outcome-based measures for both the nutrition education
and health referral services in its fiscal year 2000 performance plan.
However, it was unable to implement these measures and did not include
outcome-based measures for the two services in its 2001 performance plan.
FNS has identified an outcome measure for WIC nutrition education
services, but has not implemented it primarily because of resource
constraints. FNS intended to evaluate the impact of its nutrition education
services by assessing the nutritional quality of meals consumed by a sample
of WIC participants. FNS planned to implement this measure by extracting
data from a national dietary survey of the U.S. population. This periodic
survey, the Continuing Survey of Food Intakes of Individuals, includes WIC
participants. However, FNS determined that the survey did not contain a
large enough sample of WIC participants to establish national estimates.
FNS lacked the resources needed to develop another sample with a large
enough number of WIC participants to establish national estimates. FNS
officials said that the agency is exploring ways to overcome the sample size
problem for future implementation of this measure.
In addition, FNS has not established a measure for WIC health referral
activities that appropriately links the service’s activities with a desired
outcome. Initially, FNS intended to use the percentage of WIC child
participants who were immunized as the outcome measure for its health
referral services. To implement this measure, FNS planned to use data
gathered by the federal Centers for Disease Control and Prevention (CDC)
from its National Immunization Survey. In its 1999 Survey, the CDC began
reporting immunization data for children participating or not participating
Page 7 GAO-01-339 Measures for Three WIC Services
in WIC. However, with its 2001 performance plan, FNS discontinued this
measure. Agency officials gave two reasons for this decision. First, they
noted that FNS is not responsible for ensuring that children participating in
WIC are actually immunized, but rather those children needing
immunizations are referred to doctors as appropriate. Second, they pointed
out that immunization data provided by CDC on WIC participants did not
permit FNS to measure the extent to which WIC health referrals actually
contributed to the immunization of WIC child participants.
FNS officials noted that the problems they faced in finding a measure that
appropriately linked WIC health referral activities to a desired outcome are
not unique to the health referral services. They pointed out that it is
difficult to measure the impact of any of the individual services WIC offers
because, among other things, other variables also influence the behavior
these services are intended to promote. For example, there are several
other state and local programs that, like WIC, are aimed at improving
health through nutrition education. Separating the effects of these efforts
from those of the WIC program is difficult at best.
FNS Relies on a Variety
of Output Measures to
Monitor Program
Performance
FNS employs a large number of program output measures to gauge
performance of the WIC nutritional services it offers. Generally, these
measures are used to examine the types and quantities of services the state
agencies provide and whether the agencies are in compliance with grant
expenditure and other program requirements. (A detailed list of output
measures can be found in app. I.) To monitor these measures, FNS depends
in large part on data provided by state and local agencies through routine
financial reports, its biennial Characteristics Study, and FNS’ management
evaluations of state and local agency operations. Of these monitoring
methods, FNS relies heavily on its management evaluations to measure
state and local WIC agency compliance with program requirements.
However, weaknesses in the procedures for conducting evaluations of WIC
agencies reduce its value as a performance oversight tool. FNS recently
completed an examination of its practices for conducting management
evaluations for all FNS programs, including WIC, and made several
recommendations for improvements.
FNS uses several financial reports submitted by state agencies to measure
the extent to which state and local agencies meet expenditure
requirements. The principal measure is whether states are adhering to a
legislatively established requirement for spending on nutrition education
activities and breastfeeding promotion and support activities. Combined
Page 8 GAO-01-339 Measures for Three WIC Services
spending for these two activities must equal at least one-sixth of the NSA
expenditures plus a target amount for breastfeeding promotion and
support established by FNS at the beginning of each fiscal year. The
Addendum to the WIC Program Annual Closeout Report, prepared by each
state, serves as the primary source of data on the state’s compliance with
this expenditure requirement. For fiscal year 1999, the most recent year for
which complete data were available, FNS reported that two states and four
Indian tribal organizations did not meet this minimum spending
requirement. In addition, the WIC Program Annual Closeout Report each
state prepares provides data on the final status of the state’s grant and costs
for the report year.
To measure the nature and amount of program services WIC agencies
provide, FNS primarily uses data obtained from state and local agencies
and reported in FNS’ Characteristics Study.6 As noted earlier, the
Characteristics Study provides information on a variety of program
activities. Table 2 lists some of the key program activities in the three areas
of service included in the Characteristics Study as well as information on
the percentage of local WIC agencies offering the activity.
Table 2: Selected Program Activities Used by Local Agencies
6FNS also utilizes special reports to measure the nature and amount of program services
provided to participants. For example, FNS uses such reports to evaluate the success of
state adoption of FNS’ national breastfeeding promotion campaign, Loving Support.
Program activitya
Percentage of local
agenciesb
Nutrition education
Nutrition education sessions for pregnant participants provided
during
Certification
Food benefit issuance
Dedicated nutrition education sessions
Other health care appointments
96.8
65.8
44.7
20.2
Nutrition education offered in a language other than English
Spanish
French
Vietnamese
Laotian
Various Native American languages
Hmong
50.0
5.7
5.3
4.6
3.9
3.8
Page 9 GAO-01-339 Measures for Three WIC Services
aA total of 460 local WIC agencies responded to the survey. Responses were weighted to reflect the
universe of 2,203 local agencies, the estimated universe of 8,932 service delivery sites, and the
universe of 8,042,758 enrollees.
bFigures may not sum to 100 percent because respondents could provide multiple responses.
cResponses for these services are by percentage of WIC service delivery sites providing the service.
According to FNS officials, survey data from local agencies, of the sort seen
in table 2, may no longer be collected for the Characteristics Study for 2002
and beyond because research funds are unavailable for that purpose. These
officials explained that the Agriculture, Rural Development, Food and Drug
Administration, and Related Agencies Appropriations Act of 1998 (P.L. 105-
86) effectively transferred primary responsibility for research on food
assistance and nutrition programs from FNS to the Department’s Economic
Research Service, which has not subsequently funded the collection of the
local WIC agency data. FNS officials believe that these data are important
Nutrition education topics especially for pregnant women
Importance of folic acid
Using WIC foods for a healthy diet
Strategies to prevent or manage overweight
81.0
82.9
47.6
Methods or materials always used in providing nutrition education
Counseling/discussion
Food models
Videos, films, slides
Testing participant’s knowledge
90.7
25.5
7.4
4.1
Average length of group nutrition education sessions for non highrisk
participants, in minutes
Less than 10
10-19
20-29
30-39
40-49
50 and longer
18.1
28.8
21.2
5.1
2.1
3.1
Breastfeeding promotion and support
Group education sessions devoted solely to breastfeeding
Breastfeeding support groups
Provision of breast pumps
Individual counseling
Home/hospital visit
Peer counseling
97.4
80.7
43.6
31.4
30.9
16.7
Health referralsc
Type of referral or service provided
Well-baby care and immunizations, referral for all participants
Well-baby care and immunizations, referral as needed
Immunizations only, referral for all participants
Immunizations only, referral as needed
22.6
60.6
10.6
48.1
(Continued From Previous Page)
Program activitya
Percentage of local
agenciesb
Page 10 GAO-01-339 Measures for Three WIC Services
for monitoring the nature and level of services being delivered by local
agencies. To address this and other research concerns, FNS has attempted
to regain direct responsibility over research funding for food assistance
and nutrition programs.
To measure overall compliance with program requirements, including
those related to nutrition education, breastfeeding promotion and support,
and referral services, FNS regional offices conduct management
evaluations at each of the 88 state WIC agencies and at a sample of local
agencies. In general, each of the state agencies and selected local agencies
are reviewed every 3 years. According to FNS, management evaluations are
the primary means of measuring state and local WIC agency compliance
with program requirements. As part of the management evaluations, staff in
FNS’ regional offices conduct interviews, review planning documents,
observe interactions, and examine participants’ files. Listed below are
some of the types of nutrition education, breastfeeding promotion and
support, and health referral service program requirements that can be
assessed during management evaluations:
• Nutrition education
• State agency WIC plan includes nutrition education goals and action
strategies,
• Local agencies offer each participant a minimum of two nutrition
education contacts during each 6-month certification period and
documents the provision of service or the participant’s refusal or
inability to participate,
• State agency has developed or identified nutrition education
resources and materials for use by local agencies, and
• Nutrition education material provided is easily understood by
participants.
• Breastfeeding promotion and support
• State and local agencies have a designated breastfeeding promotion
and support coordinator who provides technical assistance to local
agencies,
• State agencies have developed standards for local agencies for use in
providing breastfeeding promotion and support, and
• Local agency staff has received training on promotion and
management of breastfeeding promotion and support.
• Health referrals
• Local agency staff refers participants, as appropriate, to private or
public health care services.
Page 11 GAO-01-339 Measures for Three WIC Services
FNS has not made full use of management evaluations to measure program
performance and provide needed oversight. In large part this is due to
inconsistencies in the procedures used to conduct the evaluations, which
have caused problems in interpreting evaluation results as well as in
comparing and analyzing results across evaluations. FNS recently
examined its practices for conducting management evaluations for all FNS
programs, including WIC. Results of this review were highlighted in a
January 2001 report to the Administrator. The report noted that (1)
management evaluations at state and local levels were not being conducted
uniformly across FNS regional offices; (2) standard vocabulary did not
exist for identification of certain concepts; (3) standard definitions were
needed for such key terms as deficiency (a judgment that the agency being
reviewed is not in compliance with program requirements and must take
corrective action); (4) additional staff time needed to be allocated to
testing and training prior to implementation; and (5) a learning, feedback,
and modification period would be needed to support consistent data
collection and the development of reporting methods. The task force
recommended a number of actions aimed at obtaining more consistent
information that, if implemented, could provide FNS with better data for
identifying national trends, quantifying deficiencies, developing policies,
and measuring mission effectiveness. Agency officials told us that a
response from the Administrator is expected by April 2001.
Agency Comments and
Our Evaluation
We provided a draft of this report to USDA’s Food and Nutrition Service for
review and comment. We met with FNS officials, including the Director,
Office of Analysis and Nutrition Evaluation; the Director, Supplemental
Food Program Division; and officials in the Grants Management Division.
These officials generally agreed with the information presented in this
report. However, they cautioned that the report addressed performance
measures as they pertained to three nutrition service components of the
WIC program and not the program in its entirety, which includes
supplemental foods and other aspects of nutrition services. FNS officials
believed that research on the effects of the WIC program as a whole
indicated that it has contributed to a number of desirable outcomes
including improved diets and infant feeding practices, reduced incidence of
anemia among children, reduced incidence of infant mortality and low
birthweight infants, and reduced postpartum medical expenses. Also, these
officials provided us with a number of technical comments that we
incorporated into the report, as appropriate.
Page 12 GAO-01-339 Measures for Three WIC Services
Scope and
Methodology
In developing information for this report, we spoke with and obtained data
from officials at FNS’ headquarters in Alexandria, Virginia, and the seven
FNS regional offices—Northeast in Boston, Massachusetts; Mid-Atlantic in
Robbinsville, New Jersey; Southeast in Atlanta, Georgia; Midwest in
Chicago, Illinois; Mountain Plains in Denver, Colorado; Southwest in
Dallas, Texas; and, Western in San Francisco, California. We reviewed
legislation, policies, procedures, and practices; reviewed reports issued by
GAO and others related to WIC; and interviewed various officials
responsible for administering the WIC program. We also reviewed FNS’
strategic and annual performance plans and other documents developed as
part of implementing the Results Act. We conducted our work from
September 2000 through January 2001 in accordance with generally
accepted government auditing standards.
We are sending copies of this report to the appropriate congressional
committees; interested members of the Congress; the Honorable Ann M.
Veneman, the Secretary of Agriculture; and other interested parties. We will
also make copies available to others upon request.
If you or your staff have any questions about this report, please contact me
at (202) 512-7215. Key contributors to this report are listed in appendix II.
Robert E. Robertson
Director, Education, Workforce
and Income Security
Page 13 GAO-01-339 Measures for Three WIC Services
Page 14 GAO-01-339 Measures for Three WIC Services
Appendix I
Appendixes Output Measures for Monitoring Program
Requirements and Activities for the Three
Services AppendxiI
Performance measure Description of requirement or activity Source of data
Nutrition education
Amount of Nutrition Services and
Administration (NSA) funds spent for
nutrition education by state agency
Requirement that state agencies spend an
amount not less than the sum of one-sixth the
total amount of its NSA expenditures on nutrition
education and breastfeeding promotion and
support plus a breastfeeding promotion and
support target amount established by FNS at the
beginning of each fiscal year
Financial report
Extent to which participant actually
receives nutrition education
Requirement that each participant will be provided
with at least two nutrition education contacts
during each 6-month certification period
Management evaluation review of
participants’ files
Extent to which the nutrition education is
in individual or group settings
Requirement that nutrition education contact will
be provided to each participant in individual or
group settings
Management evaluation observations of
staff-participant interactions, and review
of participants’ files and documents
Number of state agencies with a nutrition
coordinator
Requirement that each state WIC agency will
designate a nutrition coordinator responsible for
nutrition education
Special reports to the agency and
management evaluation review of
documents
Number of state agencies that have
developed nutrition goals and action plans
Requirement that state agency will include in its
state plan a nutrition plan with education goals
and action plans, including a description of
education methods
Management evaluation review of state
agency’s State Plan of Operations
Extent to which nutrition education is free,
easy for participant to understand, and
bears a practical relationship to
participant’s needs
Requirement that the participant will be provided
with free nutrition education that is easily
understood and bears a practical relationship to
participant’s needs situation and cultural
preferences
Management evaluation observations of
staff-participant interactions, and review
of participants’ files and documents
Number of local agencies offering nutrition
education in foreign languages
Requirement that, in order to meet the needs of
non-English-speaking participants, WIC agencies
offer nutrition education in languages other than
English, such as Spanish, Laotian, and French
Management evaluation observations
and review of documents; survey of local
agencies through WIC Participant and
Program Characteristics Study
Extent to which nutrition education
emphasizes the relationship between
proper nutrition and good health
Requirement that state agency will provide
nutrition education that emphasizes the
relationship between proper nutrition and good
health
Management evaluation observations of
staff-participant interactions, and review
of participants’ files and documents
Extent to which nutrition education
reinforces positive changes in food habits
that result in participant’s improved
nutritional status
Requirement that state agencies will assist
nutritionally at-risk individuals in achieving positive
changes in food habits, resulting in improved
nutritional status and in prevention of nutritionrelated
problems
Management evaluation observations of
staff-participant interactions, and review
of participants’ files and documents
Extent to which state agency provides the
local agency with nutrition education
resources and materials
Requirement that each state agency will identify
or develop nutrition education resources and
educational materials for use in local agencies
Management evaluation review of state
agency’s State Plan of Operations and
other state documents; review of
resources at local agency
Appendix I
Output Measures for Monitoring Program
Requirements and Activities for the Three
Services
Page 15 GAO-01-339 Measures for Three WIC Services
Number of visits made to FNS’ nutrition
education Web site
Measurement of local agency program activity to
provide nutrition education strategies to WIC staff
and increase staff nutrition education knowledge
through FNS-implemented Web site, WIC Works
Special report
Percent of WIC staff who are Registered
Dieticians (or RD-eligible) or Registered
Dietary Technicians
Measurement of FNS’ recruitment and retention
efforts to increase the number of registered
dieticians, which is critical to the delivery of quality
nutrition services, including nutrition education
Special report (biennial survey by public
health and nutrition program directors)
Breastfeeding promotion and support
Amount of nutrition services and
administration funds spent on
breastfeeding promotion and support
Requirement that state agencies must spend a
target amount of nutrition services and
administration funds on breastfeeding promotion
and support
Financial report
Number of state agencies with a
breastfeeding coordinator
Requirement that the state agency will designate
a breastfeeding promotion and support
coordinator to provide support and technical
assistance to local agencies and to develop state
agency activities
Special report; management evaluation
of state agency’s State Plan of
Operations and other documents
Number of local agencies that have a
breastfeeding coordinator
Requirement that every local WIC agency will
designate a breastfeeding coordinator responsible
for providing promotion and assistance
Special report; management evaluation
of local agency documents
Number of states that implement all or
part of FNS’ National Breastfeeding
Promotion Campaign
Measurement of the success of FNS’ campaign—
Loving Support—designed to promote
breastfeeding in the WIC program
Special report
Extent to which state agency provides the
local agency with breastfeeding education
resources and materials
Requirement that state agency will develop
breastfeeding education resources and materials
and provide technical assistance to local agencies
Management evaluation review of state
agency’s State Plan of Operations and
other documents
Number of state agencies that have
developed standards for local agency use
in providing breastfeeding support
Requirement that the state agency develop
standards for use by local agencies for
breastfeeding support for prenatal and
postpartum women
Management evaluation review of state
agency’s State Plan of Operations and
other documents
Number of local agencies offering
education sessions devoted to
breastfeeding
Measurement of program activity to encourage
pregnant participants to begin and continue
breastfeeding their infants
Survey of local agencies through WIC
Participant and Program Characteristics
Study
Number of local agencies that have staff
that make home or hospital visits to
mothers of newborns
Measurement of local agency activity to
encourage and support breastfeeding by mothers
of newborns through home or hospital visits
Survey of local agencies through WIC
Participant and Program Characteristics
Study
Number of local agencies offering peer
counseling for breastfeeding
Measurement of local agency activity to
encourage and support breastfeeding through
peer support and discussion groups
Survey of local agencies through WIC
Participant and Program Characteristics
Study
Number of local agencies that have
received training on promotion and
management of breastfeeding promotion
and support
Requirement that the state WIC agency develop
and provide to local agencies training on
promotion and management of breastfeeding
promotion and support
Management evaluation review of state
agency’s State Plan of Operations and
other documents
Performance measure Description of requirement or activity Source of data
Appendix I
Output Measures for Monitoring Program
Requirements and Activities for the Three
Services
Page 16 GAO-01-339 Measures for Three WIC Services
Health referrals
Number of participants provided with
information on health care providers
Requirement that WIC staff provide each
participant with information on available health
care providers
Management evaluation review of
participant file data and other documents
Number of participants referred to a
health practitioner when appropriate
Requirement that WIC staff refer participants to a
health practitioner when appropriate, such as
when a pregnant participant has not yet received
prenatal care or a child participant has not
received age-appropriate immunizations
Management evaluation review of
participant file data and other documents
Number of WIC agencies with on-site
pediatric care
Measurement of WIC agencies that are collocated
with pediatric care facilities to assist in meeting, in
part, the requirement that WIC staff help
participants obtain and use preventive health care
services and local agencies serve as a link
between participants and health-care providers
Survey of local agencies through the
WIC Participant and Program
Characteristics Study; management
evaluation of local agency operations
Number of WIC agencies offering wellbaby
care and immunizations
Measurement of WIC agencies that offer wellbaby
care services to assist in meeting, in part,
the requirement that WIC staff assist participants
in obtaining and using preventive health-care
services
Survey of local agencies through the
WIC Participant and Program
Characteristics Study; management
evaluation of local agency operations
Performance measure Description of requirement or activity Source of data
Page 17 GAO-01-339 Measures for Three WIC Services
Appendix II
GAO Contact and Staff Acknowledgments Appendix II
GAO Contact Robert E. Robertson, (202) 512-7215
Staff
Acknowledgments
In addition to the contact named above, Abiud Amaro, Clifford Diehl, Judy
Hoovler, Angela Miles, and Ron Wood made key contributions to this
report.
(150291) Letter
Ordering Information The first copy of each GAO report is free. Additional copies of
reports are $2 each. A check or money order should be made out to
the Superintendent of Documents. VISA and MasterCard credit
cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013
Orders by visiting:
Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC
Orders by phone:
(202) 512-6000
fax: (202) 512-6061
TDD (202) 512-2537
Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.
Orders by Internet:
For information on how to access GAO reports on the Internet,
send an e-mail message with “info” in the body to:
info@www.gao.gov
or visit GAO’s World Wide Web home page at:
http://www.gao.gov (http://www.gao.gov/)
To Report Fraud,
Waste, or Abuse in
Federal Programs
Contact one:
• Web site: http://www.gao.gov/fraudnet/fraudnet.htm
• e-mail: fraudnet@gao.gov
• 1-800-424-5454 (automated answering system)
United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Correction Requested
Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00