- Home
- Agencies
- Department of Agriculture
- Department of Housing and Urban Development
- General Services Administration
- Department of Commerce
- Department of the Interior
- National Aeronautics and Space Administration
- Department of Defense
- Department of Justice
- National Science Foundation
- Department of Education
- Department of Labor
- Office of Personnel Management
- Department of Energy
- Department of State
- Small Business Administration
- Environmental Protection Agency
- Department of Transportation
- Social Security Administration
- Department of Health and Human Services
- Department of the Treasury
- U.S. Agency for International Development
- Department of Homeland Security
- Department of Veterans Affairs
- Goals
- Initiatives
- Programs
Primary tabs
Amy McIntoshActing Assistant SecretaryOffice of Planning, Evaluation and Policy Development
FY 14-15: Agency Priority Goal
Enable evidence-based decision making
Priority Goal
Goal Overview
GOAL OVERVIEW
Through its mix of grants, contracts, and internal analytic work, the Department of Education (ED) will support the use of research methods and rigorous study designs that provide evidence that is as robust as possible and fit for the purpose. This goal will track whether ED is increasing its internal capacity to make competitive grant awards based on the existence of (and amount of) evidence in support of projects, where appropriate.
KEY BARRIERS AND CHALLENGES
The process to collect data and track progress against the goal is still under development, and using evidence to award competitive grants entails a shift in culture and capacity building across ED to do it well. Additionally, goal targets are based on reasonable projections about which competitive grant programs may make new awards in this fiscal year, but the actual dollar amount awarded will depend on final appropriations amounts and other funding decisions and trade-offs. Grantees vary in their comfort with and understanding of evaluation and use of evidence, yet ED has limited resources to support grantees in conducting rigorous evaluations that would produce evidence of effectiveness.
EXTERNAL STAKEHOLDERS
There is an increasing emphasis among stakeholders on the importance of using evidence to support government program funding decisions, and ED regularly engages the field on this topic. A number of outside organizations have convened experts to discuss how to encourage such decisions. In addition, philanthropic and congressional actors prioritized using evidence to support decision-making and have encouraged the field to do the same. Finally, ED has worked with the National Science Foundation to develop a common evidence framework around which to organize research investments and grants. Indeed, ED is considered a leader on the issue among federal agencies, and external groups are eager for ED to deepen and broaden its efforts.
Strategies
IMPLEMENTATION STRATEGY
The Education Department General Administrative Regulations (EDGAR) provides all discretionary grant programs across ED with a common framework regarding the use of evidence within ED competitions. Establishing this common framework has been an ongoing endeavor – building off of evidence concepts created in FY10 through the Investing in Innovation program, ED established common evidence standards for all discretionary programs to use in FY13, and began including them in competitions in FY14. The evidence framework includes four tiers of evidence and EDGAR allows programs to use the tiers of evidence as priorities or selection criteria, as applicable (see 34 CFR 75.210 and 75.226), which facilitates appropriate matching between the evidence available and intended program outcomes. This is valuable for the strategic impact of the competitions and will help ED track how different programs are using evidence to inform discretionary funding decisions. In Q3, programs using evidence in their competitions began to finalize their funding decisions in order to make awards by the end of Q4. The Evidence Planning Group (EPG) at ED continues to meet with these programs, as needed, to provide guidance on questions as they come up, and to begin planning discussions for competitions in FY16.
Since beginning to use the common evidence framework in FY14, ED has continued to work on sustainable, scalable solutions to the challenge of completing evidence reviews and providing technical assistance to applicants. For example, historically, for discretionary grant programs that used an evidence framework, ED staff has explained the evidence requirements to applicants via webinar. ED is now exploring more sustainable solutions that can be used for multiple competitions, including a series of videos or an interactive infographic to ensure applicants better understand the evidence framework.
Also, in FY13, ED piloted an evidence review process using WWC contractors to review studies cited by applicants. IES staff used these WWC reviews to provide user-friendly documentation to discretionary grant program offices and to enter the results of the studies put forward by applicants into the WWC’s public database of reviewed studies. The process worked smoothly. The Policy and Program Studies Service (PPSS) has taken on an expanded role in conducting literature reviews in preparation for grant competitions. The purpose of these reviews is to summarize the state of research in a field in order to determine levels of evidence required for “entry” and “exit.” In addition, in Q3 IES announced a new WWC contract that would add additional resources to support evidence reviews. In FY14, evidence reviews were estimated to cost programs about $2,500 per study, but the new contract has allowed IES to bring that estimate down to $1,350 per study, regardless of the level of rigor. Finally, in planning for FY16, EPG has begun to design a strategy that would allow programs using less rigorous evidence to use a panel of external experts, rather than IES contractors. While still in early planning stages, we think this approach, if implemented well, would further reduce the costs for programs, making evidence-based grant making a more affordable option. Evidence-based grantmaking is an iterative process, and ED has consistently focused on refining the process over the course of FYs 14 and 15 in order to improve performance toward this APG.
EXTERNAL FACTORS THAT ED CAN AND CANNOT INFLUENCE
Some ED programs are specifically designed to produce evidence, and more evidence is appearing on a regular basis. ED can and will use its grant making authority to incentivize and, when appropriate, require evaluations designed to produce evidence of what works. (EDGAR enables grant programs to do this through selection criteria aligned to EDGAR’s evidence framework.) However, the amount of evidence available in specific fields is highly variable. Accordingly, some grant competitions will enjoy a more robust evidence base supporting proposed interventions than others, especially in the short and medium term. To some extent, ED can influence the evidence base through its continued work to support rigorous evaluations of projects designed to answer key policy questions. However, rigorous research done well can take time, and it may be several years before ED-funded studies are available to push the field in new directions. In addition, ED is dependent on Congress to appropriate funding for discretionary grant programs.
Progress Update
- Progress toward goal: The Department of Education (ED) surpassed the goal of increasing the percentage of select new (non-continuation) discretionary grant dollars that reward evidence to nine percent. In Fiscal Year (FY) 2014, 15.92 percent of ED’s discretionary dollars was awarded to new projects with supporting evidence of effectiveness. Data for FY15 will not be available until Quarter 1 of FY16, but the Department thinks, based on internal projections and on past performance, that it will again meet its target.
- The strategic objective to which this agency priority goal contributes aims to improve the use of research. As such, in addition to the APG metrics, ED includes metrics specific to the Education Resource Information Center (ERIC) and the What Works Clearinghouse (WWC). As of Q4, the number of full-text of peer-reviewed education research resources reached 36,197 and the number of reviewed studies in the WWC reached 10,889. As of Q3, we have already surpassed our FY15 target for both metrics.
- ED’s Evidence Planning Group (EPG) continues to identify opportunities for discretionary grant programs to use evidence-related priorities or selection criteria in competitions. In FY15, five competitions in the Office of Innovation and Improvement (OII), the Office of Elementary and Secondary Education (OESE), and the Office of Postsecondary Education (OPE) are planning to use evidence in competitions through a mix of eligibility requirements, absolute or competitive preference priorities, and selection criteria. In addition, EPG continues to meet with each grant making office in ED to discuss appropriate uses of evidence in FY16 competitions. In Q4, the following discretionary grant programs rewarded evidence in their FY15 competitions:
- Supporting Effective Educator Development
- First in the World
- TRIO -- Student Support Services
- Strengthening Institutions Program
- In addition, in Q1 of FY16, the Investing in Innovation (i3) program will reward evidence.
- In FY15, ED moved toward a new standardized process for collecting input data relevant to this APG. EPG collected relevant data on each contributing throughout the year, and followed up with those programs early in FY16 to confirm that the amounts awarded to evidence-based projects was accurate. EPG continues to engage internal stakeholders to critically assess validation and verification processes.
- As of the end of FY15, ED surpassed this APG. However, the work is far from over. We note that EPG continues to meet with program offices in ED to discuss each program’s strategy for incorporating use of evidence in their work, and how EPG can be supportive in that process. ED has proposed an updated version of this APG for FY16-17, signaling its continued commitment to this work.
Next Steps
No Data Available
Expand All
Performance Indicators
Increase the percentage of select new (non-continuation) discretionary grant dollars that reward evidence
Contributing Programs & Other Factors
CONTRIBUTING PROGRAMS
Major Discretionary Programs Supporting Goal 5 (http://www2.ed.gov/about/reports/annual/2015plan/2013-2015-apr-app-plan-...):
- National assessment
- Research, development, and dissemination
- Statistics
- Investing in Innovation
First in the World
TRIO – Student Support Services
Strengthening Institutions Program
Supporting Effective Educator Development
For additional programs see Appendix C of the Department’s FY2013 Annual Performance Report and FY2015 Annual Performance Plan (http://www2.ed.gov/about/reports/annual/2015plan/2013-2015-apr-app-plan-...).
Expand All
Strategic Goals
Strategic Goal:
Continuous Improvement of the U.S. Education System
Statement:
Enhance the education system’s ability to continuously improve through better and more widespread use of data, research and evaluation, evidence, transparency, innovation, and technology.
Strategic Objectives
Statement:
Facilitate the development of interoperable longitudinal data systems for early learning through employment to enable data-driven, transparent decision-making by increasing access to timely, reliable, and high-value data.
Description:
Statement:
Provide all education stakeholders, from early childhood to adult learning, with technical assistance and guidance to help them protect student privacy while effectively managing and using student information.
Description:
Statement:
Invest in research and evaluation that builds evidence for education improvement; communicate findings effectively; and drive the use of evidence in decision-making by internal and external stakeholders.
Description:
Statement:
Accelerate the development and broad adoption of new, effective programs, processes, and strategies, including education technology.
Description:
To achieve the president’s 2020 college attainment goal, the nation’s education system will need to graduate many more college-ready students from high school, ensure they have access to postsecondary education, and support them as they complete their degrees—all while facing resource constraints. When other sectors of the economy need to become better, faster, or more productive, they innovate, often relying on technology for help. The education sector is no different, and the need for innovation—and its benefits—spans grade levels, curricular areas, and student needs.
A 21st-century infrastructure that harnesses modern technological advances and provides easy access to high-speed Internet can serve as a platform for greater innovation in education. Accordingly, the Department will continue to focus on ways to improve schools’ technology infrastructure and effective use of technology. It will also continue to work with Congress to establish a new advanced research projects agency for education that will use directed research and development activities to pursue breakthrough technological innovations in teaching and learning.
Technology holds the potential to expand all students’ opportunities to learn, including by supporting personalized learning experiences, providing dynamic digital content, and delivering more meaningful assessments. Technology can also help districts and schools support teachers in becoming more effective and better connected to the tools, resources, and expertise students need and help them meet more rigorous college- and career-ready standards. Technology can also help schools by providing students and school library media specialists with increased access to academic tools and other resource-sharing networks. Technology can also help schools by providing students and school library media specialists with increased access to academic tools and other resource-sharing networks. Technology-enabled instructional and assessment systems will be pivotal to improving student learning and generating data that can be used to continuously improve the education system at all levels. Innovative technology must be matched by innovative educational practices to maximize its potential to improve learning and instruction for all students, and it must be accessible to all students, including students with disabilities. Leadership is essential to ensure that innovative applications are disseminated and brought to scale.
Agency Priority Goals
Statement:
By September 30, 2015, the percentage of select new[1] (non-continuation) competitive grant dollars that reward evidence will increase by 70%.
[1] “New competitive grant dollars that reward evidence” includes all dollars awarded based on the existence of at least “evidence of promise” in support of a project, per the framework in the Education Department General Administrative Regulations (34 CFR Part 75). Consideration of such evidence appears through: eligibility threshold (e.g., in the Investing in Innovation program); absolute priority; competitive priority (earning at least one point for it); or selection criteria (earning at least one point for it). The percentage is calculated compared to the total new grant dollars awarded.
Description:
GOAL OVERVIEW
Through its mix of grants, contracts, and internal analytic work, the Department of Education (ED) will support the use of research methods and rigorous study designs that provide evidence that is as robust as possible and fit for the purpose. This goal will track whether ED is increasing its internal capacity to make competitive grant awards based on the existence of (and amount of) evidence in support of projects, where appropriate.
KEY BARRIERS AND CHALLENGES
The process to collect data and track progress against the goal is still under development, and using evidence to award competitive grants entails a shift in culture and capacity building across ED to do it well. Additionally, goal targets are based on reasonable projections about which competitive grant programs may make new awards in this fiscal year, but the actual dollar amount awarded will depend on final appropriations amounts and other funding decisions and trade-offs. Grantees vary in their comfort with and understanding of evaluation and use of evidence, yet ED has limited resources to support grantees in conducting rigorous evaluations that would produce evidence of effectiveness.
EXTERNAL STAKEHOLDERS
There is an increasing emphasis among stakeholders on the importance of using evidence to support government program funding decisions, and ED regularly engages the field on this topic. A number of outside organizations have convened experts to discuss how to encourage such decisions. In addition, philanthropic and congressional actors prioritized using evidence to support decision-making and have encouraged the field to do the same. Finally, ED has worked with the National Science Foundation to develop a common evidence framework around which to organize research investments and grants. Indeed, ED is considered a leader on the issue among federal agencies, and external groups are eager for ED to deepen and broaden its efforts.
