Absence of conflict of interest.
Citation
Public Policy Associates. (2016). Evaluation of the AMP PLUS program: Final report. Lansing, MI: Public Policy Associates, Incorporated.
Highlights
- The study’s objective was to examine the impact of the Advanced Manufacturing Pathways (AMP) PLUS program on education and employment outcomes.
- The authors used a nonexperimental design to compare the outcomes of students who participated in the AMP PLUS program to a historic comparison group using the college’s administrative data.
- The study found no statistically significant relationships between participating in the AMP PLUS program and student retention, degree completion, credit hours completed, or employment.
- The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the AMP PLUS program; other factors are likely to have contributed.
Intervention Examined
The Advanced Manufacturing Pathways (AMP) PLUS Program
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Fox Valley Technical College in Appleton, Wisconsin was awarded a TAACCCT grant to implement the Advanced Manufacturing Pathways (AMP) PLUS program. The AMP PLUS program was designed to increase student retention and completion, as well as accelerate the time to receive industry-recognized credentials. The program began in early 2013 and focused on three technical areas of need in the region: Welding, Automation/Electronics, and Machine Tool Technology. The AMP PLUS program created technology-enhanced apps to ease scheduling and shorten the time it took for students to begin career pathways, provided enhanced wraparound support services for career navigation, and developed stackable credits and additional employer-recognized non-degree certifications, in addition to providing other flexible learning opportunities (e.g., flexible scheduling).
Features of the Study
The nonexperimental study took place at Fox Valley Technical College and compared the outcomes of students participating in the AMP PLUS program to students who did not. The treatment group consisted of 371 students who first enrolled in Welding, Automation/Electronics, or Machine Tool Technology programs between January 1, 2013 and January 1, 2015. The comparison group included 566 students who were enrolled in the same programs from 2010-2012 and left the college or graduated before the grant was implemented. Using the college’s administrative records, the authors conducted statistical models to examine differences in outcomes between the groups. Outcomes included student retention, degree completion, credit hours completed, and post-program employment.
Findings
Education and skills gain
- The study found no statistically significant relationships between the AMP PLUS program and student retention, degree completion, or credit hours completed.
Employment
- The study found no statistically significant relationship between the AMP PLUS program and employment.
Considerations for Interpreting the Findings
The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. Specifically, the authors did not account for differences in gender or baseline education outcomes. These preexisting differences between the groups—and not the AMP PLUS program—could explain the observed differences in outcomes. In addition, the authors used a comparison group from previous enrollment years. Because the outcome data on the two groups were collected from participants at different times, observed outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the AMP PLUS program; other factors are likely to have contributed.