Absence of conflict of interest.
Citation
Ho, H. Y. (2016). Advanced Manufacturing Education (AME) Alliance evaluation: Final evaluation report (Grant Number TC-23753-12-60-A-27). Denver, CO: McREL International.
Highlights
- This study’s objective was to examine the impact of the Advanced Manufacturing Education (AME) Alliance on education, earnings, and employment outcomes.
- The author used a nonexperimental design to compare education, employment, and earnings outcomes of AME participants to a matched comparison group.
- The study found that AME participants were significantly more likely to complete a program of study relative to the comparison group.
- The quality of casual evidence provided in this study is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the enhanced AME program; other factors are likely to have contributed.
Intervention Examined
The Advanced Manufacturing Education (AME) Alliance
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Advanced Manufacturing Education (AME) Alliance was formed with three community colleges and one state college in Minnesota upon receipt of a four-year TAACCCT grant. The AME redesigned advanced manufacturing degree programs at these colleges to better meet the needs of their students by developing a hybrid curriculum, providing additional support mechanisms to students, collaborating with employer and workforce partners, and utilizing technology across programs. This included purchasing technology for students, developing four hybrid courses and five modularized courses, hiring six educational and employment advisors, establishing a website providing general information about grant programs for students, and expanding employment partners.
Features of the Study
The nonexperimental study was conducted across the AME alliance colleges in Minnesota (Central Lakes College, Pine Technical and Community College, Saint Cloud Technical and Community College, and Minnesota’s 360 Center for Manufacturing Excellence). The author matched participants enrolled in enhanced AME courses to nonparticipants, using propensity scores developed from baseline demographic information. Study participants included 180 participants enrolled in an enhanced AME program in the fall of 2014, as well as 180 comparison students who attended a non-enhanced AME or similar program from 2005-2010. Using data from student database records and Minnesota’s Department of Employment and Economic development, the author used statistical models to examine differences in outcomes between the groups. Outcomes included degree completion, and wage increases and employment six months after completion of an AME program.
Findings
Education and Skills Gain
- The study found that AME participation was significantly associated with program completion, with program participants being 1.6 times more likely to complete their program of study than the comparison group.
Earnings and Wages
- The study found no significant relationship between participation in the AME-enhanced program and earnings increase 6 months after completing an AME program.
Employment
- The study found no significant relationship between participation in the AME-enhanced program and employment 6 months after completing an AME program.
Considerations for Interpreting the Findings
The author created a matched group of non-participating students to compare to students who were enrolled in an AME-enhanced program. However, the author did not appropriately control for other factors that could have affected the differences between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the enhanced AME program—could explain the observed differences in outcomes. Additionally, the author used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention and did not include sufficient controls. This means we are not confident that the estimated effects are attributable to the enhanced AME program; other factors are likely to have contributed.