Absence of conflict of interest. This study was conducted by staff from ICF, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.
Citation
Horwood, T., Campbell, J., McKinney, M., & Bishop, M. (2018). Heroes for Hire (H4H) program evaluation final report. Fairfax, VA: ICF.
Highlights
- The study’s objective was to examine the impact of the Heroes for Hire (H4H) program on education and earnings outcomes.
- The authors used a nonexperimental design to compare the education outcomes of H4H program participants to a matched comparison group of students enrolled at the same college, at the same point of entry. and in similar programs of study. The authors also compared the earnings outcomes of students before and after participating in the H4H program.
- The study found that H4H participants were significantly more likely to complete the program and attain credit hours than the comparison group. In addition, participation in the H4H program was significantly associated with higher post-enrollment salaries compared to initial enrollment salaries.
- The quality of causal evidence presented in this study is low for the education outcomes because the authors did not ensure that the groups being compared were similar before the intervention nor include sufficient control variables. The quality of causal evidence is also low for the earnings outcomes because the authors did not account for trends in outcomes before the intervention. This means we are not confident that the estimated effects are attributable to the H4H program; other factors are likely to have contributed.
Intervention Examined
The Heroes for Hire (H4H) Program
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
A three college consortium in West Virginia (Mountwest Community and Technical College, Blue Ridge Community and Technical College, and Southern West Virginia Community and Technical College) received TAACCCT funds to administer the Heroes for Hire program. The program was designed to help veterans and trade-affected workers receive training and upgrade their skills for employment in various fields. The consortium targeted multiple occupations tailored to each college's job market. These occupations included 38 industry-certified credentials across the healthcare sector and manufacturing service industries with four specific curriculum pathways: Health Information Management (i.e., Medical Billing and Coding), Health Professions (i.e., Patient Care Technician, EKG and Phlebotomy, EMT, and Paramedic Science), Chemical Technology, and Geospatial Technologies. An important element of the H4H program was hands-on applied learning. A variety of professional development was also offered to the faculty and grant staff throughout the life of the grant.
Features of the Study
The study took place across the three colleges in a consortium in West Virginia. The authors used a nonexperimental design to compare the education outcomes of students enrolled in one of the H4H pathways programs to a concurrent cohort of students. The treatment group consisted of 444 students that were enrolled in a H4H Pathway program from Fall 2015 through Spring 2018 and received grant funded services. The comparison group consisted of a cohort of 444 students who were enrolled at the same colleges, during the same entry point between Fall 2015-Spring 2018, in similar programs of study, but presumably did not receive grant funded curriculum, resources, or student support services. Treatment students were matched with comparison students using demographic characteristics. The study used administrative data records to examine the impact of the H4H program on educational outcomes by comparing differences between the treatment and comparison cohorts. To examine the earnings outcome, the authors compared pre- and post-enrollment data for the treatment group only.
Findings
Education and skills gains
- The study found that H4H participation was significantly associated with credit hour attainment, with program participants earning more credit hours than students in the comparison group (43.1 vs. 41.1).
- The study also found that H4H participation was significantly related to program completion, with program participants having higher completion rates than comparison students (20.2% vs. 12.2%).
Earnings and wages
- The study found that on average, salaries significantly increased by over $6,000 for students enrolled in the H4H pathway programs when comparing their pre- and post-program enrollment salaries.
Considerations for Interpreting the Findings
The authors used propensity score matching to create a comparison group; however, they did not account for preexisting differences between the groups in baseline education outcomes or include sufficient control variables as outlined in the protocol. These preexisting differences between the groups—and not the H4H program—could explain the observed differences in outcomes. Moreover, the authors compared the earnings outcome of participants measured once before and once after they participated in the H4H program. CLEAR’s guidelines require that the authors observe outcomes for multiple periods before the intervention to rule out the possibility that participants had increasing or decreasing trends in the outcomes examined before enrollment in the program. Without knowing the trends before program enrollment, we cannot rule this out. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this study is low for the education outcomes because the authors did not ensure that the groups being compared were similar nor include sufficient control variables. The quality of causal evidence is also low for the earnings outcome because the authors did not account for trends in outcomes before the intervention. This means we are not confident that the estimated effects are attributable to the H4H program; other factors are likely to have contributed.