Citation
Hollenbeck, K., & Huang, W-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. [DRS]
Highlights
- The study’s objective was to examine the impact of the Vocational Rehabilitation program administered by Department of Rehabilitative Services (DRS) on participants’ employment and credential completion.
- The study used administrative records to compare the outcomes of low-income adults who took part in the DRS program with outcomes of a nonexperimental matched group of adults who did not take part in the program.
- The study found that DRS participants had higher employment and credential completion rates compared with those of people who did not participate in the program.
- The quality of causal evidence presented in this report is low because the authors did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to DRS; other factors are likely to have contributed.
- This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies
Intervention Examined
The Vocational Rehabilitation (VOC) Programs
Features of the Intervention
The Vocational Rehabilitation program administered by DRS is designed to help disabled veterans find and keep employment. Program services include job training, skills coaching, and job search assistance.
Features of the Study
Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared education and employment outcomes of those who took part in the Vocational Rehabilitation program administered by DRS with outcomes of those who did not take part in the program (both groups participated in an employment services program). The authors compared the two groups on employment two and four quarters after program exit, and on the percentage of each group that had earned an educational credential during the program or within one year of program exit. This analysis included 6,128 DRS participants in Virginia who exited the program from July 2004 to June 2005.
Findings
- Employment. The study found that DRS participants were significantly more likely to be employed than members of the comparison group, with an employment rate that was 20.0 percentage points higher in the second quarter and 18.2 percentage points higher in the fourth quarter after program exit.
- Education and/or training attainment and completion. The study found that DRS participants were significantly more likely than members of the comparison group to earn an educational credential during the program or within one year of program exit (a difference of 6.2 percentage points).
Considerations for Interpreting the Findings
The authors presented unadjusted treatment effects in the study. This profile reports adjusted effects obtained directly from the authors.
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the DRS participants and comparison group members were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in DRS services was six months, whereas that for the employment services program was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe DRS participants’ earnings about 12 months after they started receiving services and comparison group members’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the DRS participants earned more 6 months after their exit dates. However, this would not be attributable to receiving DRS services; it would be caused by the different elapsed time across the groups (12 months for DRS participants versus 7 months for comparison group members). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to DRS; other factors are likely to have contributed.
Additional Sources
Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.