Citation
Hollenbeck, K., & Huang, W-J. (2014). Net impact and benefit-cost estimates of the workforce development system in Washington State. (Upjohn Institute Technical Report No. 13-029). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. Retrieved from W.E. Upjohn Institute for Employment Research website: http://dx.doi.org/10.17848/tr13-029. [VOC]
Highlights
- The study’s objective was to examine the impact of Vocational Rehabilitation (VOC) programs on the employment rate, earnings, and benefit receipt of adults with disabilities in Washington State.
- The authors used a nonexperimental method to compare the short-term (3 quarters after program exit) and long-term (9 to 12 quarters after program exit) employment, earnings, and Unemployment Insurance benefits between those who took part in VOC programs relative to those who were eligible, but did not receive services from VOC programs.
- The study found that, compared with those who did not receive services from VOC programs, VOC program participants had higher employment and earnings. Receipt of benefits was lower in the short-term for VOC program participants relative to those who did not receive VOC program services.
- The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before program participation. This means we are not confident that the estimated effects are attributable to VOC programs; other factors are likely to have contributed.
- This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.
Intervention Examined
The Vocational Rehabilitation (VOC) Programs
Features of the Intervention
VOC programs offered customized services to assist people with disabilities in finding employment, including part-time employment, self-employment, and sheltered or supported employment. The services offered included assessment, counseling, vocational training, physical and restorative services (which can include corrective surgery), job search assistance, and placement assistance. To be eligible to participate in these programs an individual must have provided certification to show that they had a physical, mental, or sensory impairment which substantially impeded their ability to find employment. They must also have shown that they required and could have benefitted from the provision of vocational rehabilitation services.
Features of the Study
The authors used a nonexperimental statistical approach to compare participants of VOC programs with people whose disability status made them eligible for VOC services but who did not receive them. Eligibility was based on certification that the individual had a physical, mental, or sensory impairment that limited employment and required VOC services to improve employment outcomes. The authors used a difference-in-difference model to compare the two groups on employment, hourly wages, hours worked per quarter, quarterly earnings, and receipt of benefits before and after participation. The authors collected Unemployment Insurance records for those who had exited a VOC program from July 2005 to June 2006 to estimate the long-term impacts of the program in quarters 9 to 12 after program exit. They also collected Unemployment Insurance records for those who exited from July 2007 to June 2008 to estimate the short-term impacts in the 3 quarters following program exit. There were 4,208 VOC participants who exited in 2005–2006 and 3,502 who exited in 2007–2008. The comparison groups included 4,258 nonparticipants in 2005–2006 and 1,298 nonparticipants in 2007–2008.
Findings
Employment
- The authors reported that, compared with those who were eligible but did not take part in a VOC program, the percentage of quarters employed for VOC program participants significantly increased by 8.3 percentage points more in the third quarter after program exit and by 10.2 percentage points more in quarters 9 to 12 after program exit. The average number of hours worked per quarter also significantly increased in quarters 9 to 12, with VOC program participants working an average of 24.8 more hours than those who did not take part in the program.
Earnings
- There was no significant association between VOC program participation and earnings in the third quarter after exit, but there was a significant increase in quarterly earnings and hourly wages in quarters 9 to 12 after program exit; the average quarterly earnings of VOC program participants increased by $257 more and hourly wages increased $0.86 more than those who did not receive VOC program services in quarters 9 to 12 after exit.
Public benefit receipt
- The authors found that, compared with those who did not receive VOC program services, the likelihood of benefit receipt significantly decreased by 2.5 percentage points in the third quarter after exit, but there was no significant association in quarters 9 to 12 after program exit.
Considerations for Interpreting the Findings
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the VOC program participants and nonparticipants were on identical wage trajectories before receiving services and that the average length of participation in the VOC program was six months, whereas the nonparticipant group did not receive services. At the conclusion of participation, the VOC group exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would be looking at VOC program participants’ earnings about 12 months after they started receiving services and nonparticipants’ earnings 6 months after they registered at the Labor Exchange. If the VOC program was completely ineffective, and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the VOC program participants earned more 6 months after their exit dates. However, this would not be attributable to receiving VOC program services; it would be caused by the different elapsed time across the groups (12 months for VOC program participants versus 6 months for Labor Exchange participants). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before program participation. This means we are not confident that the estimated effects are attributable to VOC programs; other factors are likely to have contributed.