This study was conducted by staff from Mathematica Policy Research, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.
Citation
Peikes, D., Orzol, S., Moreno, L., & Paxton, N. (2005). State Partnership Initiative: Selection of comparison groups for the evaluation and selected impact estimates: Final report. Princeton, NJ: Mathematica Policy Research.
Highlights
- The study’s objective was to measure short-term impacts for the three State Partnership Initiative (SPI) sites whose effectiveness was evaluated using a randomized controlled trial. SPI was meant to increase employment and earnings for Supplemental Security Income (SSI) and Social Security Disability Insurance (SSDI) beneficiaries through the provision of direct services.
- The study’s data were collected from Social Security Administration (SSA) administrative data and the Summary Earnings Record (SER). The authors estimated program impacts on employment and earnings.
- The study found that SPI had negative and statistically significant impacts on employment and earnings, contrary to the initiative’s intent.
- The quality of causal evidence presented in this report is high because it is based on a well-conducted randomized controlled trial. This means we are confident that the effects estimated in the study are attributable solely to the SPI, and not to other factors.
Intervention Examined
The State Partnership Initiative
Features of the Intervention
From 1998 to 2004 the SSA and Rehabilitation Services Administration provided five-year funding for 12 SPI states to implement innovative projects to help people with disabilities find jobs. Although each state’s intervention design and target population varied, all SPI states aimed to increase beneficiaries’ understanding of how work affects benefits through activities such as providing information and referrals, problem solving, benefits assistance, benefits planning, and long-term benefits management. Additional services that states could provide included placing funds in an individual career account, changes to SSI regulations allowing working recipients to save more, employment services, psychosocial rehabilitation, and situational assessments.
The three state projects covered in this report targeted working-age adults. New Hampshire accepted SSI and SSDI beneficiaries with any type of disability. New York and Oklahoma both targeted people with mental illness, and Oklahoma included only SSI recipients who were not already employed. The New York SPI project included two separate programs, one offering benefits counseling and SSI regulation changes and the other offering those services plus employment services.
Features of the Study
The three SPI state projects covered in this report were evaluated using a randomized controlled trial. Each state conducted randomization slightly differently: Oklahoma conducted random assignment before contacting participants; New York asked eligible beneficiaries if they were interested in working and randomly assigned those interested in the project; and New Hampshire randomly assigned people who had already volunteered for the intervention. Those randomly assigned to the treatment group could receive information about benefits and additional services offered as part of the SPI. Those randomly assigned to the control group could not receive SPI services.
The evaluation used SSA administrative data for six-month employment rates and the SER, an extract of the Master Earnings File, for one-year employment rates and earnings. The authors used an adjusted, difference-in-differences model to estimate impacts on employment and earnings in the six months following random assignment and in the year following the year of random assignment, relative to employment and earnings in various pre-random assignment periods.
Study Sites
Three SPI state projects implemented random assignment.
- New York (Buffalo and New York City)
- New Hampshire
- Oklahoma
Findings
- In New York, those randomly assigned to the SPI benefits counseling without employment services group earned $343 less overall relative to two years before random assignment, compared with the control group’s change in earnings over the same period. This difference was statistically significant. There were no other impacts on employment rates or earnings measured six months or one year after random assignment.
- There were no statistically significant impacts on employment or earnings in any follow-up period for the SPI group randomly assigned to receive benefits counseling plus employment services in New York.
- In New Hampshire, those who received SSDI benefits (but not SSI benefits) and were randomly assigned to the SPI group were 29.6 percentage points less likely to be employed and earned $1,633 less in the year after random assignment, relative to the year before random assignment, compared with the control group’s change in employment and earnings over the same period. These differences were statistically significant.
- The SPI state project in Oklahoma had no impact on earnings or employment at any follow-up period.
Considerations for Interpreting the Findings
Due to the data’s configuration, employment and earnings impacts were estimated at the end of the calendar year that followed the calendar year of random assignment. Because each state project started at a different time, exposure lengths varied: the SPI group in Oklahoma was enrolled an average of 217 days; those in New York were enrolled 407 and 399 days for the benefits counseling and the counseling plus employment services groups, respectively; and those in New Hampshire were enrolled an average of 549 days. In Oklahoma, there was a lag between when random assignment occurred and when the project actually started.
The authors indicated that a number of similar interventions aiming to improve employment and earnings for people with disabilities were implemented at the same time and in many of the same states as SPI. Although it is not clear to what extent these other interventions affected the SPI treatment and control groups, the study seems to suggest that some SPI control group members might have received very similar services through the Benefits Planning, Assistance, and Outreach project.
Causal Evidence Rating
The quality of causal evidence presented in this report is high because it is based on a well-conducted randomized controlled trial. This means we are confident that the effects estimated in the study are attributable solely to the SPI, and not to other factors.