Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.
Citation
Schirm, A., & Rodriguez-Planas, N. (2004). The Quantum Opportunity Program Demonstration: Initial post-intervention impacts. Washington, DC: Mathematica Policy Research.
Highlights
- The report’s objective was to evaluate the impact of the Quantum Opportunity Program (QOP)—which includes case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives—on educational attainment and labor market participation approximately seven years after program enrollment. Other reports presented impact findings from different time periods.
- About 1,100 9th-grade students from 11 high schools were randomly assigned to the treatment group, which could enroll in QOP, or to the control group, which could not participate in QOP activities. The study team administered a survey to treatment and control group members about seven years after program enrollment (three years after their scheduled graduation from high school) to collect information on postsecondary educational attainment and labor market outcomes.
- Participation in QOP had no statistically significant impact on most measures of postsecondary attainment and employment. The exceptions were that participation increased enrollment in college or vocational/technical school, an apprenticeship, or the military by 10 percentage points. It decreased the likelihood of working at least 35 hours at a main job by 11 percentage points.
- The quality of the causal evidence presented in this report is low. This means we do not have confidence that the estimated effects are attributable to QOP; other factors likely contributed.
Intervention Examined
Quantum Opportunities Program (QOP)
Features of the Intervention
QOP was an intensive program composed of case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives. Services were provided year-round for up to five years to enrollees who had not yet graduated from high school. After high school graduation, participants received some continued mentoring and assistance applying to postsecondary education or training. According to the program model, case managers were to serve 15 to 25 students, and annual participation goals were 750 hours per enrollee who had not yet graduated from high school. For this evaluation, QOP was operated by seven community-based organizations, each affiliated with one to three high schools (11 high schools in total).
Eligibility requirements for students included being in 9th grade for the first time during the 1995–1996 academic year (except at one site, for which the relevant academic school year was 1996–1997) and being in the bottom two-thirds of the grade point average distribution for their school in 8th grade. In addition, the students could not be so physically or learning disabled that, according to the school, the program was inappropriate for them.
Features of the Study
Students who met eligibility requirements were randomly selected to participate in the study. Those who consented (about 1,100) were randomly assigned to either the treatment or control group. The treatment group was allowed to participate in QOP activities, whereas the control group was not. Participants in both groups were surveyed about three years after their scheduled graduation from high school—when they were entering their early 20s—to collect information on their educational and labor market outcomes. High school transcripts for both groups were also collected.
Study Sites
- Cleveland, Ohio
- Fort Worth, Texas
- Houston, Texas
- Memphis, Tennessee
- Philadelphia, Pennsylvania
- Washington D.C.
- Yakima, Washington
Findings
- Participation in QOP had no impact on high school completion or most measures of postsecondary attainment and employment.
- Participation in QOP increased enrollment in college, vocational/technical school, an apprenticeship, or the military by 10 percentage points, a statistically significant difference. Participation in QOP also increased the likelihood of ever attending a two- or four-year college by 8 percentage points.
- QOP participation decreased the likelihood of working at least 35 hours at a main job by 11 percentage points, a statistically significant difference.
Considerations for Interpreting the Findings
Although it was based on a randomized controlled trial, the study had high differential attrition across the study groups: the treatment group’s response rate to the follow-up survey was substantially higher than the control group’s. This made the study ineligible to receive a high causal evidence rating. In addition, the analysis did not demonstrate equivalence of the two groups on a measure of their financial disadvantage, which is required for studies reviewed in this topic area, nor did it include a control for this characteristic. Therefore, the study could not receive a moderate causal evidence rating.
Causal Evidence Rating
The quality of the causal evidence presented in this report is low. This means we do not have confidence that the estimated effects are attributable to QOP; other factors likely contributed.