Citation
Scrivener, S., Sommo, C., & Collado, H. (2009). Getting back on track: Effects of a community college program on probationary students. New York: MDRC.
Highlights
- The study’s objective was to examine the impact of the Opening Doors and Enhanced Opening Doors programs at Chaffey Community College in California on progress toward completing a degree. Students in the Opening Doors and Enhanced Opening Doors programs took a college success course that covered topics such as personal goals, study skills, and college rules, and had to visit the college’s student success centers for tutoring sessions.
- The study was a randomized controlled trial. Eligible students were randomly assigned to either the treatment or control group. In the original Opening Doors program, students assigned to the treatment group were offered the opportunity to participate in the program. In the Enhanced Opening Doors program, students in the treatment group were required to participate in the program. The control groups for both the original and Enhanced Opening Doors programs were not required to participate in the program. The primary data sources were a baseline survey on the background characteristics of students, students’ transcripts, and degree attainment information from the National Student Clearinghouse.
- The study found no significant impacts on students’ course registration, the number of semesters enrolled, or regular credits accumulated for the original Opening Doors program. For the Enhanced Opening Doors program, the study found that the treatment group earned significantly more nondegree credits and developmental credits during each of the two program semesters and cumulatively across both program semesters, compared with the control group. Additionally, at the end of both program semesters, a higher proportion of students in the Enhanced Opening Doors program were in good academic standing (that is, earned a grade point average of 2.0 or higher and were not on probation due to attempting fewer than 12 credits or completing fewer than half of attempted credits) compared with students in the control group.
- The quality of causal evidence presented in this report is high because it was based on a well-implemented randomized controlled trial. This means we are confident that the estimated effects are attributable to the Enhanced Opening Doors program, and not to other factors.
Intervention Examined
The Opening Doors and Enhanced Opening Doors Programs at Chaffey Community College
Features of the Intervention
In fall 2006 and spring 2007, Chaffey Community College implemented a mandatory two-semester program called Enhanced Opening Doors. The Enhanced Opening Doors program was built upon an earlier Opening Doors program, which was a one-semester intervention in fall 2005 that included three related components: (1) a voluntary three-credit college success course, taught by a college guidance counselor, which covered personal goals, college rules and regulations, and study skills; (2) visits to the student success center, where students would complete exercises addressing skills and personal learning styles, time management, and test preparation; and (3) the provision of counseling services both during and outside the college success course. The Enhanced Opening Doors program shared the same three core features as the original Opening Doors progam, but included three substantial changes. First, Enhanced Opening Doors was expanded to a two-semester intervention that mandated enrollment in the three-credit college success course in the first semester and offered a voluntary success course in the second semester. Second, the Enhanced program required that students complete at least five visits to the student success centers in the first program semester. Third, the counselors who taught the college success course were more experienced than those in the original program, and greater emphasis was placed on the importance of meeting with students outside of class.
To participate in the Enhanced Opening Doors program, students had to be on academic or progress probation, have earned fewer than 35 credits toward a degree or credential, hold a high school degree or general educational development certificate, and be part-time or full-time students ages 18 to 34.
Features of the Study
This study was a randomized controlled trial. Randomization occurred at the student level, separately for each cohort. Eligible students who consented to participate in the study were randomized either to the treatment group, which, for the original Opening Doors, was offered the opportunity to enroll in the program, and for the Enhanced Opening Doors, was required to participate in the program. Students assigned to the control group could voluntarily enroll in the student success course and access existing college services. There were 898 students in the original Opening Doors study, with 448 in the treatment group and 450 in the control group. There were 444 students in the Enhanced Opening Doors study, with 224 in the treatment group and 220 in the control group.
The authors used data from the baseline data form, student transcripts from Chaffey Community College, attendance records from Chaffey College Success Centers, and information from the National Student Clearinghouse. Impacts were reported as the difference of means between the treatment and control groups, adjusted for the timing of random assignment. For the Opening Doors program, the study reported outcomes for four semesters: the program semester and three subsequent follow-up semesters. For the Enhanced Opening Doors program, the study reported outcomes for the two program semesters.
Findings
- The study found no impacts of the original Opening Doors program in any of the post-program semesters, or cumulatively over the four semesters following random assignment, on the number of semesters enrolled in college, course registration, regular credits earned, or the proportion of students who were ever in good academic standing at Chaffey.
- For the Enhanced Opening Doors program, the study found no cumulative impacts over the two program semesters on the number of semesters enrolled in college, course registration, or regular credits earned. However, the study found that the treatment group earned 2.6 more nondegree and 0.5 more developmental credits during the two program semesters than the control group. This difference was statistically significant. In addition, the study found that a significantly higher proportion of the treatment group (30.4 percent) was ever in good academic standing cumulatively across the two program semesters compared with the control group (15.9 percent). Similarly, a significantly lower proportion of treatment group students (58.0 percent) was never in good academic standing compared with the control group (71.8 percent) through both program semesters.
Considerations for Interpreting the Findings
A key difference between the original Opening Doors and Enhanced Opening Doors programs was the requirement in the Enhanced program that students in the treatment group take the first-semester college success course. The authors noted, however, that Chaffey College decided not to enforce this component of the Enhanced program. Nevertheless, the authors suggested that by sending the message that the course was required, the college might have been able to engage students who would otherwise not have taken part on their own.
The authors noted that differences in implementation might also have contributed to the observed differences in outcomes between the treatment and control groups in the Enhanced Opening Doors program compared with the original program. The authors indicated that the College Success course in the Enhanced Opening Doors program was taught by more experienced instructors, and more than half of these instructors had taught the course in the original Opening Doors program. The authors also suggested that the visits to the Success Centers that were part of both the original and Enhanced Opening Doors programs were better integrated with themes from the College Success course in the Enhanced program.
The study authors performed multiple statistical tests on related outcomes in the progress toward degree completion domain, which makes it more likely that some impacts will be found statistically significant purely by chance and not because they reflect program effectiveness. The authors did not perform statistical adjustments to account for the multiple tests, so the number of statistically significant findings in this domain is likely to be overstated.
Causal Evidence Rating
The quality of causal evidence presented in this report is high because it was based on a well-implemented randomized controlled trial. This means we are confident that the estimated effects would be attributable to the learning communities, and not to other factors.