Absence of conflict of interest.
Citation
Franco, E. O. (2016). Final evaluation report: Summative evaluation TAACCCT round 2 accelerated, blended – online, 2 + 2 TAACCCT training project. Cayey, PR: Colegio Universitario de San Juan.
Highlights
- The study’s objective was to examine the impact of the Colegio Universitario de San Juan’s (CUSJ) modified electronics, engineering, and technology programs on education outcomes.
- The study used a nonexperimental design to compare education outcomes of CUSJ participants to a comparison group. Using CUSJ’s embedded course assessments, the author conducted statistical tests to examine differences between the groups.
- The study found that CUSJ participation was significantly associated with higher approval rates (defined as passing a course).
- The quality of causal evidence presented in this report is low because the author did not use sufficient controls in their analysis. This means we are not confident that the estimated effects are attributable to the CUSJ course modifications; other factors are likely to have contributed.
Intervention Examined
The Colegio Universitario de San Juan’s (CUSJ) Modified Courses
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Colegio Universitario de San Juan (CUSJ) in Puerto Rico used TAACCCT funds to improve technological infrastructure; create online and blended courses in Instrumentation Technology, Electric Power Technology, and Electronics Engineering Technology; add virtual educational modules; improve faculty's use of technology; improve student services and counseling; and implement online tutoring. The program was aimed at TAA-eligible workers, unemployed workers, and other university students.
Features of the Study
The study used a nonexperimental design to compare the outcomes of students who participated in the CUSJ modified courses to students who did not participate. The treatment group included 192 CUSJ students who enrolled in and rated select modified core courses from the Instrumentation Technology, Electric Power Technology, and Electronics Engineering Technology programs from 2013-2016. The comparison group included 2,495 students who enrolled in and rated the traditional formats. The outcome of interest was course approval rates defined as passing a course. Using embedded assessment data from CUSJ’s courses, the author used statistical models to examine differences between the treatment and comparison groups.
Findings
Education and skills gain
- Relative to the comparison group, the study found that participation in CUSJ modified courses was significantly associated with higher approval rates (passing rates) in five courses: Physics I Lab (78% vs. 62%), Pre-Calculus I (51% vs. 25%), Alternate Current (AC) Lab (93% vs. 83%), Introduction to Electronics Lab (92% vs. 83%), and Advance Electronics (100% vs. 67%).
Considerations for Interpreting the Findings
The author did not account for preexisting differences between the groups before program participation or include sufficient control variables, such as age, race/ethnicity, gender, a pre-intervention measure of financial disadvantage, or a pre-intervention measure of education. These preexisting differences between the groups—and not CUSJ modified courses—could explain the observed differences in the outcome. Additionally, the author does not provide information on when the comparison group completed the courses, does not account for self-selection into TAACCCT-modified courses, and conducted a one-tailed t-test as opposed to a two-tailed t-test when assessing statistical significance. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the CUSJ course modifications; other factors are likely to have contributed.