Absence of conflict of interest.
Citation
Highlights
- The study’s objective was to examine the implementation of the Urban Alliance High School Internship Program, a program that offers skills training, internships, mentoring, coaching, and alumni services to help high school seniors in low-income areas pursue college or careers.
- The study authors conducted an implementation evaluation using data from a variety of sources including program staff, program participants, observations, program documents and neighborhood and school data from secondary sources.
- The study found that skills training, workshops, and internships progressed as planned with internships happening from 2:00-5:00 Monday through Thursday during the school year and all-day Monday through Thursday after participants' senior year.
- The key strength of the study is the variety of quantitative and qualitative data collected at the individual, household, school, neighborhood, and state levels. Additionally, researchers collected data from a variety of individuals with different connections to and perspectives on the program. One limitation was the lack of information regarding specific analytic techniques.
- The companion impact study was reviewed by CLEAR in June 2024 and can be found here: Evaluation of the Urban Alliance High School Internship Program (Theodos et al., 2023).
Intervention Examined
Urban Alliance High School Internship Program
Features of the Intervention
- Type of organization: Nonprofit
- Location: Multi-site
- Population served and scale: Youth, 533 program participants
- Industry focus: Not included
- Intervention activities: Training; Internships; Coaching; Mentoring
- Organizational partnerships: Employers; Schools
- Cost: Across the four sites and two program years, the per internship cost was between $10,000 and $15,000
- Fidelity: Included
The Urban Alliance High School Internship Program was started in Washington, DC in 1996, and expanded to Baltimore, MD in 2008, Chicago, IL in 2012, Northern Virginia in 2013, and Detroit, MI in 2018. The program aimed to support high school seniors from struggling communities, particularly those with GPAs between 2.0 and 3.0. In certain locations like Baltimore, Chicago, and Northern Virginia, participants could earn high school credits, and in Baltimore and Northern Virginia, they could also receive college credits. Students were typically referred to the program by their school counselors or teachers. Urban Alliance provided training, coaching, and alumni services through program coordinators. Internships were primarily office or clerical roles, with tasks supervised by a mentor at each job site. Job sites included private firms, nonprofits, and government agencies. About 65% of the job sites involved in the program pay a fee for interns, which varied by affordability.
Urban Alliance’s logic model includes activities in four areas: skills training, direct work experience, case management, and alumni services. Each component has specific metrics, such as 90% of interns receiving 30 hours of pre-work training and a 75% workshop attendance rate in each region. For mentorship, the goal is for 75% of interns to report high engagement during quarterly surveys. Short-term outcomes are tracked through skills assessments and immediate links to college, training, or living wage jobs, with targets such as 75% showing skills growth and 85% connecting post-program. Long-term metrics include 75% of alumni being employed or in school a year after completing the program, 75% persisting at least one year in college, 50% of disconnected alumni reconnecting, 70% of four-year college students graduating within six years, and 70% of two-year college students graduating within three years.
Features of the Study
The study included the sites in Washington, DC, Baltimore, Chicago, and Northern Virginia. Detroit was not part of the study as it opened in 2018, after the evaluation began. Participants came from 80 public and charter schools across the four sites, with 19 schools in Baltimore, 22 in Washington, DC, 34 in Chicago, and 5 in Northern Virginia. The authors examined various components of the Urban Alliance High School Internship Program, including work-based learning, youth job training, school support systems, and non-school programs that provide case management and mentoring aimed at preparing students for college and careers.
Data on individuals came from program applications and the Urban Alliance case management system. Community data were obtained from the American Community Survey, the National Center for Education Statistics, and the U.S. Department of Education. The evaluation team interviewed 71 people, including program staff, school staff, job mentors, and students who did not complete the program. Focus groups were held with program participants. Surveys were completed by 134 youth, 249 job mentors, and 94 school counselors. Additionally, the evaluation team gathered audited financial records, observed 27 training or workshop sessions, and reviewed training materials. To measure program fidelity, data from Urban Alliance's case management system were used to assess training and workshop attendance, hours of training provided, hours worked, completion of post-high school plans, connections to alumni services for disconnected youth, and mentor engagement.
Findings
Intervention Activities/Services
- The study found that program coordinators interacted the most with participants because of their coaching and case management role. They mainly offered support through weekly check-ins and occasional one-on-one meetings. Coordinators used performance indicators to identify areas for support, coaching, and merit-based increases in pay.
- Skills training, workshops, and internships progressed as planned. Internships took place from 2:00 to 5:00 PM, Monday through Thursday during the school year, and all day Monday through Thursday participants' senior year.
- The program staff cultivated relationships with employers and job site mentors. Approximately 80% of job sites were involved both years of the study and 80% of the mentors surveyed indicated they would be interested in having another intern. Mentors were encouraged to discuss the intern's performance with program coordinators.
Fidelity
- The study found that the program met or exceeded their target in four of seven indicators: 91% of interns (target=75%) received at least 20 hours of training, 75% (target=75%) worked at least 350 hours, 98% (target=90%) completed a post-high school plan, and 81% of disconnected alumni (target=50%) were connected to alumni services.
- The program did not meet three indicators: 59% (target=75%) attended at least 80% of workshops, 55% (target=75%) met both pre-work and workshop training attendance targets, and 68% (target=75%) reported high mentor engagement on quarterly surveys.
Implementation Challenges and Solutions
- The study found that program attrition was a challenge. Only 41% of the youth offered the program completed it. Among those who began the skills training, 54% finished the training, and of those who completed the training, 73% completed the overall program. Higher attrition rates were noted among individuals from high poverty neighborhoods. Key reasons for dropping out included after-school commitments, competing employment, transportation issues, and poor job performance.
- Another challenge was the high turnover rate of program coordinators, with 5 out of 14 leaving in the first year and 13 out of 20 in the second year.
- To address these issues, Urban Alliance staff suggested offering course credit for the unpaid skills training, increasing communication with school partners, and building stronger relationships with youth early in the program.
Cost/ROI
- The cost per internship ranged from $10,000 to $15,000 across the four locations and over two years of the program.
Considerations for Interpreting the Findings
The key strength of the study is the wide variety of quantitative and qualitative data collected at the individual, household, school, neighborhood, and state levels. Additionally, the evaluation team collected data from a variety of individuals with different connections to and perspectives on the program. While the authors describe the time ranges and media for qualitative data collection, they do not describe recording or notetaking for interviews or focus groups. The study authors describe the notetaking guide they used for observations, but do not describe any quality assurance steps related to these observations. Fidelity was assessed by the study authors and not by the CLEAR team.