Absence of conflict of interest.
Citation
Highlights
- The study’s objective was to examine the extent to which program implementation strategies, services, and activities were implemented as planned and how well they were implemented.
- The study authors conducted an implementation evaluation using a mixed-methods design that allowed them to triangulate data from various sources, including project records, the Adherence of Implementation Self-Assessment, interviews, participant and partner surveys, and participant focus groups.
- The study found that the consortium implemented all seven of its core components as outlined in its logic model with a high level of fidelity, despite encountering some challenges. Partners and participants both expressed satisfaction with the quality of the program and their involvement with it.
- The implementation study was comprehensive in its design, data collection, and analyses. The findings aligned with the research questions and were supported by the data, although the authors cited low response rates to online surveys from partners and participants as a challenge.
- The embedded impact study was reviewed by CLEAR in May 2020.
Intervention Examined
The Orthotics, Prosthetics & Pedorthics (HOPE) Careers Consortium Program
Features of the Intervention
- Type of organization: Community and technical college
- Location/setting: Multi-site in Florida, Michigan, Minnesota, Oklahoma, Washington
- Population served and scale: Adults; Dislocated workers; 1,873 participants
- Industry focus: Health care and social assistance
- Intervention activities: Accelerated learning; Career pathways; Student support services; Technology
- Organizational partnerships: Employers; Industry partners; Workforce Investment Boards; One-Stop Centers; Four-year colleges
- Cost: Not included
- Fidelity: Included
The HOPE Careers Consortium, consisting of five colleges, was awarded a Round 3 Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant in 2013, aimed to expand and improve the delivery of orthotics, prosthetics, and pedorthics (O&P) career education and increase students’ attainment of industry-recognized credentials. To do this, the Consortium recruited participants, created technology-based and online learning opportunities, accelerated training pathways, supported retention and job placement, and developed stackable credentials and articulation pathways.
Seven core components of the intervention are outlined in the logic model and guided implementation. The Consortium also collaborated closely with a variety of partners. Industry, workforce, and community partners contributed to the grant through participation in an advisory committee, job placement, experiential learning opportunities, curriculum reviews, and leveraging of resources. The intervention targeted TAA-eligible, displaced, job-threatened, and other adult workers. However, the Consortium also provided education and training opportunities to O&P professionals and others in the healthcare field through a virtual platform. The HOPE project served 1,873 unique participants in its 42 months of implementation, with 1,350 participants completing a grant-funded program of study.
Features of the Study
This evaluation used a mixed-methods design to gather quantitative and qualitative data from a variety of sources, including project records, a fidelity assessment tool (Adherence of Implementation Self-Assessment), interviews with key stakeholders, participant and partner surveys, and participant focus groups. Evaluators conducted interviews annually with project staff. In Years 1 and 2, a total of 12 and 13 staff members participated, respectively. In the last year, 54 individuals participated, including faculty and staff, Consortium leadership, and partners. The web-based partner survey was administered twice throughout the grant. Partners were identified for participation by each of the five Consortium members. The evaluators received 36 responses in each round. Student Entrance Surveys were administered to participants who signed a consent form and enrolled in a program that was one year or longer. Overall, 172 students participated in Entrance Surveys. From Spring 2015 to Spring 2017, 104 participants completed Student Exit Surveys, although demographic information was not included. Participant focus groups involved 68 students across the five Consortium members.
The fidelity assessment embedded in the implementation evaluation spanned the grant’s implementation. It included three indices: the Adherence Index, the Quality Index, and the Participant and Partner Responsiveness Index. Descriptive statistics were conducted for the Partner and Student Exit Surveys and other relevant project records. Qualitative data were analyzed using qualitative analysis software. The fidelity assessment tool, the Adherence of Implementation Self-Assessment, contained 28 indicators, organized by the seven core components of the project. It was completed by the consortium project director and site project managers on a semi-annual basis. The self-assessment was developed collaboratively by the evaluators and the HOPE project site managers.
Study Sites
- Baker College in Flint, Michigan
- Century College in Mahtomedi, Minnesota
- Oklahoma State University Institute of Technology in Okmulgee, Oklahoma
- Spokane Falls Community College in Spokane, Washington
- St. Petersburg College in St. Petersburg, Florida
Findings
Intervention activities/services
- The study found that partnerships were an integral part of HOPE project activities, and partners contributed through participation in advisory committees, curriculum review, recruitment and outreach, hiring graduates, and donating equipment.
- The study found that the HOPE project institutions developed and enhanced 19 short-term certificates and degrees in the O&P field. They also implemented a rigorous quality review process for their curricula and programs, made significant upgrades to their equipment and labs, and collaborated effectively across the consortium and with external partners.
- The study found that connecting with workforce centers, recruiting TAA-eligible participants, and identifying opportunities for job placement in a saturated labor market were challenging for the Consortium. Curriculum development also took longer than anticipated.
- The study found that HOPE students perceived the quality of instruction they received to be high and that they were satisfied with the academic advising they received. The majority of students were also pleased with the lab and training equipment and felt that it helped facilitate their learning experience.
- The study found that the partners who completed the Partner Survey reported moderate to low levels of involvement in project activities. However, the majority indicated satisfaction with their level of involvement and agreed that the project was having a positive impact on the O&P industry.
Fidelity
- The study found that the Consortium implemented all seven of its core components according to its logic model with a high level of fidelity (93 out of 100 possible points, according to the Adherence of Implementation Self-Assessment).
Implementation challenges and solutions
- The study found that curriculum development took longer than anticipated and connecting with workforce centers was a challenge for the Consortium, often due to the workforce centers’ lack of familiarity with O&P. Further, low employment levels made TAA-eligible participants difficult to recruit. Two participating institutions also struggled with turnover of career navigators.
- The study found that, to address the challenges noted above, the Consortium used diverse recruitment strategies, and concentrated outreach at the local level resulted in the increased prominence of O&P.
Considerations for Interpreting the Findings
The implementation study was comprehensive in its design, data collection, and analyses. The findings aligned with the research questions and were well-supported by the data. The authors cited low response rates to online surveys from partners and participants as a challenge. They did not note any additional limitations, although the challenge of following up with students who exited the program prior to completing their program of study may also be considered limiting. Fidelity was assessed by the study authors and not by the CLEAR team.