Assessment and evaluation of microcredentials: What success looks like and to whom
Published by: Lindsey Downs | 9/29/2022
Tags: Credentials, Distance Education, Managing Digital Learning, Student Success
Published by: Lindsey Downs | 9/29/2022
Tags: Credentials, Distance Education, Managing Digital Learning, Student Success
Today, we conclude the WCET blog series on microcredentials which set out to address the inception, implementation, and evaluation of microcredentialing approaches across the higher education industry. We learned that there is a wide range of reasons for an institution to begin a microcredential initiative, an even broader spectrum of microcredential terminology, and yet two primary outcomes: “to improve student recruitment and retention” and “to align learning with shorter-term outcomes that can more immediately align with career progression” (Niles and Lazarewicz, 2022). Institutions and training providers use microcredentials to adapt to a number of trends such as decreasing student enrollment and workforce degree requirements and increasing acceptance of online learning, employer demand for skills training, and public demand for greater return on educational investment (Higher Learning Commission, 2022). As professionally trained and former instructional designers, we begin by defining what success looks like. In this blog post, we will present what we learned about how to assess and evaluate the efficacy of microcredentialing approaches through a variety of stakeholder perspectives.
“One of the positives from the pandemic has been the acceleration of innovation and change in higher education – microcredentials have grown in scale and areas of focus to help provide current students or adult learners additional knowledge to help them be more competitive in the job market. The assessment and evaluation of whether these microcredentials are making a difference for students and adult learners and whether employers value these micro credentials. The next phase of the growing area of microcredentials should seek to answer the questions about employer value, industry microcredentials, or higher ed developed micro credentials, measuring learning outcomes, and more,” shared Melissa Vito, Vice Provost of Academic Innovation at the University of Texas at San Antonio.
Microcredentials should create positive labor market outcomes in terms of employment and earnings. “If done correctly, microcredentials help students develop skills that today’s employers are expecting from their entry-level talent” stated Scott Shireman, Global Head of Coursera for Campus. “When microcredentials are built and recognized by industry leaders like Google, IBM, and Meta, they reflect the skills and tools most in-demand by top employers.” Shireman also invited us to consider how microcredentials impact students. “From a student recruitment perspective, the fact is that students today, and their parents for that matter, are thinking more critically about the ROI of the traditional four-year degree. We’re seeing that shift in mindset reflected in university recruitment numbers. It’s a very competitive landscape, and universities are increasingly leveraging microcredentials and job-relevant skills training to differentiate their brand and show students that they’ll benefit from multiple signals in a competitive and fast-evolving market – the two or four-year degree they graduate with, and the career certificates they earn along the way.” The new Coursera survey published shows high demand for industry microcredentials from students and employers in tight labor market.
Kate Smith, President of Rio Salado College, reinforced that student success is at the core of an effective microcredential: “Evaluation of microcredentials is tied to outcomes of students – retention, completion, post-completion employment, and wages, along with structured qualitative feedback from learners, employers, and related faculty and staff.” Microcredentials can provide students with early exposure to career choices, helping students to “build confidence to continue their post-secondary education” and also to reinforce or redirect career choices.
For former Mesa Community College student, Sam McFadden, a two-week, full-time “wire harness bootcamp quite literally changed my life, and has put me in a position to be successful in my career, and complete my education. Three years ago, it would have been difficult to imagine that I would be where I am today.” Today, Sam is employed by Boeing, the world’s largest aerospace company.
Jude Chudi Okpala, Professor of Instruction at the Department of Philosophy and Classics at The University of Texas at San Antonio explained what success should look like for students. “Success in a critical thinking microcredential, for example, consists in application of critical thinking on a concrete and real problem. Looking at students’ work in such microcredentials, one sees a correspondence between the objectives and the learning outcomes for the microcredential: for example, evidence of logic in addition to effective understanding of complexity and the relationship between the problem and solution. Invariably, a successful microcredential in critical thinking should reveal independent thinking and the students’ ability to solve a concrete problem. The microcredential should demonstrate without ambiguity the students’ skill in critical thinking/problem-solving.”
Lisa Larson, President of Eastern Maine Community College, Ed Design Lab, Designer in Residence believes “micropathways are the gateway to transformation” because this demonstrates “we are committed to transforming our institution to meet the needs of those who need us most.” Larson emphasized the need to measure the engagement, student and employer value proposition, and impact after completion. For example, getting employers to include microcredentials in their recruitment, hiring, and promotion practices are metrics of success. In short, Larson asks “Who did the microcredential make a difference for, and how can we tell the story of impact?” Microcredentials may also increase access for non-traditional students to “break through ceilings” and “earn a living wage” contributing to equity and economic growth in our communities. Larson advised that microcredentials should be “digitally discoverable”, so students can “own the learning upon receiving validation of the competency and skill.” The digital nature of a microcredential should “create the visibility for students to take every ounce of learning and experience with them.”
“Ideally a feasibility study includes a market review of demand, wages, stakeholders, and resources needed to build and implement the credential, combined with a SWOT analysis. The microcredentials are then developed with input and co-design principles when feasible, with local or national industry partners” (Kate Smith, 2022, personal communication). Data analysis plays a key role in evaluating the effectiveness and impact of scope, resources, risks and feasibility of a microcredential project. Consensus-building with campus stakeholders and employer stakeholders is critical to a mutual definition of what success looks like.
Lydia Riley, Chief of Staff of The University of Texas System (UT System) invites institutions to consider the “AGILE planning framework with frequent feedback loops that lay out the foundations of the microcredential programs, identifying strategic goals for revenue-generation or mission-enhancing endeavors”. All stakeholders expressed a desire for marketable skills, relevant, contextualized curriculum, and increased perspective of employers into curriculum. While success may be measured by the value proposition to learners, the microcredential must also be validated by employers. One of the most challenging elements when planning and assessing microcredential approaches “is the personnel capacity and stakeholder capacity; the people on the ground really designing and implementing into the institutional structures” (Lisa Larson, 2022, personal communication).
Leah Palmer, Executive Director at Maricopa Community Colleges Arizona Advanced Manufacturing Institute (AzAMI) emphasized the importance of data tracking and benchmark metrics to measure the impact on student success, costs to the institution, and sector demands for talents. Palmer recommends tracking the number of hits on your microcredential webpage, enrollments and trajectory, value propositions such as course completions, earned badges or certifications, and follow-up emails for employment-related status. We must create opportunities for enhancement and/or implementation of corrective actions during the microcredentials planning phase.
Micheal Voss, Dean of Instruction at Mesa Community College recommends an “annual threshold review of Key Performance Indicators (KPIs) and a more robust program review at least every 5 years. Ideally, program completers should receive follow-up surveys 12-18 months post-completion to gain feedback as to how the knowledge, skills, and behaviors learned have impacted the person’s employment or next step education status.”
Lori Lindenberg, District Director of Data Analytics and Strategy at Maricopa Community Colleges explained how their system of 10-independently accredited colleges shares data through the Student Clearinghouse Postsecondary Data Partnership (PDP) to assess the leading and lagging indicators typically associated with the guided pathways framework, such as credit momentum and time to completion. Measuring economic outcomes has proven a little harder for some institutions. Employment data is notoriously difficult to acquire yet increasingly important to track the impact of microcredentialing. Maricopa Community Colleges has a partnership with Equifax to study wage data (pre- and post-completion), annually for up to 10-years, but this has been traditionally focused on degrees and certificates. They are working to transcribe recognition of microcredentials into their student information system and also experiment with digital badging platforms. The country’s largest accreditor, the Higher Learning Commission, sees an opportunity to provide a quality assurance service for non-degree credentials, yet they note their current limitations in capacity and scope (HLC, 2022). Perhaps third-party validation will promote shared standards of quality and transferability of microcredentials.
Through the testimonies and insights gathered in this process, we have identified that different data points will be indicators of success based on stakeholder perspectives (Larson, personal communication, 2022). Institutions and employers should base their microcredential partnership on key values and standards such as transparency, validation, recognition, and reward of skills. Table 1 illustrates this point that the measures of success may be complex and individually defined. The value proposition of a microcredential often depends on one’s perspective.
|Affordable||Meet a specific workplace need that directly support industry job roles and job descriptions that are in demand||Engagement||Meet a specific workforce development need||Quality assurance|
|Short-term||Recruit skilled workforce||Learning||Student’s success||Curricular alignment|
|Re-careering||Diversity equity and inclusion strategies for recruitment||Authentic assessment||Enrollment||Employability of graduates|
|Upskilling||Retain and reskill/upskill current employees||Retention||Retention||Transparency of tuition, fees, and costs|
|Recognition of skills||Career pathways||Completion||Complete||ROI for students|
|Career progression||On-ramps to educational pathways||Completion|
|Higher wages||Off-ramps to employment||Revenue|
|New job or different job||Stackable educational and career pathways||Digitally discoverable|
|Pathways to economic self-sufficiency||Curricular alignment to employment||Portable for the learner|
|Gain more confidence in career path and future||Career-ready||Valuable to the learner|
|Confidence to continue engaging in post-secondary education||Industry engagement and advisory||Validated by the employer|
|Effective demonstration of the target skills||Equity opportunities for students|
|Authentic alignment of the learning objectives and outcomes||Community engagement|
|Understanding of how to apply the skills|
|Reflection on acquired competence and its corresponding outcomes–what the students should be able to do|
|Concrete professional growth|
|Relevance of the assessment and partnership with the industry/community|
It became quite evident that assessing and evaluating microcredentials will require rapid prototyping with inclusive feedback loops from students, faculty and employers. Microcredentials have shorter lifecycles and therefore continuous evaluation of KPIs and lead measures are critical. Table 2 shares examples of different KPIs for assessing and evaluating microcredentials that align to the ADDIE instructional design process.
|Needs Analysis||Design & Development||Implementation||Evaluation|
|Institutional capacity||Stakeholder buy-in||Time-to-market||Student surveys|
|Personnel capacity||Employer engagement||Minimally viable product|
|Feasibility analysis||Co-design between institutions and employers||Marketing leads||Student completion|
|SWOT analysis||Standards||Enrollments||Student intent fulfilled|
|Identification of skills gaps||Signaling completion||Alignment to industry jobs||Feedback about the positive outcomes of the microcredential|
|Technology needs||Transparency, validation, recognition and reward of skills||Employment opportunities|
|How and where the student shares the microcredential|
|Employer integrates into recruitment, hiring, retention, and promotion processes|
|Improve the design of the credential|
A heartfelt thank you to all the expert interviewees lending their time, leading this transformative movement in higher education, designing these critical learning experiences, and for helping us to measure what matters most. Interviewees included:
Higher Learning Commission. (2022, April). Evolving: Accreditation and credential landscape. Retrieved on September 20, 2022 from https://download.hlcommission.org/initiatives/2022StakeholdersRoundtablePapers.pdf
Lazarewicz, K., Niles, G. (2022, May 26). Considering Your “Why”- How Developing a Microcredential Strategy Will Help Your Students (and Your University) Thrive. WCET Frontiers. https://wcet.wiche.edu/frontiers/2022/05/26/how-developing-a-microcredential-strategy-will-help-your-students-and-your-university-thrive/
Shireman, S. (2022, September 27). New Coursera survey shows high demand for industry microcredentials from students and employers in tight labor market. Coursera Blog. https://blog.coursera.org/from-higher-education-to-employment/
Executive Director of Teaching and Learning Experiences, UTSA Academic Innovation
WCET Steering Committee, Vice President, Strategy & Advancement, Rio Salado College