Author: WCET Frontiers

Today we learn with Deb Adair, Managing Director and Chief Planning Officer, Quality Matters and Julie Porosky Hamlin, Executive Director, MarylandOnline; Member, Quality Matters Board of Directors their perspective on quality assurance for alternative higher education.  Thank you Deb and Julie for lending your many years of experience in assuring the quality online education to the continuing discussion of alternative higher education.

In the fall of 2013, the Presidents’ Forum (operating out of Excelsior College) and the Council for Higher Education Accreditation (CHEA) created a Commission on Quality Assurance and Alternative Higher Education.  They invited 26 individuals to discuss, over a three-month period, the idea of a quality review for the growing number of non-institutional providers of (largely online) courses.  In August of 2014, they published a paper, Quality Assurance and Alternative Higher Education: A Policy Perspective.

The group identified six questions for further inquiry.   Quality Matters (QM) did not participate in the discussion and we would like to contribute perspectives from our 10+ years in the field of quality assurance for online education.  Up to now, the QM rubrics and peer review process for certification of online quality have focused on the course level, but a program-level certification will be launched this year.

It’s a priority for us to be sure QM meets the needs of alternative providers; fits our tools and processes to their teaching and learning formats; and includes the voices of their representatives in our QA—quality assurance—community.

We’ve organized our thoughts around the six questions.

Would a Review of Alternative Providers Provide a Viable Public Service?

Q: Would a quality review process for alternative providers of postsecondary education offer effective documentation of quality and credibility to the public, including students, policy makers, and employers, providing a useful and viable public service?

A: Effective documentation must be a goal, perhaps the most important goal, of a quality review process that would serve the needs of alternative providers and provide QA for the consumers of their educational products.  Consider how useful it would be if we had a set of format-agnostic quality benchmarks that enable all stakeholders to compare courses offered by non-institutional providers with those offered by academic institutions. For that matter, comparisons within these sectors would be enabled as well. Using comparative data, students would be able to make informed choices, and policy makers would be provided with a common understanding of what constitutes a threshold of quality.

A common definition of online course quality would also enhance the transcript review process used now by academic institutions to accept and assign credit for courses from other providers.  It would offer academic institutions an efficient way to move beyond a review of course content and instructor credentials and to include other factors known to be important to student success.

We at QM have seen the benefits of creating such a threshold for quality and a process for benchmarking across institutions. Very often, the QM course review process and certification have been instrumental in intra-institutional direction setting and collaboration across units.

Many inter-institutional academic collaborations also rely on the QM standards and course reviews to support course-sharing initiatives.  In fact, the fastest-growing segment of QM subscribers are those participating as systems of institutions.  Currently, more than 60% of all subscribers do so as part of a consortium or system of institutions. We expect that the large number of QM subscribers who have invested significant time and effort to ensure the quality of their own courses will advocate for similar standards to be required of “credit-intended” courses offered by competing non-traditional providers.

What About a Cost-Benefit Analysis of a Quality Review Process?

Q: Would a preliminary cost-benefit analysis of a model quality review be informative? If so, how might this be done?

A: Once agreement is reached on the factors to be included in such a review, we can compare different review models already in existence to better understand the tradeoffs to be considered in understanding costs and benefits.  There is the traditional model of higher education accreditation, with significant expense in both time and money for on-site review.  The affordances offered by online education, the format utilized by a majority of alternative education providers, support a more cost-effective process of online, rather than on-site, review.  We offer Quality Matters’ approach as one model used to ensure the quality of online course (and soon program) design.

A QM higher education Peer Review team is composed of three trained and certified online instructors, at least one of whom is a subject matter expert and at least one of whom is external to the institution hosting the review.  The team, chaired by a Master Reviewer who has additional training and experience, is charged with taking the student perspective, applying the 43 QM standards with guidance from the annotations for each, and writing recommendations for improvement.

Recruiting, screening, and training the “talent” for conducting quality reviews is a challenge in itself.  QM has had more than a decade to figure it out, and we’re still fine-tuning and continuously seeking improvements.  Our online database contains more than 4,000 Peer Reviewers and Master Reviewers from across the country, and now internationally. A review typically takes three to six weeks for the first assessment and report; however, faculty/course developers have a total of 20 weeks to get through the review, including, if needed, time to make the recommended course improvements.  QM can and does manage these reviews for a fee; however, institutions may self-manage these reviews with proper training and certified reviewers.

As a result of a QM review, which is open and collegial, the faculty/course developer receives a report with improvement recommendations as well as certification of the course.  Using the Rubric and experience from benchmarking reviews, many institutions are now routinely training faculty and designing their online courses to fit the standards, an approach that prepares the courses for a successful review.  In these ways, institutions have been able to manage the time and dollar expense of course-level quality assurance.

Are You Planning to Pilot the Proposed Quality Review Process?

Q: Would development of an experimental model provide a means to demonstrate and test a workable quality review process? What might that model look like?

As obvious as it may sound, an effective quality review process must be anchored by assumptions about what constitute quality and those assumptions must be codified or captured in what we most often call “standards.”  Perhaps somewhat less obvious is that the standards themselves must have a clearcut focus or target to support consistent application.

The QM Standards, as an example, are focused on supporting student success.  QM has identified factors that are “important,” “very important,” and “essential” to student success.  The QM certification mark is intended to ensure:

• a course is coherent and aligned with the course and module learning objectives:
• that the purpose of the course, course components, and their relationships is made clear to students;
• that the course is easy to use–in navigation, technology, and setting and communicating expectations– ensuring intra-course mechanics and extra-course requirements are not a barrier to learning;
• that the course is built to engage learners and promote active learning;
• that the content and assessment support the appropriate levels of learning;
• that the learner is guided to technological, academic, and student support; and
• that the course is accessible to all learners.

How did we come up with this particular set of standards?  Our flagship Rubric for Higher Education is now in its fifth edition, a clue that the standards have evolved.  Through a widely participative and in some ways messy process, QM standards are developed and regularly updated from a ongoing review of the research literature on student learning (see QM Research Library); from evidence-based practices; from analyses of review outcomes for the last 10 years; and from a large and growing community of practitioners actively engaged in using the standards.

QM’s mantra is continuous improvement, and our review process aligns with the quality assurances practices adopted in industry and other sectors with which education interacts. QM’s core elements are clear standards of quality for course and program design, training on the standards for faculty serving as course reviewers, and a review process focused on continuous improvement.

QM’s emphasis on continuous improvement is suggested in Figure 1, a back-of-napkin graphic that dates to QM’s earliest days as a grant project funded by the U.S. Department of Education.

Figure 1: QM Course Review Process

What About Competencies and Student Outcomes?

Q: Inasmuch as the offerings of many alternative providers are designed to enable the student to master or demonstrate specific knowledge or skills, would a quality focus that measured competence (student outcomes) be a productive approach?

A: The ability to determine quality in a competency-based approach to education is as important for traditional academic institutions developing competency models as it is for alternative providers.

QM has been actively following the activities and research surrounding the competency-based education (CBE) movement. Our initial response is demonstrated in the Fifth Edition of the QM Higher Education Rubric, which includes guidance for evaluating competency-based learning. Much more work is needed to understand and guide approaches to CBE quality. QM looks forward to participating in the QA effort through collaborative initiatives to survey the field of CBE and identify the issues that need to be addressed through policy and practice.

Would It Lead to Alternative Providers Qualifying for Federal Student Financial Aid?

Q: Would an external quality review process for alternative providers offer a potential pathway for these organizations to qualify to participate in federal student financial aid programs, if such an opportunity were available?

A: Postsecondary education now includes more teaching-learning formats (MOOCs, CBE, adaptive learning, gamification) than it did just a few years ago, formats that are being embraced and experimented with by even the most traditional institutions, and credentials themselves are undergoing a rapid evolution.  External quality review, with its assurance of objectivity, could be an important component in a trend already begun.  The recent granting of financial aid for students earning degrees in CBE programs at collegiate institutions suggests the federal government in the near future may consider supporting other alternative formats from alternative providers.

How Would QM’s Work Lead to Greater Acceptance by Traditional Higher Ed?

Q: How would greater cooperation or adoption of some form of third-party verification or certification of standards of practice shared among organizations that review courses or student learning for credit improve wider understanding, acceptance, and utilization of the work of these organizations by colleges and universities?

A: This question asks, in part, how to broker a deal between external organizations that have developed principles addressing educational quality (QM, ACE, OLC are examples) and the colleges and universities that must be convinced these principles and the processes for applying them are sound: sound enough to warrant entering into a college transcript a “course” or other package of learning from an alternative provider. For quality assurance of alternative education to be effective, all parties must agree on standards and buy in to an evaluation process.

QM was developed as an answer to the need for inter-institutional quality assurance in online learning for a consortium of higher education institutions. Participating institutions had worked out how to run a seat-sharing program to allow students to enroll in one another’s courses, but the problem remained to convince stakeholders that students would be receiving an equivalent quality learning experience regardless of where they took their course. A common metric, one that is valid, consistently and rigorously applied, and collaboratively developed, was the answer that worked for the consortium, and later for other consortia around the country.

In Conclusion

We at QM are excited about the new educational options brought by alternative providers and excited to participate in a broadened and inclusive conversation about quality.

Deb Adair,
Managing Director and Chief Planning Officer,
Quality Matters

Julie Porosky Hamlin,
Executive Director, MarylandOnline
Member, Quality Matters Board of Directors