Confessions of a MOOC Reviewer

In our on-going series of blog posts on MOOCs, Elizabeth Allan shares her experiences as a reviewer of the “credit-worthiness” of a science MOOC.  Previously, Patricia Book gave us her insight into the overall ACE review process.  In this post, Elizabeth gives us the point-of-view of a faculty member who conducted the review.  WCET thanks them both for the insights into this process.

As an American Council on Education (ACE) CREDIT reviewer, I had the unique privilege of being on the team that reviewed the first MOOCS (Massive Open Online Course).  The experience was unique due to the course delivery.  In the end, the process was the same used in every other review for ACE.

I’m often asked about the process and how the review was done.  Most of the time the person asking is really wanting to know how a faculty member can be confident in the recommendation.  The answer, I believe, is in the process.  The faculty review team was led by a national coordinator with extensive experience in teaching and evaluation and consisted of faculty current in the academic discipline under review.  We all had extensive experience in reviewing online courses and in essence, this was a review of an online course – with obvious differences.  More on that later.

What Happens In A Review?
For all ACE reviews, we consider student learning outcomes, the intensity of the course, pre- and post-course assignments, qualifications of faculty, and academic and work-related experience of the participants.  Reviewers work together to review the Content, Scope, and Rigor of the course including: course syllabi, textbooks, assessment methods, student and instructor guides, student projects, instructional materials, and instructor qualifications to name just a few.

One thing we keep central in our focus is that we have the charge to consider recommendations based not on our institution, locale, or region but to critically appraise materials from a national/professional perspective.  It’s not our job to review a program based on what we do at our local campus, but instead to review each course in light of the ACE guidelines and best practices.  It is not an easy thing to do, but that’s where having a team and coordinator is vital.

How Did MOOCs Differ From Other Courses?
So what was different about the MOOC? Not that much. The course was reviewed using the ACE review criteria, no special considerations were given – the courses had to meet the requirements for Content, Scope, and Rigor.  A credit recommendation was given because the team determined that the course met the requirements.

There were, of course some obvious unique issues to address.  With so many students, how was feedback given to students? How were student issues addressed? How do you ensure the integrity of the assessment process?  And I’m sure you can imagine the discussion the team had as well as the questions we had for the course designers and instructors.

Most of the answers to the above are part of the huge proliferation of articles, etc. about MOOCs and don’t need reviewing here.  What is significant is that the MOOCs I reviewed were science courses and did not have to address much more difficult issues that would have arisen from a course requiring essays, etc.  As is probably true for a great number of science courses (both online and face-to-face) on your campus and mine, the grade was predominately based on test scores and homework.  There are companies that can be used to help validate identities and help in test security and the MOOCs employed one for these courses.

I Was Pleased With What I Saw, But the Proof Is In The Student Learning Data
So, in the end, the review was very similar to those conducted on other online courses.

There was certainly pressure (to be sure) to be thorough and exact in our review as well as being sure to have as much data and evidence to support our recommendation. But, in the end, the course did a good job with instruction in the content, the content was accurate and at a level that matched the stated learning objectives, was of an appropriate scope for an introductory course, and was a rigorous course.   To be honest, the content and delivery of the courses were such that I would be happy to see these elements in more courses that we review.

Some will say that MOOCs are the ‘death” of teaching as we know it, some say they are going to ‘transform’ education.  Maybe. Maybe not. Those are questions that have yet to be determined.

In reviewing these courses, I saw an online course that gave students the opportunity to learn content in an engaging manner.  The downside was the limited student interaction with faculty. Perhaps the answer for faculty, as has been discussed many other places (i.e., George Mehaffy’s Red Balloon Project) is something in between.

As a faculty member, I’m waiting to see what the student learning data tell us.  For me, the proof is in the pudding.

Elizabeth Allan
Associate Professor
Biology
University of Central Oklahoma

Dr. Elizabeth Allan is an Associate Professor of Biology at the University of Central Oklahoma and the Coordinator of the Secondary Science Education Program. She serves as the Director of the Central Oklahoma Science Fair. Dr. Allan has developed and taught on-line, blended, and face-to-face courses in both Biology and Science Teaching Methodology. 

Before returning to Oklahoma in 2005, Dr. Allan was the Director of the Western Carolina Center for Mathematics and Science Education, a part of the North Carolina Mathematics and Science Education Network.  She has been a classroom teacher in Oklahoma, California, and North Carolina.

Dr. Allan is the Retiring President for both the National Science Education Leadership Association and the Oklahoma Science Teachers Association. She has been an ACE reviewer for the MIVER program, ACE CREDIT, and a Military Evaluation reviewer. She was part of the team that reviewed the first MOOC to be awarded an ACE CREDIT recommendation.

Dr. Allan earned her Bachelor of Science at the University of Oklahoma and her Masters and PhD from the University of California, Riverside.