April 16, 2014

An explanation of how ACE accredits MOOCs

Listen with webReader

ACE2 Book, P. (2103) ACE as Academic Credit Reviewer–Adjustment, Accommodation, and Acceptance WCET Learn, July 25

Over the next few weeks, WCET will publish a series of blog posts on Massively Open Online Courses. This, the first in the series, provides a detailed explanation of how ACE (the American Council on Education) assessed five Coursera courses.

ACE represents the presidents of U.S. accredited, degree-granting institutions, which include two- and four-year colleges, private and public universities, and nonprofit and for-profit entities.

Pat Book, the author of the post, is a Former Assistant Vice President at the American Council on Education, and led the process for assessing the courses.

Highlights from the process

For those of you who wonder how the accreditation process works in the USA, this is fascinating reading. Here are some highlights from Pat Book’s post:

ACE finds itself in the awkward position of advocating for the best interests of their institutional members while at the same time serving as a shadow accrediting body distributing the ACE imprimatur (defined by those very member institutions) to a host of newly emerging for-profit ventures whose mission and goals are very different.

Reviewing academic courses taught by faculty at top tier universities was a new venture for ACE as its CREDIT recommendation service was not designed for nor … ever had been deployed for this type of review.  ACE leadership was anxious, as was Coursera, to address the major topic of discussion last year about whether or not MOOCs were credit-worthy.

The initial courses subject to ACE review were selected by Coursera in consultation with their partner universities (which included the University of California at Irvine, and Duke).  Coursera and the partner universities chose courses that were already offered on campus or were using content similar to an on-campus course.

All five courses reviewed received credit recommendations based on ACE’s review criteria.  The five courses received math and science recommendations, one at the developmental math level, that is, three-credits of pre-college, three at the lower division baccalaureate level, all three credits, and one two-credit recommendation at the  upper division baccalaureate level.  Faculty reviewed all course exhibits including learning outcomes, competencies, and assessment methods.  Faculty made suggestions regarding perquisites and offered other notes.  While ACE has recommended academic credit, it is up to each university or college to review these credit recommendations and determine how they may align with their general education requirements or degree programs.  There is no guarantee that any university of college will accept the ACE credit recommendations.

 ….it seems like a foregone conclusion that the courses Coursera self-selected for review would be highly likely to receive an ACE CREDIT® recommendation.  They were courses developed by faculty and already reviewed for credit in their university system in some cases and just being offered in a new delivery method albeit to a massively scaled audience.

The review process doesn’t evaluate learning outcomes, but is a course content focused review thus obviating all the questions about effectiveness of the pedagogy in terms of learning outcomes. 

 MOOCs currently serve largely an international audience who already hold college degrees and have reasons other than degree attainment motivating them.   The jury is still out on the value for the vast majority of American students who need developmental education and/or are seeking affordable access to college credentials.

Comment

First, thanks to Pat Book for making this process transparent. We are better informed about the meaning of ACE’s accreditation for MOOCs as a result.

My concern though is that ACE accreditation misleadingly suggests that Coursera courses have been approved by the American post-secondary system (represented by ACE). In fact what the ACE accreditation does (as explained by Pat Book) is merely accredit courses from institutions that are already accredited. However, it seems that a commercial organization (Coursera) has consequently received enormous marketing value for almost no cost (the article makes it clear that reviewers are paid almost nothing to do the reviews.)

More importantly, the article makes it clear that the MOOCs were accredited solely on the quality of the content. This though does nothing to address the main criticisms of MOOCs: that they employ unsuitable pedagogy for online delivery, and that the student assessment process is fundamentally flawed. ACE accreditation in essence does nothing to assure learners that they might actually be able to complete successfully such courses, or that if they do so their certificate will be transferable for credit within regular programs.

I think that we really need to squash the idea that Coursera MOOCs offer a meaningful, radical alternative to conventional higher education, and focus on their value as educational broadcasting, notwithstanding their important value in forcing many elite institutions to take much more seriously for the first time the potential role of credit-based online learning.

MOOCs, MIT and Magic

Listen with webReader

MOOC panel: Dan Hastings, Anant Agarwal, Tony Bates, Sanjay Sarma, John Daniel

In my previous post, there were two sessions at the LINC 2013 conference that referred specifically to MIT’s own strategies for technology-enabled learning within MIT. These resulted in my asking the following question towards the end of the conference:

Why is MIT ignoring 25 years of research into online learning and 100 years research into how students learn in its design of online courses?

This post then will discuss both why I think this is the case, based on MIT’s own presentations at the conference, and the broader implications for educational research and instructional design.

MOOCs, MIT and Magic

The first session at the LINC conference was on four perspectives on MOOCs. There were four speakers before the coffee break, then the four speakers formed a panel to respond to questions from the audience after the coffee break. All four presentations are available in full from here, so I will provide a very brief summary of the main points made by each presenter. The session presenters were introduced by Richard Larson, Director of LINC.

First though, MIT’s Chancellor, Eric Grimson, laid out the reasons why MIT is making such a large commitment to OpenCourseWare, MOOCs and edX, and these reasons were reinforced by other MIT speakers:

  • to rethink the campus experience in the light of developments in online learning
  • increase access to learning worldwide by making MIT resources and courses available to anyone, anywhere
  • to conduct research on learning, especially by mining and analyzing the large amount of data generated by MOOCs
  • Anant Agarwal, the Director of edX, also later added: to develop an open source platform for (massive) online learning.

Sanjay Sarma, Director of MIT’s Office of Digital Learning, opened the session. He made the distinction between MOOCs as open courses available to anyone, reflecting the highest level of knowledge in particular subject areas, and the ‘magic’ of the on-campus experience, which is distinctly different from the online experience. He argued that it is difficult to define or pin down the magic that takes place on-campus, but referred to ‘in-the-corridor’ conversations between faculty and staff, hands-on engineering with other students outside of lectures and scheduled labs, and the informal learning that takes place between students in close proximity to one another. Not mentioned but implicit was also the very high standard of students admitted to MIT, and the impact of continuous contact on campus between student and professor, none of which of course is available to MOOC students. MOOCs however as well as providing a route to high quality learning for self-directed learners can be also be re-used and incorporated by other instructors in other institutions for credit.

Sir John Daniel took a much more critical view of MOOCs, suggesting, using the Gartner ‘hype cycle’, that MOOCs would soon enter the ‘trough of disillusionment’ and reflected on whether or how MOOCs will reach the “plateau of productivity”. He also pointed out that open and virtual universities in both developed and developing countries have been providing open and distance learning on a massive scale for over 40 years, and these initiatives have provided high quality and recognized qualifications.

Professor Anant Agarwal, the President of edX, provided some facts and figures about edX MOOCs, and mentioned that MIT had awarded a scholarship to the 15 year old Mongolian student who scored 100% on the final exam of an MIT MOOC course (although he will not receive credit for it). He pointed out that although over 150,000 learners enrolled in edXs first MOOC, 26,000 did the first activity, and 7,000 went on to complete successfully the certificate based on an online exam. (This woud provide a completion rate of approximately 28%, which is probably the most valid way to calculate completion rates for MOOCs.) More importantly, Agarwal defined the pedagogical ‘innovations’ in MOOCs as follows:

  • active learning: short video lectures interspersed with student tests/activities
  • self-paced learning
  • instant feedback
  • simulations/online labs to teach design of experiments
  • peer-to-peer learning.

Some of this has been made possible by MIT engineers building original software for automatic grading or feedback, including enabling students to write formulae as answers.

Me, MOOCs and pedagogy

I was the last speaker in this session and focused on the pedagogy of MOOCs, and suggested some ways in which they could be improved, based on 25 years of research in online learning. In summary the basic points I made are as follows:

  • MOOCs face several challenges, in particular low completion rates, problems with student  assessment, especially for assessment that requires qualitative or essay-type answers, and poor Internet access in developing countries
  • there is 25 years of experience and research into what works and what doesn’t in online learning
  • by and large, this knowledge is not being applied to the design of edX or Coursera MOOCs, which are based mainly on video recordings of classroom lectures
  • paying more attention to pedagogical issues and instructional design could help mitigate some of the challenges
  • in particular more attention needs to be paid to skills development, knowledge construction/deep learning and learner support
  • research should focus on course designs that focus on skills development rather than the transmission of information, on how to scale up learner support and oncosting models that provide resources for improved learner support
  • MOOCs should not be ‘second best’ for developing countries, replacing more locally based provision
  • for all this to happen, computer specialists and educators/instructional designers need to work together as equals

A copy of my presentation can be obtained by sending me an e-mail (tony.bates@ubc.ca) and I will send you an invitation via Dropbox to download the slides.

Technology-enabled learning: what’s going on at MIT?

This was the title of another session that described in more detail MIT’s other technology-enabled activities besides MOOCs. First I need to describe how MIT organizes its technology-enabled teaching and learning, based on the Executive Director of OpenCourseware, Cecilia d’Oliveira’s, clear presentation about 10 years history and the organizational structure of educational technology initiatives at MIT.

The Office of Digital Learning

Most of the better known MIT activities in this area come under the umbrella of the Office of Digital Learning, whose Director is Dr. Sanjay Sarma, a Professor of Mechanical Engineering.

Within this division is MIT OpenCour.seWare, which collects and provides a portal for video recordings of lectures and support materials that faculty have agreed can be shared openly. Currently MIT OCW offers materials from 2,150 MIT courses, plus courses from more than 300 universities worldwide. However, these are open educational materials (OERs), not full courses. Cecilia d’Oliveira, whose background is mainly in IT, is the Exective Director of MIT OpenCourseWare.

Also within the Division of Digital Learning is MITx, which works with faculty and academic departments to develop MOOCs (massive online courses, including currently 16 available at the moment through edX), and is responsible for the platform used not only for its own online courses but also for other edX courses. Some of these courses are available to MIT students for credit, as well as being open to other learners (but without credit).

While edX uses the MITx platform (which is open source and open to other developers) for its courses, edX is a ‘portal’ or stage for bringing together the MOOCs from MIT, Harvard and other partners in edX, such as UC Berkeley. There are currently 26 universities contributing MOOCs to edX, which is a non-profit organization supported mainly by a grant of $60 million from MIT and Harvard. Professor Anant Agarwal is the President of edX, and is Professor of Electrical Engineering and Computer Science and was formerly Director of the Computer Science and Artificial Intelligence Laboratory.

Media Production Services has approximately 25 staff who help with the video capturing and production for online courses, OCW, and other technical services.

Lastly, the Office of Educational Innovation and Technology (OEIT) is also within the Office of Digital Learning. OEIT works with faculty, staff and students to enable and promote the development and dissemination of innovative uses of technology in teaching and learning. Its focus is on innovative software and hardware to support learning and teaching. For instance, the ARTEMiS project is developing high-quality visualizations by applying the principles of visual communication and using the tools of modern computer graphics to create visualizations that accurately portray scientific and technological concepts. OEIT also maintains four physical Experimental Learning Environments (ELE) and a small pool of laptops for flexible deployment for innovative curricula.  These spaces are intended as incubators for testing new or different technologically enhanced pedagogical paradigms.  These physical spaces host a suite of technologies, applications and tools. The Director of OEIT is Dr. M.S. Vijay Kumar, who has a doctorate in academic computing in education.

Office of the Dean for Undergraduate Education

Also, within the Office of the Dean for Undergraduate Education, the Teaching and Learning Laboratory (TLL) collaborates with faculty, teaching assistants, and students to promote excellence in teaching and learning throughout the Institute, assisting with MIT-wide innovations in pedagogy, curriculum, and educational technology in STEM teaching and learning.  It also conducts research in teaching at MIT. Dr. Leslie Breslow, Senior Lecturer, Sloan School of Management, is the Director.

The Office of Faculty Support, provides support to help the faculty develop and coordinate the undergraduate curriculum and educational programming and provides grants and opportunities for faculty development . Professor Diana Henderson, Professor of Literature, is the Director.

Ten years of learning technology innovation at MIT

Research into teaching and learning at OEIT

There has been over 10 years of research and experimentation in teaching with technology at MIT. Brandon Murumatsu of OEIT described two current research projects. The first was the development of an online version of an MIT ‘hard’ course in mechanics and materials, traditionally delivered in class mainly by one hour 25 minute lectures and supported by problem sets that are done as homework. In the distance version, the lectures are recorded with a TA taking notes on the different topics or ‘chunks’ addressed during the lecture. This enables  the online video to be embedded in an interactive web page that is indexed and linked to the different ‘chunks’ of the lecture (see diagram below). Students can therefore search quickly for different parts of the lecture, slow down, speed up or repeat each ‘chunk, etc. No information was given on how successful this was.

The second experiment was a take a ‘flipped’ lecture class and embed assessment with immediate feedback into the lecture, through the use of simple multiple choice questions. This enabled a large set of data to be collected and anlyzed about how students responded to the different parts of the lecture. It was reported  that students love or even dream of getting the green checkmark when they get the answers right.

MIT student responses to online learning

The last session was in some ways the best of the conference. Three MIT students presented on their experiences of online learning. Sam Shames reported how he used online learning for his project work, exploring the ‘universe’ of online, open resources (in particular OCW) to help with problem-solving and project work, and in particular the opportunity it provides for students to find individual pathways through online OERs.

Ethan Solomon reported on his experience of taking four MOOCs. The good:

  • the ability to go over materials again and again
  • ability to go at one’s own pace
  • immediate feedback

The bad:

  • the limitation of multiple choice questions
  • MOOCs are mainly just lectures
  • difficulty of organizing massive numbers of students, especially in discussions.

Comments

I found the conference fascinating, for many reasons, but here are the main points I came away with.

1. MIT is making genuine efforts to open up its teaching, its materials and opportunities for learning across the world. It has invested very heavily in this, and many institutions, instructors and learners outside MIT are taking advantage. The quality of the content is often outstanding.

2. MIT is still tied though to the lecture as the main means of delivery for online learning. In fact, the MIT students on the panel showed that they understand the need to adopt a different approach to online learning better than the faculty.

3. MOOCs are the consequence of lecture capture technology. This technology makes it easy to move teaching online, but without changing the design of the teaching. This usually results in information transmission becoming the primary pedagogy, without addressing the many limitations of lectures, except the ability for asynchronous access, which is an important improvement on the ‘live’ lecture.

4. MIT  is using a behaviourist approach to its online learning, based mainly on Skinnerian thinking and research. Long lectures are still a core part of its campus pedagogy as well, but there is additional ‘magic’ provided on campus (informal and experiential learning and close contact with faculty) which is not available to its online learners. In my view, it is a mistake to believe that such ‘magic’ cannot be created online. It can, but it needs good course design based on sound educational principles.

5. If instructional designers exist at MIT, they play a minor role or have little power. This shows in both the design of its MOOCs and in the research being conducted.

6. In my view, MIT will struggle to make an impact on educational research if it continues to ignore the potential contribution of educators. It is as if researchers such as Piaget, Bruner, Vigotsky, Carl Rogers, Gagné, and many later researchers had never existed. Can you imagine anyone trying to develop a new form of transportation while deliberately ignoring  Newtonian mechanics? Yet this is what MIT is doing in its educational research. In fact, as the research described above shows, they are re-inventing the wheel. It was admiited that many of the results they are getting are not new but have been known for many years.

For me this is a tragedy. MIT’s engineers have so much to offer in helping to improve educational technology but it needs to be informed and embedded in theories of learning, and must take account of prior research, for it to gain traction and be of value. This means working in a team with educators who have the design and research knowledge and experience, and working with them as equal partners.

Of course, MIT does not need this advice. It is immensely successful and will continue to produce great engineers. But it could also do so much more.

Having said all that, I learned a great deal from the conference, was treated with immense courtesy, and I am very grateful for the invitation to attend.

 

Who is giving credit for MOOCs?

Listen with webReader

Fitzgerald, M. (2013) Coursera courses approved for college credit InformationWeek Education, February 7

Short answer

Not many organizations – but the list is growing

This is a useful article that lists a number of schemes to give credits for MOOCs. Unfortunately it’s likely to be out of date already, given the pace of MOOC developments.

And be careful. A reputable organization such as the American Council on Education may certify certain MOOCs for credit but it does not necessarily mean the college or university that offers the MOOC will accept the ACE certification if a student wants to use that for credit.

A very small number of institutions have said that if students pass a recognized exam, then they may recognize their own MOOCs for credit, but they will not accept MOOCs from other institutions.

This highlights a really important issue about open education. For those looking to access post-secondary education, or to take their own choice of courses, it is no good courses such as MOOCs, or resources being open, if institutions won’t recognize them.

University admission policies are probably the biggest barrier to open education. There’s a reason for that – many institutions are funded on a capped FTE basis – in other words, if they take more students than the state or province has set for the institution, they don’t get the state or provincial funding for the extra students. Recognizing MOOCs may ease that in that there is no direct cost in accepting credits from another provider, but then the institution may lose that funding if the student is already part of the recognized FTEs. Whether a MOOC will be recognized for credit within a particular institution ultimately comes down to the view of individual academics and their view of a MOOC and how it fits with the requirements of a particular program, which is not terribly encouraging. You can see how messy this can all get, and it’s why universities in particular have such complex and niggly admission requirements.

It will be interesting to see whether, in the long run, MOOCs make a dent in the shortage of university places, if at all, or whether they find their value outside the system as general education programs, much like educational broadcasting in the past.

What is clear is that they are likely to muddy an already murky process for admission to university programs for those who don’t fit the standard GPA system, which surely is long due for an overhaul (why should my score in a physics exam when I was 14 deny me access to a science program when I am 30 and have work experience and later qualifications?). Students need clarity rather than confusion about what they need to do to get a university place, but above all they need a rationale and fair system with as few barriers as possible. MOOCs are not particularly helping with this, but they are opening up the discussion about what should count for credit in or admission to a university program, and it’s about time.

 

Keeping up with MOOC developments

Listen with webReader

 

How are you keeping up with MOOC developments? If you are like me, you are probably feeling swamped and not a little overwhelmed by all the coverage and news about MOOCs. Here’s what I’ve come across over the last couple of weeks.

MOOCs as an entry to a degree program

Kim, J. (2013) Q&A with Randy Best on MOOC2Degree Inside Higher Education, January 24

Joshua Kim interviews the CEO of Academic Partnerships about its plan to underwrite MOOCs that will guarantee entry to a university degree program with institutions that have partnered with Academic Partnerships, so long as the students successfully pass an examination set by the institution at the end of the MOOC. This project will use Instructure’s Canvas as the platform. Eight universities have so far signed up. The MOOCs will be existing online courses in the degree program. Students will receive full credit for successful course completion.

In this model, a university makes its own, already designed online, for credit programs, open to anyone, and if students successfully pass the course, they become admitted to the full program. This is really a big step towards opening up previously highly selective institutions.

I think this is a very interesting model. I pushed for a somewhat similar model for graduate courses at UBC, whereby we offered an ‘open’ online certificate program, but pushed for students who succeeded to carry over the credits into a full masters program. I was unsuccessful – the Faculty of Graduate Studies insisted on students meeting graduate entry requirements – even though these ‘open’ students were getting the same or better grades than other graduate UBC students taking the same courses (the certificate courses were also available as credit courses to the full-time graduate credit students).

Institutions have unnecessary and often arbitrary restrictions to entry and any model that breaks this open is to be welcomed. This one is tied to learner performance, which is properly measured and assessed.

MOOCs for credit

Fain, P. (2013) As California goes? Inside Higher Education, January 16

San Jose State University has signed a deal with Udacity:

‘to create a pilot program of three online, entry-level courses that will cost students $150 to take and lead to university-awarded academic credits if passed….The university will cap enrollment at 100 for each of the three courses, with half of the slots going to students from San Jose State. Priority enrollment for the remaining 150 openings will go to high school and community college students, members of the military and.’ veterans, and wait-listed San Jose State students.’

My question is: why is this a MOOC? It’s not massive and it’s not open and it’s not free. This sounds very similar to many existing programs aimed at enabling more open access to otherwise ‘closed’ programs, such as prior learning assessment.

Preparation for challenge exams

California community colleges, faced with a shortfall of 500,000 place in its campus-based colleges because of state funding cuts, are considering:

creating examinations for remedial courses and core general education courses for an associate degree aimed at students who want to transfer to a California State University campus. Students could use MOOCs to prepare for challenge exams, and community colleges could steer them toward the free online courses. And MOOC providers could tailor their offerings to the exams and gateway courses.

Then what? Having passed the challenge exam, there are still no places on further courses.

The problem is that California is in a financial mess, and there is a lot of flailing around to find cheap ways of providing post-secondary education. MOOCs are seen as a possible answer, but the issues of quality, learner support and assessment for MOOCs are not going to be resolved by wishful thinking. For more on the California situation, see ‘California buzzing’, which suggests other, and in my view, better ways in which online learning can help.

One last comment

MOOCs are a very interesting development, and have some potential to bring about major changes in the post-secondary education system.

However, they are only a side show to most online educational developments. Many other interesting things are happening and these are being drowned out by the hysteria and hyperbole surrounding MOOCs. It seems any new development in online learning has to be called a MOOC to get any recognition (even if it is neither massive nor open).

We need to get back to a sense of proportion here. It’s not the number of enrolments that matters, but the learning that takes place. For-credit online programs have had to prove that students can learn just as well online as on campus. There is over 20 years experience of what works and what doesn’t in credit-based online learning that is being ignored in most (but not all) MOOC developments. Not a single MOOC has been able to demonstrate clear learning gains for the students (or a viable financial model, for that matter). When that happens, they deserve to be taken seriously. Until then, I suggest you focus on the real world.

 

Another way to rank universities?

Listen with webReader

Oxford University

Mundell, I. (2013) EU rolls out university ranking European Voice.com, January 24

The European Union is promoting a new system of rankings for Europe’s universities to encourage international comparison. The system, called U-Multirank, aims to correct a perceived bias towards research performance in other international rankings, and so present a more balanced picture of university activities.  The perceived research bias of existing rankings is seen as problematic because it fails to recognise that universities may have other goals and their users may have other priorities. The EU initiative aims to bring out these other university activities and allow institutions to be compared accordingly.

The possibility that U-Multirank offers an alternative has been cautiously welcomed in some quarters, but rejected in others. The League of European Research Universities, which represents 21 elite institutions including Oxford and Cambridge, left the U-Multirank pilot project and remains opposed to its implementation, particularly with public funds.

Comment

First, I welcome a move to provide any alternative system to the fundamentally flawed university ranking models currently in use, which certainly overvalue research and undervalue teaching. The main concern though is that current rankings are deliberately rigged to promote the standings of existing elite universities in the USA and to a lesser extent in the UK. The EU is trying to redress that.

However, I doubt whether it is feasible or even desirable to reduce a complex organization such as a university to a single numerical ranking. In any university, some departments will be better than others. Different students will want different things from a university. Rankings tell you nothing about the flexibility they provide for learners. University rankings are an attempt to rig a simple metric to drive high international fee payers to certain institutions. Don’t play or support this stupid game.

See also: World University Rankings: A Reality Based on a Fraud