July 5, 2015

Measuring the success of an open textbook

Listen with webReader

Book table of contents 2

I have just come back from two days at one of my favourite communities of practice, the British Columbian Educational Technology Users’ Group (ETUG) annual workshop, this year at Simon Fraser University, Burnaby, BC. (More on this workshop in another post).

I was there to report on the early response to my online open textbook, Teaching in a Digital Age. I thought it might be of interest to share some of my presentation as a blog post, because it raises some questions about how to measure the success of an open textbook.

Student savings

This is the obvious and most important measure of success: how much does an open textbook save by reducing one of the major costs of education? Here in British Columbia, the average annual cost of textbooks for BC post-secondary institutions is $1,200 per student, if they bought all the required textbooks new (which of course, many don’t).

BCcampus, which has an extensive open textbook project, with over 60 open textbooks currently available, has been tracking their adoption by post-secondary institutions in BC. They have found that to date (actually, April 15, 2015):

  • 146 known adoptions of Open Textbooks at 14 of the 25 public post-secondary institutions across British Columbia
  • A student cost savings ranging from $475 K- $700 K (see their post on how they calculate student savings to see why they report the savings as a range).

More importantly, other studies have shown that when open textbooks are available, students make greater use of them, and students tend to perform better, as they have them from the first day of the course.

However, my open textbook is aimed at instructors and faculty, and my main aim is not to save them money, but to make the book more accessible than if it had been commercially published. So I need to use other criteria of success. I also have to compare the response to the book to the effort and cost of doing the book.

Quantitative responses

There were several deliberate marketing initiatives:

  • March 2014 – April 2015: the use of my blog for draft chapters. However, my blog readership, which consists of mainly professionals in the field of online learning, is a secondary market, although a very important one, as they can bring the book to the attention of the main market I am trying to reach, instructors and faculty;
  • April 7, 2015: I posted the final version of the book to the BCcampus Open textbook web site on April 6, 2014, and wrote a blog post the next day announcing that the book was finished and now fully available in various formats;
  • April 22, 2015: the Western Inter-State Co-operative for Educational Technology (WCET), is a national (U.S.), member-driven, non-profit which brings together colleges and universities, higher education organizations and companies to collectively improve the quality and reach of e-learning programs. They had invited me to write a blog post for their newsletter, promoting the book, and they published this on April 22;
  • April 29, 2015: Contact North in Ontario organised a world wide media release about the book
  • May 5, 2015: Academica.ca picked up on the Contact North media release (as did some other specialist media) and published a short announcement about the book.

The BCcampus open textbook web site tracks the number of visits to the book. The following shows the site traffic between April 27 and May

Figure 2: Book visitors April 29 - May 28

Figure 2: Book visitors April 29 – May 28

Between April 7 (my blog post) and April 29 (Contact North’s media blitz) the site traffic had been averaging about 1,000 visits a day, but with considerable daily variations. You can see that after a peak around April 30, the number of visits started to decline. Figure 3 illustrates why:

Figure 3: Book downloads, April 1- May 24

Figure 3: Book downloads, April 1- May 24 (source: BCcampus Open Textbook project)

Up until April 7, most people had been accessing/reading the html version of the book direct from the BCcampus web site. From April 7, though, visitors started downloading the book and reading it off-line or via a mobile version. Figure 4 shows the total number of downloads

Figure 4: Book download statistics

Figure 4: Book download statistics, April 7-May 24 (source: BCcampus)

Thus there were 8,205 downloads of the whole book between April 7 and May 24, i.e. over a seven week period. Interestingly, although the book is available in several mobile or tablet formats, 80% of the downloads were as pdfs.

Although the quantitative data is interesting, I’m not quite sure what to make of it. I struggled to exceed 10,000 copies of my best selling commercial book over a period of 10 years or so and 8,000+ downloads suggest a lot of readers, but it is easy to download something that is online and free then never read it or just a few pages.

In any case there are over half a million people in North America alone in the market I’m trying to reach. Although marketing is important, it also drives the numbers without indicating the level of engagement. So I am looking increasingly to qualitative feedback to measure the book’s success.

Qualitative responses

These can be classified in the following ways:

1. Individual faculty and instructors

This is probably my key market, but also one that is the hardest to reach, because these instructors are discipline and subject-based, and are unlikely to stumble across this book except by accident. To date, I have had a small number (less than 10) of individual instructors contact me to say that they are reading the book and finding it useful, but it is early days yet. I suspect that, like one of my other books, on effective teaching with technology in higher education, it will take time for the word to get out to this market. The earlier book in fact was selling more copies five years after publication than in its first year.

2. Faculty development

This is where someone has read the book, and wants to use it for the professional development of other instructors, usually within the same department. I have so far heard from the following who want to do this:

  • Neuroscience, CalTech
  • Professional development for digital education, Loma Linda University
  • College of Engineering, Drexel University
  • Bachelor of Technology, McMaster University
  • Faculty of Agriculture, Dalhousie University

Usually the person contacting me has been a dean or head of department, which is particularly rewarding for me.

3. Adoption as a text book for a course

The book has been quickly adopted by at least one instructor as part of either teacher training or graduate programs in education on courses at the following universities:

  • UBC (Master in Educational Technology)
  • Simon Fraser University (Professional Development certificate)
  • Royal Roads University (Master of Arts in Learning and Technology)
  • Vancouver Island University (Online Learning & Teaching Graduate Diploma (OLTD) Program)
  • Vancouver Community College (Provincial Instructor Diploma Program)
  • Seattle Pacific University (MEd in Digital Education).

Again, although these all lead to some form of qualification, they are also providing professional development opportunities for instructors and faculty. And in this case it is saving students money, although the students are mainly professionals.

4. Accreditation agencies

This was even more unexpected. I have received notification from two accrediting agencies that they are either using or recommending the book for continuing professional development:

  • Accrediting Bureau of Health Education Schools, USA
  • Texas Higher Education Coordinating Board

5. Graduate students use in theses/dissertations

I have heard from a few graduate students who are using the book to help with their dissertation or thesis. I’m hoping this will increase over time.

6. External reviews

One graduate student wrote to me to say she wanted to quote me in her thesis, but her supervisor warned her not to do this, as the book had not been formally peer reviewed. I have in fact commissioned three independent reviews, which will be published at the end of this month (June) alongside the book. It will be nearly a year before any formal reviews will be published, at least in traditional journals, but these will be important, so I am wondering which journals to send it to (as a pdf) – any suggestions?

Conclusions

It is still early days to be evaluating the success of my open textbook, but there are some surprises here for me.

  1. I didn’t expect so many people to download the whole book, and I certainly didn’t expect those that did download to use the pdf version. I actually designed the book for online use on a laptop or desktop computer, as the book has embedded multimedia and is meant to be interactive, and a resource that is dipped into rather than read from cover to cover. In another post, I discussed the problem of formatting graphics in the different versions, and I am hoping to have a slightly redesigned version for tablets soon, but maybe one should design for a pdf version from scratch.
  2. It may or may not be significant that I haven’t heard anything yet from units that are professionally responsible for faculty development, such as teaching and learning centres. I will be more than happy if I can reach directly individual faculty and instructors and their deans and heads of department. Teaching and learning centres are probably at their busiest now and already had their workshops designed before the book came out. However, I am hoping that it will eventually be seen as an essential resource by such centres, but maybe I need to network more with organisations such as the Society for Teaching and Learning in Higher Education to make them also more aware of the book.
  3. Authors need to make sure they are getting regular reports on the utilisation of their open textbooks. The software is there to do this, but it is not always accessible by the authors themselves without running a special report, so there is a bit of work to be done on the interface of Pressbooks to make this information more comprehensive and readily available to authors directly. However, interpreting such data is also tricky, even when it is available, and needs to be balanced with qualitative assessments as well.

I will write about the cost of doing an open textbook in another post, then end with a final post which will discuss whether I felt the whole exercise was worth it. In the meantime, I am wondering if any readers have suggestions for better ways to evaluate the success of an open textbook.

Investments in game-based learning and learning technology continues to grow

Listen with webReader

Growth in learning technology investment 2

Banville, L. (2015) Reports Highlight Strong Growth and Investor Interest in China, U.S. Game-based Learning Games and Learning, May 31

This is an interesting report that summarizes three major market research reports about investment in game-based learning particularly, and learning technologies in general.

It covers recent reports from the following three market research organizations:

Main results:

  • worldwide sales of game-based learning products hit $1.7 billion in 2013 (Ambient)
  • future growth in game-based learning products is expected to grow between 7%-16% per annum
  • global private investment in learning technologies generally topped $2.4 billion in 2014 (Ambient)
  • consumers are the top buyers of edugame packaged content, particularly in the early childhood market (Ambient)
  • the emergence of easy-to-use mobile game-building tools supports the cultural shift towards game-creation as an educational experience (Ambient).

One driving factor in the most recent growth has been investment in mobile learning companies in China.

Comment

Ambient Insight in particular has been extremely accurate in identifying investment trends in learning technologies.

Mobile learning and game-based learning look to be the main bets for commercial growth, followed by learning analytics.

The big question is though whether these investments will drive change in education, or whether the education market will reject one or more of these developments, either because they are too costly or difficult to implement (e.g. very high training costs to get teachers or learners to use them well) or because such technologies do not meet the actual learning needs of students.

Another question is whether the level of investment in any single educational game will be large enough to bring about major changes in learning. The danger is of spreading investment too thinly across too many games to have a major impact, focusing on low levels of learning such as memorization, rather than developing critical thinking or problem solving skills. Ambient Insight’s comment that easy-to-use mobile game-building tools are increasing suggests though that this will be an exciting area that is ripe for growth – and for research and evaluation. I just hope that educators and learners will be as involved as software developers in designing such educational games.

Update on online learning in Africa

Listen with webReader
One of the AVU’s new distance-learning centres is launched at the University of Education, Winneba in Ghana. Photograph: AVU

One of the AVU’s new distance-learning centres is launched at the University of Education, Winneba in Ghana. Photograph: AVU

Anderson, M. (2015) Out of Africa: e-learning makes further education a reality for tens of thousands The Guardian, May 20

The opening this week of the 10th e-Learning Africa international conference prompted this informative report by the British newspaper, the Guardian, about the state of virtual learning in Africa. I have used this to pull together a number of different strands about online learning developments in Africa.

The e-Learning Africa conference

Only 6% of Africans continue to any form of higher education (compared with a world average of 26%). Thus this year’s e-Learning Africa conference is particularly significant as it is taking place in Addis Ababa, the HQ of the African Union,which has prioritized virtual learning in its long-term development strategy.The conference is also hosted by the government of Ethiopia. Rebecca Stromeyer, one of the driving forces behind e-Learning Africa, has done a tremendous job in using the conference to promote the development of virtual and online learning in Africa.

The African Virtual University

The African Virtual University, a Pan African Intergovernmental Organization established by charter with the mandate of significantly increasing access to quality higher education and training through the innovative use of information communication technologies, is a major force in promoting virtual learning in Africa.

It is still relatively small in terms of student numbers, with a total of 43,000 students since it started in 1997. So far, 19 African countries signed a charter establishing AVU as an intergovernmental organisation. The AVU offered its first MOOC to 1,700 African students in March this year. Perhaps more significantly it is opening 29 new distance learning centres in 21 African countries at a cost of $200,000 each.

The AVU at the moment does not offer its own degrees, but works in partnership with other African universities to deliver online programs across Africa, sometimes in partnership also with foreign universities such as Indiana University in the USA and Laval University in Canada. AVU plans to start offering its own degrees next year.

UNISA

South Africa has been a leader in distance education in Africa for many years, with over 300,000 students a year currently enrolled in UNISA (the University of South Africa), but although it has some programs offered online, it has been somewhat reluctant to invest heavily in online technologies, because as an open university it has been concerned with the high cost and difficulties of access to the Internet for many Africans.

However, the AVU is considering making lectures accessible on mobile phones, which would tap into Africa’s estimated 112-million smartphones, and UNISA will need to move more quickly if it is to stay relevant in South African online and open education..

Fibre optics

Another major factor that is impacting on virtual learning in Africa is the spread of fibre optics. The first map shows the submarine networks and their international links and the second shows the internal, terrestrial fibre optic networks.

African submarine fibre optic networks Image: © African Politics Portal, 2010

African submarine fibre optic networks
Image: © African Politics Portal, 2010 

African terrestrial fibre optic networks Image: AfTerFibre: https://manypossibilities.net/afterfibre/

African terrestrial fibre optic networks
Image: AfTerFibre: https://manypossibilities.net/afterfibre/

The key factor here is capacity. Fibre optics enable much higher Internet speeds and bandwidth than mobile technologies (although of course the two will be used in combination) but the end result will be much cheaper Internet connectivity in Africa in the coming years.

Comment

I hesitate to suggest solutions for Africa – I’m too far away and the best solutions will be African originated. However, here’s my opinion, for what it’s worth.

Those institutions and organisations that are moving now into virtual learning will have a major competitive advantage as Internet access widens and the cost of access drops dramatically. Bakary Diallo, the rector of the AVU, believes that the AVU can drive down the cost of higher education in Africa, without losing quality. Timing will be critical though – too early a move and the large market will not be ready; too late and other providers will have moved in.

The key challenges though will be the following:

  • appropriate content: African developed OERs (such as OER Africa’s and the OERu’s) will be an essential component of a low cost, high quality, virtual learning system in Africa; at the same time, actual courses and programs available online will also be critically important and this will need substantial investment, mainly in teachers and instructional designers;
  • political recognition of the integrity and quality of virtual learning: African politicians have been very conservative in the past in recognising the value of online and distance learning. Nigeria, the major economic nation now in Africa, for instance, has almost no publicly funded online learning at a higher education level., because the government won’t recognise such qualifications. It is good that 19 countries have signed on to the AVU and the African Union has made virtual learning a priority. This though now has to be accepted by other African countries, and policies and strategies for virtual learning and above all recognition of qualifications now need rapid implementation by African governments;
  • institutional management. Even in highly developed countries, university administrators have struggled to manage well the development and maintenance of online learning. African universities will struggle even more with this challenge;
  • lack of qualified professionals: Africa has few professional instructional designers, although countries such as Kenya do have very good IT professionals and web designers. However, the private sector can offer much better salaries;
  • lack of funding: there is a high cost of investment in adopting online learning, and it will take political courage to put aside the funds needed at the level of magnitude to drive real change. However, this is no longer impossible for many African countries such as Nigeria, South Africa and Kenya, whose economies are rapidly growing. It is therefore more a question of political will than resources, for at least some African countries, although others will take much longer to catch up;
  • corruption: this has two aspects, open corruption, where government funds for online learning are diverted to individuals (usually politicians, but also sometimes local administrators), but probably much more significant will be the influence of major technology-based multinational corporations, who will lobby for money to be spent on (their) technology rather than on the human resources needed to sustain online learning (i.e. well qualified teachers).

Lastly, the challenge for Africa is to walk two paths at the same time. Online learning should not be used as a replacement for a high quality campus-based higher education system but as an integral part of a comprehensive system of higher education that includes face-to-face teaching, blended learning and fully online learning. Getting that balance right will be a mjor challenge.

Overall, though, I am very optimistic that the future belongs to Africa, and that online learning will be a critical component of that future.

Lessons about researching technology-enhanced instruction

Listen with webReader
Meiori, Amalfi Coast

Meiori, Amalfi Coast – when it’s not raining

Lopes, V. and Dion, N. (2105) Pitfalls and Potential: Lessons from HEQCO-Funded Research on Technology-Enhanced Instruction Toronto ON: Higher Education Quality Council of Ontario

Since it’s raining heavily here on the Amalfi Coast today for the first time in months, I might as well do another blog post.

What this report is about

HEQCO (the Higher Education Quality Council of Ontario) is an independent advisory agency funded by the Ontario Ministry of Training, Colleges, and Universities to provide recommendations for improving quality, accessibility, inter-institutional transfer, system planning, and effectiveness in higher education in Ontario. In 2011, HEQCO:

issued a call for research projects related to technology-enhanced instruction…. Now that the technology studies have concluded and that most have been published, this report draws some broader conclusions from their methods and findings.

What are the main conclusions?

1. There is no clear definition of what ‘technology’ means or what it refers to in many studies that investigate its impact on learning:

One assumes that the nature of the tools under investigation would have an impact on research design and on the metrics being measured. Yet little attention is paid to this problem, which in turns creates challenges when interpreting study findings.

2. There is no clear definition of blended or hybrid learning:

The proportion of online to face-to-face time, as well as the nature of the resources presented online, can both differ considerably. In a policy context, where we may wish to discuss issues across institutions or at a system level, the lack of consensus definitions can be particularly disruptive. In this respect, a universal definition of blended learning, applied consistently to guide practice across all colleges and universities, would be helpful.

3. Students need orientation to/training in the use of the technologies used in their teaching: they are not digital natives in the sense of being intuitively able to use technology for study purposes.

4. Instructors and teaching assistants should also be trained on the use and implementation of technology.

5. The simple presence of technology will rarely enhance a classroom. Instead, some thought has to go into integrating it effectively.

6. New technologies should be implemented not for their own sake but with a specific goal or learning outcome in mind.

7. Many of the HEQCO-funded studies, including several of those with complex study designs and rigorous methodologies, concluded that the technology being assessed had no significant effect on student learning.

8. Researchers in the HEQCO-funded studies faced challenges encouraging student participation, which often led to small sample sizes in situations where classroom-based interventions already limited the potential pool of participants.

9. The integration of technology in postsecondary education has progressed to such a point that we no longer need to ask whether we should use technology in the classroom, but rather which tool to use and how.

10. There is no single, unified, universally accepted model or theory that could be applied to ensure optimal learning in all educational settings.

Comment

I need to be careful in my comments, not because I’m ticked off with the weather here (hey, I live in Vancouver – we know all about rain), but because I’ve spent most of my working life researching technology-enhanced instruction, so what appears blindingly obvious to me is not necessarily obvious to others. So I don’t really know where to start in commenting on this report, except to say I found it immensely depressing.

Let me start by saying that there is really nothing in this report that was not known before the research was done (in other words, if they had asked me, I could have told HEQCO what to expect). I am a great supporter of action or participant research, because the person doing the research learns a great deal. But it is almost impossible to generalise such results, because they are so context-specific, and because the instructor is not usually trained in educational research, there are often – as with these studies – serious methodological flaws.

Second, trying to define technology is like trying to catch a moonbeam. The whole concept of defining a fixed state so that generalisations can be made to the same fixed state is entirely the wrong kind of framework for researching technology influences, because the technology is constantly changing. (This is just another version of the objectivist vs constructivist debate.)

So one major problem with this research is HEQCO’s expectations that the studies would lead to generalisations that could be applied across the system. If HEQCO wants that, it needs to use independent researchers and fund the interventions on a large enough scale – which of course means putting much more money into educational research than most governments are willing to risk. It also means sophisticated design that moves away from matched, controlled comparisons to in-depth case studies, using though rigorous qualitative research methodology.

This illustrates a basic problem with most educational research. It is done on such a small scale that the interventions are unlikely to lead to significant results. If you tweak just a little bit of a complex environment, any change is likely to be swamped by changes in other variables.

The second problem in most of the studies appears to be the failure to link technology-based interventions to changes in learning outcomes. In other words, did the use of technology lead to a different kind of learning? For instance, did the application of the technology lead students to think more critically or manage information well rather than reproduce or memorize what was being taught before? So another lesson is that you have to ask the right kind of research questions that focus on different kinds of learning outcomes.

Thus it is pointless to ask whether technology-based interventions lead to better learning outcomes than classroom teaching. There are too many other variables than technology to provide a definitive answer. The question to ask instead is: what are the required conditions for successful blended or hybrid learning, and what counts as success? The last part of the question means being clear on what different learning outcomes are being sought.

Indeed, there is a case to be made that it may be better not to set firm outcomes before the intervention, but to provide enough flexibility in the teaching context to see what happens when instructors and students have choices to make about technology use. This might mean looking backwards rather than forwards by identifying what most would deem highly successful technology interventions, then working back to see what conditions enabled this success.

But fiddling with the research methods won’t produce much if the intervention is too small scale. Nineteen little, independent studies are great for the instructors, but if we are to learn things than can be generalized, we need fewer but larger, more sophisticated, and more integrated studies. In the meantime, we are no further in being able to improve the design of blended or hybrid learning than before these research studies were done, which is why I am depressed.

UBC develops an institutional strategy for learning technologies

Listen with webReader
The Amalfi Coast

The Amalfi Coast, Italy

Bates, S. et al. (2015) UBC’s Learning Technology Ecosystem: Developing a Shared Vision, Blueprint & Roadmap Vancouver BC: University of British Columbia

I’ve not been posting much recently as I am taking a three week holiday in Europe (the photo is my view as I write this), but this report from the Provost’s Office at UBC is too significant to ignore.

What is it about?

The report basically sets out a vision and a set of strategies for the future development and management of learning technologies at UBC, a large Tier-1 research university in Canada. Although produced by a small Project Committee, it is the outcome of extensive discussions throughout the university and also externally with other institutions with successful learning technology strategies.

What is in the report?

1. Recognition of learning technology as an eco-system

A learning technology ecosystem represents faculty, staff and students interacting with their learning technology environment, composed of tools and services. There are dependencies in this ecosystem; between technologies, between technologies and services but also between users, technologies and services. The ecosystem is self-organizing, dynamic, constantly changing and evolving. Technologies are birthed, and they also are removed as new ones take their place.

2. Assessment of the current state of learning technologies

UBC uses a very interesting way of assessing the current state of learning technology within an institution, using the following conceptual framework:

UBC's current state assessment process framework

UBC’s current state assessment process framework

This has enabled the team to identify gaps in services, governance, funding and infrastructure.

Another interesting outcome of this process is that the report estimates that UBC is currently spending almost $10 million annually on supporting its LMS, of which 78% is incurred at a Faculty/academic department level, mainly in technical support for the LMS, the rest centrally, including licensing. Thus one technology tool is costing almost as much as the rest of the LT eco-system.

2. Vision and principles for LTs at UBC

UBC's LT vision and principles 2

3. Functions and services

Working group members identified functional gaps in the LT ecosystem, along with their relative importance. Similarly, members of the Working Group identified both phase-specific support required during LT life cycles, as well as support services required across the lifecycle. They identified which of the gaps required the most improvement and also prioritized them according to their relative importance.

4. Support models

UBC uses both central and local/departmental support models and because of the size and complexity of the organization, no major changes were suggested for support models (but see Governance below)

5. Governance

The working group found significant shortcomings in the current governance structure for LTs. In particular there was inadequate academic input into priorities for the selection and use of LT tools and services, and the student voice was not heard. The Working Group proposed a stronger governance model as a result.

6. Other issues

The report goes on to cover a number of other issues, such as a roadmap and success metrics and resource issues such as the need for better learning analytics and increased bandwidth.

Why this report?

Good question, Tony, and here I will have to speculate a little, as I no longer work at UBC. UBC has a long history in both distance education and learning technology development. In the early 1990s it received government funding of over $2 million to explore the use of learning technologies, one outcome of which was WebCT, the first learning management system to be widely adopted. Blackboard Inc eventually bought WebCT, and UBC still uses Blackboard Connect as its LMS.

In the early 2000s,  a ‘nascent’ governance structure for learning technologies was suggested, and in recent years governance has focused mainly on the transition from Blackboard Vista to Blackboard Connect. However, over the last couple of years, UBC has also developed a major flexible learning strategy which is now being extensively implemented throughout the university. There has been considerable frustration and dissatisfaction with the implementation of Connect which has been getting in the way of the flexible learning strategy, so I see this report as a way of fixing that disconnect (sorry for the pun.) Or, as the report puts it:

Faculty desire a greater choice of tools, so that the one with the best fit for the pedagogical purpose can be selected….the functional footprint of the LMS is shrinking over time though the footprint of the entire [LT] ecosystem is arguably increasing. We anticipate a shrinking LMS footprint while still envisaging the need for a core within the ecosystem.

Comment

Although specific to UBC, this report will resonate with many other institutions. It should be essential reading for any Provost concerned with moving their institution forward into digital learning, as institutions struggle with legacy technology systems. The report adopts a clear, evidence-based analytical approach to sensitive issues around management, technology choice, and pedagogy, even if occasionally the business-speak language grates a little.

So back to my glass of Prosecco on the sun-drenched terrace.