Way off the mark, a new ranking in the making
Photo by Bango Renders on StockSnap
At the end of last year, I wrote about a new online learning ranking announced by Times Higher Education (THE). At that stage, details were somewhat fuzzy, which, as might be expected, stimulated a lot of questions. On the whole, my conclusion was that an online learning ranking has the potential to be something positive, but there are challenges to doing this effectively and in a way that is trusted by the sector.
We’re further down the road now with the ranking, and more details have emerged that help with evaluating this new initiative. I’ve also had the benefit of attending several information sessions run by THE to further explain their initial approach and rationale.
A tricky calculation for HEIs
My overall sense, based on conversations I’ve had with a number of people, is that there’s a great deal of uncertainty about participating in this ranking. As a consequence, several HEIs have decided not to take part.
This is undoubtedly a tricky decision for HEIs to make, especially given this is a new endeavour from an organisation with no previous pedigree in online education.
The opportunity presented to HEIs is that a favourable ranking may elevate online programmes as it can be used in marketing and promotion, and may directly or indirectly lead to increased enrolments and lead generation. This is not an inconsiderable opportunity, given some of the key drivers for online education activity is to open up a much-needed alternative revenue stream. However, there’s also a risk that comes with a new ranking with an embryonic methodology, because a negative positioning of an HEI may have the opposite effect.
There’s also a risk in not taking part, that isn’t simply a sense of FOMO. If a competitor takes part and you don’t, this may enable them to steal a march on your institution and gain market share.
On the whole, the further details I’ve seen and engagements I’ve had have not changed my overall opinion that a ranking has the potential to be positive. However, recent details and engagements have raised, rather than allayed concerns. It’s evident that there are some significant issues with the approach to this new ranking that are likely to influence HEIs' decision-making on taking part.
Institution versus course level approach
When the ranking was announced, there was some ambiguity as to whether this was a ranking of courses or institutions. This is a foundational decision for a ranking of this nature, and THE have decided to develop it at the institutional level. They have given the following reason for their approach:
“We are looking at the university level, rather than the course level, because we believe that the core approach of an institution will best reflect repeatable quality.”
This statement fundamentally sets some alarm bells ringing, as it is the first of many indications that THE’s approach shows some naivety and a lack of understanding of the online education space.
My most recent research highlights that over 90 UK HEIs have some type of partnership with an online education company. This number is growing, and in several instances, HEIs have more than one partnership.
As a consequence, there are a decent number of UK HEIs that essentially have significant divergences in their online course portfolios, which go well beyond the differences you might see between different departments for on-campus programmes.
There are fundamental differences in where online degrees are situated and how they are approached. There’s a world of difference between online degrees delivered through an online programme management (OPM) company partnership or equivalent and degrees that may have developed in a more traditional manner within faculties.
These differences play out across the different dimensions that this ranking looks at, including, for example, the financial resources allocated to online students.
The costs associated with degrees offered through an OPM partnership will be much higher due to the upfront investment companies make and the sales & marketing activity undertaken to achieve scale in terms of student numbers. The cost per acquisition of online degree students can be considerable, and there are other, different longitudinal costs aligned to certain services.
There will be a huge gulf between financial resources available for online degree students on OPM supported programmes and those offered exclusively by the HEI, such that the assumption made by THE that an aggregate approach is sufficient looks misguided.
One other aspect of the challenge here in gauging and evaluating institutions based on financial resources in the aggregate is the expectation that private online education companies are willing to effectively disclose income and expenditure. Although this is top-level information rather than line-by-line detail, there may be some deliberation here, and companies may be keen to understand how this data might be used across the wider THE group, which through private equity clout, has been swallowing up an array of companies in recent years.
OPM partnerships are simply one kind of online education company partnership that exists in a landscape in which private partnerships are common and increasing. Ultimately, this means in a number of cases there is an uneven level of resource across online degrees that really calls into question the judgment of an aggregated approach to gauging the quality of an institution's online degree provision.
One of the most staggering aspects of this calculation by THE is that this is something you would expect an organisation in the business of higher education to know, and it’s certainly something that online education professionals are only too well aware of.
The right people in the room
This brings me onto another key concern, which is the level and quality of consultation THE have engaged in when developing this ranking. They have stated that this ranking was developed through multiple roundtables with university leaders. Again, this raises questions, particularly about whether the right people were in the room and consulted.
I know university leaders who would openly acknowledge that they lack a deep knowledge and understanding of online education, and it’s unfamiliar to them. They will often, understandably, rely upon the knowledge and expertise of those working with online education within their HEI. In fact, in one of THE’s information sessions, there was a perhaps unintentional acknowledgement that consultation had been solely with people who had gone through in-person education.
What is evident in the UK context is that THE have not consulted with the growing number of senior online education professionals and leaders when developing this ranking. This is both disappointing and detrimental to them, because the UK has a small but excellent community of these professionals within HEIs, that I’m sure would’ve willingly and constructively contributed to enabling THE to make a much more effective first stab at this ranking.
Business in or out?
When the rankings were announced, beyond the question of courses or institutions being ranked, there were other questions associated with the depth and breadth of the ranking. Some of those were around subject breadth as at the outset THE stated that:
“To be eligible to participate in THE’s Online Learning Rankings, an institution must provide an accredited degree-level programme (undergraduate or postgraduate excluding business schools/degrees).”
The exclusion of business schools and business & management degrees was something that really stood out when this ranking was first announced. This subject category is a key segment in the online degree space and their exclusion risks diluting an institution aggregated ranking considerably.
However, THE in their recent information sessions have provided more clarity on this and are not excluding business & management degrees but they are excluding MBAs. Their stated reasons for excluding MBAs have however, come across as a bit disingenuous.
In information sessions, they have stated that one of the reasons is due to a belief that MBA students are more prone to rank their institution highly because this is advantageous to them. Ultimately, that proneness to being driven by that motive is something all students are susceptible to, and the exclusion of MBAs is clearly much more to do with the fact that the company Poet and Quants that THE recently acquired already offers an MBA ranking.
I think it is reasonable to consider whether or not to include online MBAs in this ranking, especially given online MBA rankings like the Financial Times exist. However, if this is an aggregate institutional-based ranking, this exclusion will impact that. This also highlights a degree of naivety on THE’s part because it feels as though the assumption is that online MBAs are a case of one title per institution, however, some institutions have multiple online MBA titles that can sometimes comprise a significant proportion of their portfolio.
There’s this cool new thing, it’s called online learning….
One of the overarching impressions the information sessions gave was that this sense in which THE have just discovered an unpolished stone that they want to buff and put on display. This is manifested in odd statements about online education like:
“Unlike in-person learning, there is no universally agreed approach (to online education)”
This is a bizarre statement as there is variety in all forms of teaching, and online education has several relatively well-established approaches. There are key models such as asynchronous online degrees and those that mix asynchronous and synchronous elements. There is also a wealth of good practice, research, sharing of practices, frameworks, and models out there—THE might be at year zero with this stuff, but online education isn’t.
The dichotomy between asynchronous-only and mixed asynchronous and synchronous degrees is an interesting one to reflect on with this ranking. As someone commented in a recent information session, they had found that online students often express a strong preference for more synchronous online learning activities. Given the weighting towards student feedback for this ranking, there’s a risk of an unintended consequence that this ranking overly privileges synchronous online learning as a benchmark of quality. There is a reason why some online degrees are offered as asynchronous only, and it would be to the detriment of online education if the current diversity of models was unintentionally diminished.
Another issue raised in information sessions was around the progression of online students, as this is something that is asked for in the submission. Someone quite helpfully pointed out that online education often offers greater flexibility with step-on and step-off points. Progression to the next academic year is not always a definitive measure as students may step back on again. In fairness, this is not just a problem for this ranking but problematic for the Office for Students (OfS) too.
Moral posturing…
One of the troubling aspects of this ranking is the sense of moral posturing from THE. The data that institutions are asked to submit includes the number of online students with a disability, from a low-income household, first-generation students, as well as questions such as whether connectivity support is provided to students, whether devices or subsidies for devices are offered, and the amount of offline content.
A great affordance of online education is that it can provide access to those who might not otherwise have been able to access education. However, there’s a risk here that online education is othered, as it has been for many years. Online degrees are an educational vehicle for a diverse range of people from different backgrounds to study. My worry is that, by virtue of THE's poor grasp of online education, they might perpetuate the idea that it is simply the "poor door" of higher education.
The focus on technology is very much framed around the digital divide and digital inclusion. No one would dispute that this is incredibly important and an area where more needs to be done. However, as far as I am aware, the type of connectivity support that THE seems to be describing isn’t typically provided, nor are devices/device subsidies and extensive offline content. It feels as though THE has somewhat plucked out what they feel institutions should be doing without much knowledge of the space, rather than engaging a bit more deeply with the realities and nuances of online education.
Rankings have come under a lot of scrutiny in recent years, and it would be naïve not to acknowledge they have problematic components. One obvious response to that is for those engaged in rankings to overly indulge in messaging that overstates a company's commitment to social and environmental causes and the relationship of their activities to advancing those things. This though, is perhaps less grating if you’re selling smoothies as opposed to rankings.
Things can only get better….
On the whole, I think there is a decent amount of goodwill towards an online education ranking of some sort. I don’t believe people inherently want to see a ranking like this fail, but it’s clear that this first attempt is a bit of a mess.
That assessment takes into account the fact that first attempts are just that. However, THE have really done themselves no favours here. Had they consulted more widely and more wisely, they would have had an imperfect but far better formulated and constructed first attempt.
It feels as though this initiative so far has been hampered by some lazy hubris, which has served to convey that this is an organisation that knows and understands little about online education. To a certain extent, this is acceptable if there was robust evidence of genuine attempts to engage with the right people with credentials in online education working at HEIs.
Ultimately, if I were in the position of deciding whether or not to take part, I would have considerable misgivings about the credibility of this ranking as it stands. Several others hold similar views and have decided not to participate, and that would certainly be my position too.
While the focus here has been on HEIs, I also have concerns about students taking this at face value. Let’s be honest, students are not going to pore over the methodology; many will simply take at face value what this ranking says. As I’ve mentioned before, in this sense, rankings are always somewhat doomed to succeed.
Ultimately, the issue here isn’t that this ranking isn’t on the money the first time out — no one would expect it to be — the issue is it could have been a far better first attempt if THE had approached it differently. One hopes that whether it's this company or someone else, we might have a credible online learning ranking in the not too distant future.
