Is adaptivity a qualitative or quantitative problem?

One common criticism of adaptive learning is that by tailoring instruction so closely to students’ needs, it doesn’t challenge them enough. As embodied by James Paul Gee’s critique:

People who never confront challenge and frustration, who never acquire new styles of learning, and who never face failure squarely may in the end become impoverished humans. They may become forever stuck with who they are now, never growing and transforming, because they never face new experiences that have not been customized to their current needs and desires.

While I agree with the dangers of what he describes, I question the causal attribution.

First, adaptive learning systems that indulge in too much customization may instead be guilty of relying on a too-narrow prescription for the student’s “zone of proximal development (ZPD)”. Individualized learning does not require giving only incremental steps; it can (and should) include more ambitious steps to occasionally challenge students, perhaps just beyond their conventional ZPD (or at the limits of their ZPD when defined by “lots of help”). Students need to struggle—manageably—as part of their learning. Adapting to students’ needs can include optimizing the nature and amount of that struggle based on past experiences and future expectations.

Second, this can also be overcome by building a certain amount of variability into the system, for the sake of both the students and the system. Occasionally presenting students with problems that may or may not lie within their ZPD can help them learn “what to do when you don’t know what to do” (in the words of a dear colleague of mine, Joe Wise). Whether framed as desirable difficulties, germane cognitive load, preparation for future learning, or the development of adaptive expertise rather than just routine expertise, unexpected challenges can offer invaluable learning opportunities. Further, adaptive learning systems need to reach beyond what is already known in order to improve themselves. A truly intelligent system should be discovering new knowledge about its particular learners and even about learning in general. The possible paths a student might take are infinite, and the system’s designers don’t know what’s best—only what tends to be better compared to other paths that have already been examined. That is, adaptive learning must itself be an adaptive learner.

Both of these issues point to a quantitative problem due to adapting too narrowly or too often. The deeper question is whether adaptivity is a fundamental, qualitative problem: Does having any adaptivity at all invite complacency among students accustomed to having their learning experiences at least partly tailored to their needs? Given the well-established importance of scaffolding instruction according to students’ needs, I would argue that adaptive learning is a valuable tool not simply for accelerating but also for enriching instruction.

Personalized instruction: The other half of personalized learning

As I have explained in a previous post on personalized learning, an important dimension along which personalized learning goes beyond merely adaptive learning is to personalize the experience on the instructional side, not just the learner side. Amidst all the excitement about adaptive learning, teachers remain an often-forgotten yet crucial part of the equation. Well-designed personalization takes advantage of the human intelligence embedded in expert instructors, including opportunities for them to exercise their professional judgment in deciding which activities will work best for their students given their particular contexts and constraints.

This EdSurge report mentions Rocketship’s upcoming changes, as “New model attempts to bring teachers closer to students’ online learning experience” by returning some classroom control back to the teacher:

Rocketship’s new model will shift focus from running purely adaptive programs, to using programs that give teachers greater control over content that gets assigned.

What this highlights is the need for the design of personalized learning programs to identify when to allocate decisions to teachers (possibly with recommendations among which to choose) and when to adapt the students’ learning experience immediately, without need for waiting for additional human input. While this depends in part on the professional knowledge of the instructors implementing the system, some decisions may be straightforward or simple enough to automate. Decisions best left to expert human intervention are likely to be more complex, to depend on more contingencies, to require interpersonal contact, or to have more uncertainty in their effectiveness. Where that balance lies is subject to continual readjustment, but since there are always unknowns and since social interaction is fundamental to the human experience, there will always remain a need for personalization.

MOOCsperiments: How should we assign credit for their success and failure?

That San Jose State University’s Udacity project is on “pause” due to comparatively low completion rates is understandably big news for a big venture.

We ourselves should take pause to ponder what this means, not just regarding MOOCs in particular, but regarding how to enable effective learning more broadly. The key questions we need to consider are whether the low completion rates come from the massive scale, the online-only modality, the open enrollment, some combination thereof, or extraneous factors in how the courses were implemented. That is, are MOOCs fundamentally problematic? How can we apply these lessons to future educational innovation?

Both SJSU and Udacity have pointed to the difficulties of hasty deployment and starting with at-risk students. In an interview with MIT Review, Thrun credits certificates and student services with helping to boost completion rates in recent pilots, while the inflexible course length can impede some students’ completion. None of these are inherent to the MOOC model, however; face-to-face and hybrid settings experience the same challenges.

As Thrun also points out, online courses offer some access advantages for students who face geographic hurdles in attending traditional institutions. Yet in their present form, they only partly take advantage of the temporal freedom they can potentially provide. While deadlines and time limits may help to forestall indefinite procrastination and to maintain a sense of shared experience, they also interfere with realizing the “anytime, anywhere” vision of education that is so often promoted.

But the second half of “easy come, easy go” online access makes persistence harder. Especially in combination with massive-scale participation that exacerbates student anonymity, no one notices if you’re absent or falling behind. While improved student services may help, there remain undeveloped opportunities for changing the model of student interaction to ramp up the role of the person, requiring more meaningful contributions and individual feedback. In drawing from a larger pool of students who can interact across space and time, massive online education has great untapped potential for pioneering novel models for cohorting and socially-situated learning.

Online learning also can harness the benefits of AI in rapidly aggregating and analyzing student data, where such data are digitally available, and adapting instruction accordingly. This comes at the cost of either providing learning experiences in digital format, or converting the data to digital format. This is a fundamental tension which all computer-delivered education must continually revisit, as technologies and analytical methods change, as access to equipment and network infrastructure changes, and as interaction patterns change.

The challenges of open enrollment, particularly at massive scale, replay the recurring debates about homogeneous tracking and ability-grouping. This is another area ripe for development, since students’ different prior knowledge, backgrounds, preferences, abilities, and goals all influence their learning, yet they benefit from some heterogeneity. Here, the great variability in what can happen exaggerates its importance: compare the consequences of throwing together random collections of people without much support, vs. constraining group formation by certain limits on homogeneity and heterogeneity and instituting productive interaction norms.

As we all continue to explore better methods for facilitating learning, we should be alert to the distinction between integral and incidental factors that hinder progress.

Expensive assessment

One metric for evaluating automated scoring is to compare it against human scoring. For some domains and test formats (e.g., multiple-choice items on factual knowledge), automation has an accepted advantage in objectivity and reliability, although whether such questions assess meaningful understanding is often debated. With more open-ended domains and designs, human reading is typically considered superior, allowing room for individual nuance to shine through and get recognized.

Yet this exposé of some professional scorers’ experience reveals how even that cherished human judgment can get distorted and devalued. Here, narrow rubrics, mandated consistency, and expectations of bell curves valued sameness over subtlety and efficiency over reflection. In essence, such simplistic algorithms resulted in reverse-engineering cookie-cutter essays that all had to fit one of their six categories, differing details be damned.

Individual algorithms and procedures for assessing tests need to be improved so that they can make better use of a broader base of information. So does a system which relies so heavily on particular assessments that the impact of their weaknesses can get magnified so greatly. Teachers and schools collect a wealth of assessment data all the time; better mechanisms for aggregating and analyzing these data can extract more informational value from them and decrease the disproportionate weight on testing factories. When designed well, algorithms and automated tools for assessment can enhance human judgment rather than reducing it to an arbitrary bin-sorting exercise.