twosigma

If the idea of “blended learning” — combining elements of traditional classroom instruction with software-based supplements — sounds appealing in theory, it’s worth inquiring about how it plays out, in practice. There are a few dozen pilots in progress right now, and the first few findings have been trickling in. One of these was shared last month in the form of a white paper “Lessons Learned from a Blended Learning Pilot”, summarizing key findings from a pilot project at Envision Academy in Oakland, CA in which Khan Academy was incorporated into an Algebra I summer school course. While the report authors — Brian Greenberg, Leonard Medlock, and Darri Stephens — do present some preliminary quantitative analysis from a small controlled study, the sample size is said to be too small (although, strangely, size is never actually specified in the paper) to be able to draw any meaningful conclusions about learning gains. Stephen Downes, Seb Schmoller, and Alfred Essa have focused on the reported non-effect, but since it’s really a non-test, I’m not as concerned with the results. The real value that I find in this report is the qualitative feedback presented from teachers, students, and partners (from Stanford’s d.school and Google’s Chromebook team.) Here’s why: The ways in which students used (and misused) the Khan Academy software highlights opportunities for how “blended learning” software could (and perhaps should) work in the future. Edtechies, take note.

Below are a few of the passages that I found most interesting. What I find interesting is, of course, colored by my own research focus, so the highlights for you may be entirely different than these. </disclaimer>

By observing the data screens, a teacher can easily see that a group of three or four students are all struggling with the same concept. The teacher can call these students together and provide a targeted mini-lesson. Even better, the teacher can call over a student who has proven mastery on the topic, and ask the student to provide the instruction to his/her peers.

Absolutely. In the context of a classroom, the actionable feedback would be suggesting ways to rearrange where students are sitting and who they are encouraged to talk with. There’s a relatively new startup doing just that: Learning Catalytics (Eric Mazur, Garry King, and Brian Lukoff) looks to have built a system to do this quite nicely, and their approach is rooted in Mazur’s extensive work on Peer Instruction. I’ve been kicking around designs for an entirely-online venue for peer instruction with peer assessment for the past few years, so stay tuned for announcements on that front.

A related (and encouraging) result of the blended learning model was that students began to work together much more collaboratively than usually observed in high school classrooms. Although the instructional videos were present, most students preferred to work though the practice problems themselves, with the help of the teacher, or by soliciting peer assistance. Students were surprisingly comfortable asking each other for help. One student told us, “Because we are all working on different things, it’s easier to ask for help.”

Reading these observations is encouraging. We recently introduced a new mode of study in parts of Grockit, dubbed “study hall”, that lets each student work through their own adaptive/personalized problem-solving study session, but have access to a real-time public chat channel with all of the other students. As soon as a student gets stuck, they can click a button to signal to other students that they want assistance. Others can then hop into that student’s personalized session to work through the problem with them. Complementing personalized study sessions with affordances for on-demand peer assistance is one of many ways to bridge the personalization-collaboration divide. A paper describing our first set of experiments using this model is headed to the Intelligent Tutoring Systems conference for peer review later this month.

In one example, we observed two students working on the same section at the same time. They worked individually but conferred before submitting their answers. If they disagreed on a solution, they tried to convince one another or they looked for possible errors together. Other times students would tutor each other on different sections as needed. We see tremendous potential in this peer-coaching model and are interested in thinking about ways for students to signal to peers that they need additional help or to identify themselves as coaches on given topics.

There’s a great study from a team of physics education researchers at the University of Colorado, Boulder that looked at the effects of peer discussions (in a lecture hall) among students who disagree on the correct answer to a clicker-style question: Why Peer Discussion Improves Student Performance on In-Class Concept Questions. Definitely worth reading if you’re interested in the topic.

Most people are drawn to Khan based on its massive video library and Sal’s own charming and engaging teaching style. Like many, we assumed the videos would be the predominant learning mechanism for students tackling new material. In fact, the students rarely watched the videos. This result is consistent with some of the observations in the Los Altos pilot. The students greatly preferred working through the problem sets to watching the videos. Students turned to their peers, the hint, and the classroom teacher much more often than they did the linked Khan video.

There’s a difference between learning something (for the first time) and reviewing/reinforcing what you’ve previously learned. Video lessons, as a form of direct instruction, are a good fit for students who are trying to learn something that they do not yet know. Problem sets, as a mechanism for practice and assessment, are a good fit for reviewing and reinforcing what a student knows. When a student starts a new topic, it makes sense to me to start with video and then move to the exercises (as is traditionally done in a textbook). If a student already knows a topic, skipping directly to exercises makes sense. Once you’re in an exercise, a topic-focused video feels less relevant. A progression of prepared hints and assistance from friends can help both a student directly work through the challenge in front of them, in a way that a topical video cannot.

In a paper that we published last summer, “A Comparison of the Effects of Nine Activities within a Self-Directed Learning Environment on Skill-Grained Learning”, the opportunity to watch a topic-focused video after answering a question incorrectly was both not very popular and not effective. Again, this is not to say that video lessons themselves are not valuable, but just that the placement of a topical video in the context of practicing specific exercises on a known skill isn’t the best spot for it. As a result, Grockit has moved towards is offering video focused on instruction in one part of the application, and offering videos focused on solving specific questions as screencasts available after answering that specific question. These worked-solution videos are immediately relevant at the time that they are made available, more similar to question-specific hints or raising a hand to ask for targeted help from a teacher. (In this case, since there is no classroom teacher to ask, we developed Grockit Answers to fill the void (needless to say: YouTube comments don’t cut it), and Answers now powers every video lesson and question explanation, so students can ask specific confusing moments in a video and get answers from other learners. So I’m arriving at a different conclusion than the white paper authors about the students’ lack of interest in viewing Sal’s videos from within the exercises: the videos aren’t necessarily too long or not sufficiently interesting, they may just be presented in the wrong place or wrong time in the context of the exercises. Timing and context often matter a lot more than we realize.

I look forward to seeing how feedback from blended learning classroom pilots such as this one ultimately affects how these educational software systems continue to evolve.

Web video is playing an increasingly prevalent role in online learning: university open courseware initiatives, flipped classrooms experiments, and a growing number of virtual schools all rely heavily on video for direct instruction. And while video is itself a broadcast medium, situating video within a website opens the possibility for peer-to-peer interactions. Most video-hosting websites support some peer interactions through a shared comment stream, but this inherently treats the video as a two-dimensional object, ignoring its most meaningful dimension: time. Particularly for long-form presentations of complex concepts that are difficult to grasp simply by viewing, a venue for peer dialogue is incredibly valuable, but a venue that lacks a notion of temporal context is inherently limited. What could peer interactions around web video look like if temporal context was treated as critically-relevant? I’ve been lucky to have the opportunity to explore this question over the past few months while building Grockit Answers.

While YouTube, Vimeo, and several other video-sharing services provide programmatic access to a hosted video’s timeline, the new technology that really opened up the possibility of interacting with the temporal dimension of web video is Popcorn.js, a JavaScript event system for HTML5 media. Grockit Answers structures peer interactions as a set of Question & Answer discussion threads, each one anchored to a particular point in a video’s timeline. When a viewer asks a question, perhaps about something specific that they found confusing, Popcorn is used to automatically time-stamp the question to associate it with the particular moment in the video at which it was asked. When others viewers reach that point in the video, the question appears on the screen, presenting it at the exact moment that these viewers have the same context as the person who asked the question. Discussion is no longer about a video, it can now be scoped to a particular comment, explanation, or demonstration within a video. My absolute favorite part of this design is the often-surprising result of watching a video that has been the subject of many questions: As you start to become confused by a complex or vague explanation, the questions forming in your head pops onto the screen. Perhaps even more surprising, answers to your questions — perhaps clarifications or alternative explanations — also start appearing. While this is an asynchronous peer interaction, the time-triggered displays give the feel of just-in-time live help. I’m calling this a pseudo-synchronous interaction.

Many thanks to the Mozilla Popcorn project and the Popcorn.js development team for providing the layer of abstraction that makes this type of learning environment such a joy to build. For another great example of new forms of social learning around web videos, check out Popcorn Macbeth and some of the other great Popcorn demos.

While watching Peter Norvig, Sebastian Thrun, and Sal Khan field questions on their experiences and goals with online education, one set of comments by Peter Norvig absolutely resonated for me (at time 18:52):

I think for us, the key that we’re trying to figure out is how to combine a personal experience with a group experience. We really want to enable mastery on the part of the individual, to say you should keep interacting with the material until you’ve really gotten it, that’s going to be at your own pace. But at the same time we find that it’s really motivating to have a group that’s working together. That’s the advantage of having a class that’s run synchronously is that everybody is there, you can go into the discussion forums, you can get help from your peers. So we want to keep some group together so that they can help each other while allowing people to work at their own pace, and getting that right is the trickiest part.

Yes. My biggest fear with the momentum behind personalized and adaptive approaches to instruction is that learners are isolated from one another, no longer connected enough to provide the moral, social, and academic support to one another. If we give up collaboration on the road to personalization, have we actually moved the ball forward? Two steps forward and one step back. Or worse, perhaps one step forward and two steps back. The good news here, as Peter suggests, is that the two are not mutually exclusive. We can design scalable learning experiences that incorporate both personalization and collaboration. We just need to be thoughtful (clever?) and approach it as a design challenge. After being involved in designing three collaborative+personalized learning activities over the past few years at Grockit, I’m finally starting to understand it. Peter alludes to it above: It’s all about the timing.

Let’s start by assuming that you’ve created a learning environment (online or offline) that can adequately support both synchronous and asynchronous forms of interaction among learners. We can then ask: For a particular activity, what would be the benefits and drawback in structuring it around synchronous vs. asynchronous interactions? What mode of interaction synchronicity makes the most sense? Recognizing practical logistical constraints answered this question 80% of the time. We know that different people read at different speeds, for instance, and so a synchronous activity would leave some people waiting for others to finish. The larger the potential discrepancy in time required by different students, the more logical it is to shift to an asynchronous design. But while individuals may differ in the speed that they read, a group discussion still progresses at the shared pace of conversation. This is why the notion of a “flipped classroom” seems logistically reasonable: It’s essentially a model offering instruction at the pace of comprehension — which varies from student to student — and application at the pace of discussion — in-class group problem-solving and Q&A activities.

At a high level, bridging the personalization-collaboration divide is also a matter of timing. Personalization generally implies a self-paced progression through a sequence, decoupling the student’s own timeline from that of other their peers. Collaboration generally implies tightly-coupled timelines, where synchronized progression enables rich real-time interactions. To be able to draw on both of these simultaneously requires some creative thinking. We’ve explored three solutions so far: (1) structuring asynchronous interactions in a way that simulates or approximates the just-in-time nature of synchronous feedback (a la Grockit Answers), (2) supporting self-paced personalized learning to take place in a common space, allowing for easy access to others who have previously mastered a topic to jump in to help when needed (a la Grockit’s GRE “study hall”), and (3) simply by providing learner control over when they study alone and when they learn with others. These three are just a start. In what other ways can creative activity design bring together the best of collaboration and personalization in learning?

[Some thoughts following Badge Working Group #2]

A badge — which I think of as a publicly-displayable symbol awarded by a group to an individual as a way to recognize an achievement meeting certain criteria — can be used in a learning context in a few different ways. Gamification is a hot topic right now, but while incentivizing participation can be a powerful tool for engaging learners in non-compulsory learning activities, it needs to be done carefully to avoid substantially undermining intrinsic motivation. Gamification aside, there’s another role for badges in a learning context: When a respected organization publicly recognizes the accomplishments of an individual, this says something about what that individual is capable of doing in the future. Badges can act like mini-certifications, exactly the sort of thing we aim to convey in a resume.

But if we already have grades and degrees on resumes, why would we need badges? Badges tell a different story. Badges identify strengths without focusing on weaknesses, whereas grades average both. Badges are designed to be displayed publicly and proudly, whereas grades are generally read privately and hidden promptly. What about degrees? While the presence of a college degree on a resume can be a signal of qualification to a potential employer, this signal is (a.) slow and binary, (b.) opaque, and (c.) only issuable by a small number of formal educational institutions. In comparison, badges are (a.) designed to recognize finer-grained accomplishments and proficiencies, (b.) make transparent both the criteria for award recognition and the raw evidence of student work meeting these criteria, and (c.) be issuable by any organizations that support learning, inside or outside of the classroom. In short, badges offer a way to make out-of-class learning more relevant by making it more visible. To find out, we have to test this out.

Several learning environments already award badges to recognize accomplishments (e.g. StackOverflow, Grockit, P2PU, Khan Academy), but each one has had to recreate a similar underlying software infrastructure to support this. If we, as a community, think that badge systems are worth exploring and evaluating, we can accelerate this by offering a simple, shared infrastructure that handles basic functionality: issuing, storing, sharing, and displaying. The Mozilla Foundation recently began a badge project and Open Badge Infrastructure to do just this. This could be incredibly useful, but it does hold two risks. First, sharing a single infrastructure may result in the community prematurely settling on one approach to badges before other promising alternatives have really been explored. Second, a software-based framework for badges can’t be leveraged by groups lacking the technical know-how or organizational infrastructure to interface with the framework. One proposed goal raised during the meeting was to build a system in which it is as straightforward to create and issue a badge as it is to receive and display one. When Mozilla’s infrastructure is in place, I’ll set aside some time to create a small website that anyone can use (without writing code) to create one-off badges that flow through the system. Nothing fancy, just functional enough to let anyone who is interested in testing out web-based learning badges within their own community actually be able to try it out.

Is “badge” even the right word? Should we use different words to differentiate between incentivization and certification?

Milton Chen, on minds-on learning:

This fall, I humbly propose a new national campaign to teach basketball with textbooks. If the ensuing expressions of outrage by parents and demonstrations at school board meetings lead to energetic discussions about active hands-on, minds-on learning in academic subjects, this short-lived campaign will be very worthwhile. It will make us smarter about learning and move us closer to creating the kind of curriculum an Education Nation needs.

The LA Times did something controversial this week: They published a database of the “value added” ratings of thousands of Los Angeles elementary school teachers and hundreds of schools, searchable by name. This is a Big Deal.

I’ve been working my way through the technical paper that describes the study (PDF), but my mind keeps wandering back to a passage from the “Who’s teaching L.A.’s kids?” newspaper article:

On visits to the classrooms of more than 50 elementary school teachers in Los Angeles, Times reporters found that the most effective instructors differed widely in style and personality. Perhaps not surprisingly, they shared a tendency to be strict, maintain high standards and encourage critical thinking. But the surest sign of a teacher’s effectiveness was the engagement of his or her students — something that often was obvious from the expressions on their faces.

I think student engagement is more than just a good indicator of effective teaching. I’m interested to know if there’s any value-added work in which student engagement is the performance measure, rather than standardized test scores. How effective is each teacher at helping his or her students develop a love of learning? And can this intrinsic motivation help a student succeed in subsequent years, even when assigned to less “effective” teachers?

I’m now on Twitter: @aribadernatal and @studiosketchpad

There’s a new series of articles being posted on the O’Reilly Radar, and I’m liking it: Education 2.0. The posts touch on several different topics: the role of the DIY ethic in the Maker classroom, ways in which schools will change as a process of disintermediation sets in, and reflections on plans for change, areas for change, models for change, and how to shape projects for change.

The posts are chock-full of interesting ideas, such as this one from Rob Tucker:

Ed 2.0 isn’t only or even primarily about technology. It is about arming students with the tools and the fierce determination they will need to learn with and through that technology. It is about encouraging intellectual entrepreneurship – creating of our students hundreds of millions of one-person startups, kludging their way to happiness and success.

Even when it’s exciting, a one-person startup sounds like a lonely endeavor. Will the same be true for the Ed 2.0 learner?

In DIY U: Edupunks, Edupreneurs, and the Coming Transformation of Higher Education, Anya Kamenetz discusses the economics, sociology, and history of higher ed in the US (“how we got here”) and outlines a variety of promising new trends, techniques, and initiatives (“how we get there”, e.g. OER/CoP/PLN). My favorite passage appears towards the end of the book:

…forget about giving the guy a fish, or teaching him how to fish, either. Teach him how to teach himself, and he’ll always be able to acquire the skills he needs to find food, skills you haven’t even thought of yet for things you didn’t know you could eat. … Today, 90 percent of fish species are over-exploited… The world needs people who can figure out new ways to repair the oceans and to find or grow renewable sources of food.

But if his learning is self-initiated, how does he convince others that he is capable of acquiring these new skills? Portfolios can be a powerful way to supplement (or replace) credentials. For software developers, open-source projects are ideal fodder for portfolios: a GitHub profile can actually be more informative than a traditional resume.

More DIY on Anya’s DIY U blog.

The Mozilla Foundation and the Shuttleworth Foundation are jointly offering an Education for the Open Web Fellowship:

We invite applications from individuals interested in developing innovative approaches that educate people how to promote the open web. …Ideas can connect the open web with learners of any age: opening up the world of web citizenship to include kids; encouraging high school students to learn from the bendable and hackable world of the Internet; helping people in their 20s learn the skills they need to create wealth or find work in the web era. …ideas need not focus on formal learning within the education system. In fact, informal learning approaches that draw on the fluid nature of the Internet are highly encouraged.

Kudos to Mozilla and Shuttleworth on a great program. In it, they pose the question: Can you imagine your idea reaching thousands of people? Hundreds of thousands? Millions?

Also, I should mention:

The fellow must reside in either Europe or Brasil, and be fluent in English.

Found via OpenEducationNews