«   |   »

Khan Academy classroom pilot declared a ‘success’ based on no data

by Grace

A frothy piece in the Utopianist* declares that a Khan Academy pilot to teach math to 5th and 7th grade students in the Los Altos School District of California is a “colossal success”.

Their test run has so far yielded nothing short of colossal success, with both students and teachers alike more engaged and fulfilled.

The problem is that the article presents nothing beyond abstract anecdotes of exaltation.  It’s fine that students are engaged and fulfilled, but are they learning?  Where’s the evidence?  Where’s the data showing improved achievement levels?  Instead, the basis for declaring success is stuff like this.

  • It’s meaningful – Khan hopes to “humanize” education by providing students and teachers with the opportunity to spend some meaningful time together
  • Kids teach each other, replacing expert teachers – students better at one subject can tutor their peers who are struggling with the same concept
  • Remember, it’s meaningful –… meaningful classroom time will do more … than the ritual of silent students …
  • Fun tools with cool names – the Academy has recently introduced badges … The badges have cool names like Sun and Black Hole.
  • Students love it – Students are raving about the Khan Academy’s videos
  • A famous billionaire approves – Bill Gates makes an appearance … applauding Khan’s program …
  • As long as it’s fun – …  will allow classrooms the time to finally spend on those fun educational projects … 
  • Teachers approve because kids have fun – Teachers also talk about how they love the program because their kids are having fun 

The clincher is that they label daydreaming as a learning style, especially laughable given that the very concept of “learning styles” is a myth.

Students also love that the program supports individual learning styles – like daydreamers.

I am a big fan of Khan Academy, but give me data before you jump to hasty conclusions  and promote the use of yet another unproven educational experiment on our public school children.

* In fitting with this article, utopia is defined as  an imaginary and indefinitely remote place.

Posted on November 28, 2011 at 5:09 am in K-12 education, preparing for college   |  RSS feed

Tags: , ,

21 Responses to “Khan Academy classroom pilot declared a ‘success’ based on no data”

    
  1. Grace
    November 28, 2011 at 7:49 am

    Someone IRL told me this post makes me sound like a shill for teachers unions. 🙂

    Like

  2. Lynn
    November 28, 2011 at 8:44 am

    A shill for teachers unions? I don’t see that at all — and I’ve yet to hear of any teacher union supporting an approach to learning that values data and results over fun and engaging. Keep at it, Grace – there are more than two sides to the education reform movement. I think you sound like a shill for parents, personally.

    Like

  3. Jo in OKC
    November 28, 2011 at 1:15 pm

    It seems strange that they would have no test score data to back this up. Even if the kids just took standardized tests at the end of the prior year and then at the end of the Khan Academy year, they could compare national percentile results both on average and by kid and see what the improvement was. I’m not saying that that would be the best measure, but it would be *something*.

    I’m a big fan of different ways of learning, but are we at risk of losing good teachers because they become nothing more than babysitters to a bunch of kids plugged into a computer? Maybe that’s ok because they find the computer is equivalent to having the best teacher possible. Or not. In my experience in business, it’s the best who will leave first when the environment goes downhill.

    My guess is that computerized learning is good for some kids and some situations and not so good in others. I don’t think we know enough to figure that out yet, though.

    Like

  4. Grace
    November 28, 2011 at 1:49 pm

    I hope that the pilot program will be using real data to assess its results, but this fluff piece based its opinions on what passes for “data” among some educators. I’ve read so many pieces about how various educational programs/innovations are great because the students are “engaged”, yet actual achievement levels are never brought into the discussion. There is so much hype about Khan Academy, I fear schools will rush headlong into using it without any attempts to learn if it really works in the classroom.

    And it does seem that the best teachers will likely be the first to leave if their jobs become glorified babysitters to students plugged into computers. And I say this as someone who believes that KA does have a place in classrooms, but it needs to be tested to know for certain.

    Like

  5. Anna Loza
    November 28, 2011 at 3:59 pm

    Utopianist author here. I love that you are also on the frontlines of arguing for educational reform, but please allow me to quote some other pieces of the article which you bashed as fluff:

    “Khan believes that video teaching can help “flip” the classroom by having kids watch videos at home and doing what used to be homework in the classroom. This way, students can pause, repeat and re-watch any material that they need – a concept that will solve what fear and embarrassment may prevent in the live classroom. Since students have access to all the videos in the database, they can fill in what Khan calls “Swiss cheese gaps” in their education, by studying troublesome topics as much as they need to. Students can spend some extra time with concepts like fractions, derivatives or cell biology, making the next level of information be received on solid ground.”

    “In addition, by doing what used to be homework in the classroom, Khan hopes to “humanize” education by providing students and teachers with the opportunity to spend some meaningful time together [notice how teachers are in no way taken out of the equation, but instead relied on for more solid expertise?]. Students can ask specific questions and get direct, personalized help from the teacher when it matters most. In addition, students better at one subject can tutor their peers who are struggling with the same concept, thereby creating a mutual exchange of information and interdependence. This meaningful classroom time will do more, Khan hopes, than the ritual of silent students meant to absorb information within one minimally-interactive session.”

    See how you took that “humanize”, “silent ritual” and more, out of context? The Utopianist does not pretend to be a blog solely about educational reform — we simply mean to bring meaningful (woops, there’s that word again!) developments to the forefront of busy reader’s repertoire, so that more people can glean information about the *good* that is being done in our world, and not simply about the terrible degradation that is apparently happening on all fronts of society. This piece was not meant to be all about how students have “fun” and “love it”, but one cannot deny that that is a feature that some classrooms are simply missing. What’s the point in learning chemistry if students fall asleep or learn nothing, when it should really open their eyes to the wonders that exist all around them in the daily world?

    Yes, there is no “hard evidence” about the Khan Academy, but that is because it is relatively new. We cannot deny that the program is trying to correct some wrongs that the founder himself has witnessed, and hey, he is doing more than some others to fix what he thinks is necessary. Please don’t degrade a decent introductory article — if a reader is interested in the topic, we encourage them to take it and go further with their own research. It is our aim to shed light on positive action, giving hope to those that may be discouraged from reading what is considered the daily news.

    We support bloggers like yourself who are also on the forefront of positive developments, but I ask that you don’t simply highlight small segments which may prevent your readers from even giving a solid blog — the Utopianist — a second thought.

    Just some more pieces of the article, in case I haven’t already made my point about it being more than “fluff”:

    “The software uniquely tracks all students’ performance, showing which problems the student struggled on, including which videos they’ve completed and where they paused. Such a system is surely revolutionary, showing teachers all the problem areas – and strengths – at a bird’s eye view.”

    Oh, and those “just for fun” badges?

    “And to top it all off, the Khan Academy knows how to motivate their students; the Academy has recently introduced badges, which range from ones that are easy to get, providing simple recognition, to badges that are said to take years of work to achieve. The pilot project has seen 5th graders tackle college-level math in order to get a certain badge, as well as watch hours of physics videos; in short, nothing to scoff at. The badges have cool names like Sun and Black Hole.”

    Is the following not something we all want for our children and our students?

    “During a TED talk [go to full article for link], Khan says that when students are allowed to learn at their own pace, what can sometimes be branded as gifted or slow during one session will often reverse at another time – suggesting that those who have been granted the labels can be nothing more than victims of the right, or wrong, timing. By allowing all students to learn at their own pace for a protracted period of time, true and stable patterns can be better spotted.”

    Thanks for mentioning Bill Gates and those “FUN projects” correctly, as well:

    “By allowing each learner to progress at their own speed, monitoring real progress with real data, and motivating students in more ways than one, the Khan Academy is poised to flip the way we currently run our educational system. Bill Gates makes an appearance at the end of the TED video, applauding Khan’s program – a program Khan feels can be started around the country, anytime we’re ready. Khan sees this flip as what will allow classrooms the time to finally spend on those fun educational projects – like building a robot or measuring the height of a hill from its shadow.”

    I know the decrepit educational system of today is an emotional and heated topic for all of us, so I completely understand where you are coming from, but I do believe that you portrayed the article — and more importantly, the Academy itself — in an undeservedly negative light.

    We should all be free to make our own opinions with as much information as possible. The article also includes screen-grabs from the program’s interface, so we can all see how computers and data collection can truly revolutionize a classroom.

    Thanks for the mention 🙂 I wish you all the best and success with the educational crusade!

    Like

  6. Anna Loza
    November 28, 2011 at 4:16 pm

    Just a couple more quotations I forgot to include, to give the truly full picture:

    “Profiled in Bill Gates’ personal blog, videos of the participants show what they’re thinking about the new curriculum. Here, students talk of watching videos on rainy days to get their “energy” up and earn more points; they list badges they want to get, and speak of math being a necessary tool for life, not just a subject to get through.

    Students also love that the program supports individual learning styles – like daydreamers; and that students at any level can work at a difficulty they want, producing an extremely personalized result. As one teacher points out, teaching to the average level of understanding leaves out two-thirds of the class; the Khan Academy overcomes this, providing students with appropriate challenges and enhanced opportunity.”

    And:

    “Teachers also talk about how they love the program because their kids are having fun [key justification of this phrase follows, fluff would just stop there!] – they’re telling their parents of what they learned in class, and are also able to show them. They say the majority of their class now loves math, instead of just a couple kids. Most importantly, the teachers love how the program allows them to visualize the kids’ success, and how they can give the students themselves this information to look over. Apparently, it would take the teachers hours to compile the amount of data the software gathers and displays. The Khan Academy shows personal and group data very coherently, allowing in-depth analysis of progress as well as where each student spends their time; this way, teachers know exactly how to help each student.

    After such raving reviews of the Khan Academy [from those involved, not someone else!!], it’s only a matter of time before our educational methods are transformed, and technology figures larger in the classroom with optimal results.”

    Just making sure your readers get a clear picture to base their opinion on, and hoping that you, too, may realize that there are some parts that you seem to have left behind when making your opinion on both this article and the Academy.

    After browsing your blog and seeing what your main topic is — affording college — I am hard-pressed not to point out that the Academy may very well be an *extremely* valuable tool for students who are disadvantaged both monetarily and in attention — either in school or at home. Imagine a child that cannot afford tutors, but has regular access to a computer, or a child that is unfortunate enough to be going to one of those schools which we all know exists — classrooms where teachers couldn’t care less. Or students who need an extra push to stay motivated, whether it be to review concepts and not fall behind, or to work ahead of the curriculum and stay sharp. There is no reason to disapprove of free, highly educational, motivational (and did I say free?) videos.

    Here’s hoping all readers with students at home will take a look at the free videos, easily found on Youtube, and make their own opinion. And to be honest, everyone is very excited about this program because it has *huge potential* for success, and has already changed *at least some* lives. Is that not something to be hopeful about? Perhaps even ecstatic?

    Thanks for hearing me out!

    Like

  7. Anna Loza
    November 28, 2011 at 4:24 pm

    (OK, I can’t seem to stop!)

    The program charts success internally, compiling mounds of data about each student and their activities, as well as their progress (or lack thereof). It also shows data on the whole classroom! Something less attentive teachers may need to give them an edge when it comes to lesson planning.

    Whew, I think I’m good now. I guess I feel pretty passionate about this, as well!

    Like

  8. Jo in OKC
    November 28, 2011 at 5:29 pm

    I understand flipping the classroom, I really do. It’s just normally, when you flip the classroom you send kids home with a video or textbook and then you come back and the class does problems in small groups or as a class. The teacher then interacts, gives hints, identifies misunderstandings and clarifies what people have misunderstood (or missed completely). It’s easy for the teacher to step into or out of a group based on what the teacher hears.

    This is different because the instruction is computer-based and the problems are also all computer-based.

    I just wonder if you observed the classroom for a couple of hours, what would you see? Would you see kids raising their hands for help when they get stuck? Would you see teachers watching a monitor and actively seeking out kids who have trouble? Would you see kids accepting that help or viewing it as negative attention? Are kids celebrating their success and others successes in earning badges?

    Would you see and hear a dead silent classroom where all the sound is in headphones and all the kids are focused on the screen? Would kids be resentful of being separated from the “game” of earning badges?

    What lesson plans are there for the teacher to prepare if the kids are working on different topics and the instruction is done via computer at home?

    I’m all in favor of self-paced learning and subject acceleration. For these things Khan Academy seems great. It’s just that Khan Academy doesn’t require a teacher at all most of the time. Implementing online chat tutors seems easier than making sure that every teacher is fully up on every topic the kids are studying (since some could be WAY ahead) and the tutors could easily specialize in certain courses or topics since they’d be used over a large group of schools/students. That means the “real” teachers would be online and the people in the classroom are just there for crowd control. How many kids do you think would be in the classroom in that case? Can you envision a large gym full of kids on computers with just a few adults?

    I thought there were also concerns about screen time (even instructional screen time) for young kids and for kids with ADD.

    Like

  9. Anna
    November 28, 2011 at 8:14 pm

    Personally, as I understand it the bulk of the instructional videos can be done at home — turning that portion into ‘homework’ — and then what used to be called homework will be done in the classroom, sans computers, with the teachers. That way kids can learn at their own pace at home, but then be fully present with the teacher in the classroom, for polishing the concepts and working in groups, or what have you.

    Of course, the videos can be implemented in any way teachers or schools decide, so it may very well be that they can be watched in the classroom. So more research is definitely needed. But I personally am in favour of the first scenario I described.

    Like

  10. Grace
    November 29, 2011 at 7:15 am

    Anna, thank you for commenting!

    First, let me repeat that I am a big fan of Khan Academy, seeing the potential for it to improve efficiency and raise achievement levels in our schools. I value the KA features that engage students and allow them to work at their own pace, along with the easy to use progress tracking reports.

    However, my objection and the point of the post is that labeling a program a “colossal success” based only on its positive features is exceedingly premature and can lead to schools implementing wasteful and harmful “innovations” on students. This has been done repeatedly, leading to wasted money and lost opportunities.

    It’s important to innovate and look for ways to improve education, but I do not support paying for “new and improved” programs on a widespread basis unless they are shown to raise achievement levels. I want to see that data, along with a full consideration of all the variables involved. It’s that simple to me.

    Like

  11. Grace
    November 29, 2011 at 8:58 am

    “So more research is definitely needed.” Agreed!

    Like

  12. Jo in OKC
    November 29, 2011 at 9:17 am

    “learn at their own pace at home” is a great idea, but it’s not really your own pace if you have to do the same classwork as everyone else the next day.

    Anna, it sounds like your idea of flipping and my idea are the same. However, what gets show in pictures and talked about is kids in the classroom on the computer.

    Frankly, if the kids are going to go to school to be on the computer, why go at all? Why not do it all from home?

    Just wondering, what do they do for kids without a functional computer or internet at home?

    Like

  13. Anna Loza
    November 29, 2011 at 9:59 am

    Of course there is always more data needed — but I do not think that any school was chomping at the bit to completely throw their standard style out the window. The point is that KA shows a lot of potential, and that there are several, if not many, ways that it can be utilized in a classroom — even as an extra resource that kids can rely on.

    And of course there will always be questions and potential problems, which is why I am quite sure that no one is going to change anything in their school instantly, if at all. This is just an innovative possibility. My original point was that KA was being put down for reasons that I do not feel it deserved; any new program can and should be subjected to rigorous testing and all scenarios should be considered, but in the end we should not reject a potential positive step forward because there is not enough data — that data can take years to collect, but in the meantime, there is absolutely no harm, in my opinion, to suggest KA to the kids on their own time, and perhaps even use it for a classroom video here and there.

    It all comes down to this: the videos are well done, the data collection software is there, and educational videos are already utilized in classrooms to varying degrees — these are just potentially better ones and ones that are consistent in their structure. They are also free.

    The point is that it’s out there, schools don’t have to pay for it, collecting data is an inherent feature of the program itself, and…well, that both kids and teachers gave it a good rating. We don’t need to throw out our regular teaching styles and classroom structure, but we should acknowledge when positive and yes, successful, tools are designed.

    Like

  14. Grace
    November 30, 2011 at 6:46 am

    The problem is that the educational establishment has a history of throwing out existing “teaching styles and classroom structure” in favor of experimental programs, only to find out later that the new stuff is worse than the old is. This has happened in math, for example. TERC, widely considered a colossal failure, as well as several other math programs were inflicted on thousands of public school students because they were thought to hold promise as engaging, innovative curricula. After the fact, it was essentially acknowledged that the students had been used as guinea pigs.

    “These 19 curricular projects essentially have been experiments.”
    http://concernedcamdentonparents.blogspot.com/2008/11/terc-considered-experiment-by-national.html

    Much more concrete data needs to be collected and assessed before I will accept a new math program as a “colossal success”.

    Like

  15. Anna Loza
    November 30, 2011 at 9:40 am

    I definitely agree with you. As for KA, it has videos on every subject, and other than this pilot program in the classroom, I do believe one of its main purposes is to simply exist as educational content for all ages, with a focus on students, of course. It’s definitely tough to know one way or another, and I doubt the classroom-based version will be implemented anytime soon, but the utility and value of the videos themselves cannot be debated, I believe. Heck, I’m thinking of brushing up on some chemistry, myself 😉

    Like

  16. Jo in OKC
    November 30, 2011 at 9:48 am

    I’ve heard that the math and science Khan Academy videos are pretty good, but that the humanities ones are not so good.

    I think that the classroom-based version has already been implemented some places. There’s always a placebo effect (doing something improves things more than doing nothing, plus the more motivated teachers/admins are the ones to try things first), so I think we’ll see some good PR and then places will follow suit if they can get grants for the computers (which is where the Bill Gates Foundation will come in).

    It’ll be interesting to see what companies which have been in this market for a long time like Thinkwell (which essentially does the same thing — mini lectures on a topic with some practice, but no badges) and Aleks (which has no videos — all instruction is text) do to compete.

    Like

  17. Anna Loza
    November 30, 2011 at 11:34 am

    Hopefully competition will not be based on money (ha! one can wish, right?) but on innovating for better education delivery and that all-too-elusive success. There’s just so much going on that we can’t even begin to know what to use, what to keep and where to go from here with any certainty. But here’s to hoping great minds keep focused on the education of our children.

    Like

  18. Joanne Jacobs
    November 30, 2011 at 10:16 pm

    I thought about linking to the article, but was put off by the claim of “colossal success” with no supporting evidence. It’s fine to write about a pilot program that’s too new to evaluate. Just don’t jump to conclusions. Many “promising” ideas don’t pan out.

    Like

Trackbacks