Stop Worrying About Shanghai, What PISA Test Really Tells Us About American Students

Sputnik replica
Sputnik replica

The latest results from the Program for International Student Assessment (PISA) are public, and already some pundits are declaring it “a Sputnik wake-up.” Others shout back that international comparisons aren’t valid. Rather than wade into that debate, I’d rather look more closely at the questions in the PISA test and what student responses tell us about American education. You can put international comparisons aside for that analysis. 

Are American students able to analyze, reason and communicate their ideas effectively? [Think Common Core standards] Do they have the capacity to continue learning throughout life? Have schools been forced to sacrifice creative problem solving for “adequate yearly progress” on state tests? For more on that last question see my post “As NCLB Narrows the Curriculum, Creativity Declines.” 

PISA provides some answers to those questions and offers an insight into the type of problem solving that rarely turns up American state testing. FYI: PISA is an assessment (begun in 2000) that focuses on 15-year-olds’ capabilities in reading literacy, mathematics literacy, and science literacy. PISA assesses how well prepared students are for life beyond the classroom by focusing on the application of knowledge and skills to problems with a real-life context. For more examples of PISA questions and data click here. 

Do American students learn how to sequence or simply memorize sequences

Here’s one insight into what American students can (and cannot) do that can be gleaned from the 2003 PISA test results. We spend a lot of time in school getting students to learn sequential information – timelines, progressions, life cycle of a moth, steps for how to. Typically the teacher teaches the student the sequence and the student correctly identifies the sequence for teacher on the test. Thus we treat a sequence as a ordered collection of facts to be learned, not as a thinking process for students to use.  This memorization reduces the student’s “mastery” of the chronology to lower order thinking. I was guilty of this when I first started teaching history “Can someone give me two causes and three results of WWII?” 

Sample sequencing problem from PISA

The Hobson High School library has a simple system for lending books: for staff members the loan period is 28 days, and for students the loan period is 7 days. The following is a decision tree diagram showing this simple system:

Pisa-1

The Greenwood High School has a similar, but more complex library lending system:
All publications classified as “Reserved” have a loan period of 2 days.
For books (not including magazines) that are not on the reserved list, the loan period is 28 days for staff, and 14 days for students. For magazines that are not on the reserved list, the loan period is 7 days for everyone.
Persons with any overdue items are not allowed to borrow anything. 

Task

Develop a decision tree diagram for the Greenwood High School Library system so that an automated checking system can be designed to deal with book and magazine loans at the library.  Your checking system should be as efficient as possible (i.e. it should have the least number of checking steps). Note that each checking step should have only two outcomes and the outcomes should be labeled appropriately (e.g. “Yes” and “No”).

Student Results

Only 13.5% of US students were able correctly answered the question. Does it really matter if students in Shanghai did any better? (The student results were rated on a rubric scale.) 

When students are asked to observe a process and develop a sequence they have an opportunity to use a full spectrum of higher-order thinking skills – they must recognize patterns (analyze), determine causality (evaluate) and then decide how they would communicate what they’ve learned to others (create). Sequencing can be taught across the curriculum at a variety of grade levels – we simply have to ask the students to observe and do the thinking.

In case you’re wondering,  correct response should look like this.
Click image to enlarge.

pisa answer
pisa answer 
 

Image credit/ NASA
 

21 Replies to “Stop Worrying About Shanghai, What PISA Test Really Tells Us About American Students”

  1. “Only 13.5% of US students were able correctly answered the question. Does it really matter if students in Shanghai did any better?”

    Well said.
    Ryan

  2. Peter,
    This is incredibly important because this is exactly the kind of thinking required of software developers, for example. I am not an educator, however I do work for a software company, and virtually nothing we deal with is sequential. This example alone explains a lot of the issues we have hiring people out of college- many cannot grasp complex interactions.

  3. Martin,
    Thanks for making the connection to the adult work environment. I once heard a similar lament from a project supervisor who asked her team of engineers to analyze a situation. They came back with bullet points which simply condensed the information. They didn’t recognize that a summary should be a creative process that includes analysis, evaluation and the creation of something new. Their approach mimics the student who summarizes by copying the bold face text out of their book. Students learn early on that summarizing is not much more than guessing what the teacher thinks is important.

  4. “Only 13.5% of US students were able correctly answered the question. Does it really matter if students in Shanghai did any better?”

    Yes, it does. Given that Asian countries teach procedures and skills (and yes, sequences) in the manner decried by reformists in the U.S., we maybe should not be so quick to negate the effectiveness of foundational knowledge. Blaming the AYP requirements is a red herring. Education schools and the current edu-thought world is permeated with the idea that students can be taught “critical thinking” and “higher order thinking skills”. Those come about by mastery of procedures that are too often given short shrift by those who deem them to be “rote learning”. Anyone familiar with the textbooks used in Singapore to teach math will recognize that what was once used successfully in the US over 50 years ago has been thrown out in favor of student-centered and inquiry-based learning.

    It is ironic that in the Asian countries that use the techniques derided as ineffective here, the students there are able to use higher order thinking skills to answer questions like the library.

  5. Barry,

    Thanks for your thoughtful reply. I think you make a strong case for why Shanghai-style teaching matters.

    I plead guilty-as-charged to your comment “the current edu-thought world is permeated with the idea that students can be taught “critical thinking” and “higher order thinking skills”.

    Since I post frequently on that theme, feel free to read some of my other blog posts to see why. ~Cheers~

  6. I am also wondering what, if anything, to make of the PISA results.

    For one thing, it is a bit like comparing apples and oranges, comparing the results from Shanghai, the biggest and best city in China, in which much of the country’s wealth is concentrated and in which (along with Beijing) most of its well-educated, well-traveled parents reside, with the results from whole nations where test-takers came from all sorts of backgrounds. How would the U.S. do if its test-takers were cherry-picked from the wealthiest cities or the best schools?

    I live in Beijing and constantly hear complaints from managers of foreign companies that their local employees cannot think outside the box to problem-solve, they don’t take risks or responsibility, they need to be told exactly what to do. They are good at imitating but weak at innovating. I am talking about highly educated engineers, managers, etc.

    Beijing, though not so glitzy, is like Shanghai in many ways: it is a first-tier city in China, it is the center of political power-brokering, and full of the rich and the filthy rich. Children in Beijing have no free time. They have so much homework. Often after-school time is taken up with cram schools and tutors. What we think of as elementary school curriculum is started in kindergarten. By middle school they are learning college level courses in science and math. Children are under extreme pressure to get good grades. There are many stories of high school grads who have committed suicide when they did not get into the best university.

    There is a price to pay to achieve those high test scores, so we better make sure it is what we want before we start heading in that direction. So, lets keep our heads. Let’s not be scare mongering about the Asians and their test scores. Let’s not throw the baby out with the bath water. Yes, let’s make sure students have solid mastery of the basics, but let’s keep our environment which encourages free thinking, critical thinking, innovation, creativity, passion, and deep interests.

    It would be interesting to test the PISA test-takers 15 years from now and see how they do on some parameters like accomplishments in one’s field, career satisfaction, being a productive member of the community, general life satisfaction, to see if math and science scores at age 15 mean anything at all.

  7. I spent a great deal of time in Bejiing as well and met many young adults who had the experience of being booted out of the “free” public school system… Many were fluent in English, Spanish, as well as Mandarin… These were very intelligent youndsters but had been cast aside by the Chinese educational system because of their poor performance on a standardized test… I believe in some ways the Chinese are “whistling past the graveyard” by giving up students at such an early age… Also, much is made of the relative amounts of money budgeted for educational purposes in the US and China… A large portion of the disparity in funding revolves around Special Education… I have never found any evidence of a serious special education program in China…

  8. Julie and Jeff,
    Thanks for taking the time to post comments detailing your experience in China. You give readers a view of the PISA data through a first hand perspective that’s lost in all the hype about the “Sputnik moment.”

    Neither “good at imitating but weak at innovating” nor “‘whistling past the graveyard’ by giving up students at such an early age” would seem to be strategies worth emulating.

  9. I agree with Barry’s comments: it seems to me we fool ourselves if we think we can teach the skills used to solve this problem. Much of our thinking skill is hard-wired into our brains; there’s natural inequality in this realm that teachers can’t do much to change. But what teachers CAN to is provide the knowledge (e.g. what “staff”, “loan” and “reserved” mean, and how to decipher punctuation) that optimizes our native thinking skills’ functioning.

    Chinese schools may go overboard at drilling the “lower-order” stuff, but America goes overboard on spurious “higher-order” skill building. Our kids are ignorant –I don’t see how ignorance can be the fount of any sort of worthwhile thinking.

  10. Ben,

    Many people – some teachers included – think that need to stay focused on the basics. I agree that students need a solid foundation (you can’t think critically about nothing) but I do fear that in many classes we don’t take the kids much further.

    We also assume that lessons start with the basics and then (maybe if we have time) kids get to apply them to solve a problem. That works, but you can just as easily use a problem to generate foundation knowledge. See my post “Watch Problem Based Learning in Action: Apollo 13” for an illustration. It turns the normal lower to higher order thinking sequence on it’s head.

    “… it seems to me we fool ourselves if we think we can teach the skills used to solve this problem.”
    I do think sequencing can be taught along with all the higher ordered thinking skills – analyzing, evaluating and creating. If I didn’t, then I wouldn’t have been a teacher.

    Watch an infant organize toys or household items into groups and you will see them analyzing elements (color, shape, or function), evaluate what’s important to them (shape not color?) and create a classification schema. This would equate to higher order skills done by a toddler without them receiving any instruction. (Want to test their system? Give them another item and see if they purposely add it to one of their groups.) So why can a toddler do this level of thinking, but a school kid can’t?

    “America goes overboard on spurious “higher-order” skill building.” Just the opposite – we spend far too much time on drill and kill. Know any good careers that look like filling out a worksheet?

    “Much of our thinking skill is hard-wired into our brains; there’s natural inequality in this realm that teachers can’t do much to change….Our kids are ignorant –I don’t see how ignorance can be the fount of any sort of worthwhile thinking.” Writing off a generation as “ignorant” rather limits our options for the future.

  11. Peter,

    Thanks for your reply. I’m puzzled by your mention of the toddler. Doesn’t this example prove my point, that critical thinking skills are innate, not imparted by a teacher?
    Such innate skills are strengthened or enhanced with additional knowledge, which is what good parents and schools give their kids. Know a lot about a particular domain and you can do superior higher order thinking about that domain. I find it hard to say anything intelligent about cricket because I don’t know anything about it. I can, however, make some insightful comments about Bay Area hiking trails because I know them well. To make good doctors we do not give them three years of thinking skill practice; we give them three years of cramming their brains with knowledge. Of course their studies do entail problem solving, but without tons of memorization, all the problem solving exercise in the world would not make them competent doctors. The knowledge (including memorized problem solving procedures) is the essential ingredient in their ability to evaluate symptoms. It seems to me you’ve bought into our culture’s bias, fueled by Bloom’s silly taxonomy, against “lower order” knowledge. Why must we denigrate knowledge? Come on, think critically!

  12. around my neck of the woods, i’d estimate 30-50% of the six year old boys are being held back a year “so that they can mature”. some parents openly admits they want to give their kids a better chance at getting on the football team.

    on the opposite side of the globe parents are struggling to afford extra private tuition in subjects like math and physics for kids the same age.

    i’m sure glad i have dual citizenship. it may come in handy.

  13. Here are some random thoughts/questions, in no particular order of importance…

    * What if education was provided by a large, independent set of content providers, each using varying methods?

    * What if parents who wanted to groom their kids to be football stars and parents who wanted to raise the next Einstein (and the vast majority in between) where free to do so with out the interference of the state?

    * What if tests were on-line, and allowed for the testing of specific content, the passage of which earned you “merit badges” to go on the next topic and direction of the parents’/child’s choosing?

    * Of what use is a school district or a superintendent? What do they do to connect neurons in a child’s head?

    * On what planet would a rational civilization create a LAUSD or a CPS, or even a small district?

    * How can there be a “neighborhood school” when the entire public apparatus is run by a conglomerate of protected (essentially privatized) interests who churn bonds, taxes, and contracts for their own financial gain?

    I suppose I could go on for days with these vignettes, but they all revolve around a simple truth that no one wants to admit.

    Education is an industry. It is regulated by those who have captured the regulatory apparatus, and this has created little mini-industries (reform, measurement, building, consulting, curriculum, transportation, etc.), all of which are playgrounds for academics and adults to bruit about never-ending debates that accomplish exactly nothing.

    The actual education of the child has become an afterthought (if that).

    Given that we have a system of “controlled chaos,” where no amount of money, “reform,” or other aspect of government force shows any significant improvement, why not allow for a system of “spontaneous order?”

    How might that look?

    1. Pick a state (nothing is going to happen nationally).

    2. Set up some rigorous, broad, sequenced standards, and create a testing system completely independent from the content providers. Give strong incentives to move testing on-line. (tech is getting good enough to allow for live coaching, oral exams, and even review of art/visual content) This should take 5-10% of the current ed budget.

    3. Take the remaining 90+% and have it follow the child to any content provider/educational institution that succeeds at training/educating the child up to the testing standards.

    4. Free up the entire system to design 100s, if not 1000s, of ways to meet standards. This means dissolving the massive political and bureaucratic apparatus currently spending entire states into bankruptcy with out showing that they can educate every child.

    Would such a system be perfect? No. Would some fall through some cracks? Yes, but nowhere near the number currently being utterly failed by the current system.

    Would there be political resistance to this transformation? Surely, but if educating such as large, diverse, nation is the goal, then it is time to ignore and/or defeat this political coalition.

    It doesn’t have to be this way. To begin the process of transformation, all you have to do is withdraw all political support from the existing system. It will rapidly start to collapse of its own weight.

  14. Bruno,

    Thanks so much for your detailed comment / analysis / proposal.

    You had me at: “Education is an industry. …The actual education of the child has become an afterthought (if that).”

    I think that while we care about our own kids, as a society we don’t care much about children. If we really cared about the next generation and their future would, as you say, “a rational civilization create a LAUSD or a CPS?.” As I only half-jokingly suggested in my post “The Classroom is a Factory, But What’s the Product?” we’ve turned our schools into factories that harness the labor of students to toil at a “bubble-test” assembly line producing “achievement” data.

    I agree with your assessment that the current educational system is ripe for collapse (Should we start a pool to see if North Korea goes down first?). Rather than a sudden “Angry Birds-like crash,” education will likely suffer the death of thousand cuts. A relentless accumulation of hopelessly bored students, departure of creative teachers, unmanageable NCLB regs, increasing numbers of parents who opt for home schooling, intractable labor-management gridlock, growing online alternatives, competition from engaging PBL schools, persistent failure to produce results, etc. Most of the legacy information gatekeepers are struggling to find a new model – Hollywood, network TV, newspapers, etc – why should education be exempt from a parallel transformation?

    I sometimes think Lewis Perelman got it right in his book “School’s Out.” He suggested the way to “nudge along” the crash of the educational-industrial complex was to amend the Civil Rights act of ’64 to add academic degree / certification to the discrimination prohibitions (race, color, religion, sex or national origin). He speculated that employers would have to screen potential employees for actual skills rather than “degrees.” That, in turn, would open the market for school that could actually produce results instead of meaningless paper. He envisioned education as sort of an academic “food court” (we get to eat together – but you picked Chinese and I got a taco.) From youth through adulthood you could cash in learning credits for “just-in-time” training at the appropriate time in your life. It’s been years since I read it – will have to dig it out and see if my memories correct.

    BTW – I’ll plead gulity to being one of “them” – a consultant – or as you put it, “little mini-industries.” But I thrived for over 25 years in the classroom by focusing creating engaging learning situations. Now, hopefully I make education slightly better by posting loads of free ideas on my blog. And if I find a thoughtful client, I do go on the road and make a few bucks. But I like to think my consulting approach models my classroom approach – designing instructional moments that provoke reflection and build capacity.

  15. Peter,

    I work at The Heartland Institute as the director for the Center for School Reform, so I’m a member of the “Reform” mini-industry.

    Perhaps the only difference is that I’m trying to make my job obsolete.

    I liked your use of the “education-industrial complex” above. I’ve used “government-education complex” for quite a while, and recently posted a definition here.

    http://somewhatreasonable.com/2010/12/the-government-education-complex-defined/

    Too bad I didn’t read this post sooner. I just got back to Chicago from Portland, where I was visiting family.

    Next time I’m in town, we can solve these problems over coffee.

  16. I agree – the best consulting does itself out of further work with a client. It’s about building capacity. Coffee in Portland sounds great! Let me know when you’ll be back.

  17. Thanks for the post Peter. I want to specifically address the following. Barry you stated…

    “Anyone familiar with the textbooks used in Singapore to teach math will recognize that what was once used successfully in the US over 50 years ago has been thrown out in favor of student-centered and inquiry-based learning.”

    I am very familiar with both Singapore and Hong Kong educational systems, classrooms, teachers, parents, and students. They have hired me to provide teacher professional development (etc.) because, among many reasons, students have difficulty tapping into their creativity and having the flexibility of mind required for solving practical problems. I have (and still do) work with teachers (administrators, parents and students) to promote student-centered and inquiry-based learning (IBL).

    Now, what I teach is a model of IBL that is balanced in its tactics, content, and context. Memorization is not a bad word, rather it serves a meaningful purpose. Students do not discover formulae by magic, they apply them and sometimes create them by analyzing data. I think that rote versus IBL is a false dichotomy. I also think that IBL needs a reasonable definition based on what we know works for learning.

    That is why I created the ‘Indicators of Inquiry’ and modified the ‘Five Essential Features of Inquiry’ for general application in all classrooms. You can see them on my blog – http://www.mininggemsblog.com

    Note that in the Five Essential Features of Inquiry there is a range of control from teacher to student. Depending on the content and context – the same classroom can be teacher-centered or student-centered and several variations.

    Singapore and Hong Kong clearly feel there is something missing and are always looking to improve. They do a wonderful job organizing and funding theirs, but I cannot say the same for our 50 plus educational systems. There is something to learn from all educational systems.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.