Share and discuss this blog

Monday, January 23, 2017

To fight a Cyber War we need to train more people

I am of the age where the major preoccupation in my youth was avoiding having to go fight in the Viet Nam War. Ironically, during those years I was supported in my work by the U.S. Defense Department (DOD). I justified doing this because I wasn’t helping anyone kill anyone. If my work was to be useful to DOD at all, it would be helping us defend ourselves.

But now I am working with DOD on offense. What happened?

In the last year I have become more involved with cyber security. Why? I, and the people who work for me, primarily build online learn by doing courses (using live mentors to help when students are confused and to provide feedback on their work. When the DOD began talking to me about building a cyber operations course for them, I was interested. Developing new ways to learn is my business after all. So, I listened. I interviewed hackers employed by DOD and other federal agencies,  attended DefCon (a hacker convention held in Las Vegas) and over time I recognized what plenty of other people already knew. 

Here are some recent news stories I found about cyber attacks:

Today: Lloyds cyber-attack details emerge

Today: As attacks grow, EU mulls banking stress tests for cyber risks

Two weeks ago: Ukraine power cut 'was cyber-attack’

Two weeks ago: London NHS hospital trust hit by cyber-attack

Two weeks ago: Indian banks are waking up to a new kind of cyber attack

Three Weeks ago: U.S. Grid in ‘Imminent Danger’ From Cyber-Attack, Study Says

This is a serious issue and I want to help. We are, right now, building a course in cyber security. The Pentagon has a serious problem. Here is what Frank DiGiovanni, the Director of Force Training in the office of the Assistant Secretary of Defense for Readiness has to say: "The security of our nation is at stake. I think it’s imperative for DoD to embrace the hacker community because of the unique skills they bring to the table. They want to serve and contribute, and the nation needs them.”

This is from

DiGiovanni built an instructor led course, but there are limits to how many people can be taught face to face. Lecturing is not a really effective method for learning how to do something. We learn how to do things with one on one mentoring and we learn from trying and failing. 

DiGiovanni knows this:
“We infused the course with sociology, ethnography and anthropology.… You don’t conduct an assault on the enemy if you don’t know the terrain they’re in, what surrounds them.”

The social science disciplines help students better understand who they’re up against and why. Those facts can then be aligned with what we know of adversary’s signature techniques, tactics, and procedures.
“Techniques give clues about who they are and could also tip off what you’re after,” DiGiovanni says. This includes the way adversaries might seek to cover their tracks. For example, Russia adapted the concept of maskirovka – literally, masking –from conventional battlefield usage and applied it to the cyber arena. Students learn to identify the tactics of different adversaries, as well as the techniques that can be employed to cover one’s tracks. They have to become adept at identifying what the adversary is doing as well as executing their own cyber missions without leaving digital fingerprints in their wake.

“The biggest complaint about journeyman-apprentice is: It doesn’t scale,” DiGiovanni says. That makes it more costly and slower, compared to traditional teaching methods. Journeyman-apprentice is another core concept built into this course.

DiGiovanni doesn’t want to ditch the approach, just find a way to make it more efficient.

So, DOD contracted with my company to build a course to train tens of thousands.

We will help the military fill its large need for hackers by creating hackers. DOD wants to teach offense and defense. We can’t just simply sit back and defend, we have to frighten the enemy to stop as well as get into their systems. This is a lot like building missiles to defend against missiles.  

It crossed my mind that I could re-employ the defense part of the course and use it to train people who work for companies that may be subject to attack. I discussed this with one of the hackers whom we rely on as an subject matter experts in our course. I was told that I had it wrong. In fact, I had a lot wrong (after all why would I know?) 

Some stuff I (and most people) had wrong.

  1. Students would need to be people who can program (not true)
  2. Companies can hire the people they need. (They can’t be found)
  3. There must be some existing courses to train more (There are but they are short, or lecture based, or generally like most courses that try to teach complex skills quickly without using learning by doing with lots of practice and help.)  
  4. Businesses need defenders not attackers (This is completely wrong because some of the best cyber people are penetration testers who break into their own company’s systems to find out where they are vulnerable.)

We have developed just enough right now to be able to try it out on people who want to help. We are finding the oddest of people who want to do this (a massage therapist, an acupuncturist (OK, she was a computer scientist before she retired), a recent H.S graduate taking a gap year, and the former head of research at a big consulting firm. They are getting good at this and love it. (You need to be someone who gets into complex puzzles and generally thinks breaking into things is fun.)

We have a public website (which is changing very day) if you want to see more.

Below is a something from the first page of the course which lists what students will learn to do:

I am excited about this because I think it matters. Personally, I like having electricity and knowing that my money is secure when I use a bank.

Tuesday, January 10, 2017

Ten Questions about AI

I have had it with the stream of articles about what an “AI” can do. Yes, machine learning works. It is possible to analyze key words, correlate them with other key words, do a massive amount of statistics, and find out some stuff. People cannot do that and computers can. Is this AI? It sure isn't anything people can do, and it also doesn't correspond to anything I understand about what it means to be intelligent.

So, to make clear what AI is really about I propose the ten question test. Here are  ten questions that any person could easily answer and that no computer can answer well. Since I consider myself an AI person I would not say that no computer will ever be able to answer these. I hope we can figure it out. But AI isn’t here yet, despite what venture capitalists, giant companies, and the media keep saying. 

These questions can help explain why.

1. What would be the first question you would ask Bob Dylan if you were to meet him?

I am using this one because IBM’s Watson “met” Bob Dylan and told him that his songs were about love fading. First, I might point out that that conversation is insane. If you were to meet Bob Dylan you might have some things you'd want to know. I’d like to know if he feels that his songs are “literature." I’d also like to know if he thinks he helped a generation feel stronger about protesting injustice and war. I would not count all the words he used and tell him which word appears most often. Watson does not behave as intelligent entities do. Intelligent entities are curious.  They have things they want to know and can recognize who can answer questions that come to their minds about different arenas of life.

Here is another: 

2. Your friend told you, after you invited him for dinner, that he had just ordered pizza. What will he eat? Will he use a knife and fork. Why won’t he change his plans?

You will notice that eating is not mentioned in question 2. Neither are utensils. So how could an “AI” understand these questions.  It would have to know about how people function in daily life. It would have to know that we eat what we order, and that when we say we ordered food it means that we intend to eat it, and it also means that we don't want to waste it. It would also have to know that pizza is typically eaten with one’s hands. It might also know that Donald Trump famously eats pizza with a knife and fork and might mention that when asked.

3. I am thinking of driving to New York from my home in Florida next week. What do you think?

In order to answer the above question, one would need a model of why people ask questions like that one. It is hard to answer if you don’t know the person who is asking. If you do know that person you would also know something about what he is really asking. Does he have a car that is too old to make the trip? Maybe he has a brand new car and he is asking your advice about whether a long trip is a good way to break it in. Maybe he knows you live in New York and might have an idea whether the roads are icy there. Real conversation involves people who make assessments about each other and know what to say to whom based on their previous relationship and what they know about each other. Maybe the asker is really asking about a place to stay along the way (if the person being asked lives in Virginia say.) Sorry, but no “AI” is anywhere near being able to have such a conversation because modern AI is not building complex models of what we know about each other.

4. Who do you love more, your parents, your spouse, or your dog?

What does this question mean and why would anyone ask it? Maybe the person being asked is hugging their dog all the time. Maybe the person being asked is constantly talking about his or her parents. People ask questions as well as answer them. Is there an  “AI” that is observing the world and getting curious enough to ask a question about the inner feelings of someone with whom it is interacting. People do this all the time. “AI’s” do not.

5. My friend’s son wants to drop out of high school and learn car repair. I told her to send him over. What advice do you think I gave him?

If you know me, you would know how I feel about kids being able to follow their own interests despite what school wants to teach. So an intelligent entity that I told this to would probably be able to guess what I said. Can you? No “AI” could.

6. I just saw an ad for IBM’s Watson. It says it can help me make smarter decisions. Can it?

My guess is that this is something Watson can do. It can analyze data, and with more information a person can make better decisions. Could Watson make the decision? Of course not. Decision making involves prioritizing goals and being able to anticipate the consequences of actions. Watson can do none of that.

7. Suppose you wanted to write a novel and you met Stephen King. What would you ask him?

I have no idea what IBM is trying to say to the general public here. Apparently IBM  is very proud that it can count how many times an author says the word “love.” If I wanted advice on writing a novel I doubt I would ask Stephen King, but here is one thing that is sure. Watson wouldn't understand  anything he said about writing a novel and Watson won’t be writing any novels any time soon. Now as it happens my AI group frequently worked on getting computers to write stories of one sort or a another. We learned a lot from doing that. I am quite sure that IBM hasn’t even thought about what is involved in getting a computer to write novels. Having something the computer wants to say? Having had an experience that the computer is bursting to describe to people? That would be AI.

8. Is there anything else I need to know?

When might yir ask such a question? You might have had a conversation with a chat bot and found out how to get somewhere you were trying go. Then you might (if you were talking to a person) ask if there is anything else you needed to know. Answering that question involves knowing whom you are talking to. (Oh, yeah, there is great Ethiopian Restaurant nearby and watch out for speed traps.) Let’s see the chat bot that can answer that question.


9. I can’t figure out how to grow my business. Got any ideas?

It is obvious why this is a difficult question. But, in business, people have conversations like that all the time. They use their prior experiences to predict future experiences. They make suggestions based on stuff they have themselves have done. They give advice based on cases in their own lives and they usually tell personal stories to illustrate their points. That is what intelligent conversation sounds like. Can AI do that? Not today, but it is possible. Unfortunately there is no one that I know of who is working on that. Instead they are working on counting words and matching syntactic phrases.

They are also working on AI document checkers that will help Word with spell check, or grammar check. “NeuroGrammar™ uses its advanced neural-network artificial intelligence algorithms in order to analyse every noun phrase and verb phrase in every sentence for syntactic and semantic errors.”

How marvelous. So here is my last question:

10. Does what I am writing make sense?

Amazingly, this is hard. Why? Because in order to understand my points you need to match them to things you already think and see if I have helped you think about things better or decide that you disagree with what I am saying here based on your own beliefs.  You already have an opinion on whether my writing style was comprehensible and whether the points I made made sense to you. You can do that. AI cannot.   Do I think we could do that someday in AI? Maybe. We would have to have a complete model of the world and an understanding of what kinds of ideas people argue for and what counterarguments are reasonable. Intelligent people all do this. “AI’s” do not. An “AI” that understood documents would  not be a grammar checker.

It would be nice if people stopped pushing AI that is based on statistics and word counts and “AI people” tried to do the hard work that making AI happen would require. 

Tuesday, January 3, 2017

College is Over

College is over. Not today, and not next week, but it’s time has come. The evidence is simple and easy to decipher. Consider this:

The U.K. offices of Ernst & Young have announced they will stop requiring degrees, but instead will offer online testing and search out talented individuals regardless of background. Why? They say there is no correlation between success at university and success in careers.

Maggie Stilwell, EY’s managing partner for talent, said the changes would “open up opportunities for talented individuals regardless of their background and provide greater access to the profession”.

Why would E&Y do this and why do I believe this is just the tip of the iceberg? Last month, the US Department of Defense awarded my company a large contract to train Cyber Attackers and Defenders. The Pentagon official in charge who awarded  this contract, told me that at the start of World War II, the US knew it needed to train 10,000 fighter pilots quickly. Do you think they were concerned about whether their future pilots had college degrees?

Today, he told me, “we need to train 10,000 cyber attackers”. Is he concerned that these attackers have college degrees? Quite the opposite. He has talked with many cyber attackers in the U,S. and has hired them to help us instead of trying to be destructive. He has found that most of them were in Special Ed in school and many are on the autism spectrum. He cares about whether they like to break into things, and if they can delve deeply and patiently into complex puzzles. They don’t need to know algebra. They don’t even need to know how to program.

Can you learn cyber security in a university? You certainly can’t major in it. You will find a few courses in it in some colleges, but you can be pretty sure that the professors will not have been hackers themselves.

The truth of this can be seen when examining most any university program anywhere. I used to be the Chairman of the Computer Science Department at Yale. So, I was amused when two years ago the computer science undergraduates there had a mini-revolution, complaining that Google wasn’t hiring Yale CS graduates. The reason was clear enough. The faculty, many of whom were still there from my day, are, in essence, theoreticians. They may know how to program but they don’t really do it anymore and they want to teach about their new ideas and their latest theories.

I didn’t want to do anything different when I was a professor. I was worried about Artificial Intelligence not programing. So, I kept my best programmer with me whenever I moved, and he the taught my students to program. He was not even a professor but all Yale students from that time will tell you he was the best programming teacher they ever had. When I moved to Northwestern he came with me. I forced Northwestern to make him tenured faculty. But he has never been promoted in the 25 years he has been there. Why? He doesn’t do research, he just builds things and teaches students how to do that. Right now he is building an AI mentor that is a critical part of training 10,000 cyber attackers and defenders. Will he publish his work? He doesn’t care. Maybe we will write something together. But this is no way to succeed in the academic world and no way to train students. Students are cajoled into majoring in history or literature with no job prospects in sight. Even in Computer Science the faculty has little interest in preparing students for the real world.

The situation is worse than you think. The Chairman of the Economics Department at Columbia told me that Calculus is a requirement for their majors. When I asked why, he responded that since there is no business major at Columbia, most of the kids who came to NewYork as a way of entering Wall Street soon discovered their only path was majoring in Economics. The Department didn’t want to deal with a large volume of students so it created a Calculus requirements as way of getting rid of the less intelligent ones.

Faculty in general are not interested in the pragmatic concerns of their students. I once proposed a job related course of study when I was at Yale and was told by President Bart Giamatti: “we don’t do training, Roger.” (I responded that we did do training at Yale. We trained professors.)

Students buy into this idea at first. I once addressed the Freshman at Yale (the various Chairs were advertising the advantages of majoring in their respective departments.) I said only one thing: “major in Computer Science, get a job.” I was booed by the Freshman class. (All of the CS graduates that year (1981) went to work at Microsoft and are all quite wealthy today I assume.)  But Yale’s tradition is to train intellectuals (who typically came from wealthy families.) The world has changed, but Yale not so much.

Last winter my cousin sent her grandson to me for advice on what college he should attend. I asked him wanted out of life, and out of college. He said he wanted to start his own business (again a practical subject not typically taught in college because the faculty have usually never started  a business.) And, he said, he wanted a college with lots of “rah rah.” We settled on one of the big state schools that is very good academically. I spent this Thanksgiving with his mother and asked how he was doing. She replied that he was having a tough time because he hadn’t gotten into the fraternity he wanted to get into. 

He will not learn how to start a business in college, but he will have fun. (His school had a pretty good football season.) When the President of the U.S. says everyone needs to go to college, all he is really saying is that the high schools have failed and college is the only way you will learn to think at all. When he says “everyone needs to learn to code” I start to wonder. Can Mr Obama code? Why does he care?

But, others care, and although I think the experiment with coding “bootcamps” has been a failure, this is not because learning to code won’t get you a good job.

Here is what people are saying about these bootcamps;
Is it really possible to become a highly employable developer in just a few months? It certainly sounds that way if you look at the Facebook ads. A new coding bootcamp at Rutgers, for instance, says that you can “become a developer in just 24 weeks,” while Full Stack Academy, a for-profit startup, goes seven better–just 17 weeks–and has a 97% job placement rate to boot. But if you have no prior coding experience, and are looking for a well-paid computer-engineering job, you should be wary of such offers. The attractions of these courses are obvious. Software jobs routinely pay some of the highest wages in the country, and founders are often heard complaining about the “battle for talent.”

What’s more, the U.S. government has joined wealthy investors in supporting these shops.  Now, it’s as easy to attend a coding bootcamp as it is to go to any other college.
But coding bootcamps are starting to garner skepticism and for good reason. They’re still very new: the oldest have been out in the field starting careers for a few years at most. There’s also an increasingly vocal group of people who say that they cut corners, and that they can’t possibly impart in just a few months the skills that a coder needs to be effective on the job.

I agree that the time spent in them is too short. I also don’t like their methods. (You can’t learn to program from listening to a lecture.) But more and more we will begin to see these competitors to colleges and the colleges will not be able to adapt. Colleges are run by tenured faculty after all, and they don’t want change. Being a professor at a top tier school means hardly teaching at all. (I taught one quarter course every other year a Northwestern.) You work with PhD students and talk with colleagues and run around the world being famous. Undergraduates are not on your mind. (It is a really good job and no one wants to really have to work at teaching.)

Where will we get our cyber security professionals? Here is a report I found:


The cybersecurity workforce shortage -- which has 1 million job openings in 2016, and is projected to reach 1.5 million by 2019 -- is especially acute at hospitals and healthcare providers, according to one industry expert.
"Healthcare IT projects are being consumed with runaway EHR (electronic healthcare records) projects" says Bob Chaput, a healthcare information risk management and compliance expert, explaining the main reason he sees for the lack of qualified cyber staff at hospitals. "Secondarily, healthcare leadership has been slow to prioritize and fund cyber programs" adds Chaput, who is CEO at Clearwater Compliance, a cybersecurity firm with a specialty practice geared to helping healthcare CIOs and CISOs.

We will have to open cyber security “bootcamps” as well. (And data analytics bootcamps.) Soon there will be many of these programs that are intended to produce professionals, and these professionals will get hired if they can show what they have produced in the way of work products.

Gradually, college will go back to what it always was: “a four year vacation funded by my parents” as one of my Northwestern students told me. But that means that only rich people will go to college. If and when college is made free, then you and I will be paying for their vacation. Kids who want jobs will go to bootcamps. (I sure hope they change that name.)

Or, we could try making high school more pragmatic and job oriented. (Of course that will never happen as long as Common Core is the rule and the Testing companies own education.)

Either way, our 3000 colleges will turn into 100 colleges soon enough. I am all for Harvard and MIT being places where cutting edge research is done and where one can learn to be a researcher. But no matter how many times we say STEM, the fact is that we have more researchers than we need at the present time. Colleges, that is those that are not in the top 100, will need to adapt, or they will die. 

Tuesday, December 6, 2016

OECD should be ashamed; PISA scores announced; doing more damage

Last week I spoke at Online Education Berlin. I thought that I was the opening speaker, but the people who arranged this meeting had set up the OECD representative of PISA, Andreas Schleicher, to be first, so that I could respond to him. I was going to talk about AI and pragmatic learning in online courses, but altered my talk to be an anti-test rant in response to his remarks. He never spoke to me at any time. Here is one the slides I used:

Now, not everyone knows what PISA is, but you should know. PISA is the single worst idea being forced down people’s throats in the world of education. It is sponsored by the OECD, which I had previously thought of as an organization that was started to do good. 

I first became aware of the effects of PISA when I was invited to speak in Bogota, Colombia because they had come in 62nd on the PISA rankings. What that means is that they were in an international competition, the World Cup of testing, and Colombia felt they needed help because they came in 62nd. At this meeting it was clear that they needed help all right, not in getting their PISA scores better but in getting some perspective on education.

Here is a current announcement from PISA: 

Coming Soon: PISA Results
On 6th December 2016 at 11.00 am (Central European Time) the results from PISA's 2015 round of testing of 15-year-olds in science, reading and mathematics in 72 countries and economies will be released. 

How sad. 72 countries are waiting to hear how they are doing in the international math competition. Why would anybody care? Here is the analysis from last year:

"Asian countries outperform the rest of the world", according to the OECD, with Shanghai, Singapore, Hong Kong, Taiwan, South Korea, Macau and Japan amongst the top performing countries and economies. Students in Shanghai performed so well in maths that the OECD report compares their scoring to the equivalent of nearly three years of schooling above most OECD countries.
Of the 64 countries with comparable data up to 2012, 32 improved their reading performance while 22 showed no change and 10 deteriorated. If you look at performance at maths, 25 show an average annual improvement, 25 show no change, and 14 show a deterioration in performance.
Qatar, Kazakhstan and Malaysia recorded an average improvement in maths performance of more than eight points per year. The OECD report also praises Brazil, Chile, Germany, Israel, Italy, Mexico, Poland, Portugal, Tunisia and Turkey who it claims have "shown a consistent improvement" over time in maths performance.

I will keep you in suspense about who won this years competition. But let's think about what these scores might mean in each country. I will start with Greece, a country I know well because I was a consultant to a Greek ship owner (to help him invent software for the shipping industry) for many years.

Greece's main industries are tourism, shipping, industrial products, food and tobacco processing, textiles, chemicals, metal products, mining and petroleum. (This is from wikipedia)

I can tell you that there is not a single course in any Greek school that teaches shipping or tourism. They do teach Ancient Greek and Greek history and, of course, algebra and science. Greece came in #42 in PISA in 2012 and I am sure this made them very anxious. What makes me anxious is that Greek schools didn't care about teaching their kids about the jobs they might actually be able to get in Greece. Instead they are engaged in a competition to answer questions like this one:

Infusions (or intravenous drips) are used to deliver fluids and drugs to patients.
  Nurses need to calculate the drip rate, D, in drops per minute for infusions. They use the formula D = dv where
d is the drop factor measured in drops per millilitre (mL)
v is the volume in mL of the infusion
n is the number of hours the infusion is required to run.

1. A nurse wants to double the time an infusion runs for.
Describe precisely how D changes if n is doubled but d and v do not change.

2. Nurses also need to calculate the volume of the infusion, v, from the drip rate, D. An infusion with a drip rate of 50 drops per minute has to be given to a patient for 3 hours. For this infusion the drop factor is 25 drops per millilitre. What is the volume in mL of the infusion?

Now, I was a math major in college and I figured out math on my own when I was about 5. I can’t answer this question. My mind glazes over. I could see teaching nurses how to do these calculations in nursing school, but everyone in the whole world?  Why? I assume the chief engineer on a ship knows how to do math. I am also sure he went to engineering school and rather sure that he actually never does much algebra.  But PISA ensures that every 15 year old student should be able to answer questions like this. 

Wonder how France did? They came in 25th. This is important because science and math matter so much in France. The Major industries in France include automobile manufacturing, aircraft production, chemicals, electronics, machinery manufacturing, metallurgy and tourism.

Surely these require math you say? Well, if they do, you can be sure that the math needed is covered in the schools that teach metallurgy or chemistry. PISA is for 15 year olds. I would be happy to see France offer metallurgy in high school, or tourism, or aircraft production, but they don’t. They teach the basics which includes Moliere, Robespierre, Louis the 16th, and lots of algebra. My daughter attended high school in Paris when she was 13. She got a B in English.  I asked her if anyone in her class spoke English and she said “no.” “Then why did you get a B,”  I asked. She said she never heard of the subjunctive case. So I am pretty sure they also teach the subjunctive case in France which matters so much to a grammarian and so little to an actual speaker who has non-conscious knowledge of their language and can use it but doesn't know what it is called (nor do they care.)

I was wondering if they tested for knowledge of the subjunctive case in PISA and found this:

In the PISA 2012 Student Questionnaire an OCT was operationalised by asking students to indicate their familiarity – on a 5-point scale from “never heard of it” to “know it well; understand the concept” – with actual mathematics concepts (e.g. “polynomial function”) and foils (e.g. “proper number”). Foils were created by combining a term from grammar (i.e. proper, as in proper noun; subjunctive, as in subjunctive mood; declarative as in declarative sentence) with a mathematical term (i.e. number; scaling; fraction, respectively).  

So they expect it to be taught so that they can use it as a foil in math exams. Pretty clever PISA.

How about Finland? Finland is considered very progressive in education. They are said to be eliminating traditional subjects which would be a wonderful thing. And they win PISA quite often which would imply that maybe they aren't eliminating algebra. But surely this is because “math teaches you to think” which everyone believes and I believe as much as I believe that the Great Pumpkin arises on Halloween eve. 

Metals and engineering now constitute the largest sector of Finnish industry, with motor vehicles and machinery driving much of the growth of the late 1990s.

So, Finland teaches machinery in high school? I hope they do but I doubt it.

As I said, I was worrying about Colombia when I was asked to visit there after they “failed” PISA.

“Colombia is the largest export partner of Aruba (39.4%). The petroleum and natural gas coal mining, chemical, and manufacturing industries attract the greatest U.S. investment interest.”

Do they teach mining in schools in Colombia? No. I was advising in Chile too recently. Mining is major there too. Do they teach mining in Chile? No. I fought with the education ministry there who wanted to teach more algebra in order to do better on PISA tests. 

Today the PISA scores were announced. Singapore won. I was recently speaking in Singapore. The cab drivers told me what an undemocratic totalitarian country it was and I hadn't even asked. I met with the Chairman of Singapore Democratic Party who told me he can never win an election because the party in charge owns all the media and he never gets mentioned. Singapore thinks it has a great education system. I spoke to the teachers there maybe 10 years ago and suggested that maybe learning to think for yourself was more important  than rigidly practicing for math tests all the time. My message was received coldly by the teachers.   And then I went outside and many people on the street were cheering for me. The talk had been televised and average people, the doorman, the taxi driver, the hotel manager were all aware that they had hated school and had been taught nothing of value for their lives. But Singapore won PISA . Hip hip hooray.

After my talk in Singapore last year I got a call from people who wanted my help on a project to teach people to get better scores on TOEFL tests. China and India do well on PISA too. How does this relate to TOEFL tests? Many people in these countries really want to go to university in the US or the UK. They need to show they can speak English well and this is shown by?  Multiple choice tests of course. Prospective students need to have great TOEFL scores and great math scores to get admitted. So congratulations PISA, and to all winning countries, on helping your people get our of you country and study in the US the UK. That will help you get rid of your future scientists, who rarely want to leave the US after they get there PhDs here. (I rememberer trying to persuade a brilliant Indian AI student of mine to go back to his country and help it. He looked at me like I was crazy. Now he is the head of AI at Amazon.)

Does the US and UK have it right? Hardly.

In Santa Fe, New Mexico I asked the people  who ran the Indian school there what curriculum I could build for them and they said “Casino Management.” One can well understand why. Could I do it. No. The Governor vetoed it and said “more algebra.”

In Kansas I asked what they needed and was told aerospace engineering. “Really?” I asked. “Yes” they said, we build Lear Jets here and we don’t have a single aerospace engineering course anywhere in Kansas.” Could I do it? Of course not. We have Common Core instead of PISA in the US. But it is all the same. The 19th century curriculum reinforced by testing companies. 

Getting people to think for themselves, becoming capable of employment, able to live happy lives, would seem to be better goals to me, but instead we have PISA. OECD should be ashamed.

Here is part of what the OECD had to say today:

Around 1 in 10 students across OECD countries, and 1 in 4 in Singapore, perform at the highest level in science. Across the OECD, more than one in five students falls short of baseline proficiency: only in Canada, Estonia, Finland, Hong Kong (China), Japan, Macao (China), Singapore and Viet Nam do at least nine out of ten 15-year-old students master the basics that every student should know before leaving school.

This underlines the challenge that all countries, including some of the wealthiest ones, face in meeting Sustainable Development Goal 4 by 2030 to achieve “inclusive and equitable quality education and promote lifelong learning opportunities for all”.

The report reveals the policies in place that successful countries share: high and universal expectations for all students; a strong focus on great teaching; resources targeted at struggling students and schools; and a commitment to coherent, long-term strategies.

Canada, Denmark, Estonia, Hong Kong (China) and Macao (China) achieve both high standards of excellence overall and equity in education outcomes.  A number of countries have improved equity, especially the United States. But in Australia, Czech Republic, Finland, Greece, Hungary, New Zealand and the Slovak Republic, the share of students performing at the highest levels fell at the same time as the share of low performers rose.

“Achieving greater equity in education is not only a social justice imperative, it also fuels economic growth and promotes social cohesion,” added Mr GurrĂ­a.

The OECD PISA 2015 Survey underlines that, in the context of massive information flows and rapid change, everyone now needs to be able to “think like a scientist”: to be able to weigh evidence and come to a conclusion; to understand that scientific “truth” may change over time, as new discoveries are made, and as humans develop a greater understanding of natural forces and of technology’s capacities and limitations.

Everyone needs to know how to think like a scientist. Really? Why? And how would these tests assess that? Let's look at a typical PISA science question:

Bird migration
Most migratory birds gather in one area and then migrate in large groups rather than individually. This behaviour is a result of evolution. Which of the following is the best scientific explanation for the evolution of this behaviour in most migratory birds?
1 Flying in large groups allowed each bird to have a better chance of finding a nesting site.

Flying in large groups allowed other bird species to join the migration.

Birds that migrated individually or in small groups were more likely to find adequate food.

Birds that migrated individually or in small groups were less likely to survive and have offspring.

I found this question in an Australian newspaper today. Here is the headline that accompanied it:

Australian school students two years behind world's best performing systems

Or, in others words, we are all losers. We need to be very afraid. Inciting fear seems to be the major outcome of PISA. I wonder why. Mr Schleiker was proud that all countries will soon be the same and his test will make them be the same in their schools. This is exactly the opposite of what needs to be the case. School should fun. School should be relevant to life after school. And every country and every state is different. Their differences should be cherished. And no one needs algebra or science in high school. What we need is people who can think clearly, understand  scientific thinking, be creative, defend one’s ideas, diagnose problems and come up with solutions, That is what PISA should test. Actually PISA should test nothing. OECD needs too get rid of this idea completely. Perhaps they could put their money into helping countries teach kids things that will useful in their own countries.