Broken Arrows of English Exams | Toru Sasaki | TEDxRikkyoU
URL: https://www.youtube.com/watch?v=GtJg70xInW0 Video ID: GtJg70xInW0 ============================================================ Transcriber: Irina Campan Reviewer: Elisabeth Buffard [Connecting the Dots] [Broken Arrows of English Exams] Good afternoon! I used to be an English teacher at a high school in Tokyo. I remember how I started my first class in April 2001, when I was 27. I began talking to my students like this: Good Morning. I will talk to you only in English for the coming three years. I will write exam questions also in English. You may have spent time for memorising Japanese translations but you shouldn’t spend your time for that. Then a student said: “That’s not a good idea, teacher! In exams, you will anyway ask us to translate English into Japanese, right?” I sort of expected that response, so I said: “What if I say I won’t ask yo any translation?” This was my first conversation with my students. As I promised, I didn’t ask any translations in exams until they graduated. We spent time for learning sentence structures, summarizing passages and writing essays in English. I came to believe exams change student’s learning style. Next month, I had the second step and I asked my students to write a longer essay. I was very happ when I collected their works. I taught 4 classes, each of 40 students, so I got total 160 essays. With a pile of that size on my desk, I thought: this is the true language teaching. I’m doing an amazing job. I’m an excellent teacher! (whispers) By the way, do you know how long it takes to mark 160 essays? Do you know? At least 15 minutes for each, to read, correct, and write comments. Then it takes 2400 minutes, which was 40 hours I worked until midnight, but after a week, had a dream of being chased by piles of essays, they were shouting at me: “Mark me tonight! Mark me tonight!” Can you guess how I finished them? (paper shredding) I told this story to my colleague. Then she said: “Oh, you did it the modern way, I went more traditional.” (paper burning) This could have happened to you. You spent hours writing an essay, submitted it, but now it turned your teacher might have shredded or camp fired them. If exams and assignments change how you spend your time for them will also change. Then, does the national exam for university entrance ask you to write an English essay? No Why? Simply, it takes too much time to mark them. No technology can make a day 48 hours. I’m an AI student. My research field is Natural Language Processing, NLP. I know many of you use NLP technology, even every day. Good examples are Google Translation and smart assistants “Hey Siri!” I believe NLP is the only technology that can reduce the marking time. If I can mark an essay in 15 seconds, instead of 15 minutes, there could be an essay writing question in the national exam, which changes how you spend hundreds of hours and your ideas of what English learning is. Now let’s talk about AI. Actually, NLP based, Automated Essay Scoring systems have already been used in the US. Some famous exams such as TOEFL or GRES tests are partially marked by AES systems. However, we don’t use that system in Japan for entrance exams. Can you guess why? Unfortunately, AES systems don’t read or understand essays. All they do is calculation. When we build an AES system, we collect many sample essays and pre-fixed scores, we convert the sentences into numerical data and let the machine learning model calculate the relationship between the numbers and the scores. We get the functional formula. Once in exams, the AES system in turn uses the functional formula and calculates the appropriate scores for the examiner’s essays. All the calculation is done based on test similarity. So, if your essay was scored at 6 on a 10 point scale, it means that the numerical data of your essay was closest to those scored at 6 in the samples. Naturally, the sample data is critically important for the validity and reliability of such AES systems. If more data is available, the stronger association the model develops between the essay and the result. So if you wrote in a conventional style, with common words and phrases found frequently in the sample, you may get a higher score even though your essay is not very good. On the other hand, if you wrote in a unique way with rare expressions, your score could be lower, even though your writing is brilliant. This is simply because the machine learning model cannot find a strong association between your extraordinary essay and a high score. What you get from an AES system remains to be a calculated value, so in some cases, the result doesn’t fit our intuition. And we already have more advanced technology. Originally, scientists decided what language features to include in the system but current, state-of-the-art systems began to do such feature selection, autonomously, with what we call deep learning technology. There, when you ask, “why was my essay scored at 6 out of 10?“, the answer would be: “Sorry, machine only knows.” A black box lies in the middle. Maybe I talk too much about what’s happening inside AI. The story is not that complicated, so let’s make it simple. What do you want to know if you tak an important exam that is marked by AI? I’m sure you want to know how AI calculates your score, not just the result. In multiple choice questions, how we get scored is clear. But in most AES systems, we see the result, but how is hidden in the black box? To bring AES system to the mainstream of education, we should change the black box into white. You should be able to see why your essay was scored at 6 instead of 8. Japan is a country where equality is bad. University entrance exams affect our life a lot. With the black box and ‘machine only knows’ situations, we cannot prove that all essays are marked with equal validit and reliability. Classroom again. Essay writing is an effective way for learning English but is not common in exams. It takes too much time for marking. That’s why they mostly ask translation and “fill in the blank” type of questions. But at the same time, you know that’s not what you need after graduation. I see a broken arrow here. What you study and what you need to be able to do are not properly connected Today, many of you are students. I’m sure you’re encouraged to write and discuss in English in classes, which is great. Then, when you send your job application, what are you required to give, regarding your English competency? In most countries in Asia, that’s TOEIC test. Let me ask a question: have you ever taken TOEIC test? Raise a hand, if you have. Yeah, most of you, as I expected. Another question: Do you think what is measured in TOEIC is well connecte to what you need for the job? Raise your hand, if you think so. Hmm, much fewer, almost zero. Unfortunately, all TOEIC questions are in multiple-choice style. So by its nature, it cannot measure your ability at work to explain your idea or persuade others. Here, what you study, and what you need for future are apart from each other. The arrow is still broken in the middle. Even though you study mostly for TOEIC test at university, once you’ve started working, you need to express yourself. No more multiple choices, you need real English ability. So how can you manage that? A friend of mine attends an after-work English school There, professional tutors assist her study by marking and commenting on her English in fine details. That’s effective, yes. But she paid almost $ 5,000 for a three-month course. That’s not an option for everyone. [Re-connecting] So, how can we re-connect the broken arrow where we study English in Japan? Re-connecting means to match what you study, first with the exam contents and next with what you need to be able to do. It’s impossible for human raters to mark over 400,000 essays in the national exam. Its also impossible for all high school students in Japan to pay $5,000 for private tutoring. Can you think of any realistic way, other than the AES system that I talked about? It is possible to build a framework of AI for educational purposes where anyone can access AES systems from anywhere in Japan. From daily feedback, to student works, to marking of entrance exams. These systems should be accepte by students and teachers whilst the AI black box turns into white. Imagine. Our mindset will change towards English learning. After finishing elementary level of grammar and vocabulary, you write paragraphs and essays regularly in class and exams. Your writing is scored in a split second by the AES system with a white box. Now, what you study daily and what you need for future are re-connected. It is difficult to directly change a mindset itself but if we change exams first, they affect how you spend hundreds of hours for them, which largely reshapes your idea of what English learning is. You may say I’m a dreamer, but I’m sure, we together, can make a change in classrooms with the help of AI. Though it may be a small butterfly flipping in the beginning, I believe it will be a powerful tool to guide the education culture of Japan to the place where its supposed to be. Now is the time to re-connect the broken arrows of Japan. Thank you! (Applause)