Every Singapore secondary student sits an oral examination, one in English and another in their mother tongue. They read a passage aloud, then hold a short conversation with two examiners about a video or a picture. It carries a real share of the final grade. It is also, for many students, the component they practise least.
The reason is not that teachers do not care. It is arithmetic. An oral practice runs perhaps five minutes a student. A class of forty is more than three hours of one teacher's undivided attention, and that is only a single round. To give a class the four or five rounds before the exam that would genuinely build the skill, a teacher would need most of a working week with nothing else in it. So oral practice tends to happen once or twice a year, bunched around the mock exams.
And when it does happen, the feedback is thin. A teacher listening to forty students in real time, clipboard in hand, cannot write each of them a paragraph. The student hears a band and maybe a sentence: work on your fluency, develop your ideas more. True, usually. Actionable, rarely. Set it beside how the same student gets feedback on a piece of writing: annotated in the margins, specific, handed back to read again. Oral feedback is almost none of those things. The skill is examined like writing and coached like an afterthought.
I teach in a secondary school in Singapore, and I had watched this gap for years without an obvious way to do anything about it: a high-stakes skill that students barely rehearse, and barely get told how to improve. What finally set things moving was not a theory about AI in the classroom. It was a small request from a colleague.
What I built
My Head of Department for Mother Tongue wanted something modest: a way to transcribe students' oral recordings so that marking them was less of a slog, and free if possible. A transcription tool, nothing more.
She already knew the official option. Student Learning Space, the national platform, has a Speech Evaluation feature that scores a student's audio for accuracy, fluency, and words correct per minute. It is a real tool doing a real job. But it did not give her the plain transcription she was after, and it works the way the platform works: each teacher builds their own rubric, their own bands and mark ranges, before any grading happens.
Two things about that stayed with me. The SEAB rubrics are public; grading straight from the official rubric, rather than a teacher's hand-built copy of it, is less work and closer to the exam the student will actually sit. And accuracy, fluency, and words per minute are the metrics of reading aloud, where there is a reference text to measure against. The harder half of the oral exam is the open conversation, where the student is not reading anything and there is no script to compare the words to. I wanted to know whether that half could be graded too.
I also wanted it to be genuinely easy to use, easy enough that a teacher could set an oral task in a couple of minutes and a student could record one without being taught how. On a tool like this, usability is not a finishing touch. It is most of whether the tool gets used at all. Building the thing myself would teach me more about every part of this than recommending someone else's ever would. So I started with the transcription she had asked for, and kept going.
Riverside Sec Oral Conversation is a website. A student opens it, sees the oral task they have been set (a reading passage, or a stimulus image with discussion questions) and records their answer in the browser. The recording goes to Google's Gemini, which transcribes it. The transcript and the audio then go to Anthropic's Claude, which grades the response against the actual SEAB rubric for that student's exam level and writes feedback: what was strong, what to work on, specific to what the student actually said. The grading takes about twenty-four seconds. Then the recording lands in the teacher's review queue.
That last step is the one that matters most, and it is the one I want to be exact about. The AI's feedback reaches the student immediately, tagged as AI-graded. The teacher reviews every AI grade, and can change any score, rewrite any comment, or accept it as it stands; once they finalise, the label updates so the student knows the mark now carries the teacher's authority. The whole system bends around a single rule: the AI's grade is provisional, always, and the teacher is the final authority, always. Somewhere in the code there is a function that grades a recording, and a comment above it that says, in effect, never alter a grade a teacher has finalised. That comment is load-bearing. It is the design philosophy compressed into one line.
Today it handles thirteen distinct SEAB exam levels across Chinese, Malay, and English, from the higher-language syllabuses down to the foundation levels, each with its own rubric, its own mark scheme, its own shape. It is used by thirty-seven teachers and more than seven hundred students at the school where I teach. It has graded close to eight thousand recordings.
None of that is the interesting part. The interesting part is what happened to the classroom, and everything that broke along the way.
What it's actually for
When people hear that an AI grades the oral exam, they ask the same thing first: is it as accurate as a human examiner? It is the obvious question, and I think it is the wrong one.
Here is the one I would ask instead. A student finishes an oral practice. What do they get, and when? Under the old arrangement: a band and a sentence, several days later, if the teacher found the time. With the tool: a detailed, specific account of their own response, about twenty-four seconds later. Here is where your pronunciation slipped; here is where your answer stopped developing the idea; this is the part of the passage you rushed.
That is the actual product. Not accuracy. Personalisation, detail, and speed. A teacher cannot give forty students a personal paragraph of oral feedback every week; there is no arrangement of the timetable in which that is possible. The AI can, not because it is cleverer than the teacher, but because it does not tire, and there is effectively one of it per student.
And once feedback is fast and specific, something shifts in how often practice can happen. This is the part I did not predict. One of the teachers using it told me she now runs at least three times as many oral practices with her classes as she used to. Not because she has more time (she has exactly what she had before) but because the practice no longer costs her the marking. The student records; the AI does the first-pass feedback; she reviews. The bottleneck moved.
Three times the practice is not a small change. Oral skill, like most skills, answers to repetition with feedback far more than to a single high-stakes attempt. A student who has rehearsed a stimulus conversation ten times, hearing each time exactly what to fix, walks into the exam a different student from the one who did it once in a mock. The title of this essay says I built an oral-exam grader, and grader undersells it. The grading is the mechanism. The frequency is the point.
I should be honest about the other side, because the rest of this essay is about things going wrong. The AI is not a perfect examiner. You are about to read, in some detail, about a stretch when it could grade a student on words they never said. It does not have to be perfect. It has to be good enough that a student's tenth practice is better-informed than their first, and it needs a teacher reading behind it to catch what it gets wrong. Accuracy is not the foundation the system stands on. The teacher is. What the AI provides is reach: feedback for every student, every time, at a frequency no single teacher could sustain alone.
The unglamorous ninety-nine per cent
Calling Claude to grade a transcript is about ten lines of code. Calling Gemini to transcribe audio is about the same. If that were the whole job, the whole thing would have taken a weekend. It did not take a weekend. Those ten lines are perhaps one per cent of the system; the rest is everything that has to be true for those ten lines to be safe to put in front of a real child. Three of the things I learned the hard way.
The transcript that was never spoken
A transcription model turns audio into text. Give it speech, it returns the words. Give it silence, you would expect it to return nothing. It does not. Give Gemini a recording that is silent (a muted microphone, or a student who hit record during setup and never spoke) and it does not hand back an empty string. It hands back a fluent, plausible, entirely invented response: words the student never said, in sentences they never built. The same thing happens when a recording is mostly noise: a fan, a slammed door, corridor chatter. The model pattern-matches the noise into language.
On its own, that is a curiosity. Inside a grading pipeline it is something worse. The invented transcript goes to Claude, which has no way of knowing it is invented. Claude grades it competently against the rubric, and writes the student feedback on their fictional answer. For a window in April, a student could open the tool, see a band score and a paragraph of specific feedback, and none of it would have had anything to do with what they had actually done, because they had not actually done anything. The system was not broken in a way that announced itself. It was broken in the way that looks exactly like working.
That is the failure I think about most now. A wrong answer that looks wrong is a nuisance. A wrong answer that looks right is a hazard. A grade of zero tells a teacher to go and check. A confident, hallucinated thirteen out of fifteen tells everyone that all is well.
The fix is not one check. It is six, layered, because no single signal can be trusted on its own. The smallest recordings, too small to hold speech at all, never reach Gemini; they are marked silent on the spot. Gemini reports its own confidence, and low confidence is rejected. A second, smaller model (Claude's Haiku) looks at the result independently and flags transcripts that smell hallucinated. When Gemini is confident and Haiku disagrees, the audio is transcribed a second time and the two transcripts are compared: genuine speech yields roughly the same words twice, but a hallucination is invented afresh each run, and the two diverge. And in a multi-part exam, if two different recordings somehow produce an identical transcript, that is not coincidence; it is the model returning the same fiction twice, and both are thrown out.
None of this is clever. It is the opposite of clever. It is six unglamorous checks standing between a model that will lie with confidence and a student who would believe it. It is also the clearest case I can make for why the teacher stays in the loop, not as a courtesy, but as the design. The AI does not know when it is making things up. Someone who does has to be reading.
The four recordings I lost
On the seventeenth of April I lost four students' recordings. Not corrupted, not misfiled. Gone.
Riverside Sec Oral Conversation keeps its data in two places. The records (who recorded what, the grades, the feedback) live in a database. The audio files themselves live in object storage, a separate system. There was a feature that let a recording be retried when its grading had failed, and the retry, by my design, deleted the old one first: the database row and the audio file both. A bug sent four real recordings down that delete path when it should not have.
The database, it turned out, I could recover. It keeps a rolling history and can be wound back to a point in time before the deletion. The object storage kept no such history. I had never switched versioning on, because I had never thought hard about the difference between the two systems. The database forgave me. The storage did not. Four students' actual oral recordings were simply gone, and there was no version of asking nicely that would return them.
What I built afterwards is the thing I should have built before. Nothing is hard-deleted now. Before any recording is removed, its audio is copied into an archive and its record is snapshotted; both are kept for thirty days and swept up automatically after that; and there is a command I can run to restore anything inside that window. It is not sophisticated. It is simply present, which is the whole point, because the distance between this existing and not existing is four students who recorded an oral and trusted that it would still be there.
The lesson I would underline for anyone building on two systems at once: they do not have the same safety nets, and you will discover which one is weaker by losing something that lived in it. The time to learn that is not afterwards. Doubly so when the thing in the system is a student's assessment record. We will add backups later is a sentence that sounds reasonable right up until later arrives on a schedule of its own.
Three languages is not one feature
Riverside Sec Oral Conversation began in Chinese. English came about three weeks later, and Malay a week after that. The landing page says it supports three languages, which makes that sound like one feature that simply runs three times. It is not. It is three features that happen to share a screen.
Here is the small version of why. Gemini, the transcription model, has a bug where it sometimes returns nothing at all: no transcript, no error, just empty. The obvious fix is a fallback: when the main model comes back empty, try a smaller, faster one. So I tested the smaller one. In English it was excellent; I ran it against the main model on a batch of real recordings and the transcripts came back all but identical. In Chinese and Malay, the same smaller model hallucinated badly; more often than not it produced transcripts that were simply unusable.
So the fallback is not a setting. It is a decision made per language. English recordings, when the main model returns empty, drop to the faster one, because for English that is a safe trade. Chinese and Malay recordings do not; they take the slower path of waiting and retrying the main model, because for those languages a fast answer is a fast hallucination, and a slow correct transcript beats a quick invented one every time.
Multiply that single decision across everything else (the rubrics, the mark schemes, the way each language's exam is even shaped) and three languages stops being a feature and becomes three parallel problems, each needing its own measurements. The honest move was not to hunt for the setting that works for all of them. It was to accept there isn't one.
There is a smaller example I am fond of, because it is the opposite kind of decision. Early on, grades were stored under one form of identifier; later, a better one. Seven hundred-odd grades still sit under the old form, and more than four hundred of those have been finalised by a teacher. I could write a migration to tidy them. I have not, and I do not intend to. Migrating live assessment records for neatness is a risk taken entirely for my benefit and paid, if it goes wrong, entirely by the students. So the code simply understands both forms, with a comment marking the old path as load-bearing, and it will likely stay that way for years. Production maturity is not always the clean fix. Sometimes it is knowing which mess is safer left alone.
The pattern in all of these is the same. The part that looks like the product, where an AI reads the audio and returns a grade, works on the first afternoon. The part that makes it safe to hand to a child works for months, is mostly invisible, and is the actual job.
If you teach, and you want to build
More teachers are going to build things now. The tools have crossed the line where a genuinely useful classroom tool is within reach of someone whose main job is teaching, and that is good. If you are considering it, here is what I would pass on.
Your advantage over a software company building the same thing is real and specific. You know the rubric from the inside: not the document, the rubric, the way it is actually applied at four o'clock with forty more to go. You know what thin feedback feels like to receive, because you have given it, against your own better judgement, for want of time. You know which sentence of feedback a student will act on and which will slide straight off. A company building an AI oral coach is guessing at all of that and testing its guesses in usability sessions. You are not guessing. Build the thing only you can build, the one that turns on knowing the classroom, and do not try to out-engineer the engineers on the parts that are only engineering.
But the responsibility is real too, and it arrives quietly. The afternoon a student records an oral into your tool and trusts it with the outcome, you are running production assessment software, whether or not you ever use those words. The four recordings I lost were the moment that stopped being abstract for me. A side project can lose data and the cost is your weekend. This cannot. Build the safety nets before you need them, and assume you will need them.
And keep the teacher in the loop, not as a stopgap until the AI is good enough, but as the design, permanently. I want to be exact about why, because it is easy to read this as caution. It is not caution. The AI and the teacher are good at different things, and the good system uses both. The AI is tireless, instant, and the same for every student; it gives you reach a person cannot. The teacher has judgement, and context, and accountability; they can tell when a confident answer is wrong, and as the invented transcripts showed, that is not a rare event. A tool that sets out to replace the teacher throws away half of what makes it work. A tool that sets out to serve the teacher keeps both halves. The product was never the grader. The product is the partnership.
The gap I began with was a high-stakes skill that students barely rehearsed and were barely told how to improve. That gap was never a mystery, and it was never about a shortage of care. It was arithmetic: too many students, too few hours, feedback that cost more time than anyone had. What has changed is narrow, and I think large: a student can now practise speaking, and find out how it went, as often as they are willing to try. The exam has not changed. The amount of rehearsal a student can get before it has. If you teach, and you can build, there is a great deal of that kind of arithmetic waiting to be redone.