Context Matters
In June, our CTO, Ryan, sent me an article from PhilSchmid.de about Context Engineering. After a quick read I realized what they were describing was the approach we/I had originally settled on several years ago when I built my first AI Assistant for a couple colleagues teaching AICE Geography.
AICE Geography, in our school, is a rigorous course for high school freshmen who had never had an IGCSE Geography course in middle school. The class, at the time, had a 48% pass rate. In prior years the pass rate fluctuated into the 50’s and back, and was a strong indicator of success in the AICE program. Students who failed the test on the first attempt, prepared, and sat for a retest often did quite well. Those students, what I call the first pass and second pass, have a strong indication of success in the program overall. They have seen what the rigor looks like, understand what is needed to pass the course, and apply themselves appropriately. Those who do not pass this initial gauntlet and fail to recalibrate their efforts do not fare so well.
I had selfish motives in designing the assistant. My colleagues, who I like very much, had very little time and were always taking work home; my middle daughter was also taking the class. I needed the assistant to be a tool that worked great from the outset. AI was relatively new then and I did something that seemed counterintuitive to many: I shut off the assistant’s access to the internet. AICE classes are rigorous enough on their own, I didn’t want to muddy the waters by allowing just any data in, I wanted only sources that the three of us had vetted. This approach, coupled with testing data, gave us a very workable assistant.
I was doing the same thing with the assistant that we do with students: providing all of the information needed to solve a given problem before asking them to solve the problem. By doing this, I don’t need to spend much time training a user on writing great prompts, the assistant is already context specific, so prompt improvement will happen naturally, by the user interacting with the assistant.
By early Spring I had something usable to show my colleagues and they used it with zeal. Now, instead of us reading and comparing all data from previous years and trying to create meaning from it, we could query the assistant for the data we needed, check for accuracy and move ahead. Study sessions seemed to become more engaging. The instructors had a new love of their subject because prep didn’t take hours each evening, they could return to the part they loved best: teaching.
The next school year, when test results came in, the pass rate from the class had moved up to 58%. We were excited, but were still not convinced. The improvement may have just been due to our excitement, after all, the model had only been used for two months before testing. I worked on improvements and my colleagues doubled down, using the assistant to help them update slides, aggregate information to create better study guides, and laser focus their beginning weeks of instruction on where the common pitfalls were for the exam. They were exceptional educators before, now they had an AI tool that put the data they needed at their fingertips. They could build on the things that were successful and rebuild the portions that were shaky.
This year, when exams came in, the pass rate was almost 70%. For freshmen that have never had an IGCSE course to prepare them, we found these results exceptional. We’re excited for this year and I have built assistants specific to other subjects. My training classes for educators this year were full of teachers and administrators alike. Everyone wants a context-specific assistant, and in the future I think that is what will be most useful. AI won’t replace good teachers. You still need someone vetting output and adjusting delivery to reach a diverse group of learners. AI will absolutely facilitate the obsolescence of teachers who don’t care about their craft, doing the same thing day in and day out and hoping for smarter students to provide them with better results.
