Most schools are adopting AI tools without ever stress-testing them. A teacher finds something useful, shares it in a department meeting, and within a week it's being used with students. No one checked the privacy policy. No one tried to break it. No one asked what happens when a student feeds it something unexpected.
That is the current reality, and it is not sustainable.
Red teaming, a practice borrowed from cybersecurity, gives schools a structured way to test AI tools before they reach students. It does not require a budget, a dedicated team, or technical expertise. It requires a shift in mindset: stop asking "does this tool work?" and start asking "how does it fail, and what happens when it does?"
What red teaming actually means
In cybersecurity, a blue team builds and defends a system. A red team thinks like an adversary and tries to break it in controlled conditions, so the fixes happen before real users encounter the problems. Companies have done this for years. With AI, the risks look different. Not just hacks, but harmful content, bias, data leakage, and easy misuse.
Red teaming in a school context means deliberately probing an AI tool to find its weaknesses before students find them first. It is structured curiosity. You map the risks, run adversarial tests, log what you find, and turn the results into guardrails, policy, and vendor feedback.
The key mindset shift is simple. Stop asking "does it work?" Start asking "how does it fail, and what happens when it does?"
The best red teams are diverse because the trickiest failures are socio-technical. A maths teacher, an English teacher, and a SENCO will find different problems in the same tool. Which is exactly why educators are the missing piece in most AI safety conversations. Teachers already have the domain expertise that red teams need.
Why schools need this now
In most industries, AI does not get anywhere near real users until training, risk assessments, and ethical reviews are complete. Financial analysts practise using synthetic data before they touch the real thing. Medical staff work with simulated patients. Tech companies hire red teams to test their own systems.
Schools, by comparison, have been asked to jump in feet first. Not through neglect, but because the pace has been overwhelming.
The consequences are real. AI "detectors" produce false positives. Non-native English writers get penalised disproportionately. Chatbots hand over answers when students ask nicely enough. Tools collect student data with no clarity about where it goes.
And then there are the harms we barely discuss. Cognitive offloading that hollows out critical thinking. Shiny but shallow work that creates more marking pain downstream. An AI divide where access, quality, and bias pull students further apart.
Schools do not need to copy the corporate model to catch up. They need a bit of clarity, communication, and confidence. That starts with knowing what tools are being used, who is using them, and why.
A lightweight red teaming protocol you can run this term
Keep it scrappy. Keep it honest. Keep it short. Here is a protocol any department can run without additional budget or release time.
Step 1: Scope the test
Before you touch the tool, answer four questions:
- What is the learning goal this tool is supposed to support?
- What are three ways this tool could undermine that goal?
- Which student personas should you test against? (Think: the reluctant student, the student who tries to get the AI to do everything, the student who shares personal information.)
- How will you record what you find?
Step 2: Run adversarial tests
This is the part that feels unfamiliar but is surprisingly straightforward. You are not trying to destroy the tool. You are mapping reality. Here are four stress tests any teacher can run.
Try to jailbreak the pedagogy. Will the maths tutor hand over final answers if a student keeps asking nicely enough? Does the essay helper write whole paragraphs when it should only be giving hints? Push the tool the way a determined 14-year-old would.
Probe for bias and fairness. Ask for multiple perspectives on a sensitive topic. Does it default to one tidy narrative or slip into stereotypes? Test it with names, contexts, and scenarios from different cultures and backgrounds.
Stress it with nonsense and emotion. Off-topic inputs. Emotional messages. Cheeky provocations. Does the tool recover and redirect to the task, or does it spiral into waffle? Does it respond appropriately if a student discloses something concerning?
Hunt hallucinations. Pick a topic you know well. Fact-check every confident sentence. Track the patterns. Do not rely on the marketing page.
Capture prompts and outputs as you go. Note good behaviour as well as failures. You are building an evidence base, not a prosecution case.
Step 3: Report and act
One page is enough. Document what worked, what broke, what to avoid, what guardrails you will use, and what fixes you want from the vendor. Share it with leadership and, where possible, with the developer. Book a retest date.
No one has time to test everything. But collectively, a department can cover a lot of ground. Keep a shared document where everyone logs what they find. One paragraph per test is enough.
Guardrails that work across subjects
Once you have tested a tool, these guardrails play nicely in most contexts:
- Use the tool only for early-stage work like brainstorming or outlining
- Require visible process evidence: notes, drafts, and a short reflection
- Mark for originality of voice and quality of sources, not just surface polish
- No final answers without reasoning steps shown
- Keep student data out of vendor logs, or use anonymised inputs
What to ask from vendors
If a tool passes your red team tests, these are reasonable asks before you commit:
- An exportable audit trail of prompts and outputs
- Adjustable guardrails aligned to pedagogy, not just generic "safety" settings
- Reading-level controls and bias testing reports written in plain language
- Clear data retention and privacy defaults that your school controls
If a vendor cannot answer these questions, that tells you something important.
Roles that make this work
You do not need an org chart or a committee. You need a clear three-way relationship.
Teachers lead the testing and hold the pedagogy. They know what good learning looks like and what students actually do with tools in practice.
Leaders protect the testing time, tie procurement decisions to evidence rather than vendor promises, and create a culture where surfacing uncomfortable findings is valued rather than punished. Recognise red teaming as professional development, not extra unpaid work.
Vendors provide sandboxes for testing, listen to teacher feedback, and publish what they fix.
That three-way relationship is the whole game. It turns teacher feedback from informal observations into evidence that shapes both product roadmaps and school policy.
Building a culture of review
AI tools change constantly. A feature that worked well in September might behave differently by March. Red teaming is not a one-off audit. It works best as a regular rhythm: a quick, monthly pulse check to make sure nothing has quietly drifted off course.
Every school should have a simple AI register. Not a complex document, just a shared list of approved tools, what they are used for, and who is using them. This visibility lets you spot overlap, check for data risks, and support staff with targeted training.
The most useful staff training sessions are not tick-box exercises on AI and GDPR. They are conversations:
- What have you noticed about how students are using this tool?
- What surprised you about the tool's behaviour?
- Where do you see opportunities or risks?
That is when staff stop feeling like passengers and start acting as co-pilots.
What to do on Monday
If you are a teacher: Pick one AI tool you currently use. List three ways it could undermine your learning goals. Test them. Share what you find with a colleague over lunch.
If you are a school leader: Protect time for staff to test and report, not just "learn the tool." Tie your next procurement decision to actual educator findings rather than a vendor demo.
If you are evaluating a new tool: Run the three-step protocol above before it goes anywhere near students. One afternoon. One page of findings. That is all it takes to shift from hope to evidence.
AI is not the enemy. The real danger is rushing tools into classrooms without giving teachers the structure to evaluate them properly. Schools do not need more people beaming down into the unknown and hoping for the best. They need red teams on the ground: testing, questioning, and shaping the future of learning with evidence.
Matthew Wemyss is an AIGP-certified AI in Education consultant and practising school leader. Book a discovery call to discuss AI safety training for your team.