Same score. Different answers.

We took a medical AI model (BioMistral-7B) and gave it 400 exam questions from MedQA, the US medical licensing exam set.

Then we did what everyone does before putting a model on a phone: we compressed it from 8-bit to 4-bit.

The exam score barely moved (47% → 50%).

But 1 in 6 answers changed.

One of them:
A 65-year-old is losing weight and gets weak on the stairs, but the weakness eases after a little exercise.
🟢 Original model: a warning sign of lung cancer ✅
🔴 Compressed model: myasthenia gravis ❌

Same score. A missed cancer.

That's the problem with judging AI by one number. A change that looks harmless on the scoreboard can quietly rewrite what the model tells people.

So when Eppy, our AI agent, tunes a model to run faster on a phone, it checks the answers, not just the score.

On a real Android foldable:
⚡ 1.4× faster
✅ Only 1 in 33 answers changed (3%)
📊 Same accuracy (50.5% → 52.0%)

If you're shipping AI into healthcare, finance or anywhere it matters:
👉 Don't just check the score. Check the answers.

Research benchmark on exam questions, not medical advice. Related work: Microsoft Research's "Accuracy is Not All You Need" (2024) found the same effect on general models.

Changing your AI? We’ll find what broke. myturn@ephard.ai