Can a Local Model Read My Inbox? A Bake-off

How much of my daily AI work actually needs to leave my machine? The frontier models still win the hard, open-ended thinking. But most of my real workload isn’t that - it’s repetitive, high-volume, and touches data I’d rather keep home: my mail, my documents, decades of personal history.

Email was the perfect test. I was building a system that reads my inbox for me - each message becomes a small structured record, so a daily brief can be written from records instead of raw mail. That per-message reading should run locally, if local is good enough.

So I ran a bake-off.

The job

Every model got the same task, one message at a time:

Read this email and return a structured record: a summary (three bullets or fewer), a type (newsletter / transactional / human / notification / junk), an importance score (1-5), and any actions or dates it contains.

Unglamorous on purpose. Faithful reading at volume is the job that decides whether the whole system is cheap and private or expensive and leaky.

The setup

  • Gold set: 26 emails picked to be representative, not easy - 9 newsletters, 6 transactional, 3 from humans, 3 notifications, 5 junk (two outright scams). The failure modes hide in the edges.
  • Judge: Claude Opus 4.8 as the quality ceiling.
  • Four axes: faithfulness (no invented facts - a confident hallucination is worse than a miss), classification, importance, and speed per email.
  • Serving: LM Studio on one Mac, one model loaded at a time, uncontended GPU only.

The result

ModelClassification (vs ground truth)Non-junk accuracyFaithfulnessSpeed / email
Qwen3.6-35b-a3b (thinking off) 🏆20/26 (77%)~96%excellent - no hallucinations~3.8s
Gemma-4-12b-it (8-bit MLX)17/26 (65%)strongstrong~13.8s
Gemma-4-e4b-it13/26 (50%)goodgood~4.5s
Qwen3.6-35b-a3b (thinking on)excellent (sampled)-excellent~50s (rejected)

Qwen3.6-35b-a3b with thinking off swept everything - fastest, best classifier, best summaries. It’s a 35B mixture-of-experts model with only ~3B parameters active per token, which is why it beats a dense 12B on the clock while out-reading it.

Classification accuracy plotted against seconds per email. Qwen 35B-a3b with thinking off sits alone in the fast, accurate corner at 77 percent and 3.8 seconds, beating the dense Gemma 4 12B - 65 percent at 13.8 seconds - on both axes at once. The mixture-of-experts model out-reads and outruns the dense one because only about 3 billion of its 35 billion parameters fire per token, which is the whole reason speed is not size. Across all three, on 26 emails, zero hallucinations
Classification accuracy plotted against seconds per email. Qwen 35B-a3b with thinking off sits alone in the fast, accurate corner at 77 percent and 3.8 seconds, beating the dense Gemma 4 12B - 65 percent at 13.8 seconds - on both axes at once. The mixture-of-experts model out-reads and outruns the dense one because only about 3 billion of its 35 billion parameters fire per token, which is the whole reason speed is not size. Across all three, on 26 emails, zero hallucinations

What I learned

Faithfulness is solved. Zero hallucinations across all three finalists - exact dollar amounts, order numbers, flight codes, version strings, all pulled clean from the source. And that’s where the privacy dividend points: the most sensitive details in my mail are exactly the ones a local model can read and a cloud model never has to see.

Speed is not size. The 35B model was the fastest. Only ~3B of its parameters fire per token, so it beat the dense 12B on wall-clock while reading better. Architecture beat parameter count on both axes - the most counterintuitive takeaway here, and the one that changed how I pick local models.

The shared weakness is junk and scams - and it barely matters. All three tend to file scam mail as “newsletter”: they read the content faithfully but miss the intent. In the real pipeline, junk gets filtered upstream before the model ever sees it, and a scam rubric in the prompt closes most of the rest.

The gotcha the whole result rests on: Qwen’s “thinking” has to be turned off in the LM Studio UI, not through the API. With thinking on, the same model burns ~2,000 reasoning tokens and ~50 seconds per email. The API flags that claim to disable it were silently ignored in my setup - the only switch that worked was the load-time toggle. The production runbook now asserts “thinking off” before any batch runs. This is the kind of detail a bake-off exists to surface.

The verdict

Qwen3.6-35b-a3b, thinking off, is the enrichment workhorse. Gemma-4-12b-it is the fallback.

The bigger result: for faithful, structured reading at volume - on the most private data I own - local inference is already good enough. Fast enough to chew through a whole inbox overnight on one Mac, accurate enough to trust, nothing invented. The frontier model stays only where it earns its keep: the synthesis, the writing, the judgment calls.

One data point, not a doctrine. The next entries in this notebook run the same question on other jobs. On the inbox, the on-device thesis held.

Method notes / reproducibility

  • Gold set: 26 emails, hand-labeled for type, importance, and key facts. Held constant across all models.
  • Grading: classification scored against the hand labels (“defensible alternative” tracked separately from hard errors); faithfulness judged by checking every extracted specific against the source; importance judged by whether action-required items outscored noise.
  • Speed: total wall-clock divided by message count, uncontended GPU only.
  • Serving: LM Studio on a single Mac; one model loaded at a time; Qwen run with thinking disabled at load time.
  • Source data is not published - the underlying messages were personal mail and stayed on the machine, which is rather the whole point.