---
title: "Can local AI help me cheat on my math test?"
description: "Checking vision capable models against college level math problems. Testing both their vision capability and reasoning ability. Correct answers first. Then total wall time, output tokens, and measured generation rate on the same five native-image problems."
pubDate: 2026-08-25
author: "discostew"
tags:
  - "local AI"
  - "vision models"
  - "math benchmark"
  - "Apple Silicon"
---

<div class="math-benchmark">
  <div class="math-hero-stats" aria-label="Headline benchmark results">
    <div class="math-hero-stat">
      <span>Fastest 5/5 result</span>
      <strong>Gemma 4 12B 8-bit · 8m 21.4s</strong>
    </div>
    <div class="math-hero-stat">
      <span>Qwen3.8 quality setting</span>
      <strong>27B Q4 · xhigh · 5/5 · 21,230 tokens</strong>
    </div>
  </div>

  <section aria-labelledby="math-test-images-heading">
    <h2 id="math-test-images-heading">The exact five test images</h2>
    <p class="math-section-deck">These are the actual PNGs supplied to every native-vision route below.</p>
    <div class="math-image-grid">
      <figure class="math-image">
        <img src="/images/posts/local-ai-math-test/01-tilted-square-circle.png" alt="Problem 1: Tilted square and circle" />
        <figcaption><span>01</span> Tilted square and circle<code>challenge-packages/geometry-area-challenges-v1/images/01-tilted-square-circle.png</code></figcaption>
      </figure>
      <figure class="math-image">
        <img src="/images/posts/local-ai-math-test/02-overlapping-circles.png" alt="Problem 2: Overlapping circles" />
        <figcaption><span>02</span> Overlapping circles<code>challenge-packages/geometry-area-challenges-v1/images/06-overlapping-circles.png</code></figcaption>
      </figure>
      <figure class="math-image">
        <img src="/images/posts/local-ai-math-test/03-nested-square-circle-square.png" alt="Problem 3: Nested square and circle" />
        <figcaption><span>03</span> Nested square and circle<code>challenge-packages/geometry-area-challenges-hard-v1/images/01-nested-square-circle-square.png</code></figcaption>
      </figure>
      <figure class="math-image">
        <img src="/images/posts/local-ai-math-test/04-quarter-circle-lens.png" alt="Problem 4: Quarter-circle lens" />
        <figcaption><span>04</span> Quarter-circle lens<code>challenge-packages/geometry-area-challenges-hard-v1/images/04-opposite-corner-quarter-circle-lens.png</code></figcaption>
      </figure>
      <figure class="math-image">
        <img src="/images/posts/local-ai-math-test/05-tilted-square-area.png" alt="Problem 5: Tilted square area" />
        <figcaption><span>05</span> Tilted square area<code>screenshots/source/tilted-square-area.png</code></figcaption>
      </figure>
    </div>
  </section>

  <section aria-labelledby="math-results-heading">
    <h2 id="math-results-heading">The results</h2>
    <p class="math-section-deck">Thirteen recorded local runs, ranked by correct answers and then summed task wall time. Model load and unload are excluded.</p>
    <div class="math-results">
      <article class="math-result-card is-perfect">
        <div class="math-result-top"><div class="math-rank">1</div><div><h3>Gemma 4 12B</h3><div class="math-tags"><span class="is-arch">Dense</span><span>8-bit</span><span class="is-thinking">Thinking on</span></div></div><div class="math-score"><small>Correct</small>5/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>8m 21.4s</strong></div><div><span>Total tokens</span><strong>20,305</strong></div><div><span>Generation</span><strong>41.21 tok/s</strong></div></div>
      </article>
      <article class="math-result-card is-perfect">
        <div class="math-result-top"><div class="math-rank">2</div><div><h3>Qwen3.8 27B</h3><div class="math-tags"><span class="is-arch">Dense</span><span>4-bit</span><span class="is-thinking">xhigh thinking</span></div></div><div class="math-score"><small>Correct</small>5/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>10m 33.1s</strong></div><div><span>Total tokens</span><strong>21,230</strong></div><div><span>Generation</span><strong>34.94 tok/s</strong></div></div>
      </article>
      <article class="math-result-card is-perfect">
        <div class="math-result-top"><div class="math-rank">3</div><div><h3>Qwen3.5 35B-A3B</h3><div class="math-tags"><span class="is-arch">MoE</span><span>4-bit</span><span class="is-thinking">Thinking on</span></div></div><div class="math-score"><small>Correct</small>5/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>11m 9.4s</strong></div><div><span>Total tokens</span><strong>56,443</strong></div><div><span>Generation</span><strong>85.83 tok/s</strong></div></div>
      </article>
      <article class="math-result-card is-perfect">
        <div class="math-result-top"><div class="math-rank">4</div><div><h3>Qwen3.5 35B-A3B</h3><div class="math-tags"><span class="is-arch">MoE</span><span>8-bit</span><span class="is-thinking">Thinking on</span></div></div><div class="math-score"><small>Correct</small>5/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>17m 7.3s</strong></div><div><span>Total tokens</span><strong>70,264</strong></div><div><span>Generation</span><strong>69.22 tok/s</strong></div></div>
      </article>
      <article class="math-result-card is-perfect">
        <div class="math-result-top"><div class="math-rank">5</div><div><h3>Qwen3.5 9B</h3><div class="math-tags"><span class="is-arch">Dense</span><span>8-bit</span><span class="is-thinking">Thinking on</span></div></div><div class="math-score"><small>Correct</small>5/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>19m 19.0s</strong></div><div><span>Total tokens</span><strong>74,240</strong></div><div><span>Generation</span><strong>64.84 tok/s</strong></div></div>
      </article>
      <article class="math-result-card is-perfect">
        <div class="math-result-top"><div class="math-rank">6</div><div><h3>Qwen3.8 27B</h3><div class="math-tags"><span class="is-arch">Dense</span><span>oQ8</span><span class="is-thinking">xhigh thinking</span></div></div><div class="math-score"><small>Correct</small>5/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>21m 39.3s</strong></div><div><span>Total tokens</span><strong>27,325</strong></div><div><span>Generation</span><strong>21.46 tok/s</strong></div></div>
      </article>
      <article class="math-result-card">
        <div class="math-result-top"><div class="math-rank">7</div><div><h3>Gemma 4 26B-A4B</h3><div class="math-tags"><span class="is-arch">MoE</span><span>8-bit</span><span class="is-thinking">Thinking on</span></div></div><div class="math-score"><small>Correct</small>4/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>3m 5.2s</strong></div><div><span>Total tokens</span><strong>13,894</strong></div><div><span>Generation</span><strong>78.65 tok/s</strong></div></div>
      </article>
      <article class="math-result-card">
        <div class="math-result-top"><div class="math-rank">8</div><div><h3>Gemma 4 26B-A4B</h3><div class="math-tags"><span class="is-arch">MoE</span><span>QAT Q4 base / Q8 MoE</span><span class="is-thinking">Thinking on</span></div></div><div class="math-score"><small>Correct</small>4/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>5m 50.0s</strong></div><div><span>Total tokens</span><strong>28,921</strong></div><div><span>Generation</span><strong>84.68 tok/s</strong></div></div>
      </article>
      <article class="math-result-card">
        <div class="math-result-top"><div class="math-rank">9</div><div><h3>Qwen3.8 27B</h3><div class="math-tags"><span class="is-arch">Dense</span><span>4-bit</span><span class="is-thinking">low thinking</span></div></div><div class="math-score"><small>Correct</small>4/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>7m 12.3s</strong></div><div><span>Total tokens</span><strong>14,694</strong></div><div><span>Generation</span><strong>36.09 tok/s</strong></div></div>
      </article>
      <article class="math-result-card">
        <div class="math-result-top"><div class="math-rank">10</div><div><h3>Muse Glimmer 30B</h3><div class="math-tags"><span class="is-arch">MoE</span><span>4-bit</span><span class="is-thinking">Thinking on</span></div></div><div class="math-score"><small>Correct</small>4/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>15m 1.6s</strong></div><div><span>Total tokens</span><strong>28,563</strong></div><div><span>Generation</span><strong>32.86 tok/s</strong></div></div>
      </article>
      <article class="math-result-card">
        <div class="math-result-top"><div class="math-rank">11</div><div><h3>Muse Glimmer 30B</h3><div class="math-tags"><span class="is-arch">MoE</span><span>8-bit</span><span class="is-thinking">Thinking on</span></div></div><div class="math-score"><small>Correct</small>4/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>17m 24.6s</strong></div><div><span>Total tokens</span><strong>20,268</strong></div><div><span>Generation</span><strong>20.08 tok/s</strong></div></div>
      </article>
      <article class="math-result-card">
        <div class="math-result-top"><div class="math-rank">12</div><div><h3>Qwen3.6 27B</h3><div class="math-tags"><span class="is-arch">Dense</span><span>4-bit</span><span class="is-thinking">Default thinking</span></div></div><div class="math-score"><small>Correct</small>4/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>29m 12.2s</strong></div><div><span>Total tokens</span><strong>60,679</strong></div><div><span>Generation</span><strong>35.13 tok/s</strong></div></div>
      </article>
      <article class="math-result-card">
        <div class="math-result-top"><div class="math-rank">13</div><div><h3>Qwen3.8 27B</h3><div class="math-tags"><span class="is-arch">Dense</span><span>4-bit</span><span class="is-thinking">medium thinking</span></div></div><div class="math-score"><small>Correct</small>3/5</div></div>
        <div class="math-metrics"><div><span>Wall time</span><strong>8m 51.8s</strong></div><div><span>Total tokens</span><strong>18,316</strong></div><div><span>Generation</span><strong>36.13 tok/s</strong></div></div>
      </article>
    </div>
  </section>

  <section aria-labelledby="math-machine-heading">
    <h2 id="math-machine-heading">Test machine</h2>
    <p class="math-section-deck">Every retained result on this page ran locally on the same Mac Studio.</p>
    <div class="math-machine-grid">
      <div><span>Model</span><strong>Mac Studio (Mac15,14)</strong></div>
      <div><span>Chip</span><strong>Apple M3 Ultra</strong></div>
      <div><span>CPU</span><strong>32-core CPU (24 performance + 8 efficiency)</strong></div>
      <div><span>GPU</span><strong>80-core GPU</strong></div>
      <div><span>Memory</span><strong>512 GB unified memory</strong></div>
      <div><span>OS</span><strong>macOS 26.6.1 (25G76)</strong></div>
    </div>
  </section>

  <section aria-labelledby="math-takeaways-heading">
    <h2 id="math-takeaways-heading">What these five problems say</h2>
    <div class="math-takeaways">
      <div><strong>Qwen3.8’s reasoning is the real upgrade here.</strong><p>At the same Q4 size, Qwen3.8 xhigh went 5/5 with 21,230 output tokens. Qwen3.6 went 4/5 with 60,679—2.86× as many—and took 29m 12.2s. That is consistent with much stronger 3.8 reasoning/post-training, although this benchmark cannot separate base training from post-training.</p></div>
      <div><strong>Qwen3.8 needs xhigh for this quality bar.</strong><p>xhigh: 5/5, 21,230 tokens. Medium: 3/5, 18,316. Low: 4/5, 14,694. Low is cheaper, but it lost an answer; medium lost two.</p></div>
      <div><strong>Q4 is the obvious Qwen MoE choice.</strong><p>Qwen3.5 35B-A3B kept 5/5 at both quantizations. Q4 used 56,443 tokens in 11m 9.4s versus Q8’s 70,264 in 17m 7.3s.</p></div>
      <div><strong>The Gemma MoE result is more nuanced.</strong><p>Its QAT Q4/Q8 hybrid matched full Q8 at 4/5, so quality held, but one long wrong trace pushed wall time to 5m 50.0s versus 3m 5.2s at Q8. Choose the smaller load unless latency is the priority.</p></div>
      <div><strong>Do not spend memory on Qwen3.8 oQ8 for this workload.</strong><p>Q4 xhigh and oQ8 both scored 5/5. Q4 used 21,230 tokens in 10m 33.1s; oQ8 used 27,325 in 21m 39.3s.</p></div>
    </div>
  </section>

  <section aria-labelledby="math-recommendations-heading">
    <h2 id="math-recommendations-heading">Best tested quantization by model</h2>
    <p class="math-section-deck">The practical choice from this retained five-problem evidence, not a claim about every workload.</p>
    <div class="math-recommendations">
      <article><span>8-bit</span><h3>Gemma 4 12B · Dense</h3><div>5/5 · 8m 21.4s · 20,305 tokens</div><p>The retained passing choice: keep the small dense model at Q8 unless a Q4 run proves parity.</p></article>
      <article><span>4-bit · xhigh</span><h3>Qwen3.8 27B · Dense</h3><div>5/5 · 10m 33.1s · 21,230 tokens</div><p>The quality choice: 5/5. Medium fell to 3/5 and low to 4/5; oQ8 kept 5/5 but took 21m 39.3s.</p></article>
      <article><span>4-bit</span><h3>Qwen3.5 35B-A3B · MoE</h3><div>5/5 · 11m 9.4s · 56,443 tokens</div><p>Kept the same 5/5 as Q8 with 56,443 rather than 70,264 output tokens and a shorter wall time.</p></article>
      <article><span>8-bit</span><h3>Qwen3.5 9B · Dense</h3><div>5/5 · 19m 19.0s · 74,240 tokens</div><p>The retained passing choice for this small dense model; it reached 5/5.</p></article>
      <article><span>QAT Q4 base / Q8 MoE</span><h3>Gemma 4 26B-A4B · MoE</h3><div>4/5 · 5m 50.0s · 28,921 tokens</div><p>Same 4/5 score as full Q8 with the smaller load; Q8 cut wall time to 3m 5.2s in this run, so use it only when latency matters more than memory.</p></article>
      <article><span>4-bit</span><h3>Muse Glimmer 30B · MoE</h3><div>4/5 · 15m 1.6s · 28,563 tokens</div><p>Matched Q8 at 4/5 while reducing wall time from 17m 24.6s to 15m 1.6s.</p></article>
      <article><span>4-bit reference only</span><h3>Qwen3.6 27B · Dense</h3><div>4/5 · 29m 12.2s · 60,679 tokens</div><p>Useful as the before case, but it missed one problem and generated far more tokens than Qwen3.8 Q4 xhigh.</p></article>
    </div>
  </section>

  <p class="math-footer">Five-problem native-image reasoning test · Local retained results</p>
</div>