can-local-ai-help-me-cheat-on-my-math-test.mdx
Post File

Can local AI help me cheat on my math test?

by discostew

Checking vision capable models against college level math problems. Testing both their vision capability and reasoning ability. Correct answers first. Then total wall time, output tokens, and measured generation rate on the same five native-image problems.

  • local AI
  • vision models
  • math benchmark
  • Apple Silicon
Fastest 5/5 resultGemma 4 12B 8-bit · 8m 21.4s
Qwen3.8 quality setting27B Q4 · xhigh · 5/5 · 21,230 tokens

The exact five test images

These are the actual PNGs supplied to every native-vision route below.

Problem 1: Tilted square and circle
01 Tilted square and circlechallenge-packages/geometry-area-challenges-v1/images/01-tilted-square-circle.png
Problem 2: Overlapping circles
02 Overlapping circleschallenge-packages/geometry-area-challenges-v1/images/06-overlapping-circles.png
Problem 3: Nested square and circle
03 Nested square and circlechallenge-packages/geometry-area-challenges-hard-v1/images/01-nested-square-circle-square.png
Problem 4: Quarter-circle lens
04 Quarter-circle lenschallenge-packages/geometry-area-challenges-hard-v1/images/04-opposite-corner-quarter-circle-lens.png
Problem 5: Tilted square area
05 Tilted square areascreenshots/source/tilted-square-area.png

The results

Thirteen recorded local runs, ranked by correct answers and then summed task wall time. Model load and unload are excluded.

1

Gemma 4 12B

Dense8-bitThinking on
Correct5/5
Wall time8m 21.4s
Total tokens20,305
Generation41.21 tok/s
2

Qwen3.8 27B

Dense4-bitxhigh thinking
Correct5/5
Wall time10m 33.1s
Total tokens21,230
Generation34.94 tok/s
3

Qwen3.5 35B-A3B

MoE4-bitThinking on
Correct5/5
Wall time11m 9.4s
Total tokens56,443
Generation85.83 tok/s
4

Qwen3.5 35B-A3B

MoE8-bitThinking on
Correct5/5
Wall time17m 7.3s
Total tokens70,264
Generation69.22 tok/s
5

Qwen3.5 9B

Dense8-bitThinking on
Correct5/5
Wall time19m 19.0s
Total tokens74,240
Generation64.84 tok/s
6

Qwen3.8 27B

DenseoQ8xhigh thinking
Correct5/5
Wall time21m 39.3s
Total tokens27,325
Generation21.46 tok/s
7

Gemma 4 26B-A4B

MoE8-bitThinking on
Correct4/5
Wall time3m 5.2s
Total tokens13,894
Generation78.65 tok/s
8

Gemma 4 26B-A4B

MoEQAT Q4 base / Q8 MoEThinking on
Correct4/5
Wall time5m 50.0s
Total tokens28,921
Generation84.68 tok/s
9

Qwen3.8 27B

Dense4-bitlow thinking
Correct4/5
Wall time7m 12.3s
Total tokens14,694
Generation36.09 tok/s
10

Muse Glimmer 30B

MoE4-bitThinking on
Correct4/5
Wall time15m 1.6s
Total tokens28,563
Generation32.86 tok/s
11

Muse Glimmer 30B

MoE8-bitThinking on
Correct4/5
Wall time17m 24.6s
Total tokens20,268
Generation20.08 tok/s
12

Qwen3.6 27B

Dense4-bitDefault thinking
Correct4/5
Wall time29m 12.2s
Total tokens60,679
Generation35.13 tok/s
13

Qwen3.8 27B

Dense4-bitmedium thinking
Correct3/5
Wall time8m 51.8s
Total tokens18,316
Generation36.13 tok/s

Test machine

Every retained result on this page ran locally on the same Mac Studio.

ModelMac Studio (Mac15,14)
ChipApple M3 Ultra
CPU32-core CPU (24 performance + 8 efficiency)
GPU80-core GPU
Memory512 GB unified memory
OSmacOS 26.6.1 (25G76)

What these five problems say

Qwen3.8’s reasoning is the real upgrade here.

At the same Q4 size, Qwen3.8 xhigh went 5/5 with 21,230 output tokens. Qwen3.6 went 4/5 with 60,679—2.86× as many—and took 29m 12.2s. That is consistent with much stronger 3.8 reasoning/post-training, although this benchmark cannot separate base training from post-training.

Qwen3.8 needs xhigh for this quality bar.

xhigh: 5/5, 21,230 tokens. Medium: 3/5, 18,316. Low: 4/5, 14,694. Low is cheaper, but it lost an answer; medium lost two.

Q4 is the obvious Qwen MoE choice.

Qwen3.5 35B-A3B kept 5/5 at both quantizations. Q4 used 56,443 tokens in 11m 9.4s versus Q8’s 70,264 in 17m 7.3s.

The Gemma MoE result is more nuanced.

Its QAT Q4/Q8 hybrid matched full Q8 at 4/5, so quality held, but one long wrong trace pushed wall time to 5m 50.0s versus 3m 5.2s at Q8. Choose the smaller load unless latency is the priority.

Do not spend memory on Qwen3.8 oQ8 for this workload.

Q4 xhigh and oQ8 both scored 5/5. Q4 used 21,230 tokens in 10m 33.1s; oQ8 used 27,325 in 21m 39.3s.

Best tested quantization by model

The practical choice from this retained five-problem evidence, not a claim about every workload.

8-bit

Gemma 4 12B · Dense

5/5 · 8m 21.4s · 20,305 tokens

The retained passing choice: keep the small dense model at Q8 unless a Q4 run proves parity.

4-bit · xhigh

Qwen3.8 27B · Dense

5/5 · 10m 33.1s · 21,230 tokens

The quality choice: 5/5. Medium fell to 3/5 and low to 4/5; oQ8 kept 5/5 but took 21m 39.3s.

4-bit

Qwen3.5 35B-A3B · MoE

5/5 · 11m 9.4s · 56,443 tokens

Kept the same 5/5 as Q8 with 56,443 rather than 70,264 output tokens and a shorter wall time.

8-bit

Qwen3.5 9B · Dense

5/5 · 19m 19.0s · 74,240 tokens

The retained passing choice for this small dense model; it reached 5/5.

QAT Q4 base / Q8 MoE

Gemma 4 26B-A4B · MoE

4/5 · 5m 50.0s · 28,921 tokens

Same 4/5 score as full Q8 with the smaller load; Q8 cut wall time to 3m 5.2s in this run, so use it only when latency matters more than memory.

4-bit

Muse Glimmer 30B · MoE

4/5 · 15m 1.6s · 28,563 tokens

Matched Q8 at 4/5 while reducing wall time from 17m 24.6s to 15m 1.6s.

4-bit reference only

Qwen3.6 27B · Dense

4/5 · 29m 12.2s · 60,679 tokens

Useful as the before case, but it missed one problem and generated far more tokens than Qwen3.8 Q4 xhigh.

Five-problem native-image reasoning test · Local retained results