Skip to main content
2,200+ contractors in the research universe benchmarked. Do you know your gross profit per labor hour? See where you stand →
Level
Strategy

68 AI Model Runs Tested on 924 Real SMB Close Checks

Sam YangEx-CFO across trades, SaaS & services · $2.5B in total PE/banking transactions · Stanford MBA
Updated November 13, 2026·Originally published October 18, 2026·10 minute read
Share

LevelBench Research (v5.0 · Nov 13, 2026)

Top frontier models from OpenAI and Google now cluster between 92.9% and 95.2% across 924 real SMB close checks, yet every unassisted model still misses 44+ cutoff, roll-forward, and missing-check items. In live client books, closing that final gap depends on accounting playbooks and controller review rather than model brand.

Sam Yang, Stanford MBA, ex-CFO across trades, SaaS, services

10 minute readStrategy

The short answer

LevelBench v5.0 is a vendor-neutral AI accounting and CFO benchmark covering 68 live model and playbook runs across 8 AI labs on 60 anonymized SMB client closes (924 checks), a 25-company close subset (372 checks), and a 374-company close vault (5,923 checks). Frontier reasoning models from Google and OpenAI cluster within roughly two points (92.9% to 95.2%), while every AI + Level Playbook setup across all 8 providers reaches 100.00% down to $0.0039 per close task.

Key takeaways

  • Across 60 anonymized SMB client closes (924 checks), frontier models sit within overlapping +/-1.4% confidence margins: Gemini 4 Carbon (95.1% to 95.2%), Argon (94.7%), OpenAI GPT-6.1 Sol High (93.3%), and GPT-6 Astra (92.9% at $0.022/task, roughly one-third Carbon's cost), while Moonshot Kimi K3.1 leads open-weight models at 81.3%.
  • Every raw model still misses 44 to 440 checks (4.8% to 47.6%) alone. Roughly 45% of frontier misses hit cross-month accrual reversals or multi-tab inventory roll-forwards, and 30% guess expense categories when check images or PDF bank statements are missing.
  • Each provider has distinct engineering trade-offs: OpenAI GPT-6.1 and Astra isolate reasoning budgets cleanly at default settings, whereas Vertex AI counts internal thinking tokens inside configured output limits (cutting off 21.3% of Carbon Medium and 100% of Carbon High tasks at 8,192 tokens unless raised to 65,536).
  • Pairing even a lightweight $0.0039/task worker model with Level Playbooks and controller review hits 100.00% across all 374 client closes (5,923 checks) in 1.1s typical speed. Download all 40 public Excel workbooks on the Agentic Accounting Benchmark page.

See how your operating numbers benchmark.

Compare your margins and cash flow against industry benchmarks or view flat-rate CFO & bookkeeping pricing.

Every week, a business owner or controller asks me the same question: "Can I just connect ChatGPT or Gemini to QuickBooks and let it run my books?"

Most software demos answer that question with a cherry-picked bank feed categorization. They show an AI model reading a clean coffee receipt and picking "Meals and Entertainment." Existing accounting benchmarks test AI on clean textbook tables where a human accountant has already removed every contradiction.

Real SMB client takeovers look nothing like that. When a finance team takes over a $3M to $12M contractor or ecommerce brand, the books carry missing context and conflicting human footprints. A prior bookkeeper posted a year-end manual Journal Entry to clear Uncategorized Expense without tagging the original rows. Paper checks lack check-stub images, walk-in supply-house debit swipes match the dollar amount of an open vendor bill, and Slack notes contradict the signed contract.

To see what actually happens on real multi-tab client workbooks, we built LevelBench v5.0. It runs 68 live multi-cloud evaluations across 60 anonymized SMB client closes (122 tasks, 924 checks), a 25-company close subset (52 tasks, 372 checks), and our 374-company close vault (752 tasks, 5,923 checks). All 40 public Excel close packages are downloadable on our Agentic Accounting Benchmark hub.

How we built a benchmark around real SMB takeover messes

Most AI benchmarks fail finance operators for two reasons: the prompts tell the model the answer, and the test data has already been cleaned up by a controller.

In real life, nobody hands an operator a clean table and says, "Please calculate the ASC 330 landed inventory cost excluding outbound shipping." Instead, you get a messy QuickBooks export and an unlinked manual journal entry from the prior bookkeeper alongside freight bills, Shopify payout exports, and a conflicting Slack note.

To mirror real operations, LevelBench v5.0 enforces three strict rules:

  1. In our unassisted zero-shot tier, models receive only the anonymized entity Chart of Accounts, realistic client onboarding memos, and unfiltered ledger exports. They never receive pre-digested accounting formulas or labels warning them which rows are traps.
  2. Every workflow embeds actual bookkeeping hazards we encounter across SMB cleanups:
    • In WF01, a 7999 Uncategorized Expense row was already reclassed by a prior bookkeeper's manual Journal Entry (JE-0899), so reclassifying the bank row double-counts the expense.
    • Also in WF01, $20,200 of paper checks lack check-stub images while a clerk guesses "probably materials" in Slack, alongside a $22,500 credit-line paydown with a misleading mobile transfer memo.
    • In WF02, a technician's walk-in debit-card supply pickup matches the exact dollar amount of an open vendor bill, tempting raw AI to clear Accounts Payable instead of booking direct COGS.
    • In WF03, wholesale orders paid by physical paper check sit inside QuickBooks Undeposited Funds (1210) outside the Stripe merchant batch.
    • In WF06, field CRM jobs mix signed base contracts with unsigned upgrade options while cross-month subcontractor bills require Monday-to-Friday workday proration rather than calendar days.
    • Across WF09 to WF11, QuickBooks groups Current Portion of Term Debt inside Other Current Liability, VIP dinners sit in 6100 Marketing, and seller brokers try adding back recurring software and rainy-day drive time.
  3. We test the exact same checks across unassisted models and playbook-governed runs so finance leaders can measure how much accuracy comes from the model versus accumulated operator guardrails.

The three setups we tested

Here is how the three setups work on real client books.

  • In Unassisted Raw AI Model runs (39 live runs across 31 model configurations), you hand raw Excel workbooks and handover notes directly to a model with zero hints. That pool includes Gemini 4 Carbon and Hydrogen, GPT-6.1 Sol and Terra, GPT-6 Astra, Claude Fable 5.2 and Sonnet 5.5, Moonshot Kimi K3.1, and DeepSeek V3.2.
  • In AI + Level Playbooks + Operator Review runs (29 live runs), the AI runs behind Level's 0.03-second workbook check, 18 audit gates, 61 controller rules, and 16 historical error catchers built across 2,200+ past SMB cleanups. Missing check images or bank statements are held for human review before posting to QuickBooks.
  • In a separate Agentic CLI Harness side evaluation (6 multi-file runs), we also tested command-line workspace tools (OpenAI Codex CLI, Google Gemini CLI, and Antigravity CLI) equipped with Level's close checklist.

The LevelBench v5.0 leaderboard: representative results across 68 live runs

Below is the headline comparison across representative runs from the 68-entry LevelBench v5.0 leaderboard.

RankSystem or ModelTest Sample & ChecksAccounting AccuracyExact FormattingMistake RateTypical Speed (s/task)Cost ($/task)
1Level Playbooks + Gemini 2.5 Flash-LiteFull 374 Closes (5,923)100.00%100.00%0.00%1.10s$0.0039
2Level Playbooks + Gemini 3.8 FlashFull 374 Closes (5,923)100.00%100.00%0.00%7.75s$0.0179
4Level Playbooks + GPT-6.1 Terra (Low)60 Client Closes (924)100.00%100.00%0.00%2.55s$0.0164
17Level Playbooks + Gemini 4 Carbon (Medium)60 Client Closes (924)100.00%100.00%0.00%11.11s$0.1076
29Level Playbooks + Claude Fable 5.225-Company Sample (372)100.00%100.00%0.00%15.79s$0.6270
30Gemini 4 Carbon (Low) · Unassisted Raw60 Client Closes (924)95.24%95.24%4.76%88.01s$0.0634
31Gemini 4 Carbon (Medium) · Unassisted Raw60 Client Closes (924)95.13%95.13%4.87%115.75s$0.0981
32Gemini 4 Argon (Low) · Unassisted Raw60 Client Closes (924)94.70%94.48%5.30%70.30s$0.0546
35GPT-6.1 Sol (High) · Unassisted Raw60 Client Closes (924)93.29%93.29%6.71%85.80s$0.0651
36GPT-6 Astra (Medium) · Unassisted Raw60 Client Closes (924)92.86%92.75%7.14%56.99s$0.0224
40Gemini 4 Hydrogen (Medium) · Unassisted Raw25-Company Sample (372)91.67%91.13%8.33%33.83s$0.0803
43Gemini 3.8 Flash · Unassisted RawFull 374 Closes (5,923)87.56%82.44%12.44%43.22s$0.0183
44GPT-6.1 Terra (Medium) · Unassisted Raw60 Client Closes (924)87.55%87.12%12.45%18.44s$0.0063
47Claude Fable 5.2 · Unassisted Raw25-Company Sample (372)86.83%84.95%13.17%135.92s$0.3412
51Moonshot Kimi K3.1 (#1 Open-Weight Raw)60 Client Closes (924)81.28%80.30%18.72%32.46s$0.0349
56Claude Sonnet 5.5 · Unassisted Raw25-Company Sample (372)73.39%70.97%26.61%46.65s$0.1325
58Qwen 3 Next 80B · Unassisted Raw60 Client Closes (924)71.10%70.89%28.90%43.12s$0.0078
59DeepSeek V3.2 · Unassisted Raw60 Client Closes (924)69.81%67.42%30.19%32.26s$0.0027
61Claude Haiku 5.5 · Unassisted Raw60 Client Closes (924)66.02%49.24%33.98%34.11s$0.0487
63Mistral Large 3 (675B) · Unassisted Raw60 Client Closes (924)58.12%53.14%41.88%11.20s$0.0073
65Gemini 2.5 Flash-Lite · Unassisted RawFull 374 Closes (5,923)52.37%50.26%47.63%20.55s$0.0030

Free benchmark review

See how your operating numbers benchmark.

Start with a free conversation about your priorities. With agreed access to the relevant records, we compare your numbers with applicable benchmarks and identify what to review first.

Four findings every CEO and finance leader should care about

1. Even at 95.24% raw accuracy, an unassisted model still misses 44 close checks

When you test AI on clean textbook problems, almost every model looks ready to replace your bookkeeper. On real multi-tab SMB close workbooks across 60 client closes (924 checks), non-truncated unassisted accuracy spans 52.38% to 95.24% (60-close median 80.58% across 24 clean runs, and 52.37% on Gemini 2.5 Flash-Lite across all 5,923 full-vault checks).

Even Gemini 4 Carbon (Low) at 95.24% still misses 44 of 924 checks (4.76%). Forty-five percent of Carbon's misses happen on multi-tab inventory roll-forwards and cross-month accrual reversals. Another 30% happen when the model guesses categories on missing check images or missing PDF bank statements instead of holding for client proof. Deeper thinking in Carbon Medium occasionally rationalizes re-booking a raw bank row on top of a prior bookkeeper's manual journal entry.

2. Frontier models from OpenAI and Google sit within ~2 points, with real cost and cutoff trade-offs

No single model vendor owns a decisive lead on real books. Across 924 checks, Gemini 4 Carbon (95.13% to 95.24%), Gemini 4 Argon (94.48% to 94.70%), OpenAI GPT-6.1 Sol High (93.29%), and OpenAI GPT-6 Astra (92.86%) cluster within roughly two percentage points inside overlapping +/-1.4% confidence margins. OpenAI's GPT-6 Astra achieved 92.86% in 57s at $0.0224 per task (roughly one-third Carbon's cost), GPT-6.1 Terra delivered 87.55% in 18s at $0.0063 per task, Moonshot Kimi K3.1 led open-weight models at 81.28%, and Claude Sonnet 5.5 scored 73.39% on our smaller 25-company subset (note: Claude Fable 5.2 and Claude Sonnet 5.5 were evaluated on the 25-company subset rather than the 60-close sweep, where only Claude Haiku 5.5 ran).

Each cloud provider also carries distinct production hazards. OpenAI's APIs separate reasoning tokens cleanly at default settings, whereas Google Vertex AI counts internal thinking tokens inside a caller's configured output limit. Because Carbon Medium averages 4,076 thinking tokens per task and Carbon High averages 23,371 thinking tokens on multi-tab Excel ledgers, a caller-configured 8,192-token limit silently cuts off 21.3% (26/122) of Carbon Medium tasks and 100% of Carbon High tasks unless raised to 65,536 tokens.

3. Why playbooks from 2,200+ past experiences reach 100.00% at $0.0039 per close task

Look at the top playbook rows in the leaderboard above. Pairing Gemini 2.5 Flash-Lite (52.37% raw across all 374 client closes) with Level Playbooks (built from 2,200+ past SMB cleanups) and controller review reaches 5,923 / 5,923 checks (100.00%, +47.63 percentage-point lift) in 1.10s typical time at $0.0039 per close task. That run automatically caught 417 rule exceptions and held 131 unverified items for human review before anything touched QuickBooks.

In a real finance operating system, playbooks run an instant 0.03-second spreadsheet check first. They check the manual JE register before reclassifying Uncategorized Expense and verify Undeposited Funds before flagging a Shopify payout variance. They also exclude uninstalled stored materials from AIA WIP progress and hold any check lacking bank proof for operator review.

4. Your software stack determines how hard your monthly close is

LevelBench-100 also exposes why AI finance projects fail in service businesses: the hardest errors happen in the handoff between operational software and QuickBooks Online.

That is why we paired LevelBench-100 with three companion benchmark hubs so operators can check their entire stack.

November 13 multi-model addition: Claude Fable 5.2, GPT-6 Astra, and Gemini 4 Hydrogen

For completeness, we expanded our 25-company close subset (372 checks), 7-company CFO automation suite (82 checks), and browser-portal safety tests to include Claude Fable 5.2, GPT-6 Astra (Low and Medium), and Gemini 4 Hydrogen (Low, Medium, and High) alongside Gemini 4 Carbon and Gemini 3.8 Flash.

Model Tier25-Close Raw (372 Checks)+ Level Playbooks (25-Close)7-Company CFO Suite (Raw → Playbooks)Multi-Step Portals (15 Runs)
Gemini 4 Carbon (Low)93.55% (105.4s)100.00% (13.2s, 60-close baseline)Not testedNot tested
Gemini 4 Hydrogen (High)Not testedNot tested92.68% → 100.00% (11.46s)Not tested
Gemini 4 Hydrogen (Medium)91.67% (33.8s, $4.18)100.00% (5.03s, $4.38)86.59% → 100.00% (5.17s)100.00% (0 unsafe, 2.67s)
Gemini 4 Carbon (Medium)91.67% (137.1s, $4.91)100.00% (11.1s, 60-close baseline)Not tested93.33% (1 unsafe, 6.70s)
GPT-6 Astra (Low)87.37% (43.2s, $8.04)100.00% (5.27s, $21.52)85.37% → 100.00% (5.71s)100.00% (0 unsafe, 2.37s)
GPT-6 Astra (Medium)87.10% (54.4s, $10.60)100.00% (6.34s, $21.68)84.15% → 100.00% (6.70s)100.00% (0 unsafe, 2.49s)
Claude Fable 5.2 (64K unclipped)86.83% (135.9s, $17.74)100.00% (15.79s, $32.60)80.49% → 100.00% (14.10s)100.00% (0 unsafe, 4.72s)
Gemini 4 Hydrogen (Low)85.75% (16.6s, $2.03)100.00% (3.86s, $4.24)75.61% → 100.00% (3.76s)100.00% (0 unsafe, 2.01s)
Gemini 3.8 FlashNot testedNot tested84.15% → 100.00% (6.80s)80.00% (3 unsafe, 4.00s)

Three practical findings came out of those additional runs:

  1. Every model hits 100% with playbooks, while unassisted raw scores diverge by 7.8 points: Unassisted raw scores on the 25-company close subset ranged from 85.75% (Hydrogen Low) and 86.83% (Claude Fable 5.2) to 87.37% (GPT-6 Astra Low) and 91.67% to 93.55% (Gemini 4 Hydrogen Medium and Carbon). Inside Level Playbooks and controller review, every tested model reached 100.00% (372 of 372 checks) with zero routing overrides on the 25-close subset (while Claude Fable 5.2 scored 93.75% raw route selection, reaching 100% gated on the 7-company CFO suite).
  2. Deterministic popup guards matter even for top frontier models: On 15 multi-step autonomous portal trajectories, Hydrogen, GPT-6 Astra, and Claude Fable 5.2 all completed 15 of 15 with zero unsafe clicks (versus 14 of 15 for Carbon Medium and 12 of 15 for Gemini 3.8 Flash). On single-step deceptive modals, however, raw models without Level's destructive-confirmation guard clicked through a disguised invoice-removal dialog (83.33% raw vs. 100.00% guarded).
  3. Token ceilings and code audits pay off immediately: Raising output limits to 64K raw / 16K structured for Claude Fable 5.2 and 4,096 tokens for Gemini 4 browser checks eliminated silent reasoning cutoffs. During a parallel codebase audit, Claude Fable 5.2 also surfaced 14 real production bugs across rule partitioning, negative bill-line spend inflation, transfer-pair boundaries, and open-invoice A/R tie-outs that are now fixed.

Three questions to ask before you let any AI touch your books

Before you buy a tool or service that claims to automate your bookkeeping or monthly close, ask three operator questions.

  1. Ask what happens when a prior bookkeeper already posted a manual Journal Entry or when a counter debit swipe matches an open AP bill. Generic prompts double-count expenses and corrupt AP aging unless takeover playbooks check the manual JE register first.
  2. Ask how the system knows when to hold and ask. When a $20,200 paper check clears with no check-stub image and a casual Slack guess from a clerk, the system must quarantine the check in Uncategorized / Suspense rather than guessing an expense account.
  3. Ask whether running the exact same export three times produces the exact same trial balance to the penny. If the answer is no, you have a chatbot rather than an accounting team.

If you want to see where your current QuickBooks setup or subledger sync is leaking margin before your next close, run our profit gap diagnostic tool or test your open jobs in the WIP schedule calculator. If you want a finance team backed by playbooks from 2,200+ past SMB experiences, see how our CFO and accounting services and software integrations work in practice.

Frequently asked questions

Frontier models from Google and OpenAI clustered tightly between 92.9% and 95.2% across 60 anonymized SMB client closes (924 checks): Gemini 4 Carbon (95.1% to 95.2%), Argon (94.7%), GPT-6.1 Sol High (93.3%), and GPT-6 Astra (92.9% at ~1/3 Carbon's cost), with Kimi K3.1 leading open-weight models at 81.3%. On the 25-company close subset (372 checks), Gemini 4 Hydrogen Medium scored 91.7%, GPT-6 Astra Low/Medium scored 87.1% to 87.4%, and Claude Fable 5.2 scored 86.8%. Every evaluated Level Playbook setup across all 8 providers (29 of 29) reached 100.00% on its tested sample.

Vertex AI counts internal thinking tokens inside a caller's configured maxOutputTokens ceiling. Because Carbon Medium averages 4,076 thinking tokens per task and Carbon High averages 23,371 thinking tokens on multi-tab workbooks, an 8,192-token cap truncates 21.3% of Carbon Medium tasks and 100% of Carbon High tasks until raised to 65,536 tokens.

Level runs an instant 0.03-second spreadsheet check, 18 pre-close audit checks, 61 controller rules, and 16 historical error catchers around every model call. Across all 374 SMB client closes (752 tasks, 5,923 checks), pairing Gemini 2.5 Flash-Lite (52.37% raw) with Level Playbooks reached 100.00% in 1.10s typical time at $0.0039 per close task.

Yes. All 40 public anonymized SMB close packages (.xlsx, .csv, .json; 84 tasks, 588 checks) are free for personal, CPA, and academic review, and web search indexing or AI citation with attribution is expressly permitted. Commercial AI labs can email sam@levelcfo.com to license our held-out private 334-company (5,335-check) close vault.

Source and claim note

Methodology, dataset, and licensing note: LevelBench (levelbench-100.v5.0) evaluates 68 live provider-executed AI model and Level Playbook test runs across 60 anonymized SMB client closes (122 tasks, 924 checks) and a 25-company close subset (52 tasks, 372 checks). It also covers our 374-company close vault (752 tasks, 5,923 checks across 19 operational and tax systems). It publishes 40 downloadable SMB close case packages (84 tasks, 588 pass/fail checks with raw .xlsx, .csv, and .json files) plus 36 SMB AI error patterns (18 Pre-Close Audit Checks, 61 Senior Review Rules, and 16 Historical Error Catchers). The "2,200+ past SMB experiences" figure reflects the founding team's pooled analysis of 2,242 SMB accounting, CFO, and diligence reviews. All published benchmark packages and .xlsx workbooks are de-identified with zero customer PII. Public files (/data/benchmarks/levelbench-100.json, /data/benchmarks/levelbench-apex-smb-worlds.json, /data/benchmarks/smb-ai-accounting-fault-taxonomy.json, /data/benchmarks/smb-ai-accounting-fault-taxonomy.csv, /data/benchmarks/levelbench-100-companyco.json, /data/benchmarks/levelbench-100-leaderboard.csv, and /data/benchmarks/cases/) ship under the Level Personal, Academic & Non-Commercial Evaluation License v1.0 (canary token LEVEL-IP-CANARY-9F4A-7C2E-PERSONAL-USE-ONLY-CONTACT-SAM-AT-LEVELCFO-COM); search indexing and AI answer-engine retrieval/citation with attribution to levelcfo.com are expressly permitted. Commercial AI labs and software vendors must contact sam@levelcfo.com to license the held-out private 334-company (5,335-check) close vault. A 100.00% benchmark score proves playbook conformance on this sample; live business accuracy depends on source document availability and timely client responses on held items.

Share

Free Excel & CSV Workpapers (No Email Required)

Download the contractor accounting & QuickBooks tie-out pack

Built from real contractor monthly close controls. Covers our 3-account clearing model, uninstalled materials cutoff, WIP balance-sheet tie-outs, plus surety working-capital haircut schedules:

+ Show 7 more specialist CSV workpapers (WH-347 Payroll, QoE EBITDA, 13-Week Cash Flow, Service Agreements)

Free profit audit

Have us reach out for a free audit

Drop your work email and our team will reach out within a few business hours to benchmark your books against 2,200+ contractors in the research universe.

Sam Yang

About the author

Sam Yang

Founder & CEO

Founder of Level, the always-on finance team for contractors, service businesses, and ecommerce brands. Ex-CFO across trades, SaaS, and service businesses. 4 years as Director of Growth Product at BuildOps, building financial tooling used by 1,000+ commercial contractors. Four years in PE and investment banking rolling up and acquiring service businesses, $2.5B in total transactions including M&A and IPOs. Stanford MBA, Brown undergrad. The Level founding team's analysis of 2,200+ contractors ($13.25B in job revenue) across operating, private-equity, and CFO roles anchors the Level Index benchmark research.

LinkedIn

See how your operating numbers benchmark.

Start with a free conversation about your priorities. With agreed access to the relevant records, we compare your numbers with applicable benchmarks and identify what to review first.

2,200+ businesses benchmarked$13.25B in revenue analyzedWeekly action cadence

No credit card. Start with a 15-min call. We only follow up if we can actually help.

No commitment. Real numbers, not generic advice.