Skip to main content
2,200+ contractors in the research universe benchmarked. Do you know your gross profit per labor hour? See where you stand →
Level

The Level Index · AI Accounting & CFO Benchmark · Updated Nov 13, 2026

AI Accounting & CFO Benchmark: 68 Model Runs on Real SMB Closes

Level provides AI-native accounting and fractional CFO services for growing service businesses and ecommerce brands, backed by playbooks from 2,200+ client cleanups. Across 68 live test runs spanning 8 AI providers, top raw frontier models cluster between 92.9% and 95.2% on 60 SMB client closes (924 checks). AI governed by Level Playbooks reaches 100.00% across every provider down to $0.0039 per close task. Built by operators, for operators.

5,923

Monthly Close Checks

374 client closes · 40 public .xlsx cases

68

Live Model Runs

39 raw model runs · 29 playbook runs

92.9% to 95.2%

Frontier Raw Tier

GPT-6.1 / GPT-6 & Gemini 4 (median 80.6%)

4.8% to 48%

Raw AI Error Rate

Missing check stubs, cutoffs & prior JEs

100.00%

With Level Playbooks

$0.0039/task worker routing · CPA sign-off

Frontier Models Cluster Near 93% to 95%, and Why the Last 5% Needs Playbooks

Across 924 checks, Gemini 4 (94.5% to 95.2%) and OpenAI GPT-6.1 Sol / GPT-6 Astra (92.0% to 93.3%, down to 1/3 the cost) sit within overlapping confidence margins, while Kimi K3.1 (81.3%) leads open-weights.

Every raw model misses 44 to 440 checks on accrual reversals, missing check images, and prior-bookkeeper entries, while shared thinking-token limits cut off high-thinking runs if uncapped.

All 29 Level Playbook setups across 8 labs hit 100.00%, including a lightweight $0.0039/task worker model reaching 5,923 / 5,923 in 1.1s.

Visual Comparison · 60 Client Closes (924 Checks) & 374-Company Vault (5,923 Checks)

Accuracy by AI Model & Playbook Setup

Higher bar = fewer mistakes · Click any row for details
Any Model + Level Playbooks (2,200+ Past Experiences) + Controller Sign-Off
100.00% Across All Vendors · 0% MistakesDetails ▾
All 29 playbook setups across OpenAI, Google, Anthropic, Moonshot, Qwen, DeepSeek, Meta, and Mistral score 100.00%, down to $0.0039/task in 1.1s with lightweight worker models.
Frontier Reasoning Models Alone · OpenAI GPT-6.1 / GPT-6 & Google Gemini 4
92.0% to 95.2% Raw · 4.8% to 8.0% Mistake RateDetails ▾
Top raw models cluster within ~2 points across 924 checks (overlapping +/-1.4% confidence margins): Gemini 4 Carbon (95.1%-95.2%), Argon (94.5%-94.7%), GPT-6.1 Sol (92.0%-93.3%), and GPT-6 Astra (92.9% at $0.022/task, ~1/3 Carbon's cost).
Fast Worker Models Alone · OpenAI GPT-6.1 Terra & Google Gemini Flash
79.87% to 87.56% Raw · 100.00% With PlaybooksDetails ▾
GPT-6.1 Terra Medium (87.55% at $0.006/task), Terra Low (85.61% at $0.004/task), Gemini 3.8 Flash (87.56% full vault; 79.87% on 60 closes), and Gemini 3.5 Flash (86.26%) trade raw accuracy for speed and low cost.
Open-Weight & Mid-Tier Models Alone · Kimi K3.1, Claude Sonnet 5.5, Qwen 3 & DeepSeek
69.81% to 81.28% Raw · 100.00% With PlaybooksDetails ▾
Moonshot Kimi K3.1 leads open-weight models at 81.28% raw, alongside Claude Sonnet 5.5 (73.39% on our 25-company sample), Qwen 3 Next 80B (71.10%), and DeepSeek V3.2 (69.81% at $0.003/task).
Smaller & Ultra-Light Models Alone · Haiku 5.5, Llama 4, Mistral & Flash-Lite
52.37% to 66.02% Raw · 100.00% With PlaybooksDetails ▾
Claude Haiku 5.5 (66.02%), Llama 4 Maverick (62.55%), Mistral Large 3 (58.12%), and Gemini 2.5 Flash-Lite (52.37% full / 52.38% on 60 closes) miss 34% to 48% alone, yet reach 100.00% inside Level Playbooks.

Scorecard · 13 Accounting & CFO Jobs Across 3 Stages

What AI Does Well vs. Where It Breaks

Click any job row to see where raw AI breaks & how playbooks fix it
Accounting / CFO Job (Click Any Row to Expand)
Frontier Raw
Fast Worker
Open-Weight
With Playbooks
Stage 1 · Daily Bookkeeping & Cash Reconciliation (Getting Data In & Tying Out Cash)
1. Pulling Monthly Reports from Accounting, Payroll & Field Portals▾
71.4%
64.3%
21.4% to 41.7%
100.0%
What Raw AI Gets RightUnassisted Playwright scripts pass 60 of 84 portal-gate trials (71.4% with Sol 6.1 / Opus 5.5; 54 of 84 = 64.3% with unassisted GPT-5.6 Terra), while vision-based browser agents (35 of 84 = 41.7%) can navigate login screens and click visible buttons on simple single-page portals.
Where Raw AI BreaksUnassisted vision browser agents fail on 49 of 84 portal-gate trials (58.3% failure rate) and silently truncate 24 of 84 trials (28.6% truncation rate on scrolling tables, reading only the 25 visible rows on screen instead of 1,400 GL transactions), while raw Playwright scripts fail on 24 of 84 trials (28.6%) and silently truncate 10 of 84 trials (11.9%) when SaaS portals update page layouts or time out on multi-year exports.
How Level Playbooks Fix It (100%)Level's browser extraction playbooks pull native CSV/JSON exports directly, split multi-year pulls into monthly windows, verify the exact client company ID before downloading, and cross-check every export against the trial balance so zero rows are ever silently dropped (84 of 84 = 100.0% extraction completeness).
2. Bank & Credit Card Feed Categorization & Missing Check Holds▾
95.2%
87.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightGemini 4 Carbon (95.2%) and GPT-6.1 Sol (93.3%) classify standard SaaS, fuel, and supply-house card charges accurately and separate merchant-processing fees from customer settlements.
Where Raw AI BreaksEven Gemini 4 Carbon still guesses categories on 13 missing-evidence items (paper checks without check images or unverified bank credits) and double-counts 2 to 5 prior-bookkeeper manual Journal Entries, while fast and open-weight models score 52.4% to 81.3%.
How Level Playbooks Fix It (100%)Level's close playbooks (built from 2,200+ past SMB cleanups) check manual Journal Entries before touching any Uncategorized row and hold every paper check over $250 missing a check image (lifting Gemini 2.5 Flash-Lite from 52.4% to 100.0% at $0.0039/task).
3. Accounts Payable (AP) Vendor Bill Matching & Negative AP▾
95.1%
87.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightFrontier models match single-invoice wire payments cleanly and separate wire fees ($46.78 Wise FX / $200 domestic wire) from principal bill paydowns.
Where Raw AI BreaksUnassisted models follow casual Slack guesses on identical-dollar vendor bills, reclassify negative AP vendor balances to Prepaid Expenses without checking for missing bills, or confuse walk-in counter debit purchases with AP bill paydowns.
How Level Playbooks Fix It (100%)Level's AP playbooks distinguish walk-in COD card swipes from open vendor bills, investigate negative AP vendor ledgers against unentered invoices, and pause for remittance advice on tied bill amounts.
4. Accounts Receivable (AR), Undeposited Funds & Lender Draws▾
95.2%
87.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightFrontier models calculate standard promotional allowances, early-pay discounts, and merchant fee deductions on matched customer invoices.
Where Raw AI BreaksRaw AI books wrong-entity wholesale ACH deposits ($84,250) and 50% consumer-lender draws ($17,500) straight to Sales Revenue, accepts Opening Balance Equity plugs in Undeposited Funds, and misses dealer financing fees ($470 to $570).
How Level Playbooks Fix It (100%)Level's cash-application playbooks match deposits across sister entities and property addresses, unwind Undeposited Funds plugs against bank batches, and hold un-invoiced lender draws in Customer Deposits.
5. Bank & Corporate Card Statement Tie-Outs & Zero-Activity Continuity▾
94.7%
85.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightFrontier models spot duplicate bank-feed deposits, unrecorded wire fees, and post-cutoff November card charges bleeding into October exports.
Where Raw AI BreaksUnassisted models certify $4.26M of QuickBooks cash as '100% reconciled' when zero PDF bank statements are attached, zero out legitimate second bank accounts at the same bank, or leave card autopays in Uncategorized Expense.
How Level Playbooks Fix It (100%)Level's statement-custody gates require attached PDF bank statements before certifying cash, verify last-4 account numbers, reconcile full parent + sub-card hierarchies, and prove $0-activity months mathematically.
Stage 2 · Month-End Accrual Close (Matching Revenue & Costs to the Right Month)
6. Field Software (WIP), AIA Retainage & SaaS ASC 606 Cutoffs▾
93.5%
85.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightGemini 4 Carbon cut cross-system subledger-to-GL bridge errors by 67% vs Gemini 4 Argon (9 misses down to 3 across 924 checks) and defers flagged future-month deposits.
Where Raw AI BreaksOpen-weight and fast models (22 to 111 bridge misses) count boxed warehouse materials as completed WIP progress, net 10% subcontractor retainage off COGS, prorate labor on calendar days instead of Monday-Friday workdays, or double-count mid-year SaaS seat upgrades.
How Level Playbooks Fix It (100%)Level's close playbooks remove uninstalled warehouse materials from WIP progress, record full 10% subcontractor retainage, prorate Monday-Friday workdays, and schedule SaaS upgrades cleanly.
7. Inventory Landed Cost, Import Tariffs, 3PL Counts & Returns▾
95.1%
87.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightFrontier reasoning models separate basic inbound ocean freight from outbound parcel shipping on single-tab schedules.
Where Raw AI BreaksMulti-tab inventory roll-forwards and month-end plus/minus sign flips account for 45% of even Gemini 4 Carbon's misses (21 of 44): raw AI accepts inflated Shopify SKU costs, expenses 100% of import tariffs upfront, misses containers owned at sea, or subtracts return credits twice.
How Level Playbooks Fix It (100%)Level compares Shopify SKU costs directly against prior-year physical inventory ($17.87/unit) and factory purchase orders ($17.76/unit), keeps inbound tariffs in unsold stock, and locks 3PL physical counts.
8. Prepaids, ASC 842 Clinic Leases, Cloud Setup & R&D Tax Credits▾
95.2%
87.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightModels compute standard 12-month straight-line insurance amortization and equipment depreciation ($300/mo on an $18,000 3D X-ray sensor) cleanly.
Where Raw AI BreaksUnassisted models book $120,000 landlord TI wires as Other Income, lump 4 months of Form 8974 R&D credits into October (creating -$1,809.26 negative payroll tax), or credit insurance amortization to Accrued Liabilities instead of Prepaids.
How Level Playbooks Fix It (100%)Level's technical GAAP and tax-credit playbooks net TI incentives against ASC 842 ROU assets, enforce ASC 350-40 cloud capitalization splits, and allocate R&D payroll credits to each payroll period of origin.
9. Multi-Entity FX Eliminations, Profit First Sweeps & S-Corp Draws▾
94.5%
87.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightFrontier models match basic domestic Due-To/Due-From transfers and standard management fee eliminations across two entities.
Where Raw AI BreaksRaw AI translates monetary EUR loans at monthly average FX rates instead of spot rates, counts Profit First bank-account sweeps as revenue, leaves LLC partners on W-2 payroll, or books disproportionate personal card draws in a 60/40 S-Corp.
How Level Playbooks Fix It (100%)Level's consolidation and entity-tax playbooks enforce ASC 830 spot remeasurement, eliminate intercompany inventory markups, pair Profit First sweeps to $0 P&L impact, and protect S-Corp pro-rata distribution parity.
Stage 3 · Fractional CFO Analysis, Cash Flow & M&A Diligence
10. 13-Week Cash Flow Bridges, Debt Grouping & Stock Options▾
93.3%
87.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightFrontier models simulate weekly AR collection delays against bank line-of-credit covenant floors once working-capital accounts are cleanly separated.
Where Raw AI BreaksRaw AI sweeps QuickBooks 'Other Current Liabilities' (containing current term debt) into operating cash flow, alters prior-month Opening Cash to force Ending Cash to tie, or flips AR/Inventory cash signs.
How Level Playbooks Fix It (100%)Level's cash-flow tie-out checks lock Opening Cash to prior bank statements, separate bank debt and non-cash stock options, and tie Ending Cash to the penny.
11. Channel Unit Economics, Hidden Account Flips & Health Bill Audits▾
95.1%
87.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightModels compute baseline CAC, LTV:CAC, and channel contribution margins once account groupings and physical unit counts are cleaned.
Where Raw AI BreaksWhen a $2.06M account shift has $0.00 net impact on bottom-line Net Income (or models run out of response space on wide 109-account spreadsheets), raw AI reports 'no errors' while year-over-year ad returns and margins are distorted.
How Level Playbooks Fix It (100%)Level scans every Chart of Accounts line for year-over-year vendor and channel shifts, nets pass-through client ad spend, and matches insurance carrier bills against active payroll.
12. M&A Quality of Earnings (QoE) & Questioning Bad Add-Backs▾
95.2%
87.6%
58.1% to 81.3%
100.0%
What Raw AI Gets RightTop frontier models normalize documented excess owner compensation, personal owner boat/travel expenses, and one-time ERC grant income.
Where Raw AI BreaksUncritical raw AI models agree with whatever memo is in front of them, accepting seller-broker claims that ongoing software is a 'one-time add-back' or trusting a temp bookkeeper's note calling a $19,400 sister-company transfer a 'vendor rebate.'
How Level Playbooks Fix It (100%)Level's Quality of Earnings checks require signed contracts and bank-to-bank trace proof, rejecting normal recurring operating costs from Adjusted EBITDA.
13. Executive Board Decks, Live Excel Formulas & Clean Formatting▾
95.2%
87.1%
53.1% to 80.3%
100.0%
What Raw AI Gets RightGemini 4 Carbon achieved 0 formatting or rounding misses across all 924 checks when configured with a full 65,536-token response limit.
Where Raw AI BreaksClaude Haiku 5.5 (66.0% accounting vs 49.2% exact formatting) and Gemini 2.5 Flash (75.9% vs 69.7%) lose 6 to 17 points on rounding and table formatting, while standard 8,192-token API caps cut off 21.3% of Carbon Medium tasks mid-table.
How Level Playbooks Fix It (100%)Level's workbook checks verify live Excel formulas, tie every slide figure to the closed ledger, and run an instant 0.03-second spreadsheet check with full 65,536-token response limits.

Interactive Leaderboard

AI Accounting & CFO Leaderboard

Showing 16 of 68 systemsDownload CSV ↓
Summary of 16 popular setups: 4 Level Playbook setups + 12 raw AI models (click below to see all 68 test runs).
RankSystem / Model & Test SampleSetup TypeAccounting (%)Mistake RateSpeed & Cost / Task
#1
Level Playbooks + Gemini 2.5 Flash-Lite
Google · 374 Client Closes · 5,923 Checks · Sample #1 · +47.6 pts lift
AI + Level Playbooks
100.0%
0.0%(0 mistakes)1.10s (playbooks) · $0.0039
#4
Level Playbooks + GPT-6.1 Terra (Low)
OpenAI · 60 Client Closes · 924 Checks · Sample #2 · +14.4 pts lift
AI + Level Playbooks
100.0%
0.0%(0 mistakes)2.55s (playbooks) · $0.016
#17
Level Playbooks + Gemini 4 Carbon (Medium)
Google · 60 Client Closes · 924 Checks · Sample #15 · +4.9 pts lift
AI + Level Playbooks
100.0%
0.0%(0 mistakes)11.11s (playbooks) · $0.108
#29
Level Playbooks + Claude Fable 5.2
Anthropic · 25-Company Sample · 372 Checks · Sample #6 · +13.2 pts lift
AI + Level Playbooks
100.0%
0.0%(0 mistakes)15.79s (playbooks) · $0.627
#30
Gemini 4 Carbon (Low)
Google · 60 Client Closes · 924 Checks · #1 Raw
Raw AI Model
95.2%
4.8%(mistakes)88.01s · $0.063
#31
Gemini 4 Carbon (Medium)
Google · 60 Client Closes · 924 Checks · #2 Raw
Raw AI Model
95.1%
4.9%(mistakes)115.75s · $0.098
#32
Gemini 4 Argon (Low)
Google · 60 Client Closes · 924 Checks · #3 Raw
Raw AI Model
94.7%
5.3%(mistakes)70.30s · $0.055
#35
GPT-6.1 Sol (High)
OpenAI · 60 Client Closes · 924 Checks · #5 Raw
Raw AI Model
93.3%
6.7%(mistakes)85.80s · $0.065
#36
GPT-6 Astra (Medium)
OpenAI · 60 Client Closes · 924 Checks · #6 Raw
Raw AI Model
92.9%
7.1%(mistakes)56.99s · $0.022
#40
Gemini 4 Hydrogen (Medium)
Google · 25-Company Sample · 372 Checks · #3 Raw
Raw AI Model
91.7%
8.3%(mistakes)33.80s · $0.080
#44
GPT-6.1 Terra (Medium)
OpenAI · 60 Client Closes · 924 Checks · #9 Raw
Raw AI Model
87.5%
12.4%(mistakes)18.44s · $0.0063
#45
GPT-6 Astra (Low)
OpenAI · 25-Company Sample · 372 Checks · #6 Raw
Raw AI Model
87.4%
12.6%(mistakes)43.20s · $0.154
#47
Claude Fable 5.2
Anthropic · 25-Company Sample · 372 Checks · #8 Raw
Raw AI Model
86.8%
13.2%(mistakes)135.94s · $0.341
#51
Moonshot Kimi K3.1
Moonshot AI · 60 Client Closes · 924 Checks · #12 Raw
Raw AI Model
81.3%
18.7%(mistakes)32.46s · $0.035
#56
Claude Sonnet 5.5
Anthropic · 25-Company Sample · 372 Checks · #10 Raw
Raw AI Model
73.4%
26.6%(mistakes)46.65s · $0.133
#59
DeepSeek V3.2
DeepSeek · 60 Client Closes · 924 Checks · #19 Raw
Raw AI Model
69.8%
30.2%(mistakes)32.26s · $0.0027

Accuracy, Cost & Safety Comparison

Accounting Accuracy vs. Cost per Close Task ($ USD)

Level Playbooks lift Gemini 2.5 Flash-Lite from 52.37% raw to 100.00% across all 5,923 checks at $0.0039 per close task (1.1s typical speed).

AI + Level Playbooks (100.0%)Unassisted Raw AI Models
Highlighted Box = Best Zone (High Accuracy · Low Cost & Risk)
TOP OPERATOR QUADRANT (100% PLAYBOOK ZONE)50%65%80%95%100%Playbooks + Gemini 2.5 Flash-LitePlaybooks + GPT-6.1 Terra LowPlaybooks + Gemini 4 Carbon MedGemini 4 Carbon LowGemini 4 Carbon MedGemini 4 Argon LowGPT-6.1 Sol HighGPT-6 Astra MedGemini 4 Hydrogen MedGemini 3.8 FlashGPT-6.1 Terra MedGPT-6 Astra LowClaude Fable 5.2Moonshot Kimi K3.1Claude Sonnet 5.5DeepSeek V3.2Gemini 2.5 Flash-LiteMeasured API Cost per Close Task ($ USD, lower is further left)Accounting Accuracy (%)

Closed-Month QuickBooks & Subledger Diagnostic ($2M to $50M+ Operators)

Want us to test your actual QuickBooks and subledger exports against these 100 checks?

We run one closed month of your general ledger, bank statements, and operational subledgers through Level's takeover playbooks (built from 2,200+ SMB experiences) to show where prior-bookkeeper adjustments, unbilled WIP, duplicate AP bills, or MSO intercompany entries are drifting.

Part 2 & Part 3 · Anonymized Client Books & Error Guardrails

Explore the Client Case Studies & Common AI Mistakes

Jump into our 40 downloadable Excel close packages in Part 2 or our catalog of 36 common AI accounting mistakes in Part 3. Commercial AI labs can license the held-out 334-company (5,335-check) close vault via sam@levelcfo.com:

How We Tested & Which AI Model Fits Each Monthly Close JobHow to choose the right AI model for daily bookkeeping, multi-file reconciliations, browser portal exports, and CPA sign-off.

1. Which AI Model Fits Each Accounting & CFO Job

Across our 60-company (924-check) multi-file SMB close sample, unassisted foundation and open-weight models score 52.38% to 95.24% (median 80.58%), still missing 4.76% to 47.62% of checks on missing check images, cross-month reversals, and prior-bookkeeper manual JEs. When paired with Level Playbooks (built from 2,200+ past SMB closes) and controller review, all 29 playbook setups reach 100.00% accuracy:

Monthly Close JobBest Model + Playbook SetupRaw vs. With PlaybooksCost / TaskWhy Operators Use This Setup
Messy Client Takeover & Subledger BridgesGPT-6.1 Sol / GPT-6 Astra or Gemini 4 + Playbooks92.9%-95.2% → 100.0%$0.022 to $0.105Frontier reasoning models sit within ~2 points on multi-tab closes (Astra offers ~3x lower raw cost; Playbooks eliminate all remaining cutoff misses).
Daily Recurring Close & Schedule Tie-OutsGPT-6.1 Terra (Medium / Low) + Level Playbooks85.6%-87.6% → 100.0%$0.017Runs in 2.6s to 3.1s inside Level Playbooks with 47 to 62 unverified items held for human review before posting.
Multi-Entity Consolidation, SaaS & M&A QoEGPT-6.1 Sol or Gemini 4 Carbon + Level Playbooks92.0%-95.1% → 100.0%$0.104 to $0.108Best for multi-currency intercompany eliminations, ASC 606/340-40 SaaS contract upgrades, M&A QoE add-backs, and board narratives.
High-Volume Bank Feeds & 374-Company Vault RunsGemini 2.5 Flash-Lite or Gemini 3.8 Flash + Playbooks52.4%-87.6% → 100.0%$0.0039 to $0.018Reaches 5,923 / 5,923 checks (100.0%) across all 374 client closes in 1.1s at $0.0039 per task with 417 automated rule checks.
Fast Close, Code Audits & Unseen Browser PortalsGemini 4 Carbon / Hydrogen, GPT-6 Astra, or Claude Fable 5.2 + Playbooks85.8%-93.6% → 100.0%$4.24 to $32.60 / 25 closesAdded Nov 13 across 25 close cases and 7 CFO suites: all tested models reach 100.0% with Level Playbooks (Carbon Low 93.55% raw; Hydrogen Medium 91.67% raw in 33.8s; Astra Low 87.37% raw in 43.2s; Fable 5.2 86.83% raw in 135.9s) and pass 15/15 multi-step portals with 0 unsafe clicks.

2. Browser-Use AI Across 14 Accounting, Payroll & Field Portals

Many SMB workflows require pulling reports from web portals where APIs are missing or incomplete. We tested three browser AI approaches across 14 production portals (QuickBooks Online, BuildOps, ServiceTitan, Housecall Pro, LMN Landscape Management Network, Roofr, MarketSharp, Rippling, Gusto, Ramp, Brex, Settle, Finaloop, Shopify) to see whether AI agents pull complete ledger and subledger data:

Unassisted Browser Bot

Unassisted Vision & Screenshot Browser Agent

Complete Exports
41.7%
Dropped Rows
28.6%
Avg Speed: 184.5s · Cost: $0.420/run

Passes 35 of 84 portal-gate trials (41.7%): scrapes only visible rows in scrolling tables on 24 of 84 trials (28.6% silent truncation rate, missing 80%+ of off-screen GL lines), clicks wrong export buttons during modal popups, and times out on multi-year date ranges.

Unassisted Browser Bot

Unassisted Playwright Script (Sol 6.1 / Opus 5.5; Terra 5.6 passes 54/84 = 64.3%)

Complete Exports
71.4%
Dropped Rows
11.9%
Avg Speed: 48.2s · Cost: $0.050/run

Passes 60 of 84 portal-gate trials (71.4%; 54/84 = 64.3% with GPT-5.6 Terra): brittle page selectors break when QBO, BuildOps, or Rippling ship UI updates; silently truncates 10 of 84 trials (11.9%) on scrolling tables or gateway timeouts.

Level Playbook Workflow

Level Browser Playbooks (2,200+ Past Experiences) · Terra 5.6 / Argon 4

Complete Exports
100.0%
Dropped Rows
0.0%
Avg Speed: 19.4s · Cost: $0.012/run

Passes all 84 of 84 portal-gate trials (100.0% observed completion, 0/84 = 0.0% observed silent truncation): pulls native CSV/JSON exports directly, splits multi-year pulls into monthly windows, verifies the client company ID before every action, and cross-checks row counts against prior closes so partial exports are never posted.

November 13 Follow-Up: Multi-Model Unseen Portal & Autonomous Workflow Safety

We expanded our browser safety suite across 6 model tiers from 4 model families: 6 popup scenarios (3 benign banners and 3 deceptive auto-pay, bank-feed disconnect, and billing upsell traps across 5 trials each), 8 unseen portal scenarios (40 runs), and 5 multi-step autonomous portal workflows (15 full trajectories). Without a deterministic destructive-confirmation guard, raw frontier models (including Claude Fable 5.2 and GPT-6 Astra) clicked through a disguised invoice-removal dialog (83.33% raw popup accuracy vs. 100.00% with Level's guard).

Model TierPopup Traps (Raw → Guarded)Multi-Step Portals (15 Runs)Unsafe ClicksMulti-Step Step Speed
Gemini 4 Hydrogen (Low)100.00% (guarded)100.00% (15/15)02.01s
GPT-6 Astra (Low / Medium)83.33% → 100.00%100.00% (15/15)02.37s to 2.49s
Gemini 4 Hydrogen (Medium)100.00% (guarded)100.00% (15/15)02.67s
Claude Fable 5.283.33% → 100.00%100.00% (15/15)04.72s
Gemini 4 Carbon (Medium)96.67% (guarded)93.33% (14/15)16.70s
Gemini 3.8 Flash100.00% (guarded)80.00% (12/15)34.00s

First-time portal operating rule: Fast models can safely navigate unseen read-only portals on the first try only inside a strict wrapper that verifies the exact company header before clicking, allows read-only navigation and export actions only, halts immediately on two-factor or anti-bot prompts, and ties downloaded statements to the penny before saving a repeatable recipe.

3. Multi-File Close CLI Harnesses (6 Production CLI Runs · 16 Checks)

Evaluates 6 retained production CLI harness invocations when guided by Level's multi-file close checklist across 16 deterministic checks on a multi-file landscaping close package (9 Uncategorized Expense rows totaling $51,948.71, $20,350.00 of paper checks missing images, $211.69 Intuit merchant fees, $7,075.00 Owner Bridge anti-double-count check, and uncertified payment-link write block).

#CLI Harness & ModelTriage ScoreFull Close ScoreTime & CostOperator Verdict
#1Codex CLI · GPT-5.6 Terra on Azure OpenAI9 / 9 Triage (100.0%)16 / 16 Total (100.0%)27.22s · $0.018Fastest 16/16 multi-file close run (27.22s), holding $20,350 of missing check images and avoiding the $7,075 Owner Bridge double-count.
#2Codex CLI · GPT-6.1 Sol on Azure OpenAI9 / 9 Triage (100.0%)16 / 16 Total (100.0%)36.46s · $0.06516/16 checks passed in 36.46s with full double-entry tie-out and uncertified payment-link block.
#3Antigravity CLI · Gemini 4 Argon (Low) on Vertex AI9 / 9 Triage (100.0%)16 / 16 Total (100.0%)37.5s · $0.028Fastest Vertex AI 16/16 run (37.50s), 2.8x faster than High thinking with identical 100% rubric conformance.
#4Antigravity CLI · Gemini 4 Argon (Medium) on Vertex AI9 / 9 Triage (100.0%)16 / 16 Total (100.0%)49.6s · $0.03216/16 checks passed in 49.60s across bank feed triage, Owner Bridge reconciliation, and check holds.
#5Gemini CLI · Gemini 4 Argon on Vertex AI9 / 9 Triage (100.0%)16 / 16 Total (100.0%)68.83s · $0.03516/16 checks passed in 68.83s with structured 6-part operator close summary.
#6Antigravity CLI · Gemini 4 Argon (High) on Vertex AI9 / 9 Triage (100.0%)16 / 16 Total (100.0%)104.97s · $0.05516/16 checks passed in 104.97s; illustrates higher thinking-token latency on multi-file CSV schedules.

4. How We Graded the 100 Monthly Close Checks (5 Practical Categories)

Exact Math & Tie-Outs

41 / 100 checks

Exact column footing, gross-to-net invoice cash math, 13-week cash bridge arithmetic, and fully loaded CAC/LTV calculations across messy SMB takeover exports.

Real Accrual Cutoffs

23 / 100 checks

Completed-not-billed WIP revenue recognition, upfront customer deposit deferral, Monday-Friday business-day subcontractor proration, FIFO landed cost capitalization, and M&A QoE add-back filtering.

Knowing When to Hold & Ask

11 / 100 checks

Holding unverified paper checks without check stub images, pausing on tied-amount vendor bills instead of guessing, isolating unreferenced wire overpayments in Unapplied Cash, and refusing cash-plug journal entries.

Clean Books & Subledger Matching

17 / 100 checks

Cross-referencing prior-bookkeeper manual Journal Entries before reclassifying Uncategorized Expense rows, matching AP bank outflows to open bills without double-counting, and reconciling MSO/PC intercompany balances.

Board Deck & Excel Quality

8 / 100 checks

Rejecting hardcoded Excel SUM total cells, catching unpinned figures in board slides, enforcing headline word limits, and blocking AI self-attribution.

Bad Book-Entry Rate (Safety & Subledger Pool)

28 safety checks

Measures how often an AI model posts a harmful entry across the 11 Knowing When to Hold & Ask checks and 17 Clean Books & Subledger Matching checks (such as guessing a vendor category on a $20,200 paper check without a check image or double-counting a prior bookkeeper's manual journal entry).

Fair, Apples-to-Apples Multi-Cloud Protocol: Every model received the exact same prompts, anonymized Charts of Accounts, and raw multi-tab Excel (.xlsx) and CSV exports across 60 core SMB client closes (122 tasks, 924 checks) and our 374-company close vault (752 tasks, 5,923 checks). All 40 public SMB close packages are downloadable for non-commercial review, while 334 multi-step client closes (5,335 checks) remain held out in a private offline vault for commercial AI lab licensing.

Configured 8,192-Token Cutoff Stress Test (Ranks #66 to #68, Labeled "[Cut Off at 8K Limit]"): Production runs allocate 65,536 output tokens so reasoning models never run out of response space. Ranks #66 to #68 (tagged "[Cut Off at 8K Limit]") test a caller-configured 8,192-token limit on Vertex AI, where internal thinking tokens share that limit; cut-off JSON responses receive 0 credit on unparsed checks.

Sam Yang

Lead Benchmark Author · Accounting, Fractional CFO & AI Systems

LinkedIn →

Sam Yang · Founder & CEO

Ex-CFO across trades, SaaS & services · $2.5B in total PE/banking transactions · Stanford MBA

Founder of Level, the always-on finance team for contractors, service businesses, and ecommerce brands. Ex-CFO across trades, SaaS, and service businesses. 4 years as Director of Growth Product at BuildOps, building financial tooling used by 1,000+ commercial contractors. Four years in PE and investment banking rolling up and acquiring service businesses, $2.5B in total transactions including M&A and IPOs. Stanford MBA, Brown undergrad. The Level founding team's analysis of 2,200+ contractors ($13.25B in job revenue) across operating, private-equity, and CFO roles anchors the Level Index benchmark research.

Frequently asked questions

No single model vendor wins every dimension: across 60 SMB client closes (924 checks), frontier models from Google and OpenAI cluster within ~2 percentage points (Gemini 4 Carbon 95.1% to 95.2%, Argon 94.7%, GPT-6.1 Sol High 93.3%, and GPT-6 Astra 92.9% at roughly one-third the cost), while Moonshot Kimi K3.1 leads open-weight models at 81.3%. Pairing any evaluated model across all 8 providers with Level Playbooks and controller review reached 100.00% on its tested sample.

Every raw foundation model misses 4.8% to 47.6% of monthly close checks when run without accounting guardrails. Across all providers, roughly 45% of frontier misses happen on cross-month accrual reversals and multi-tab inventory roll-forwards, 30% guess expense categories when check images or bank statements are missing, and the rest double-count prior-bookkeeper manual journal entries.

Frontier reasoning models sit in a tight 92.0% to 95.2% band on 60 closes (overlapping +/-1.4% 95% confidence margins), with OpenAI GPT-6 Astra ($0.022/task, 57s) and GPT-6.1 Terra ($0.006/task, 18s) offering major cost-and-speed advantages over heavier reasoning runs. Kimi K3.1 (81.3%), Qwen 3 (71.1%), and DeepSeek V3.2 (69.8% at $0.003/task) lead open-weights, while Claude Sonnet 5.5 scored 73.4% on our 25-company subset.

On platforms like Vertex AI where internal thinking tokens share a caller's configured output limit, multi-tab spreadsheets consume 4,000 to 23,000+ thinking tokens per task (cutting off 21.3% of Gemini 4 Carbon Medium tasks and 100% of Carbon High tasks under an 8,192-token limit). OpenAI GPT-6.1 and Astra isolate reasoning budgets more cleanly at default settings, while raising Vertex limits to 65,536 tokens prevents cutoffs.

Level runs an instant 0.03-second spreadsheet check, 18 pre-close audit checks, 61 controller rules, and 16 historical error catchers around every model call. Across all 374 client closes (752 tasks, 5,923 checks), pairing Gemini 2.5 Flash-Lite (52.37% raw) with Level Playbooks reached 100.00% in 1.1s typical speed at $0.0039 per close task.

Across 14 production portals (QuickBooks Online, ServiceTitan, BuildOps, Gusto, Rippling, Settle, Finaloop, and Shopify), unassisted screenshot bots complete only 41.7% (35/84) of trials and drop off-screen rows on 28.6% of scrolling tables. Level Browser Playbooks pull native CSV/JSON exports and tie totals to the trial balance for 100.0% (84/84) completion, and fast-model unseen-portal stress tests (15 multi-step trajectories) reached 100.0% with zero unsafe clicks at 2.0s to 2.7s median latency inside our read-only wrapper.

Field platforms track operational job status and gross invoices, whereas QuickBooks, Sage Intacct, and Spectrum track net bank settlements, merchant fees, vendor bills, and accrual cutoffs. Unassisted AI breaks when excluding uninstalled stored materials ($34,000) from AIA WIP, grossing up 10% retainage ($9,000), or deferring unstarted customer deposits.

Under state Marketplace Facilitator laws, delivery apps like UberEats, DoorDash, and Grubhub remit sales tax directly to the state. POS sync tools (Stream POS, Toast, and Square) often push delivery orders into QuickBooks Sales Tax Payable alongside in-store orders, double-charging sales tax unless delivery tax is separated.

When ecommerce brands switch from periodic inventory to Shopify perpetual COGS, teams often upload landed unit costs (factory price plus freight and tariffs) into Shopify while still expensing freight bills in the general ledger. In Case 14, that double-count inflated COGS by $2.19M (546 bps) until reconciled to factory purchase orders.

Yes. All 40 public anonymized SMB close packages (.xlsx, .csv, .json; 84 tasks, 588 checks) and 36 AI mistake patterns are free for personal, CPA, and academic review, and search indexing or AI citation with attribution is expressly permitted. Commercial AI labs can email sam@levelcfo.com to license our held-out private 334-company (5,335-check) close vault.

Content reviewed .Read the revision history

Grow without losing control of the numbers

Drop your info and we’ll show your real margin across jobs, crews, products, and customers, where profit leaks to rework or hidden costs, and how much cash is sitting in unbilled work or inventory.

2,200+ businesses benchmarked$13.25B in revenue analyzedWeekly action cadence

No credit card. Start with a 15-min call. We only follow up if we can actually help.

No commitment. Real numbers, not generic advice.