llm-meter

Measure LLM provider mechanics — time-to-first-token, sustained tokens/sec, and cost at list rates — one line of JSONL per probe.

The problem it solves

Most LLM benchmarks measure task quality: can this model fact-check, plan, follow a schema. That number moves when a new model ships. But there is a second number that moves much faster — the provider's mechanics. How long until the first token comes back, how many tokens per second once it starts, and what the call costs at published list rates. Those shift weekly with provider load, and if you run any production workload on an API, they are the numbers that actually hit you. llm-meter exists to measure that second thing, deliberately separate from quality benchmarks, with results stored apart and joined on a shared schema later.

How it works

It is a small Python CLI (3.10+, installed with uv) with three moves. llm-meter run fires a probe at any OpenAI-compatible endpoint — either a sustained-generation prompt (a ~1k-token essay, for real tok/s) or a one-word prompt that isolates time-to-first-token. A pi adapter measures a full agent loop instead, tool calls and history replay included, as a coarser kind: "agent" number the report never mixes into completion rows. Every probe appends one JSON object to results.jsonl — timestamp, provider, model, tokens in/out, TTFT, seconds, tok/s, cost. The schema is the contract: jq, pandas, or another benchmark's report can read the file directly without importing the package. Costs are priced at published per-token list rates from llm_meter/pricing.py — list price on purpose, because subscriptions don't price a production workload — and models without a known rate get null, never a silent 0. SDK retries are disabled during probes, since a hidden retry would fold two attempts into one timing. Token counts are the provider's own, streamed with include_usage, not estimates.

What's real today

Version 0.1.0, MIT license, 17 tests across five files covering the probe, pricing, report, schema, and the pi adapter. The repo ships a dated snapshot of real probes in BENCHMARKS.md — stamped 2026-07-21, deepseek and z.ai, three runs each, medians reported. Two rows from that week, sustained-generation probes: deepseek-v4-flash returned first token in 0.42s and sustained 61 tok/s at $0.00031 per probe; z.ai's glm-5.2 took 0.88s to first token, 44 tok/s, $0.00463 per probe. The same file shows why the tool exists: under load, sustained probes on those same models logged 6–9s TTFT, while short prompts minutes later answered in 0.5–1.8s. That drift is the whole point — the snapshot is an example of the output, not a leaderboard.

Run it yourself

Python 3.10+ and uv; keys come from environment variables (DEEPSEEK_API_KEY, ZAI_API_KEY, OPENROUTER_API_KEY, ANTHROPIC_API_KEY) or a keys file:

git clone https://github.com/eanderson4/llm-meter
cd llm-meter
uv tool install -e .        # puts `llm-meter` on PATH

# sustained-generation probe: TTFT + tok/s over a ~1k-token essay
llm-meter run --provider deepseek --model deepseek-v4-flash -n 3

# any OpenAI-compatible endpoint
llm-meter run --base-url https://api.example.com/v1 --key-env MY_KEY --model whatever

# pivot everything measured so far
llm-meter report

git clone https://github.com/eanderson4/llm-meter
cd llm-meter
uv tool install -e .        # puts `llm-meter` on PATH
# or run in place:
uv run llm-meter providers

# sustained-generation probe: TTFT + tok/s over a ~1k-token essay
llm-meter run --provider deepseek --model deepseek-v4-flash -n 3

# TTFT-dominated probe (one word of output)
llm-meter run --provider zai --model glm-5.2 --prompt short -n 5

# any OpenAI-compatible endpoint
llm-meter run --base-url https://api.example.com/v1 --key-env MY_KEY --model whatever

# agent-loop probe: one pi one-shot mission, tokens read from the session file
llm-meter pi --pi-provider zai --model glm-5.2

# pivot everything measured so far
llm-meter report

kind        provider    model                 prompt     tag        n err   ttft   tok/s    out   $/probe
----------------------------------------------------------------------------------------------------------
completion  deepseek    deepseek-v4-flash     sustained  -          3   0   0.42      61    980   0.00031
completion  zai         glm-5.2               sustained  -          3   0   0.88      44   1010   0.00463

{"ts": "...", "kind": "completion", "provider": "deepseek", "model": "deepseek-v4-flash",
 "prompt_id": "sustained", "tag": null, "ok": true, "error": null,
 "tokens_in": 31, "tokens_out": 987, "cached_in": 0,
 "ttft_s": 0.42, "seconds": 16.6, "tok_per_s": 61.1, "cost_usd": 0.00028}

Repo post · repo created 2026-07-20, posted 2026-08-06. The vibes-based alternative is picking whichever provider felt fastest last Tuesday. It benchmarks at one vibe per second.