easiapiaiLLM API prices

Provider accountability

Check what your LLM provider actually serves

A price page cannot tell you whether the model behind it is the one it names. Run 10 questions with exact answers on the provider you already use, with your own key, on your own computer; our server grades the answers. Many small runs from many people add up to a test nobody has to pay for alone. Why and how.

Ask your coding agent

Paste this line into Claude Code, Codex, Cursor or any agent that can run code. It asks which provider and model you use and where your key is, sends our questions to your provider from your computer, and sends us only the answers. Your key never leaves your machine.

test my LLM provider using easiapiai.com/test
For the agent: how to do it
  1. Ask the user, and nothing more: the seller they use (its name as on easiapiai.com), the OpenAI-compatible base URL they call (the part before /chat/completions), the model (its name on easiapiai.com, and the id they send to the provider), and the name of the environment variable that holds their API key. Never ask for the key itself, never print it, never write it anywhere.
  2. Get the questions: GET https://easiapiai.com/api/test/questions?model=<model name or id on easiapiai>&n=10. Keep the run_id (it is valid for 6 hours) and follow the instructions field. If the model is not found, the answer lists suggestions: show them to the user, do not guess.
  3. Ask the provider, on the user's machine, one request per question: POST {base}/chat/completions with the header Authorization: Bearer and the key read from the environment variable at that moment, and the body {"model": "<id at the provider>", "messages": [{"role": "user", "content": "<the prompt, exactly as given>"}], "temperature": 0, "max_tokens": 2048}. No system prompt, no hints, one try per question: do not retry a wrong answer or keep the best of several.
  4. Measure: latency_ms from sending the request to the end of the answer; ttft_ms only if you streamed, otherwise null; output_tokens from the response's usage.completion_tokens if it has one. If a request fails, send an empty response and "error": "timeout", "http_429", "http_5xx" or "other".
  5. Show the user what you will send (the seller, the model and the answers), then send the raw answers, as the provider returned them, one entry for every question in the run (a run with a missing question is refused; a reply longer than 4000 characters is sent as its last 4000), to POST https://easiapiai.com/api/test/submit: 
    {"run_id": "<from the questions>", "seller": "<seller name as on easiapiai>",
 "base_url_host": "<host of the base URL, optional>", "model_requested": "<id sent to the provider>",
 "official": false, "install_id": "<a random UUID you keep>", "client": "agent",
 "answers": [{"qid": "<qid>", "response": "<raw text, up to 4000 characters>",
              "latency_ms": <int>, "ttft_ms": null, "output_tokens": <int or null>, "error": null}]}
    official is true only when the seller is the model's own developer. Make the install_id once and reuse it if you can keep it: it lets us count people rather than runs, and we store only its hash.
  6. Show the user the result: the score (correct of total, and errors) and the results_url. Say plainly that 10 questions are a small sample: the run adds to the shared count for this model and seller, and nothing about them is published before 300 answers from 5 people.
  7. Never send the API key, the user's own prompts or conversations, files, file paths or project names. Only the test answers, the timings and the fields above leave the machine.

How we grade: for a number question, the last number in the answer; for a choice question, the letter A to D. Answers that failed (a timeout, an error) are counted as errors, not as wrong.

Use the command line

A small open-source tool, easiapiai on GitHub (Python, standard library only, MIT), does the same steps for you. It is not on PyPI yet, so it installs straight from GitHub:

uvx --from git+https://github.com/brokonado/easiapiai easiapiai test --model "<model on easiapiai>" --base-url <OpenAI-compatible base URL> \
  --provider-model <model id at the provider> --seller "<seller name>" --key-env <VARIABLE_WITH_YOUR_KEY>

Or pipx run --spec git+https://github.com/brokonado/easiapiai easiapiai test ... with the same options. -n sets the number of questions (10 by default); add --official when you test the developer's own API. The key is read from the environment variable and goes only to your provider.

easiapiai bill adds up your Claude Code usage (Codex too, experimental) on your machine and prices it at every seller, like the bill page.

Results so far

No results yet. The table fills in as people test the providers they already use. Give your coding agent the line at the top of this page, or use the command-line tool once it is out: your 10 answers would be among the first here.

Agreement is the share of answers that match the reference answer. It is shown for a model at a seller only once there are 300 graded answers from at least 5 people, with a Wilson 95% interval; before that, only the count. It measures these questions, not the company: a low number is a reason to look closer, and the seller hears from us before anything negative is published. This table is rebuilt with the site every 6 hours; agents can read the live numbers at GET /api/test/results.

What we keep

What we store

  • the ids of the questions you were asked;
  • an excerpt of each answer, at most 500 characters, and whether it was right;
  • the timings and output token counts;
  • the seller and model you tested, and the provider's host name if you send it;
  • a one-way hash of your IP address and of a random install id your tool keeps.

What never reaches us

  • your API key: it stays in an environment variable on your computer and goes only to your provider;
  • your own prompts and conversations;
  • your files, file paths and project names.

Results are published only as totals per model and seller, never per person. More on the privacy page.

Test questions: GSM8K (grade-school math word problems), test split (MIT), Karl Cobbe et al., OpenAI. MMLU (Measuring Massive Multitask Language Understanding), test split (MIT), Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt; screened with MMLU-Redux 2.0 (CC BY 4.0). Used under their licences; the answer keys stay on our server.