AI model routing: compare cost per successful task before chasing cheap tokens

Read the announcement as an experiment idea

Ming Lu, Andreas Jansson and Harrison Harnisch published Cloudflare’s Auto Router announcement ↗ on 2026-09-30. The public beta chooses a model for a request rather than making every user select one. The educational question is how to judge the resulting work, not how to promise savings from a headline.

Below is a self-designed evaluation plan. We have not run it or measured savings. Use it to decide what you would need to record before changing an existing AI workflow.

Define success before comparing prices

Invent a small batch of document tasks: extract a delivery date, list three stated product constraints, and return a record in a fixed JSON shape. Write the expected values yourself. Include a few documents where the requested value is missing, so the correct response must explicitly preserve that uncertainty.

Keep a separate task that needs more reasoning, such as identifying a contradiction between two invented requirements. Give that task its own rubric. A cheap model may suit one class of work while requiring additional review in another; an average score can hide this difference.

Judge outputs without looking at the model name when practical. Otherwise, a famous model or a low price can influence how generously you grade an ambiguous answer. Preserve the raw output so an unexpected verdict can be revisited.

Count the complete attempt

Record the cost of all requests used for the task, including retries and unsuccessful attempts. Divide the batch’s actual request cost by its number of outputs that pass your rubric. Call this the measured request cost per successful task, and report the failure count beside it. If no output passes, the ratio is undefined; do not show zero as a good result.

Keep human review time in a separate column. You can later estimate its value under an explicitly stated assumption, but it is not the same thing as API charges. A low invoice and a long cleanup session can coexist.

For example, consider invented results where one route costs 2 units and yields 8 accepted outputs, while another costs 3 units and yields 15 accepted outputs. Their costs per accepted output are 0.25 and 0.20 units. This arithmetic is a teaching example, not Cloudflare’s results or current currency pricing.

Track routing conditions as well as answers

Cloudflare’s Auto Router documentation ↗, updated 2026-10-02, describes eligible model restrictions, routing response headers and session identifiers. These settings help make the selected provider and conversation boundary observable. A route can only choose from models that your configuration permits.

For your experiment, save the chosen model, task class, total requests, latency and rubric verdict. Compare like-for-like tasks, and record the allowed model pool at the time of the run. In a multi-turn exercise, keep the conversation boundary stable so a different session definition does not become an unrecorded variable.

Decide by the failure pattern

Finish with a short error log: missing fields, unsupported claims, malformed JSON and unresolved contradictions. Write a fallback rule for one common failure and decide which tasks still need human review. Run that rule on the saved examples before proposing a larger trial. The useful outcome is an inspectable operating decision; whether it saves money remains a question for actual measurements.