AI Gateway adds confidence-based decision fallbacks
By Steven Van ·
The beta supports Choice and Score confidence thresholds and Boolean probability ranges, with both stages billed when a fallback runs.
An uncertain answer can now trigger a second model to rerun the complete decision in Vercel AI Gateway, even when the primary request succeeds. Decision fallbacks are in beta and let developers set the conditions for escalation.
Set the uncertainty threshold
Choice and Score questions use confidenceBelow, which checks the concentration of the answer's probability distribution. A threshold of 0.6 triggers a fallback when confidence falls below that value, or when finite confidence is unavailable. Boolean questions use probabilityBetween instead, such as an inclusive range from 0.4 to 0.6 for the probability that an answer is true.
Conditions can target a named question or, with question omitted, check every question of the matching type. Developers can combine conditions using any, all or atLeast to require one, every or a specified number of signals to match.
Configure one fallback stage
To opt in per request or virtual model, add a conditional object containing model and when as the first entry in providerOptions.gateway.models. AI Gateway supports one conditional object and one successful-response fallback stage per request.
Plain model-name entries can follow the object and continue to handle execution errors. Requests without a conditional entry keep their existing behaviour. When an uncertainty condition triggers a second decision, both stages are billed. The decision fallbacks documentation provides configuration examples.