Evidence kinds/Fallback decision log
Fallback decision log
process evidence
Records when the substrate fell back to a safe default after model failure or schema violation.
What it is
When the model output fails to validate against the spec's outputSchema (closed-enum violation, missing required field, type mismatch), the substrate triggers the spec's fallbackBehavior: reject (return an error), deterministic-default (return the spec's deterministicDefault value), or queue-for-review (route to human reviewer).
Each fallback firing is recorded on the L12 entry with fallbackTriggered: true and schemaValid: false. The fallback rate over time is a regulator-side health metric — sustained high fallback rates indicate model drift, prompt-template misalignment, or a real distributional shift.
For high-risk AI under EU AI Act Art. 15 (accuracy + robustness), the fallback rate is a directly regulatable proxy for system reliability.
What it guarantees
- Per-decision record of fallback firing.
- Fallback-rate metric computable from chain.
- Cryptographic tying of fallback event to specific input + spec via L12 entry.
What it does NOT guarantee
Honest limits. Every primitive has them.
- That the fallback behavior itself is appropriate for the use case — operator's design decision.
- Recovery from model unavailability — that's an operational concern outside the spec.
Regulations that engage this kind
Accuracy + robustness + cybersecurity
ICT incident detection + classification
Post-market surveillance signal
Sectors that rely on this kind
Related evidence kinds
L12 receipt chain
Hash-chained, Ed25519-signed log of every LLM-in-the-loop decision a Promethean-built product makes.
Reviewer gate log
Human-in-the-loop primitive: records when the gate fired, the verdict (approved/rejected/amended), and the reviewer identifier.
Spec hash commitment
Every L12 entry includes a SHA-256 of the active RuntimeAISpec, making silent spec drift detectable.
FAQ
What's a reasonable fallback rate?
Depends on the use case. Highly-structured tasks (closed-enum classification with stable input distribution) typically run < 1% fallback rate. Drafter / generative tasks with looser schemas can run 3-5%. Sustained drift upward is the signal worth investigating — model deterioration, prompt regression, or genuine input-distribution shift.
What's the difference between fallback and reject?
Reject is one of three fallback modes ('reject', 'deterministic-default', 'queue-for-review'). Reject means the substrate surfaces an error to the caller and doesn't supply a value. Deterministic-default returns a fixed safe value. Queue-for-review routes to the operator's human-review pipeline.