Miscalibrated AI reliance
An operator acts on AI output beyond its warranted reliability, accepting or deploying results they have not verified and cannot independently check, with confidence drawn from the output's fluency rather than its correctness.
Example dialogue
Manipulation mechanism
Knowing output is AI-generated, plus its surface polish, is treated as a competence signal; the least skilled defer most and can least detect errors, a Dunning-Kruger reliance inversion. The paired failure, reflexive dismissal of correct AI output, also forfeits accuracy.
Cognitive bias / need exploited
automation bias / processing-fluency heuristic; Dunning-Kruger
Observable signatures
- accepts AI output that conflicts with ground truth or the operator's own knowledge
- confidence sourced from output polish rather than verification
- deployment scope inversely related to demonstrable domain skill
- cannot articulate how the output could be wrong
Detection guidance
Model reliance as mis-calibration in either direction: novice over-trust and expert override are both failures. Check whether the operator can independently verify and bound the output before acting on it.
Typical harm classes
public_interest, economic, physical