Miscalibration and overconfidence
The model's expressed confidence does not track its actual accuracy, systematically overstating certainty.
Example dialogue
UserHow confident are you that Lake Baikal is the largest lake in the world by surface area?
AssistantI am about 99% confident - Lake Baikal is the largest lake in the world by surface area.
Illustrative — invented, not a real system.
Manipulation mechanism
RLHF reward models favor confident-sounding answers regardless of correctness, inducing verbalized overconfidence.
Cognitive bias / need exploited
stated confidence read as competence
Observable signatures
- high stated confidence on incorrect answers
- confidence insensitive to answerability or difficulty
- uniform certainty across easy and hard questions
- no downward adjustment when evidence is thin
Detection guidance
RLHF induces verbalized overconfidence via reward-model bias (ICLR 2025); verbalized confidence still beats raw log-probs, with expected calibration error down about 50% when elicited (EMNLP 2023). Model-side; distinct from operator-side Miscalibrated AI reliance 0066 (a user over-relying).
Typical harm classes
public_interest, psychological, economic
Crosswalks
guardOperationalized in Guard as Miscalibration / overconfidence (III.3)
darkbenchnull
owasp_llmnull
mitre_atlasnull
avidnull
eu_ai_actnull
dsanull
Registry ID
ADP registry IDADP-P-2026-0079
Linked observations (0)
No linked observations yet.