The AI also has the tendency inherited from the broad human tendency in training.

So you get overconfident human + overconfident AI which leads to a feedback loop that lands even more confident in BS than a human alone.

AI can routinely be confidently incorrect. Especially people who don’t realize this and don’t question outputs when it aligns with their confirmation biases end up misled.