If you use humans to fine tune and judge the quality of output, then in some sense, that’s pretty much all the AI can possibly do.

Everyone can see the output, “I don’t know.” and mark it zero (dude). But if the model instead generates some bull then the meat bags on the other side are absolutely going to reward that behavior with a sounds-good-to-me thumbs-up.