Obviously, scanning the content of messages and content is controversial, but Apple says it performs the scan on the device. Users can also turn the feature off in the settings. “Sensitive Content Warning uses on-device machine learning to analyze photos and videos. Because they’re analyzed on your device, Apple doesn’t receive an indication that nudity was detected and doesn’t get access to the photos or videos as a result,” its website adds.

I’d rather these kind of systems be set up to run locally like this, even better that we are given control to disable it. For the ML model it’s a balance between too many false positives and too many false negatives.