When iCloud photo is enabled, all photos are uploaded to iCloud and only a low-resolution copy is saved on the phone for browsing, the high res photo is download on-demand if the user clicks it.
Photos app will perform on-device machine learning when the device is not in use and is charging(typically while you’re sleeping) to detect faces in your photo library so that you have “people albums”. This is just one case that the app is using ML to achieve.
What confuses me is that the low res images are just not good enough to perform ML on, especially for face detection. So how can it analyze those low-res photos and giving the correct prediction?
I have two guesses:
Don’t care low or high res, just analyze what is on the device. (I don’t really believe this is the case as the face detection is too accurate to be believed that they were a result of ML on 480p photos)
On-demand loading, download the original photo from iCloud while the ML algorithm is running, and delete the original photo from the device when it is done. (Isn’t this a bandwidth waste for iCloud servers?)
Edit 1: I’m interested in how will it handle the case when I manually added some photo to one person(this action causes the model to change, well, eventually change after it learns the new photos that weren’t detected previously), all photos must be analyzed again using the new model. At this time, most high-resolution images (let's say 199GB/200GB) are in iCloud only, so will the system pull them one by one to perform ML using the new model again? What if I “edit” the ML regularly? (by adding an un-detected photo to people album but not all at once) If so, this could be a huge drain to the iCloud servers... don’t forget some people do have more than 1TB of photos in iCloud.
Don't take me wrong, I'm purely interested in how this whole thing works because to me it is done very nicely. And to be clear, I completely trust Apple on the "on-device" promise, and this is exactly why I asked this question to get an understanding of the technical details that make it both respecting privacy and giving accurate results.