I’ve experienced something similar and I always suspected that they were the photos on the phone before iCloud Photo Library was enabled. The only way I know to delete them is manually or through a manual sync on my Mac using Image Capture which gives you the option to delete all the photos copied to the Mac. Note that I’m syncing to a folder on the Mac, not to Photos.
I’m not sure what’s supposed to happen but I was trying to force my iPhone to only have optimized copies, so I turned off iCloud Photo Library, deleted the photos that remained, and then turned iCloud Photo Library back on. Photos proceeded to delete all the photos I’d manually deleted from my iPhone from iCloud Photo Library. That was unexpected but not a huge crisis because I restored them from “Recently Deleted”. However, I did learn not to assume I understood iCloud Photo Library well enough to think I could bend it to my will.
After those experiences, if I wanted to take a device completely out of iCloud Photo Library, I’d erase it and rebuild it form scratch rather than a backup. Painful but the only way to be sure.