If I give an app access to my camera roll, what exactly does that imply? Can the app in theory upload all my photos to their server? E.g. when I upload an image to Instagram I can scroll through my camera roll and choose a photo to share. Even though I only press share on this one photo, can Instagram in theory upload my other private photos to their server? (Please add sources for your statements)
You have nothing to worry about.
An app will always be required to ask permission to access your Camera Roll - no matter what it needs it for. Apple's photos system is built in a way that prevents an app from simply retrieving a photo and sending it to any remote service without first having your consent to do this AND it will have to display the process to the user.
Every app that is the App Store has been thoroughly examined by Apple and the binary (the code) is tested to ensure it doesn't do anything sketchy - in which case Apple would simply not allow it to be published on the App Store.
An app will only ask permission to access your Camera Roll if it wants to save a photo, or retrieve a photo for a specific use case (for example, for a profile picture). In most cases, the app will show you what it is doing with your image(s) as that is how Apple wants developers to build apps - so that they're user friendly and the user has as much control as possible. However, even if the app was built in a way that meant it would fetch images in the background (for example, Google Photos), you will always have to provide consent before it initiates anything.
Also, remember you always have the ability to decline access to your Camera Roll when the app asks, or you can go to settings and remove access from there.
If you have any further questions let me know.
I don't have any sources to list, but I'm an app developer myself.