You can disable the auto start of Android File Transfer, and only open it in case you need it.
Here are the steps you need to do:
- Open up User & Groups and remove Android File Transfer from the Login Items
- remove the agent from Google's app's library:
rm -r ~/Library/Application\ Support/Google/Android\ File\ Transfer/Android\ File\ Transfer\ Agent.app
- remove the backup agent from Google's app:
rm -r /Applications/Android\ File\ Transfer.app/Contents/Resources/Android\ File\ Transfer\ Agent.app
The removing steps are needed, as otherwise the app will unfortunately reinstall itself to the login items the next time you start it. Also instead of removing, you can just move them away for backup in case you might need them.
After doing these steps the app will not start when you connect a phone, and you need to open it manually (CMD+Space plus start typing "Android File Transfer") when required.