Have you enabled two step verification? If not that would be the first thing I do after reading this. This makes it so if someone trys to sign into your account it sends a message to all the devices you have associated with your account, presenting you with the name of the device and the location of the device (based on IP)
Try to hack your own account, and find out the steps you need to gain access.
For me when I try to recover my password (the first step a hacker might take) it asks me to confirm my phone number. If you don't know your phone number you're out of luck. If you do know the phone number then it doesn't matter much anyways as a confirmation message is sent to your phone.
I also believe they cannot present your security questions without you not being logged in for 24 hours. That could be Google I'm thinking of though.
After that information here are a couple steps I would take if I was worried about security.
Enable two step verification.
Be sure you use a different password for your Apple Id than you do on any other account (good practice for everything anyway) Never give out your password, be sure to always confirm where you are on the web. Ex. "Google.com" not "Gooogle.com" (Even though this example doesn't work because Google owns both domains, you get the point)
Be sure you have access to the phone number and email address on file with your account.
Ensure the email you have listed for your Apple ID is secure, and uses it's own password as well so a hacker couldn't gain access to that either.
I would recommend using Gmail if you don't already use it with your Apple ID, as I find them to be the most secure email to use.
Don't worry too much, you're in good hands!