Your passphrase isn't being stored anywhere, but your decrypted private key is stored (in memory) by a process called
ssh-agent (man page). This process, which OS X starts when it boots up, stores and manages private keys so they never have to be exposed to other processes that use SSH connections.
When you enter in your password, your computer decrypts your private key and
ssh-agent gets a copy to hold on to until it is killed (e.g. on shutdown) or the key is manually removed using
ssh-add (man page):
ssh-add -l lists all currently held keys
ssh-add -D forces
ssh-agent to forget all currently held keys
ssh-add ~/.ssh/newkey_rsa adds the private key
ssh-add -t 3600 ~/.ssh/newkey_rsa adds a new private key with an expiry time, so
ssh-agent will only remember
newkey_rsa for (say) 3600 seconds.
It may satisfy your concerns to know that your passphrase isn't stored anywhere. But if you really want your computer to prompt you for your passphrase every time, you could use
ssh-add to make
ssh-agent forget your key and then re-add it with a short expiry time.
Keep in mind that other solutions — like requiring a password to unlock your workstation when you're away from your desk — may also address your underlying security needs.