Your passphrase isn't being stored anywhere, but your decrypted private key is stored (in memory) by a process called ssh-agent
(man page). This process, which OS X starts when it boots up, stores and manages private keys so they never have to be exposed to other processes that use SSH connections.
When you enter in your password, your computer decrypts your private key and ssh-agent
gets a copy to hold on to until it is killed (e.g. on shutdown) or the key is manually removed using ssh-add
(man page):
ssh-add -l
lists all currently held keys
ssh-add -D
forces ssh-agent
to forget all currently held keys
ssh-add ~/.ssh/newkey_rsa
adds the private key ~/.ssh/newkey_rsa
to ssh-agent
.
ssh-add -t 3600 ~/.ssh/newkey_rsa
adds a new private key with an expiry time, so ssh-agent
will only remember newkey_rsa
for (say) 3600 seconds.
It may satisfy your concerns to know that your passphrase isn't stored anywhere. But if you really want your computer to prompt you for your passphrase every time, you could use ssh-add
to make ssh-agent
forget your key and then re-add it with a short expiry time.
Keep in mind that other solutions — like requiring a password to unlock your workstation when you're away from your desk — may also address your underlying security needs.