First, Allan is right to enter "UseKeychain yes" in your .ssh/config file.
Second, my method comes also handy in another way
# Use Apple Keychain for ssh-key passphrases
# automatically load keys into ssh-agent
# Automatically forward ssh-agent to destination
If you know add the key to your ssh-agent by issueing
and enter your passphrase, you can start. BUT you also add your passphrase to you keychain:
-K Store passphrases in your keychain.
With -d, remove passphrases from your keychain.
ssh-add -K <keyfile>
you not only load your key but also save the passphrase in your keychain.
You have to issue the "-K" only once.
With that configuration you have a save key and the passphrase safely stored in your keychain and whenever you open a terminal using ssh, the key is automatically loaded into you ssh-agent and the passphrase is taken from your keychain.