When you encrypt iPhone backup on Mac, a password entry dialog will pop up to ask you to enter your password and display a "Remember this password in my Keychain" check box. If you checked the checkbox, your iPhone encrypted backup password was saved in your Keychain files. So you can open the Keychain to find the encryption password once you forget it. 

This post would be helpful: http://www.isumsoft.com/it/find-iphone-encrypted-backup-password-in-keychain-on-macbook/