It is possible to brute force a password or recover the recovery key from memory if someone is determined enough. The guest account by default is a Safari only account. It would be to your advantage for them to log in to the Guest account as it activates "Find my Mac" if they jump on to an internet connection. From the guest account they will have tono access to the machines files.
In the future, best practice would be to set a firmware password. Without a firmware password set they can still have access to the machine. To be the most secure you should set both filevault 2 and a firmware password.