This is because you are booting into a separate partition on your hard drive. FileVault identifies volumes via partitions so when you encrypt your "hard drive" it is the partition that contains your boot volume and data only. The recovery partition and any other additional partitions are never encrypted.
This isn't cause for certain though since booting to the Recovery Partition does not expose your encrypted volume nor does doing so grant you access to it. As a matter of fact, if you need to repair your user folder's ACL's and permissions; which you would do through your recovery partition, you would have run some terminal commands to unlock your encrypted partition before repairs can be done.
Boot Camp would serve as a good example of this. With Boot Camp, your hard is now partitioned into two volumes; One for Mac OS and another for Windows. This would add up to three partitions on your hard drive. FileVault would only encrypted your Macintosh HD partition and not touch your Windows and Recovery partition; but at no point would you be able to access your Macintosh HD's content via Windows or Recovery Mode.