You could see about getting a quote from a data recovery team to attempt to reverse engineer your situation. Since you are fairly certain that the key is 12 characters long, that significantly reduces the number of passphrase that must be attempted to unlock the volume.
Also, paying a professional to do this means you are only renting the equipment and/or software to try each combination rather than needing to fund that yourself.
Since the encryption takes time and is reversible, it clearly is possible to brute force this, but I would get a quick feeling for how much of the drive is encrypted by saving the output of
diskutil list and
diskutil cs list and then ejecting the drive and powering it off until you can get it in the hands of a pro.
Also, you do have to re-type the entire password manually and then confirm the operation, so perhaps you just had a normal software error and didn't establish the encryption. Best of luck with the recovery. Losing data is no fun and having baby pictures only on one drive is not something I ever like to confront - yet it does happen and the pain is awful in most cases to lose those artifacts of the time and memory.