Is it possible to edit some code in Notification Center to allow Facebook messenger notifications to have a "Reply" or "Close" similar to iMessage?
No. As of 10.9.1, Apple has not added the ability to do so, and unfortunately, you're going to need to wait on Apple to do this natively.
I can, however, suggest an app that you can try. It's called Chat Heads and it works really well. It's available in the Mac App Store for free.
Here's the link: https://itunes.apple.com/ca/app/chat-heads/id670724330?mt=12