How to use plain text smileys (disable automatic emoji substitution) in Messages.app? In OS X 10.8 there was a setting for this in the app’s Preferences pane, but in OS X 10.9 that doesn’t seem to be the case anymore.
Go to the Edit menu, submenu Substitutions, then uncheck Emoji.
Run the following command (source) in Terminal:
defaults write com.apple.messageshelper.MessageController SOInputLineSettings -dict-add "automaticEmojiSubstitutionEnablediMessage" -bool false
Then restart Messages.app.