Referring to iMessages, not SMS. Pasting text ~20,800 characters long gives an error that the message is too long on my Mac. What's the limit?
As of macOS 10.12 iMessage (Messages app) seems to be able to send arbitrarily long messages without breaking them up or returning an error.
I have managed to send very large messages over 25k characters at this point. iOS then displays these in a larger message format that makes it easier to read the message when it is expanded.
(This answer doesn't apply to the latest version of Messages in 10.12 which appears capable of sending VERY large messages)
TL;DR; In 10.10.5 the limit is 18996 but you can't paste in a block that large.
This isn't the answer you want but it's the answer you should listen to:
iMessage isn't designed for sending huge long messages. Write an email instead.
But if you really want an answer the max characters is 18996 but the catch is that iMessage on OSX (10.10.5) won't let you paste in a single block of text that big. Try breaking up and pasting in smaller 1000 character chunks if you must use this platform to send large amounts of text.