I'll leave the iOS version below,. as that actually works - but for OS X it's nowhere near as efficient.
Messages > Prefs > Accounts > Blocked
Click +
You can type the phone number but nothing will happen unless the number is already recognised as a Contact [which is frankly infuriating].
One workaround might be to make a Contact called "zzz spam" & keep adding bad actors to that contact, which may also immediately add them to the banlist anyway.
tbh, I'd prefer a better answer than this, if anybody has one
[Previous answer, mistakenly for iOS]
From Apple KB : Block phone numbers and contacts or filter messages on your iPhone, iPad or iPod Touch
Messages
If you're in Messages, open the conversation, tap
,
then tap the name or phone number. Scroll to the bottom of the Info
screen, then tap Block this Caller.