Your brother's Apple ID probably still has his phone number associated with it for iMessage. The smoothest way to deal with this is to have him disassociate his phone number from his Apple ID.
Apple has a self-service web site that explains how to accomplish this:
If that doesn't work, this older KB article has a helpful section "Unlink a phone number".
If he has other Apple devices (Mac or iPad) that are using his Apple ID, your iMessages may be being delivered there. If none of those devices are available when you send him an iMessage, your phone will eventually offer you the opportunity to resend the message as an SMS or MMS.
Unfortunately you can't disable iMessage on a per-contact basis.