TL;DR answers to questions
How does macOS validate and secure connections between Bluetooth
keyboards without prompting for a code?
It's not just macOS - your keyboard is connected and encrypted at the firmware level - if it wasn't you wouldn't be able to perform an NVRAM reset before the OS loads.
However, you don't need a code because modern Bluetooth keyboards use an authentication algorithm to pair and a generated authentication key based on public keys to secure the data transmissions.
Does it just use trust-on-first use?
No. It verifies its identity using an algorithm and 128-bit generated keys.
Is it possible to force macOS to use a pairing code?
Why? This would be going back to pre-2009 pairing methodologies. This would be the equivalent of enabling WEP shared keys on modern WiFi networks.
Bluetooth Pairing
The process you described for pairing a Bluetooth device (a keyboard in your example) uses an old (Bluetooth 2.1) method of authentication called Simple Secure Pairing (SSP).
Basically, SSP has 4 association (pairing) models:
- Numeric Comparison. Both devices have input and display so the user can simply select "Yes" or "No" to pair the device
- Passkey One device has input capability (like a keyboard) and the other has display capability (like a computer). The device with display capability shows a 4 to 6 digit number and the device with input capability enters it.
- Just Works This is for devices with no display or input capability (like headsets) where you can't see or enter a passkey.
- Out of Band (OoB) This is for devices that support an additional common wireless technology (NFC) where the devices have to be in very close proximity to each other. One device must "tap" the other device before pairing will occur.
The most important thing to note is that SSP is NOT the encryption key; it's merely the pairing mechanism to identify each other. Encryption is handled through a public key. The code you enter is to make sure that's the device you want to connect to; it's not the security.
Since Bluetooth 3.0 (April 2009), Bluetooth devices uses an AMP key for authentication which automates the above process.
AMP Key Derivation
AMP link key is derived from the Bluetooth link key. A Generic AMP Link
Key(GAMP_LK)is generated by the AMP Manager in the host stack whenever
a Bluetooth link key is created or changed
Authentication
The Bluetooth device authentication procedure is in the form of a
challenge–response scheme. Each device interacting in an authentication
procedure to as either the claimant or the verifier. The
claimant is the device attempting to prove its identity, and the
verifier is the device validating the identity of the claimant. The
challenge–response protocol validates devices by verifying the
knowledge of a secret key — the Bluetooth link key.
Encryption
There are 4 modes of encryption
- Mode 1 - No encryption
- Mode 2 - Individually addressed traffic is encrypted based on keys which are based on the link keys
- Mode 3 - All traffic is encrypted using keys based on the master key
- Mode 4 - (Bluetooth 2.1 + EDR) mandates all traffic is encrypted except for service discovery
Bluetooth keyboards using Bluetooth 2.1 (keyboards from 2009 and beyond) and up encrypt all their traffic.
SOURCE:
SP 800-121 Rev. 2, Guide to Bluetooth Security (May 2017)