So I was having some issues connecting to a WPA2 WiFi network and a prompt showed up asking for me to "Enter exactly 64 hexadecimal characters". I was able to simply hit Cancel and simply reconnect but I was wondering what this was. I have never seen this prompt before and was wondering if anyone could shed some light on how to reproduce it and why it occurred.
You are connecting to a WPA2 network in WPA-PSK (pre-shared key) mode. In this mode, you'll usually be able to authenticate by entering a password as a string of ASCII characters (up to 63 characters). macOS will generate a 256-bit authentication key from the password using a series of manipulations (known as salting and hashing) that ensures that eavesdroppers cannot easily grab your password midair.
However, it is also possible in this mode to authenticate by having the user directly enter the actual authentication key. This can be used if macOS cannot determine with the WiFi access point exactly how those manipulations described above should be done. The user is then prompted to enter exactly 64 hexadecimal characters that is equivalent to a 256-bit authentication key. This method is not commonly used in a home user setting.
I would say that some sort of "glitch" happened - interference, lack of signal, etc. - that made it impossible for macOS at that moment to use the ordinary password authentication, and then prompted you to allow a key authentication instead. When you reconnected, those temporary conditions that allowed for the glitch were gone, and you could authenticate as usual with your password.