The Character Viewer has two states, as seen below.
The complex state, that needs to be clicked first and doesn't seem to allow keyboard-only activation:
The simple state, that gets automatically activated when invoked and is a joy to work with using only the keyboard. Tab key is used to switch between emoji categories:
You can switch between the states ...
Unless/until Apple deigns to provide a built-in unicode keyboard (as it does for macOS), you'll need to use a keyboard provided by a 3rd party app.
The free app CharMap provides a minimal version of this, where you may have a hard time finding the symbol you're looking for.
×, for example, is about halfway through the LATIN-1 SUPPLEMENT section, near the ...