How to use CapsLock as modifier key without consuming Alt+Shift keybinding?

Question

I want to use CapsLock as a modifier key to type symbols like ({\})= using the alphabetical keys in the keyboard.

For example:

CapsLock+a is mapped to =
CapsLock+k is mapped to \

To do this, I mapped CapsLock to Option+Shift using Keyremap4Macbook following the instructions in "A Useful Capslock Key".

<item>
    <name>F19 to F19</name>
    <appendix>(F19 (capslock) to (shift+opt))</appendix>
    <identifier>private.f192f19_escape</identifier>
    <not>VIRTUALMACHINE</not>
    <autogen>
        --KeyOverlaidModifier--
        KeyCode::F19,
        KeyCode::OPTION_L,
        ModifierFlag::SHIFT_L,
    </autogen>
</item>

Note that, before that I mapped CapsLock to key code 80 in PcKeyboardHack which is key code of F19.

Then I mapped key combinations of CapsLock + alphabetical characters to symbols such as =\})( using Ukelele.

So far, everything works properly except that there is a side effect of this setup: I cannot use Alt+Shift modifier for any other key binding. This is pretty restrictive compromise.

I tried mapping CapsLock to more complex key bindings such as Ctrl+Shift+Alt or Cmd+Shift+Alt or Ctrl+Cmd+Shift+Alt. But these didn't work well because then mapping CapsLock+alphabetical keys in Ukelele changes the alphabetical keys too. For example, assume that Ctrl+Shift+Alt+a is mapped to =. Then a alone is mapped to = as well.

I wonder if this problem can be solved somehow? I think there might be two alternative paths to look for solution:

1. Creating a new custom modifier key for CapsLock without consuming existing modifier keys such as Alt+Shift.
2. Preventing the mapping of Ctrl+Shift+Alt+a to change the a key all together.

Are these solutions possible to implement?

Answer 1

Got this all from more recent Brett Terpstra posts, so full credit to him.

Using Karabiner Elements and BetterTouchTool, you can create a "Hyper" key mapped to control-option-shift-command (which probably won't be used by any apps as a default modifier). Then you can use BetterTouchTool to maps Cap Locks + "a" to type "=". Typing "a" by itself still gets the letter "a". You can also set it up that tapping Cap Locks alone functions as the escape key or even as standard Caps Lock.

Rather than reposting Brett's code here, I'll link to the post: A Hyper Key with Karabiner Elements, full instructions.

Some screenshots to help guide you along:

After adding Brett's code to karabiner.json, the Complex Modifications pane will look like so. No need to do anything else here.

In BetterTouchTool, switch to the Keyboard pane, add a shortcut, type Caps Lock + character, then choose "Insert/Type/Paste Custom Text" as the Trigger Predefined Action.

In the Insert/Type/Paste Custom Text sheet, type the character(s) you want and choose "Insert Text by Typing".