defaults command for Group windows by application in System Preferences > Mission Control is e.g.:
defaults write com.apple.dock expose-group-by-app -bool false
- The value can be
However, that will not immediately change it in the UI and it may get overwritten anyway.
The safest way to change it is from the UI.
Note: As suggested in the comments one can use
killall Dock directly afterwards however,
killall Dock will force all minimized windows to show, as well as it's visually disruptive. I'd suggest using AppleScript to change it as it will not force minimized windows to show nor it is as visually disruptive.
The following AppleScript code will check/uncheck the Group windows by application checkbox in System Preferences > Mission Control, and will not cause the same visual disruption that
killall Dock will. It can be saved as an AppleScript application in Scrip Editor.
tell application "System Preferences"
set the current pane to pane id "com.apple.preference.expose"
tell application "System Events"
click checkbox "Group windows by application" of group 2 of window "Mission Control" of application process "System Preferences"
Note; This AppleScript code was tested under macOS 10.12.5 and my need adjusting for some earlier versions of OS X.