It seems this invokes the 'App Exposé' feature, which shows all windows of a single application in the Mission Control display style.
It is by default available with a four-finger downward swipe on the trackpad; that will invoke this display for the currently active application, or, if you do it while hovering over an application icon in the Dock, for that specific application. You can change this in System Preferences → Trackpad → More Gestures:
As @user3439894 mentioned in the comments, you can also right-click an icon on the Dock and select 'Show All Windows' from the context menu:
Another option is to create a keyboard or mouse shortcut for the 'Show Application Windows' command in System Preferences → Mission Control: