Handbrake is probably not the best choice for this as it's so unfriendly and difficult to setup. The easiest way I've found is:
Select one of the Matroska H.265 Presets
Change Summary > Format to MP4 File
The only issue is if your source video does not match one of the presets.
You might have more luck, and make your life easier with Half ...
High Sierra's H.265-capable apps want hvc1 instead of hev1. Handbrake's stable release as of now (September 26, 2017) does the latter. Nightly builds should do the former.
Relevant code change: https://github.com/HandBrake/HandBrake/commit/67dfaca7ac3ec56cc4ab3c7f9f60ab3b7c43a0db