I went to the Nikon site and installed S-VCNXSP-160500MF-ALLIN-ALL, but those are just transfer and editing stuff, I want to record from photobooth the raw live video.
Is it possible?
I think you're probably going to need Camera Control Pro 2 - the only one of their suite that isn't free. Though if you buy it from a retailer, not direct from Nikon, it's about ⅓ the price.
I have a Nikon D5500 & also a Logitech HD Pro C920 webcam - neither of which are natively supported by QuickTime.
Using an app called Webcam Settings it can persuade the Logitech to show as a source for QuickTime/Photo Booth etc - but it can't persuade the Nikon to show at all.
It's perfectly happy recording from Camera Control, though, including Live View to the Mac & also file transfer.