It appears that you can't recover the file, at least at the level of the Finder. However you may be able to by using recovery software (See below as to where the file is temporarily located).
When a QuickTime recording is started using File : New Movie Recording, the temporary file sits in the folder:
~/Library/Containers/com.apple.QuickTimePlayerX/Data/Library/Saved Application State/com.apple.QuickTimePlayerX.savedState/
Once the recording is closed, without saving, the file window_(some unique number).data is automatically deleted. You can see this if you go to the location above, start a recording, and then close without saving.