If I don't have my phone on me, and it has nothing to connect to, would I still be able to make payments with just my watch?
I have an Apple Watch series 4 (no cellular)
Yes.
Per Apple, you can still do many things even if your Watch isn't connected to your iPhone, cellular, or wifi.
Thanks to @grg's comment for pointing out that the Watch uses NFC to make a connection to the payment terminal.