Your best defence is always to turn off unnecessary services. If you're not using remote desktop: turn it off. If you're not using the HTTP or FTP servers: turn them off. Fewer services running, fewer points of entry for possibly intruders to exploit.
Aside from locking it down, there are some free and open source products that are OS X friendly you can look at to do intrusion detection on your machine.
Though I haven't personally run it, I do have colleagues who know and trust it for intrusion detection. It's BSD-compatible so it makes a it a good fit for OS X. Another upside to going with Snort is it's available as a Homebrew package:
> brew info snort
Depends on: daq, libdnet, pcre
For snort to be functional, you need to update the permissions for /dev/bpf*
so that they can be read by non-root users. This can be done manually using:
sudo chmod 644 /dev/bpf*
or you could create a startup item to do this for you.
So you get a simplified path to installation and some trust in the fact that it ports well to OS X and runs there. With Homebrew installed you only need to do:
> brew install snort
And you're ready to get started with it.
Check out this Snort for OS X Lion Server setup guide that the Snort community provides to get started with rule writing for your OS X machine. That's a great document and, in addition to walking through installing Snort from source (which you don't need to do), it talks about all the things you should do your OS X Lion Server instance to help protect it. If you install via Homebrew, start at Section 5 (page 13) in the PDF since you don't need to worry about installing it from source code.
I've run Tripwire on linux machines to do rapid intrusion detection and alerting. It's effective but it's a bit of a beast to set up. It can perform actions when rules are matched against log files. Of course, a savvy hacker is going to know to disable Tripwire as soon as they break in so they don't end up with their session getting cut off.
The MacWorld hint talks about setting up Tripwire on OS X. It's not simple and the article ends with mentioning that it's not been tested.