It's hard to tell without access to your system or steps to reproduce your error, but in my experience, the OS is performing like any other UNIX when the swap device and root filesystem fills. The simplest way to avoid the dialog is ensure you have enough free space to make another swap file (they top out at 2 GB I believe on Mountain Lion).
Most people like to have 25% (or more) free space since the OS optimizes hot files and defragments on the fly and these self-disable when free space becomes low to avoid churning the IO when there isn't space to write or many free places on the drive to choose for faster file access.
One notable difference between OS X and other UNIX is that the Apple OS does late page allocation to swap, so you don't see large swap files when they are not needed. For this, you can either disable swap entirely or ensure your drive has sufficient free space to handle whatever amount of swap you see in practice.
I've not heard of anyone messing with the paging algorithm itself, but Apple has open sourced that part of the OS, so anyone who cares can make code level changes and run with it.
I would start with buying the next largest side storage drive and monitor your space usage to ensure you don't get less than 10 GB empty as a starter rule of thumb. If you're doing serious work, you might invest more in drive space to ensure you don't run out.