(Tested on OS X 10.8.2; other releases may be slightly different.)
You can change the system’s limit on ptys with the sysctl command.
sudo sysctl -w kern.tty.ptmx_max=255
This should immediately let you open more ptys.
To set this automatically after each reboot, put the following line in the
(Note: I did not try this, but, from a look at the source code, it looks like launchd does process this file.)
This file may not exist; if you have to create it, you should probably be sure it has restricted ownership and permissions:
sudo chown root:wheel /etc/sysctl.conf
sudo chmod 644 /etc/sysctl.conf
Eventually, you may run into some other per-process or per-user resource limits (e.g. total number of processes per user, and total number of open file descriptors per process).