How can I find out which algorithm OSX10.7 uses (NewReno, SACK, Tahoe...) and if possible, how can I change it?
This Stack Overflow question should help: How to load a different congestion control algorithm in Mac OS X?
The LEDBAT congestion control algorithm is built into OSX Lion (10.7.X) and does not need to be compiled separately. I don't think it would be possible to use LEDBAT in an earlier version of OSX without alot of work. LEDBAT provides for 'background' transfers and may be selected on a socket by setting the following socket option - setsocketopt(sock, SOL_SOCKET, O_TRAFFIC_CLASS, SO_TC_BK, sizeof(int)). It also seems that the socket's traffic class may affected by the DSCP value in the IP header.
Additionally OSX has had the facility to load kernel modules for other network protocols for some time using their Network Kernel Extensions (NKE) - for example there is an NKE module that has been developed for the SCTP protocol.
Alternatively, another related question might be useful: How do you set the TCP congestion control algorithm on OS X?
According to http://www.macgeekery.com/gspot/2006-03/mac_os_x_network_tuning_guide you can actually modify the algorthims used by Mac OS X by using the sysctl command, but you cannot use anything but the New Reno algorithm