Is there an easy way to upgrade from OS X 10.6 to OS X 10.6 Server, or a workaround to eliminate the 10-user limit for ftp connections?
No - server needs additional packages to be run at install time and there is no general upgrade path or migration assistant to move a client mac to server.
Even at $499 Snow Leopard Server provides a lot of bang for the buck - but if you can compile open source and don't need this NOW a free solution might make more sense.
Once you confirm the new binary works as planned for more than 10 users, you can dig into launchd and set it up to call your ftpd instead of the built in one.
I find the smoothest, easiest to rollback method is to use disk utility to partition to the hard drive and add a volume you want to put a fresh OS on. Then do a fresh install and move over the needed data. Working DNS and Open Directory is recommended, there are docs from Apple and books available on the topic.
FTP is deprecated in Lion server, and the hacks to re-enable it severely compromised security in my testing. It's not a secure protocol by default, so I would recommend finding another way to server the resource in question.