EDIT: The following is a partial solution. It will work at the shell level as expected but in Finder it can be bypassed with the if user is admin or knows the a admin user id & password. See comments below...
Check the man page on
chmod and you will find...
The following permissions are applicable to directories:
list List entries.
search Look up files by name.
Add a file.
Add a subdirectory.
Delete a contained object. See the file delete permission
So the command you are looking for is...
$ chmod +a "staff deny add_subdirectory" test
to forbid anyone in group staff from creating sub folder and then testing should give you
$ mkdir test/subtest
mkdir: test/subtest: Permission denied
Where as creating a file...
$ touch test/blah.txt
is no problem. You should also find that Finder will have the Create Folder option disabled as well.