EDIT: The following is a partial solution. It will work at the shell level as expected but in Finder it can be bypassed with the if user is admin or knows the a admin user id & password. See comments below...
Check the man page on chmod
and you will find...
The following permissions are applicable to directories:
list List entries.
search Look up files by name.
add_file
Add a file.
add_subdirectory
Add a subdirectory.
delete_child
Delete a contained object. See the file delete permission
above.
So the command you are looking for is...
$ chmod +a "staff deny add_subdirectory" test
to forbid anyone in group staff from creating sub folder and then testing should give you
$ mkdir test/subtest
mkdir: test/subtest: Permission denied
Where as creating a file...
$ touch test/blah.txt
$
is no problem. You should also find that Finder will have the Create Folder option disabled as well.
HTH