Executive summary
A swap area (called swap partition, swap file or page file) is just a mechanism to augment the physical memory addressable by the OS.
If you are low in available physical RAM, and on today's machine, I would say if you have 4 GB or less RAM. Then you should probably have a swap. But this depends on your workload and how you use your machine.
If you have more than 4GB of RAM, there is a good chance that your system might never use the swap. So if the swap is enable or not, this won't change much.
Although this depends on other factors, the longer your machine will be running the more likely the OS will swap small unused portion of the RAM to the swap eventhough there is plenty of place left apparently. However, often the system use the "free" RAM for file buffers and other stuff, and some workload might make your file buffer use all available RAM.
Background information
Any way what would happen if you disable the swap and your memory is completely used? Well, the next time an application request more memory (like with a malloc or realloc), it will receive an error! If it is properly coded, the application will detect it, but handling such an error and terminating the program gracefully could be a challenge still! However, many developer don't test for successful memory allocation and often the application crashes as soon as this non allocated memory is going to be used. So there is a strong chance that the application will crash, and actually that many application will crash because you have more than one application usually running at the same time. Hopefully, the kernel itself should be immune, but in my experience it was not always the case.
If you have swap, you will have the same behaviour, but you have a higher limit so chances are that you won't reach that limit.
I have personnaly 2 machines with 8 and 16 GB of RAM. My personal usage of these machine should never bring me close to use all of this RAM (today! perhaps in a few years that could be different). The only reason I would hit the physical RAM limits is that a process got rogue and is consuming all the RAM. Having no swap is a relief, the kernel will promptly kill the rogue process as soon as it hit the RAM limit. If you have swap, you will get a painful experience (maybe not on SSD...) as your system will swap and swap and swap and become sometimes totally unresponsive until hopefully your kernel kills the culprit!
Advises
You should not be scared by people telling you something horribly wrong might happen if you remove swap. If you do so, your physical address space will be reduced, much like if you would go from 4 GB of RAM to 2 GB. Would you do such a thing? Then you should not remove the swap. You can however decide to limit it, and that would be wise.
If one thinks his workload might bring him to use close or more than the available physical RAM, or if one needs a lot of I/O (read/write to disks), then swap should be kept.
When swap should be kept, a user should try to set limits to the amount of memory a process can use. On a Linux system, one could use cgroups memory limits to avoid that some process gets out of control, I don't know the equivalent on OS X but that should be the way.
However, if one has more than enough available memory for his workload, then he could safely disable swap or he could keep it. It won't change much even in terms of performance because if the system does not use the swap, then you don't get hit by a performance drop (especially visible on an old rotational hard disk, less so on SSD).
Using the same trick as above to try to set memory limits per process or group of processes is also beneficiary in the case of more than enough available RAM.
My own example
I have an old PC with 1 GB RAM, there I do have a big swap: 4 GB so I can run Firefox and other application on it. I tend to use as little application (or tabs in Firefox) as possible to have a correct user experience though.
I have a small server with 8 GB RAM. There I know exactly my work load and I do not want my server to slow down because a process got rogue and is consuming all memory. There I have no swap and cgroup memory limit enforcing barrier, so applications or services can crash but not the kernel. Btw, this is a Linux machine.
Then, I have my Mac with 16 GB RAM. I am not the sole user of it, and we would not mind waiting when switching user. There we do have swap, it is difficult for me to plan the work load of other users of this computer. Even if I would be the only one I would use swap. I often run several VMs and use memory consuming applications, but not all at he same time, so I would not mind the OS swaping in an unused app so I can get faster disk I/O or launch yet another VM. Althoug up to now, I haven't seen this machine swapping!