How can I use my external hard drive on Mac OS so I can modify/edit files on both OS's: Mac and Windows? How should I format it?
FAT32 (called MS-DOS (FAT) by Disk Utility; a filesystem originally released in 1977 and updated a few times since, lastly in 1996) really is the only cross platform filesystem that is going to work fully out of the box with Windows and Mac OS X.
Be careful though, if you are using Disk Utility to format the drive, you should make sure to choose the Master Boot Record partitioning scheme (hit the "Options..." button below the "Partition Layout" control on the Partition pane). The default GUID partitioning scheme won't be recognised by 32-bit Windows XP and earlier Windows operating systems and Mac OS X versions earlier than 10.4.
Mac OS X has had support for reading NTFS formatted disk for a few versions, but still doesn't have write support. There are a few third-party products that allow Mac OS X to read NTFS formatted drives but as far as I'm aware the free ones aren't as well maintained as the commercial ones. I'd love for someone to tell me differently. For a while I've been using http://code.google.com/p/macfuse/ but as far as I can tell it hasn't been updated since December 2008.
Tuxera (who develop one of the commercial NTFS drivers for Mac OS X) have a list of free NTFS drivers that are developed from the same NTFS-3G source used by Linux to read NTFS drives. http://www.tuxera.com/community/ntfs-3g-download/
Using exFAT would be a good idea if the Windows computer runs Vista or Windows 7. This is a “simple“ filesystem yet it supports > 4 Gb files and multi-terabyte partitions.
For compatibility with 32 bits filesystems you still have to use MBR, not GPT.
FAT32 (called MS-DOS (FAT) in Disk Utility) is a cross compatible file format although you will be limited to 4GB maximum per single file. Plugins for the mac can also allow it to handle using NTFS volumes, which is a more desirable solution
If you are dealing exclusively with Windows XP and later (XP and 2003 after an update) and OS X 10.6.5 both systems will be able to read and write exFAT file systems. This will require no additional software to work and will deal with large file and storage sizes much better than FAT32.
My answer from a similar question: Best File System for Sharing Between OS X and Windows
If you're working exclusively with 10.6.6 or greater on the Mac side, try exFAT. Native read/write support under Windows and OS X, and none of the file size limits of FAT32. Disk Utility will happily format your drives using it.
It's probably your best option, as it avoids any user-space filesystem drivers, which personally make me a bit uneasy.
XP and Vista support exFAT with appropriate updates: Vista as of SP1, and XP with SP2 and the KB955704 update
Also a good point from the above posters re: MBR vs. GPT on 32bit systems.
An MBR partition scheme with a FAT32 (called MS-DOS (FAT) in Disk Utility) volume will allow both OS X (10.4 and up) and Windows (XP and up) to read and write with no additional drivers.
NTFS is a better filesystem than fat32 and is well supported by many OSes. OSX has several approach accessing NTFS read-write.
The open-source solution is to install
ntfs-3g with macports, and modify your system's auto-mount script.
the disk can be formatted with windows, or with ntfsprogs on a mac.
(filesystem operations always envolve risk, and very likely lots of command-line work.)
NTFS is the native windows filesystem. It's open-source drivers work quite stably and reliably. NTFS will work like a charm if you'll ever need linux support.
if you don't feel comfortable altering the system yourself, paid softwares and services can always be found.
i can post my ntfs auto-mount script for mac if you can't find one with google.