I've purchased some books that make epub and mobi versions available for download. What are the main differences and which is better supported in iBooks?
ePub is the newer standard and is supported by the Apple iBooks app, but Apple only supports it's own FairPlay DRM wrapper used to encrypt the ePub files to protect the books from being copied or read on unauthorised devices.
Most other (non-Apple) eBook stores sell ePub books encrypted with Adobe's Adept DRM wrapper, which you can't read with the iBooks app (although certain other eBook apps do support the Adobe DRM system)
Mobi is for other reader systems (I believe the Kindle uses this format) and is not supported by the iBooks app. I expect there are other iPhone apps that support mobi files.
The latest version of iBooks supports the PDF format. The support is excellent. You can drag ePub or PDF files to your iTunes library.
There are differences that are covered in great detail in this article:
I do not think that an end user should care about this until the seller offers the books in all formats, in your case in EPUB as that is the format supported by iBooks