It Has to do with the Clipboard; not Screen Capture
Screen capture can obviously support more than these two file formats; obviously it supports GIF and JPEG/JPG, but it also supports PDF. However, the issue here isn't what Screen Capture is capable of handling, it what your clipboard is capable of. Why? Well, you're going directly from your clipboard to pasting an inline image into a mail message. Once Screen Capture "hands over" bitmap to the clipboard, it's work is done.
Now, your clipboard stores images as a RAW format so it can preserve every single detail. How it gets saved (when Screen Capture writes it to disk), the clipboard simply doesn't care if you use a high quality format like PNG or TIFF or a lossy format like JPEG or PDF; basically, you chose it, you'll get it.
But why PNG or TIFF?
The very short answer is both are image formats that support lossless compression. In other words, they can make the images much smaller than their RAW format without losing an ounce of quality.
Apple chose PNG as it's default and I can only speculate as to why...
- PNG (Portable Network Graphic) was to be a replacement for the GIF format
- It's web friendly
- It was designed for transferring images over the network/Internet
TIFF on the other hand...
- Acquired by Adobe in 1996 for desktop publishing
- It was an attempt to standardize scanned (and faxed) documents into a single high quality file format
- Built in support for meta data
- Could hold multiple pages as a single document like a PDF
Why you get TIFF when pasting to your App
Coming out of the clipboard, you've got two ways to go - PNG or TIFF. Since you specifically told screen capture you didn't want PNG, it went to the only other choice available - TIFF.
How can I save it as JPG format?
Don't use your clipboard. Save to your Desktop (or other location) and just manually convert to JPEG. Then drag your item over to the app you're using. If you do this through the Preview App, you will have the opportunity to set the quality/size of your JPEG in case it's still too large for some receiving email systems. (Preview) File → Export → Format
If this is something you do quite a bit, you can easily create an Automator action...but that's another question for another time.