I have downloaded a flv file, and I would like to save it in iTunes library for later play. I guess I should convert it first, but is this supported by OSX or do I need an extra application. And which file format will keep the quality and the size of original video?
You can convert it to the mp4 and insert it. I use handbrake to convert or fix indexes.
Also you can give a try to Perian if it'll work in your OS.
SmartConverter is a free app available on the AppStore and that does the job. It has the added benefit over Handbrake that for
.flv files it will just change the container and not actually re-encode the video. Which is quicker and doesn't make you loose quality.
The downside of SmartConverter is the ads displayed and that you cannot do batches of conversions. For that you need to buy the pro version.