I used Handbrake 0.9.9 (64 bit) on OS X 10.8.4 with iTunes 11.0.4.  I encoded the same movie using the AppleTV 2 preset and also the iPhone preset.  It encoded successfully generating two .m4v files on my desktop.  I couldn't "Add To Library" nor drag and drop into the library.  They would play fine in QuickTime.  Many hours of surfing later, the answer was simple:  rename them to .mov files.  Voila.  In they went.