Is it possible to make screenshots in vector format (SVG or PDF) instead of only in raster formats on OS X?
Solution 1: Using a vector GUI widget API with PDF backend:
Actually, it is possible. Consider the fact that on screen objects are created by a widget API, and if this widget API is based on vector elements and have appropriate vector output backend, then it can output to vector formats like PDF.
Please look at the similar question, which also contains some working samples:
This needs Gtk+ widget library and Cairo as its GUI display and PDF generation backend, which both are available for Mac OS X. But as few applications in Mac OS X use Gtk+, this is not a general solution. I wanted to show the possibility of creating vector screenshots. If you are a developer, this may be useful for you.
Solution 2: Creating vector mockups:
For general applications, you can create mockups. Here you can find "Free Yosemite Vector GUI for Illustrator", which can be used to create mockups of your GUI. It includes all main UI Components:
If you're talking about webpages, you can use the Print to PDF feature of Safari and then open the PDF in a vector program.
This should make it easy to grab some SVG resources or to make a tutorial in a video editing program where you want to zoom in on some webpages without losing quality - for example.