More info to share besides @Fake Name's answer:
After exploring all the pdfs I have that suffer from stuttering in scrolling, they are all old PDFs (I guess, generated by latex in the process ps->pdf), which uses Postscript font types (e.g. Type3). And the fonts are shown blurry in Preview.app. (No matter how small the PDFs are, e.g. <10k. However, if you only have a few pages, say 3 pages, the scrolling stutters only the first time you reach that page or during a few seconds after opening the file.)
The Preview.app on Yosemite fails to deal with these fonts appropriately (rendering, smoothing). When opening them, any scrolling on the pages firstly visited will show stuttering, lags (not slightly, quite obvious and serious).
Alternative "Solution" (alleviating scrolling lags): in Preview.app, "Preferences" -> "PDF" tab -> uncheck "smooth text and line art". Then you will see significance improvement of scrolling performance. However, fonts are still shown blurry if not zooming in. One word, these fonts are no longer well supported after Yosemite. (Note that disable smoothing will give you really bad font rendering in other normal PDFs).
If I remember correctly, the old versions of OS X didn't have this problem; maybe certain functionalities of Preview.app have been removed, since new LaTeX typesetting systems no longer use these fonts any more (while you can still configure it and to use them). No idea why Apple meant to produce these troubles.