Max resolution is primary limited by the actual chip-set type/model for graphics.
In my case it is a
If you want to read more: http://www.intel.com/support/graphics/intelhdgraphics4000_2500/sb/CS-033714.htm
Just for fun, this is how the inner life looks like.
So, it is hardware limited, not OS x.
As for your second question,why it looks smaller in higher resolution?
Lets try the simple answer. When using reading glasses out of focus everything looks big (but slightly fuzzy). Now use reading glasses in full focus (high resolution) you will see objects very sharp not washed out and they appear to be smaller, but that is they actual size.