Apple works very hard to conserve battery life, and determining location, no matter if it is through GPS, cellular, or wifi connections takes extra battery. The arrow displays any time a 3rd party app requests your location. For instance, when you open Maps, the arrow will likely turn on because Maps needs to pinpoint where you are. The arrow may also turn on for system level locations services (settings can be adjusted in Settings > Location Services > System Services), and depending on your settings, the arrow may or may not appear for that.
As for Location Services being on or off, you can set that in the settings (Settings > Location Services), but if you see the arrow, it is likely on already.
The arrow only appears when location services are active, and may or may not show when it is just on.