You need to enable automatic downloads.
In iTunes: iTunes > Preferences > click on the Store tab: there you can enable automatic downloads by separately checking the boxes next to 'Music', 'Apps' and 'Books'.
On your iOS device: Settings > Store: there you'll find switches to activate automatic downloads for 'Music', 'Apps' and 'Books'.
Note: If you only enable automatic downloads in iTunes music, apps and books downloaded on your iOS device will also download in iTunes on your computer. If you only enable automatic downloads on your iOS device the reverse is true. If you want it to work in both directions you need to enable it both in iTunes and on your iOS device.