iTunes 10.6.3 shows the AirPlay icon when playing music purchased from the iTunes Store, but not when playing TV Shows purchased from the iTunes Store.
MacBook running iTunes 10.6.3 Airport Express on the same network Speakers plugged in to Airport Express
Steps to reproduce:
In iTunes, select a song from the Music section. At the bottom of the iTunes window, click the AirPlay icon. Pick the Airport Express. Click "Play". Sound comes out of the speakers. Pause the music. Select a TV show from the TV Shows section. Click "Play". Sound comes out of the MacBook.
Extra puzzle: TV shows played with the iPad Videos application work with AirPlay, so one workaround is to buy a used iPad, sync the TV show to the iPad, then watch the TV show on the iPad instead of the MacBook.