Newer iOS devices have more system memory, which is separate from the flash memory. The OS itself has a specific memory footprint at boot time, and the rest of the system memory is used by applications. I don't have any data on this, but I can imagine that each major revision of iOS (1, 2, 3, 4, 5) has a larger system memory footprint than the previous release.
Keep in mind that the iPod Touch has a lower overall price than the iPhone released within the same year. An iPhone may cost $200 plus a two-year contract, but this is due to a subsidy paid by the wireless provider to Apple. So, the iPod Touch is expected to have a bit lower quality in terms of hardware specs (for example: system memory size, camera quality, CPU speed).
Taking the iPhone 4 and iPod Touch 4th generation as examples, Wikipedia states that the iPhone 4 has 512MB of system memory, and the iPod Touch 4th gen has 256MB of system memory. When the system runs out of memory for applications, it must make more memory available by terminating background processes or reducing their memory footprint. This may take time. You're more likely to experience this overhead on a device with less system memory, and on a likely slower CPU.