iOS is a beautifully designed operating system. Part of what makes it so great is the troubleshooting protocol, which is basically: (1) power cycle the device or (2) restore from a backup. I have an iPad and iPhone 4S that have separate configurations but are linked to the same iTunes user ID. This means that while I effectively have all the same apps available to me, I don't have to have them installed on both devices. As an example, my wife and son went out of town last week. I removed all of my apps that had important data that syncs between everything (he's 14 months old, and has an uncanny knack for manipulating the iPad in only the most unfortunate of ways), added some new videos and infant-appropriate games, and so forth. Basically I configured it to be an entertainment device specific for him, for that trip. When they return I'll simply connect it to my Mac and restore it to the state it was in the moment before they left. Gorgeous, simple, and hassle-free.
The point is that Through iTunes, you have pretty granular control over what gets synced between your computer and the device. As far as backups go, see this article for detailed information about what does and doesn't get backed up to your computer. Of course, you can also backup information to iCloud. This is similar, but not identical to, the backup on your computer. This article lists what gets sent to iCloud. Having trouble deciding which service(s) to use? Apple even provides some pros and cons to consider in this article. As far as I know and can tell, stopping app syncing does not erase the apps, it merely stops backing them up beyond the current backup state. Deleting an app directly from your iPhone, however, "...will also delete all of it's data" (per the iOS warning message). That part is fairly transparent and self-explanatory.
If you don't want an application to be re-added to your iPhone, just use the "Apps" panel in iTunes and deselect the check boxes next to the apps that you've deleted from the device. You can also de/select the check box labelled "Automatically sync new apps." This will prevent them from showing up again. Rinse and repeat for the other panels (e.g., Music, Photos). If time is a factor when it comes to syncing podcasts (this is mostly what I sync, too), I would try to do it at a time of day when it's a non-issue, like when you're eating breakfast or engaged in some other routine. Let it sync when you won't be messing with it anyway, and it is immediately a non-issue.
Hope this helps with your question!