Well I guess after accidentally stumbling onto the fix I'll explain what I found.
iTunes is sometimes bad at managing feeds. One podcast for some reason it split up into 2 feeds even though they were really one, I deleted one and it got rid of both. My unplayed count went down, but there was still like 900+ unplayed count. Then I stumbled upon it. If you go to the "List" section in Podcasts (which I've never spent any time in because I assumed it was just the podcast list in different form) I found the reason.
When I subscribe to a podcast I typically tend to change the rules for a podcast so it only keeps the most recent or only unplayed episodes. If you ever want to go back to an old episode you can right click on a Podcast feed and tell iTunes to list all episodes. From there then you can select individual old episodes to download.
For some reason iTunes can treat this episode listing as suddenly having a whole bunch of new episodes. Even though you never downloaded any of them or had long since deleted them through iTunes rules, it treats them as unplayed episodes in the listing, though if you go back to the "Unplayed" section they never show up there. Though this also happened to be the case for at least 1 podcast I am certain I never asked it to retrieve old episodes for.
So if you ever get something like this check the listing, even though you really don't have "unplayed" episodes iTunes may show a lot of episodes of your podcasts there as being "unplayed" because they are in the feed. Select all and right-click and select "mark as played." Manually mark as unplayed any episodes you forgot to de-select and suddenly your iTunes unplayed podcast count is correct.
I've attached a screenshot showing iTunes considering a whole big listing of Hypercritical episodes as "unplayed" even though you can see they aren't downloaded to my Mac.