I've noticed a strange pattern of delays when trying to fetch websites; the behavior seems extremely distinctive, and is easy for me to reproduce so I'm fairly confident of most of the details below. I'm hoping someone will recognize it, or be able to work it out.
The length of delay seems to increase with the amount of time since restarting my computer, as far as I can tell without bound. This is less carefully tested than the other points, but it I am confident that restarting fixes the problem and that it rarely appears within the first few hours after restarting (and seems manageable for a lot longer than that).
The delay appears in any of Firefox, Safari, or Chrome.
curl works fine normally. But while any of Chrome, Safari, or Firefox is loading a page, curl will experience large delays. Closing the browser will immediately cause all of the running curl jobs to finish successfully (for example, I can start up 5 instances of curl fetching different websites, all of which will halt at the progress bar showing no progress; closing Chrome will cause all of the jobs to immediately finish).
The delay for curl is substantially smaller than the delay for websites, even when it exists. Right now it is on the order of 15 seconds, because I haven't restarted in a while in the hopes of debugging it. A website will take much longer than this, perhaps 15 seconds while "waiting for www.---.com" followed by comparable lengths of time waiting for others.
Latencies (as reported by ping) don't increase during problems.
While curl is experience delays, different curl jobs will finish at the same time. That is, they will all hang for a long time, then all stop hanging at the same moment (and so finish at roughly the same time, on the scale of the 15+ seconds for which they are hanging). Sometimes this happens in browsers, but not reliably (which I think might be because a browser has several opportunities to hang before actually finishing loading the page). After the curl jobs finish, immediately restarting them results in them hanging in exactly the same way, so it's not that the problem has cleared up (at least not for very long).
The delay is quite varied across web pages. I haven't done a very detailed survey (that will be the next step I guess).
when it hangs, curl -v hangs at the progress bar with no data transmitted.
While loading pages in Chrome, most of the time is spent "waiting for" CDNs and third party services.
Often Chrome will be displaying a blank screen, and pressing "Stop" will cause the page to render (with most but not all of the assets loaded). I assume this is related to the last point, and is just a behavior of chrome that appears when we have this long delay, rather than being closely related to the problem.
The delay appears when connected to many different wireless networks with different performance characteristics (e.g. a 50mbps line). I haven't tried wired networks, though I presume it wouldn't matter.
Similarly a different DNS makes no difference (which isn't surprising given that ping is always fine).
I observe the same behavior under very different levels of stress on the system. For example, right now I have 3 / 8 GB of memory free.