Ok, so I've begun noticing this a few times myself today, so I've been browsing with a profiler running.
The latency I'm seeing is purely the ads. As they're inline JavaScript, the page is blocked until they finish loading. If the ad servers are slow or don't respond at all, the page stops and waits.
One way the site mods could combat this would be to place the ad JavaScript at the end of the page, and then use the site's own JavaScript to shift them back up to the top once everything has loaded. That way, if there's a delay in loading the ads, page rendering is halted way down the bottom where no one will notice or care.
That's all presuming the problems I'm seeing are the same as the ones you are.