surely there's a bigger risk of distortion from a smaller amp. Just as in HiFi.
If your cab can handle 800watts and you run a 300 watt amp into it at full tilt, there's a risk of the power amp distorting, producing non sine waves and wrecking your speakers. Maybe that's what your hearing.
Your safest bet is to use an amp that is too big for the job, not too small.