How long does it take to detect pitch?
At 120 beats per minute a quarter note lasts 500ms, a sixteenth note 125ms or just five wave-cycles.
The minimum must be two zero crossings - half a cycle. For E1 that's ~1/80 of a second, or 12.5ms. In practice two in the same direction is more reliable (most real world waveforms are somewhat asymmetric), so 25ms. If you guess after half a cycle then refine each half cycle after that the minimum delay is going to be 12.5 + a bit milliseconds.
For a low B you are looking at 16.7 and 33 milliseconds. Apparently the human ear can detect a delay of 40ms* so to avoid latency on low notes it looks like you ideally have to get it right in one wave-cycle.
*I did read along time ago that we are very sensitive to sounds arriving early (e.g. the bang before the flash jars) but accept slight delays (because we regularly experience these in real life whenever we see the distant source of a sound).