Just plugged it in for another quick check and this indeed appears to be it.
The noise was there from start up, checked the button, headphone setting was engaged. Button out, noise gone!
Thanks for all the suggestions, and I'm very aware that the seller wasn't obliged to do anything, I haven't hassled him, just made him aware.
Now, off to actually enjoy the new amp!