I agree. Much better to leave it.
If you remove one of the blocks there is little chance of the supplier sending you a good match for the ones that are left anyway. You might as well remove all of them and replace with a properly matched set.
Ibanez put oval abalone dots on my fretboard and one of them is similarly at odds with the rest. They are a lot smaller than yours but I still had to get used to it. I now think of it as a unique characteristic and - isn't that why we use organic materials anyway? To have something that is taken from nature because it can't be uniform and manufactured?
Like your infill, it is likely that you are the only one who will be aware of the difference once the instrument is strung. The instrument as a whole isn't going to suffer, IMHO.