...and me.
I listen to different recordings of the same material by the same bands and I have favourites. Sometimes I dislike a particular version. Other times a version stands head and shoulders above the others for no reason other than the convergence of all of the right elements in the same space and time.
Like many others, I can not define what it is that grabs me about the performances I like most. Sometimes it's the ambience of a live recording combined with the raw adrenaline coming through from the musicians involved. Other times it is the clever overdubbing and layering of pre-recorded pieces to make something extraordinary.
The loss of stems probably matters more with recordings that are heavily worked in the studio. In those cases, the magic does not come from spontaneity as much as from the crafting of a piece off-line, as it were. In those cases, I agree with you.