Same here. I'll use the B string for the occasional low D as a fill note when playing to a G chord, and for a beefier sound when I want it.
I had a fretless 5 for a while in the 90s, didn't particularly get on with it, and couldn't find a 5 with a neck that I liked anywhere near as much as the slim and shallow neck on my '87 Thumb until I happened to buy an Antoniotsai bass purely on a whim from Vietnam. It so happened that I also bought a Squier 5 at the same time, a Super Special or something, P body with J pickups. The Antoniotsai came and the Squier went straight away. That was 2007 and after a bigamous period of a year or so, I went fully 5.
Now I still have the Thumb 4 (and a fretless equivalent) and an acoustic 4, the other 20+ are 5s or 6s (plus a 7, 10, and 12). I take the 4s out now and again to open mics, which keeps me on my toes. I completely agree with @lozkerr that just because you can use 5-string fingering doesn't mean you must. In fact, playing Oasis's "Don't look back in anger", I find myself alternating between 4 and 5 fingering, partly in order to stay awake.