I have gone in completely the other direction (as a result of my Dad, who was a pro Jazz man before the war). As soon as I started playing bass 'seriously' he took me along to a jazz big band where I was sight reading all the usual big band things (I had learned piano for years and so the change to bass wasn't hard as far as sight reading went). I was also in a punk/R'n'B band at I got asked to join a modern jazz quarter so I was doing punk gigs, modern jazz (piano, bass, drums and vibes) and doing the big band too.
I have to say, playing in a big band is something else when you are literally immersed in the sheer power of a horn section. It's quite an experience.
I think it's the challenge but also you become less evangelical as you get older and are less inclined to maintain the 'tribalness' of youth. There are some simply stunning tunes in Jazz and I defy anyone to listen to Ben Webster play a ballad on the sax and not be moved.