It's a tricky one. I find myself in a similar situation (though I haven't left a band)... Jammed with a couple of nice guys - it's mostly layered, atmospheric post-rock stuff which I don't mind, but I feel a bit limited as a bass player.
I'm tempted, like you, to duck out, but the other half of me thinks that I could get something out of this. If they're cool and you enjoy their company, it might be worth giving it a go. If it doesn't work, cut your losses.