I'm a drummer and have the same kind of conversations in my band regularly.
I found that the most positive messages from band members usually went along the lines of "Look, the timing of the band belongs to *you* as the drummer, YOU have your foot on the accelerator and brake and how fast we go is completely your domain. However, we can't all play as fast as you can so we need to be able to signal that we need it to go slower."
The next thing we did was have several practice sessions where we just concentrate on 'messaging' i.e. playing our usual tracks but getting different members of the band to signal a slowing down or speeding up and listening to the drummer drive the pace from that. It's like a game where the drummer is in control and different band members make 'speed up' or 'slow down' signals. We always over-emphasise the speed change so that all band members can clearly agree that it has sped up or slowed down. It's a lot of fun as a drummer because you actually feel like the driver in terms of speed.
If the drummer is a reasonable person they'll see how important it is to practice this messaging as much as any other aspect of practising. By making it a 'game' and reassuring them that they are always in controls, it works well.
You have to be honest with yourselves though that you are prepared to trust the drummer with the tempo setting (not paying lip service and ignoring them!) - respect must work both ways.
We drummers need nurturing and ego-stroking. Otherwise we throw our toys out of the pram and have tantrums, but then we often get the blame when actually *nobody* is working as a team member.
david