I think this is the right idea. I don't think you would have played that way when you have the energy of live musicians and can feel the bass drum and snare as much as hear them.
If it was me, I'd tighten up by learning the groove with a metronome and then once I had that down learn the arrangement, using fills to articulate certain moments. IMO the fills you used didn't add to the song or performance but that's often a preference thing.