I dont think thats right on the dots point. Dots inbetween fretlines indicate where to put your finger for the string to then hit the fret behind the finger (closest to the bridge) to pitch the note. With no fret (but a lined fretless board) your finger should be on the fretline closest the bridge end of the fret to accurately pitch the note.
As far as the OP goes, its your bass so its really down to you. Is it one of those passing whims we all get? are you likely to sell the bass at some point? I have had irreversible mods done to a fairly pricey bass, I dont regret it as I dont beleive I'll ever sell it. If you love everything about it, apart from the fact it has frets, take it to a luthier who is experienced and good at de frets and get it de fretted, or buy a bass that comes close and de fret that.
Jonny