I sometimes use super glue and baking soda, you should be able to roughly see the original string depth in good light, as the original nut will be a slightly different colour to the glue and baking soda.
To file down the glue and baking soda I use an old set of strings at the same gauge your new ones are, or even use the new ones, I then do each string in turn, take about 4 inches of the string and pull it taught, then use it like a saw and saw back and forth, try keeping the string as flat as possible, apply some downward pressure and it will easily carve out new string grooves.