Tried and Failed to Remove Old Paint from Fireplace - now stuck with a mess

Hello- Our house has a brick fireplace painted white that we wanted to strip and leave it a natural brick. I used a citrus based stripper, and this got off a lot of paint, but not enough. The paint wouldn't all come off, especially in the grout, even with a lot of scrubbing. So I gave up. We have decided we will just paint over it with a color we like. The problem is there is still some residue from the citrus stripper. I have used mineral spirits to remove some of it, but its still kind of a mess. How can I either remove the rest of the stripper so I can repaint, or otherwise prep it to be painted so the stripper underneath doesn't compromise the paint on top? Any help is much appreciated. Thanks, Turaco