Soap dish removal

As I mentioned in my other response, if it is installed by smearing mortar on the back, creating a keyed system, you will need to remove the surrounding tile to get to it.

I guess another option might be to use a Dremel or oscillating saw to cut it up and take it out in pieces. That might be quite the adventure.

But what are you going to fill the hole with? Do you know if you can find something that is the exact same size? By removing the surrounding cut tiles, you can fill in the entire section with whole tiles.

It's already usable, it's just nasty looking. A replacement plastic piece isn't going to remove the rust.