If there is an article 4 restriction on the area, then they need planning permission to change the windows. So the first step is to make enquiries as to whether that has been applied for and granted.
As regards the derogatory remarks here, I am very much a "live and let live" person.
However, when you are talking about planning permission and conservation areas, if you allow things like that to take place, and fail to act, then it can devalue other properties in the vicinity.
Consequently, it is not correct to say that the issue does not affect neighbours.