leaky garden window

The idea is logical. The volume of air within the "garden window" is exposed on five out of six surfaces to outside temp. Glass does not provide as much insulation as the usual wall. Air in the "garden window", cooler than the room air is more dense - heavier than room air. Dense air flows out along the bottom and into the room. TB

More glass area is like a bigger hole in your wall to convect in cold or heat when its hot out.

It would appear what you are feeling is normal. You have a lot of glass area. Even the best glass is far worse insulator than a standard insulated wall. You feel the flow of air because as the air in the window area is cooled (the air's heat is passed through the window to the outside) it becomes heavier and flows like water would down and out of the area and onto the floor. It draws more warm air from the top into the window. It just feels like it is outside air.

If you close the curtains on other windows in your home and after a few hours feel around the area under the curtains you will also feel cold air flowing.

There is not much you can really do about it.

Yes.

If you don't mind the appearance you can reduce the problem by applying a clear window film across the opening. It isolates the cold air in the window space.