You can get smoke candles to test the integrity of your chimney - I've never used them but a proper fireplace shop should sell them and if you do the test one day when there's no fire in the chimney you may get a better indication of where the leak is ( if leak there be ). I take it the fire does not feature a damper to adjust the airflow as it goes up the chimney? I have such a thing as an integal part of my fire and if you close it too far too fast with a roaring fire some smoke spills out, though not always visibly. Another possibility is that your fireplace is badly designed, perhaps with an arched top that allows smoke out, again perhaps onlya little so it isn't seen easily, but if you get smoke upstairs then it's more likely to be a leak. At any rate, a properly designed fireplace should have a flue area/fireplace entry area ratio within certain limits ( dunno what they are ). Finally, a badly placed chimney will suffer downdraughts ( the best place for a chimney is on the top of the roof, not partway up.
Andy.