This is very true, I was not attempting to suggest you _could_ even use a conventional chuck, for just the reasons you cite.
However if you could and were to increase the effective mass of the bit by a factor of (at least) five, you will also increase energy losses at the point of transfer. You are going to get those losses at some point in the system, so better they are at the drill to wall interface.