It appears that this is happening on the holes that go against the grain.
This would indicate either a loose fit of the bit to the guide hole, a dull
bit, or the wrong type of bit. Every tool has limitations. It seems that the
holes that are drilled with the grain are very clean. ????
Just my two cents.