Pay the contractor nothing at all and do not sign the contract until you have every penny from all the special assessments. You should not need a signed contract to levy an assessment. If you do not have the money in reserves to cover all the work and any potential change orders, do not do anything.
No way. What are you getting for the deposit?
How much labor? Do not make any payment for material that has not been reviewed and approved by your inspection service (you do have one, right?) prior to its delivery. Do not accept delivery of material not approved. Ask for samples of all material and then rigorously enforce the quality standards.
Payment upon satisfactory completion including repair of all punch list items and sign off by the inspector is more appropriate.
You are right to be paranoid.