My son's 5 now; he started helping me when he was 4. He built all kinds of things out of the scrap pile with wood glue; he's big into robots, Bionicles, Transformers, etc. He glued and clamped them together and then he painted them with craft paints.
My daughter is 3, and she's doing small scraps with paint and glue, too. In our house boys and girls both get to do wood. The only time my kids aren't welcome in the shop is when the big power tools are running. They can use anything that isn't a power tool and isn't too sharp - e.g., clamps, handsaws, screwdrivers, etc are all okay.
One of the most fun projects I built with my son was a "money box." He got to pick all the materials (scrap maple, painted bright yellow of course) and he got to make all the choices for dividers and the like. He even painted the logo on the top. Now he stores his most top-secret kid stuff in it. The key is that they get to be in charge of how it is done, and then they will love it.
Cheers, Nate