Horse Chestnut tree: Tree looks good, but no chestnuts?

Our horse chestnut tree (which pretty much every year looks healthy) never really puts out any chestnuts. Some years the tree is covered in flowers, and other years not so much.
Last year there were lots of flowers, and lots of small chestnuts forming, but most never got larger than a pea or marble-size before they disappeared.
Is something missing from the soil? Fertilizer or something else?
Or are these trees know for being variable as far as chestnut production goes?
Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
Add image file
Upload is a website for homeowners and building and maintenance pros. It is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.