Yellowish in color???? If so then you will probably not like the prices of getting the stuff yourself. I have had foam sprayed into block walls before that worked well. Problem with foam is the "good stuff" is a two part mixture and requires equipment that is expensive for a single use.
I suggest you contact some local contractors and have them bid the job.
In lew of that try this site. Do the math carefully. The site is mostly board feet not cubic feet.
Is there any reason that you can not just stuff fiberglass into the areas and use some spray adhesive? Most of the heat goes up anyway.