This small tree grows along roadsides close to the bank of the Colo river west of Sydney.
formatting link
has fruits containing nuts.
The internal structure of the fruit looks something like a mangosteen, but the thin pulp has an astringent flavour.
Inside the the thin shell is a pure white nut. It tastes pleasantly mealy, but not delicious. I spat it out in case it is poisonous.
I don't think it is a native Australian species and there are a lot of garden escapes in this area.
What species is it?