Trim (Exterior) Question

Unfortunately, as I prepare more and more of my home's exterior surface for painting, I think I am finding a 'home construction deficiency' of a kind. Some of the trim seems to be made not of wood but something like a thick cardboard-y wall board. (Insulation board? I think it's that black, like 3/8" thick board that is ordinarily just under the wood or vinyl siding.) The paint has adhered to it well in some places, to the point you couldn't tell it was junk. In other places, the paint is peeling (though it sands off nicely, along with the underlying material!). When I hose certain parts of the trim down, they appear to absorb water pretty well. (Fortunately I live in a very dry climate.) But jeeze this stuff is great food for termites, carpenter ants, etc. Or better food than wood, right? I certainly can't protect it with a coat of paint the way I could protect wood.
The builder most likely took a short cut (some ten years ago), right? I've been in the house two years. Had the home inspection and everything. I never thought to check something like this. OTOH, I'm in a cookie-cutter neighborhood. All the houses were built about the same time, with very similar styles. Everyone probably has the same junk. No one's house looks awful. But I am not confident about how well this stuff will hold up.
Should I tear this stuff off (and it might be every piece of trim, vertical and horizontal), go buy some board at Home Depot etc., then paint?
I'm somewhat handy with a saw (I have a circular saw). I can hammer nails competently...
Pardon my naivete on this subject. I'm googling simultaneous to making the inquiry here.
Painting novice update: Per my thread of a few weeks ago, I have been preparing all of the exterior of my house for painting and acually painted a good deal of the back porch. Looks great! Love that Sherwin-Williams "Duration." SW's clerks were great--not pushy and happy to make suggestions when asked.
The siding is very rough-hewn, by design, so I'm having more luck applying it with a "coarse" roller.
Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload

Correction: There is a definite, pronouned grain pattern in these trim boards. Knot patterns are on them. Yet it still looks like highly compressed cardboard on the sides where the paint has come off. Plus, like 99% of the wood siding is in good shape (apart from needing a nice paint job), but only about half the trim is in good shape. The old paint peels more from the trim than it does the siding.
Maybe because the edges of the trim are exposed to water, so it soaks the water up more readily?
All experience welcome.
Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload

HomeOwnersHub.com is a website for homeowners and building and maintenance pros. It is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.