I presume the thrust of the original question is not so much whether the inspector can arrest you for code violations but rather what the consequences are of the inspector noticing evidence of criminal activity (not just code violations).
If your computer conks out and you take it to Best Buy to have the Geek Squad fix it and the guy finds your cache of kiddie p*rn on the hard drive and calls the cops you're going down and no claim of warrantless search is going to help because you voluntarily took the machine in. But if the building inspector visits your place and he sees your grow room filled with pot plants is it different because he's an agent of government and can demand admission under some rule of building code law yet doesn't have a warrant to be conducting a criminal search? I imagine he's not obligated to ignore what he sees and he can give police info leading to a full fledged search. No violation of expectations of privacy.
Now then if in the meantime you destroy every molecule of contraband and the only evidence of the crime is his testimony (let's say he's a BI only part time and he's also a PhD in botany and can testify as an expert that what he saw was pot) then I don't know.