Some bugs only come along for a short time each season. I'd tr blasting the plant from below with the garden hose sprayer. If a couple of weeks g by and you're still having problems, then look for another solution.
if it looks like webbing u might have spider mites. take a peek on th stems themselves use a magnifying glass if u have to and look to see i u might have tiny little spiders crawling around on your rosemary. if u do then i would hit them with the water as was mentioned and wai to see what happens then if they come back just treat with soapy wate spray as u had originally thought. spider mites sure do not like soa and water lol. good luck. cyaaaaa, sockiescat:)
-- sockiescat