Converting lawn mower to manual throttle?

I have a cheap MTD mower with a 5hp B&S engine. The "automatic" throttle lost a spring and the plastic piece that appears to control the engine speed (the vertical piece in this pic:

is broken so I converted it to a manual throttle cable. I connected the cable to the piece that is connected to the plastic piece in the above pic via a rod with a loop in it. (Hopefully the pic and description make sense). The thing is that if I'm sitting still the cable controls the RPMs as expected but as I move the mower around RPMs fluctuate even if I'm not cutting anything. I except the RPMs to drop when there's a load from cutting but it seems random and will surge and drop.

Any thoughts? Is there something else that controls engine speed that I haven't found on this engine?

thanks in advance!