I listened to the fan some more last night and while it's definitely a lot louder than it used to be with the flexible duct vented out the soffit, I may be misleading when I describe it as "very loud". It's all relative. The noise sounds kind of like air flow in the duct. It's a kind of echo like sound, a sort of howling sound. The damper flap is not stuck.
Regarding an exhaust fan inline, I had such a fan in my kitchen. It was extremely loud. The new panasonic fan is rated 2 sones and was VASTLY quieter than the inline fan before I connected the ducting to the roof, and still definitely much quieter with the ducting hooked up. The ducting did make it a lot louder. If the ducting is a source of noise I don't understand how moving the fan to a different site will help as you still need ducting. Both the fan in my kitchen (which is not the topic of this post) and the fan in my bathroom (which is the topic of this post) are very quiet fans when there is no ductwork. Both of them got quite a bit louder when I attached rigid ductwork.
But the bath fan (which is rated at 0.8 sones) was very quiet when it was vented via flexible ducting out the soffit and is significantly louder now. The fan was the same. Getting an inline fan is clearly not necessary to make it quiet because the fan I have used to be quiet with a different duct configuration. Could the problem be the roof cap. There was an intermediate configuration where the fan was connected to the roof cap with a serpentine run of flexible duct that made a 90 degree turn right at the fan exhaust port (bad!). This was also louder than the old setup that ran to the soffit, but not as loud as the rigid duct.
I did some testing to assess whether the fan noise due to the rigid ducting had to do with the mounting or support of the rigid duct. I had my wife listen to the sound of the fan while I held parts of the ductwork to damp any vibrations. Nothing I did to the ductwork made any difference. So attaching a rubber sheet to the ducts will presumably have no effect. (It's got less mass than I do.) It almost seems like the only thing that could have an effect would be something that alters the air flow pattern.