This is quite possibly an insane idea. As in dangerous, and won't work either. But, I'll suggest it anyway and you can decide if it's as insane as I think it is.
Get a Lazy Susan bearing:
Carefully lower the quill until it just makes contact with the workpiece. Apply just enough friction to the workpiece to let it spin at 400 RPM. The holesaw is spinning at 620, the work is spinning at 400, the difference is
120 RPM!Probably not something you really want to try, but fun to think about, in a sick sort of way.
Personally, I'd go with the idea already suggested elsewhere -- rough-cut with a bandsaw, then clean up with a router and template bit. Or maybe just a sanding drum?