The quantity in a CFL is about 1/1000th of the amount the average person has in their body.
The phosphors are probably a more significant risk, but they aren't particularly toxic. Most pronounced effect is to slow down wound healing, so keep it out of any existing wounds, and avoid cutting yourself with the tube. Highest long term exposure to rare earths was by projectionists using carbon arc lamps, as the carbon contained rare earths which were vapourised, and then breathed in. That did cause significant occupational health issues, but from long term exposure in a form in which they could be absorbed.