Singularly they are relatively harmless to most of us, but in large quantities they can be dangerous.
I had the misfortune to move a golf buggy that had been parked on a nest, when I moved it they came out of the resulting hole and attacked me, assuming that I was attacking them. They went up inside my shirt and I ended up with about 12 stings on my back. I had some amazing dreams that night, cheapest trip in ages!! But not recommended :-)
I also had a nest under a shipping container, (my garden shed), initially they ignored my trips in and out of the shed and I ignored them, but one day they decided enough was enough and again I ended up with a few stings. That was the final straw as far as that nest was concerned, they didn't survive :-(
If they are well away from me they can carry on about their business, but I'm no longer happy to share my space with them.
I went into an attic and when I turned the light on found a nest built around it, they were a bit shocked to find their home illuminated from inside and buzzed their feelings very loudly. Have you ever seen someone exit through a loft hatch without using a ladder? Apparently it was quite impressive. They didn't live much longer after that either.