Is it cheaper to leave my mini split running low all day or turn it off when I leave?

Your coworker is basially right, and it's not an old furnace myth. It actually applies more to mini splits than to furnaces. These units are most efficient cruising along at low output. When you shut it off all day, the walls, floors, and furniture cool down along with the air, and then the unit has to run flat out to bring all of that mass back up to temp. That recovery push is where the efficiency goes out the window, becasue the compressor is maxed out instead of humming along in its sweet spot. I ran into the same thing with my setup. I used to kill it every morning out of the same habit, and my evenings were spent waiting for the room to catch up. Switched to leaving it on a low setting during the day and the difference was noticable right away. House was comfortable when I walked in, and the unit barely ran while I was gone once the room settled.

One thing to know: don't leave it at your full comfort temp either. A small setback, like two or three degrees below where you'd normally keep it, gets you most of the savings without creating a long recovery. If yours has a timer, that's your friend. Set it to kick back up an hour or so before you get home at 5:30 and you walk into a warm room without paying to heat an empty house all afternoon. You're right that comparing whole bills week to week won't tell you much, weather swings too much. If you want real data, check if your electric company offers hourly usage tracking thru thier website, or just compare the runtime behavior on similar cold days before and after the change.

Also worth checking that the unit isn't low on refrigerant or has a clogged filter, since a dirty filter makes the recovery crawl even worse. But the on-off habit alone would explain everything you're describing.

I service these things for a living and I've metered it on my own unit out of curiosity. Cruising along at low output it pulls around 400 watts. Same unit doing a recovery push from a cold house was drawing over 1200 watts and still took an hour to settle. Add in the defrost cycles it kicks into when it's running hard on a cold evening, and you're losing even more time to those pauses. One thing I'd add for your situation, if your outdoor unit sits anywhere that snow drifts up against it, keep that cleared. A blocked coil chokes the whole thing when it's working hardest.

This is making me rethink my own habits. I've been doing the same shut it off thing with the wall unit in my living room, mostly because I figured a machine resting all day uses less than one running. Sounds like that logic doesn't really work here. I tried leaving mine on the low setting yesterday and honestly forgot it was even on, it was that quiet. One thing I keep wondering though, when it's just holding the temp all day does the outdoor box still hum away the whole time, or does it only kick in when the room actually drops? I couldn't tell from listening to mine.

So I tried the low setting all day after reading this. What got me was the watt numbers, running gentle instead of hammering to catch up. It worked, house was comfortable by supper. One thing though, the tech mentioned defrost cycles and now I'm wondering what temprature to leave it at. Is there a sweet spot, or does any low setting beat shutting it off?