How do I finish the edge where my new peel and stick tile hits the rug?

The best way to handle that diagonal transition is using a **Z-bar transition strip**. Since you're dealing with a height difference between the peel-and-stick vinyl and the existing carpet, you need something that anchors the carpet edge properly so it doesn't fray or pull up. You'll want to install a tack strip about a quarter-inch away from where your tile edge ends, then tuck the carpet over it and into the Z-bar. This gives you a clean, tucked edge that looks professional even on a sharp angle.

Don't worry too much about the diagonal cut making things difficult for the metal strip itself. Most aluminum transition strips are easy enough to cut with a simple hacksaw. The trick is to measure the total length of that angled line and buy a strip long enough to cover it in one single piece. If you try to piece togather two shorter strips, you'll end up with a visible seam that will catch dirt and eventually snag someone's socks.

Since you are using peel-and-stick tiles, make sure your subfloor is incredibly clean and level after you pull up the carpet and pad. Any leftover adhesive or staples will telegraph through those thin tiles almost immediately. I usually recommend a coat of primer specifically made for peel-and-stick flooring to ensure it stays down, expecially in a high-traffic area like a walkway to the kitchen.

For the actual cut on the carpet, use a fresh blade in your utility knife and a long straight edge. Take your time with the angle. Once the tile is down and the Z-bar is screwed into the subfloor, use a rubber mallet to gently crimp the metal edge down over the carpet. This locks everything in place and prevents that 'fuzzy' look you're worried about. It'll handle the heavy foot traffic much better than a standard flat transition strip that just sits on top.

You should prioritize checking the vertical clearance of the primary entryway door because, while the reduction in floor height from removing the plush pile and padding will technically increase the gap, the transition to a rigid vinyl surface often necessitates a reassessment of any existing door sweeps to ensure a proper seal against seasonal drafts. Furthermore, if your diagonal cut terminates near the door frame, you must be exceedingly precise with a jamb saw to ensure the new vinyl tiles slide beneath the casing rather than merely abutting it, which would otherwise result in an unsightly gap that collects debris and eventually compromises the adhesive's integrity at the threshold.

Spot on with the Z-bar suggestion. It realy is the only way to keep that diagonal from looking like a DIY disaster. One thing to keep in mind that's easy to overlook is the specific type of tack strip you use. Since you're transitioning to thin vinyl, make sure you get 'architectural' tack strips with the shorter pins. If you use the standard ones meant for thick subfloors, those little metal points might actualy poke thru the top of the carpet once you mallet the Z-bar down, wich is a literal pain if you're walking around in thin socks or barefoot.

I'm still a bit confused about the 'architectural' tack strip thing. I don't realy know much about flooring tools, so if I go to the store, will they just know what that is? Also, if I use a rubber mallet to crimp the metal down like you said, does that mean I can't ever move it if I mess up the angle? I'm terrified of making a crooked cut on the carpet and being stuck with it forever.

Don't sweat the cut too much. If you're worried about messin' up the angle, just leave the carpet an inch or two long at first. You can always trim more off, but you can't put it back once it's gone. Also, check your subfloor for any old staples after the pad's out. If you miss even one, your new tiles'll have a wierd bump that'll drive you nuts every time you walk over it. Just use some pliers and get 'em all out first.

Quick update—I went to the store to look for those architectural tack strips and the Z-bar. I checked my door clearance like suggested, and since the vinyl is so thin, the gap is actually huge now! My biggest worry is still that diagonal line. Since it’s not a 90-degree angle, how do I actually cut the metal Z-bar so the ends look finished where they meet the wall?