It's the whole "I claim right to" thing, even if it's all pretend nonsense, it feels oddly like being talked over.
I know neither of you meant anything by it, and I don't want to make either of you upset, but it was leaving me feeling uncomfortable and unsure how to even respond.
I want to emphasize that I'm not mad at either of you, and I didn't expect you to know how it would make me feel. Just wanted to stop it, politely, since it was making me uncomfortable.