I would agree with you. I never want my manager there. As an EM, I never want to be there. And yet, as the years go by, when I have an EM role I find my presence is requested. Again, who knew?
I don't know, at least for my current team a lot of the stuff we complain about in the retro is beyond the team level: coordination between teams, infrastructure issues, ... The intra team issues are usually talked about before (e.g. during standup, or just asynchronously).
If the EM is not there, those are not actionable. What we try to do is: do the retro by ourselves, and invite the EM at the end of there is stuff for them. Usually they are not available though...
