actually you do need another participant to take the other side of your trade. That’s not a middle man. Market makers, liquidity providers, are a key component of these markets, taking risk with their own capital (no systemic risk, no too big to fail, going to get a government is things go badly), requiring sophistication and specialization to survive in an ultra competitive environment with a high degree of uncertainty and risk exposure.
We don't need an HFT middleman.