I do exactly same as you described. Use plain servlets with embedded tomcat in application. It all works very well for me.
But now I have to support multiple Spring boot projects. I can't help noticing one thing common in these projects that it is about 10% functionality and 90% of Spring turd nuggets strewn all over project repos.
It isn't particularly balanced to assume that the spring way is the only way to do things because magicians...
If the OP has that problem with spring then he can use the base servlet API or another solution.