The main concern with using the ast module from stdlib is that you lose all formatting/whitespace/comments in your syntax tree, so writing any changes back to the original sources requires doing a lot more extra work to preserve the original formatting, put back comments, etc. This is entire point of lib2to3/fissix and LibCST, allowing large scale CST manipulation while preserving all of those comments and formatting. We do recognize the limitations of lib2to3/fissix, though, so there have been some backburner plans to move Bowler onto LibCST, as well as building a PEG-based parser for LibCST specifically to enable support for 3.10 and future syntax/grammar changes. But of course, this is very difficult to give any ETA or target for release.
Indeed! I also would suggest people to use a CST implementation (parso / LibCST) instead of refactor if they intend do large scale refactors, but from what I can see in my previous attempts (e.g teyit, a unittest assertion formatter) when you deal with small code fragments (a single expression, or a small statement) then you generally don't need to worry much about the style. The only concern is the literals (especially strings, which there are a few different variations of the same AST) where you could resurrect them back from the token stream (which the CustomUnparser representative in refactor allows).
The real start point for this project was to find / replace all type(<literal>)'s in CPython codebase with type(type(<literal>)) (e.g type('') would become type(str)) which is very light weight transformation, and I was able to write a script which did it without having any major problems about style on over 2000 files. Here it is for the reference: https://github.com/isidentical/refactor/blob/master/examples...
Also one thing to note here is that; in the last couple of years, thanks to black (and yapf), the adoptance of code formatters have really increased which is very nice for custom refactoring tools like refactor since the end-code would be refactored anyways so that means if you convert a multi line call, or a list to a single line version then the formatter you use probably reformat that segment anyways.
But thanks for authoring Bowler! It is a very cool project.
Did you consider PyCQA/RedBaron (which is based upon PyCQA/baron, an AST implementation which preserves comments and whitespace)?
https://redbaron.readthedocs.io/en/latest/
It was considered, but the initial goals of Bowler was to a) build on top of lib2to3 in an attempt to get broader upstream support for maintaining it as an "official" cst module, which fizzled out, and b) to use some of the more complex matching semantics that lib2to3 enabled, which baron and other cst alternatives don't really attempt to cover.
Things like "find any function that uses kwargs" or "find any class that defines a method named `bar`" can be easily expressed with lib2to3's matching grammar, and no other CST that I'm aware of (that isn't itself based on lib2to3) has equivalent functionality. This is something we wanted to add to LibCST, but haven't had the time to focus on given other priorities. Meanwhile, we used LibCST to write a safer alternative to isort: https://usort.readthedocs.io
The main concern with using the ast module from stdlib is that you lose all formatting/whitespace/comments in your syntax tree, so writing any changes back to the original sources requires doing a lot more extra work to preserve the original formatting, put back comments, etc. This is entire point of lib2to3/fissix and LibCST, allowing large scale CST manipulation while preserving all of those comments and formatting. We do recognize the limitations of lib2to3/fissix, though, so there have been some backburner plans to move Bowler onto LibCST, as well as building a PEG-based parser for LibCST specifically to enable support for 3.10 and future syntax/grammar changes. But of course, this is very difficult to give any ETA or target for release.