But running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)
I believe. I run it on my mac M5 pro at like 30t/s with some RAGs and let it work on stuff overnight and it's great. It isn't the same as the big models where things can be more unbounded, but if local models keep progressing there is a universe where a 200-300B model is all most of us will need to stay out of the big tech moats.
