Transformers lack recursion and are limited by the network's fixed depth, so "reasoning", IMHO, is basically a way to emulate deeper recursion. As we go through the layers, concepts are pattern-matched and refined, but at some point we have to stop and cannot refine them any further (no more layers). Usually, this refinement continues during the generation of the next token (the previous intermediate results needed to continue the refinement are still in the KV cache).
But some problems require a substantial number of pattern-matching and refinement steps. The problem is, we also have interference from the fact that the model is trained to model language using mostly non-reasoning data of varying semantic lengths. Because of that, it may stop generating text before the abstract refinements are fully completed, simply because the pretraining data tells it to.
So we have to additionally train models to produce "reasoning traces" so that the emulated recursion continues for longer than what is typically found in pretraining data, allowing the model to build richer and more complex abstractions and surface more concepts. The ability to split problems into steps and logically connect concepts is already present in non-reasoning models (the original CoT trick), because some of it exists in the pretraining data, but not enough to support much longer recursion (hence the premature stops).
As for whether it is "true reasoning" or not, I think that is just arguing semantics for the sake of it. LLMs can demonstrably solve various complex problems. Yes, they often make stupid mistakes, but don't we have the saying, errare humanum est? Don't humans make mistakes too? Don't we also have around 200 cognitive biases showing that we "simply pattern-match" too? I think we still cannot get rid of the Great Chain of Being idea.
The point is that LLMs are doing human-ish things on their own, and that's what the value in them is, so OP is bringing attention to the fact that the tradeoff is "thinks dynamically, sometimes makes mistakes". Computers have never been built to think dynamically before.
"think dynamically" is the type of discourse TFA denounces, though. The thing designated by "think" is already largely unclear when you think about it, "think dynamically" sounds like self-improvement-coach nonsense.
But some problems require a substantial number of pattern-matching and refinement steps. The problem is, we also have interference from the fact that the model is trained to model language using mostly non-reasoning data of varying semantic lengths. Because of that, it may stop generating text before the abstract refinements are fully completed, simply because the pretraining data tells it to.
So we have to additionally train models to produce "reasoning traces" so that the emulated recursion continues for longer than what is typically found in pretraining data, allowing the model to build richer and more complex abstractions and surface more concepts. The ability to split problems into steps and logically connect concepts is already present in non-reasoning models (the original CoT trick), because some of it exists in the pretraining data, but not enough to support much longer recursion (hence the premature stops).
As for whether it is "true reasoning" or not, I think that is just arguing semantics for the sake of it. LLMs can demonstrably solve various complex problems. Yes, they often make stupid mistakes, but don't we have the saying, errare humanum est? Don't humans make mistakes too? Don't we also have around 200 cognitive biases showing that we "simply pattern-match" too? I think we still cannot get rid of the Great Chain of Being idea.