I think the assumption that there is some sort of formal ‘reasoning’ process that is inherently superior to whatever ‘non-reason-based’ decision process ‘non-reasoning’ systems use is not immediately justifiable.
Most reasoning is ultimately based in heuristics: this is the right thing to do next because it has worked in the past. This is the right thing to do because our intuition tells us doing it will lead to this desirable outcome.
When you’re formulating a math proof, sure, the steps you are laying out make up a ‘reasoned’ argument but how you choose what steps to take is wholly intuitive. Why assume the contrapositive to begin this proof? It’s worked in the past and it might work here.
It’s all pattern matching trained on a reward function.
It's true that humans are not perfectly logical actors, but even so, most humans won't tell you seriously that the swimming pools in the Titanic are empty, or that glue is a great pizza topping.
At the end of the day, there's a difference between showing your work for a math problem by writing down each logical step as you work through it, vs. just blindly copying both the answer and the "show your work" part from your classmates.
That humans can fail at math, or at other kinds of logic problems, is immaterial. It's still an entirely different process, and it leads to different results.
The fact that humans often fail because they're actively prioritizing something else can also be for the best. Like, it's reasonable to get a math problem wrong because you're distracted by argueing classmates who might be about to start a fight right next to you. Or you might realize that the premise of a question itself is wrong. Or you might logically know that the Earth goes around the sun, but decide to pretend otherwise because it's not worth dying over.
Humans are always balancing priorities, and it doesn't always make sense to judge their success by a single metric like "do these math problems and show your work within 10 minutes". LLMs do not have these conflicts, even when we would want them to.
I think the assumption that there is some sort of formal ‘reasoning’ process that is inherently superior to whatever ‘non-reason-based’ decision process ‘non-reasoning’ systems use is not immediately justifiable.
Most reasoning is ultimately based in heuristics: this is the right thing to do next because it has worked in the past. This is the right thing to do because our intuition tells us doing it will lead to this desirable outcome.
When you’re formulating a math proof, sure, the steps you are laying out make up a ‘reasoned’ argument but how you choose what steps to take is wholly intuitive. Why assume the contrapositive to begin this proof? It’s worked in the past and it might work here.
It’s all pattern matching trained on a reward function.