CHATGPT DOESN'T REASON! (Top scientist bombshell)

Not only can you not tell if someone memorized the answer or reasoned from first principles to get the answer, you can’t tell if someone memorized the steps involved in reasoning from first principles when it looks like they have reasoned from first principles. When I first took calculus I had no trouble memorizing the Epsilon-Delta proof without really understanding it, so if you asked me to demonstrate that I understood the fundamental principles of calculus I could provide a proof of the principles without understanding it

I've worked with people who don't reason either. They exhibit the kind of shallow non-thinking that ChatGPT engages in.

I memorized all the knowledge of humans. I can't reason but I know everything humans have ever put online. Am I useful? Provide reason.

LLMs definitely can do transitive closure. Not sure why the guest stated otherwise. I tried it out with completely random strings as object names and Claude could do it easily. So it's not just retrieving information.

Next word prediction is the objective function, but it isn't what the model learns. We don't know what the learned function is, but I can guarantee you it isn't log-odds.

There's a difference between in-distribution reasoning and out-of-distribution reasoning. If you can make the distribution powerful enough, you can still advance research with neural models.

Have we yet defined intelligence sufficiently well that we can appraise it and identify it hallmarks in machines?

His analogy of GPT to learning a second language makes 100% sense to me.

if the title says “people don’t reason” many viewers think it makes the strong claim “ALL people don’t reason“, when it is actually making the weaker claim “SOME people don’t reason“. that title is factually defensible but misleading. one could be excused for interpreting this title to be claiming “ChatGPT doesn’t reason (at all)“, when it is actually claiming “ChatGPT doesn’t reason (very well)“. One of the beauties of human language is that the meaning of an utterance derived by the listener depends as much on the deserialization algorithm used by the listener as on the serialization algorithm employed by the speaker. the YouTube algorithm chose this title because the algorithm “knows” that many viewers assume the stronger claim. nonetheless, be that as it may, this was a wonderful interview. many gems of insight on multiple levels ; including historical, which I enjoyed. I especially liked your displaying the title page of an article that was mentioned. looking forward to someone publishing “Alpha reasoning: no tokens required“. I would watch again.

Thank you. He was my favorite guest that I watched here so far. I learned a lot.

Those people who are assuming the AGI is going to be achieved have never done long-term work inside any of the major GPT systems if you want to have a quick and dirty test, tell it to create you a fiction book first make 15 chapters and 10 sections with each chapter And then have it start writing that book look at it in detail and you will see section after section it loses sight of essentially every detail. It does a better job if you are working inside the universe, another author has already made and does the worst job if you were creating a brand new universe, even if you have it define the universe.

Could an LLM for example figure out whether a request requires a planner, a math engine etc, transform the request into the appropriate format, use the appropriate tool, and then transform the results for the user? I think that LLMs provide a good combination of UI and knowledge base. I was suspicious myself that in the web data they may well have seen joke explanations, movie reviews, etc etc and can lean on that. I think that LLMs can do better, but it requires memory and a feedback loop in the same way that embodied creatures have.

Very convincing arguments. Haven't heard it laid out this succinctly and comprehensively yet. I'm sure Yann LeCunn would be in the same camp, but I recall not being persuaded by LeCunn's arguments when he made them on Lex Fridman

I love this episode. In science, it's never about what can be done or what happen in the system, but it's always about mechanism that lead to the event (how the event happen basically). What is severely missing from all the LLMs talk today is the talk about underlying mechanism. The work on mechanism is the key piece that will move all of these deep neural network works from engineering feat to actual science. To know mechanism, is to know causality.

Sara Hooker said the same about us not fully understanding what is used in training - the low frequency data and memorization of those being interpreted as generalization or reasoning. Good interview.

Gpt is like a reactive mega wikipedia....

0:18 When I was 4 years old, I was often stuck at my parents work. The only thing for me to do that was entertaining, was play with calculators or adding machines. I memorized the times table, because I played with calculators a lot. My parents would spend $8 at the drug store to keep me from asking why is the sky blue and other pertinent questions. I was offered to skip first grade by after kindergarten, and my parents said no. Jeff Bezos is the same age from me, and also from Houston. His parents said yes to skipping first grade. I told my parents this forever until they died.

Introduction and Initial Thoughts on Reasoning (00:00) The Manhole Cover Question and Memorization vs. Reasoning (00:00:39) Using Large Language Models in Reasoning and Planning (00:01:43) The Limitations of Large Language Models (00:03:29) Distinguishing Style from Correctness (00:06:30) Natural Language vs. Formal Languages (00:10:40) Debunking Claims of Emergent Reasoning in LLMs (00:11:53) Planning Capabilities and the Plan Bench Paper (00:15:22) The Role of Creativity in LLMs and AI (00:32:37) LLMs in Ideation and Verification (00:38:41) Differentiating Tacit and Explicit Knowledge Tasks (00:54:47) End-to-End Predictive Models and Verification (01:02:03) Chain of Thought and Its Limitations (01:08:27) Comparing Generalist Systems and Agentic Systems (01:29:35) LLM Modulo Framework and Its Applications (01:34:03) Final Thoughts and Advice for Researchers (01:35:02) Closing Remarks (01:40:07)

Claude just deciphered a random biography, in rotation cipher, for me. All I told him was that it was a Caesar cipher and then gave him the text. I didn't tell him how many letters it was shifted or rotated by and I didn't use rot13. I tried it three times with three different shift values and it translated it perfectly each time. There's no way that Claude has memorized every single piece of information on the internet in cipher form. Don't know if it's "reasoning" but it is certainly applying some procedure to translate this that is more than just memorization or retrieval. ChatGPT also did it but it had some errors. Instead of criticizing other scientists for being fooled and not being analytical enough maybe you should check your own biases. I have found it true that it can't do logic when a similar logic problem was not in its training data but it definitely can generalize even when very different words are used.

I have to stop to ask if '150% accuracy' is an actual thing in LLM/GPT? Or other weird number things that I'll have to go read. Keywords?