))
“o1 is a bad idea” by abramdemski
Manage episode 449794982 series 3364758
Contenuto fornito da LessWrong. Tutti i contenuti dei podcast, inclusi episodi, grafica e descrizioni dei podcast, vengono caricati e forniti direttamente da LessWrong o dal partner della piattaforma podcast. Se ritieni che qualcuno stia utilizzando la tua opera protetta da copyright senza la tua autorizzazione, puoi seguire la procedura descritta qui https://it.player.fm/legal.
This post comes a bit late with respect to the news cycle, but I argued in a recent interview that o1 is an unfortunate twist on LLM technologies, making them particularly unsafe compared to what we might otherwise have expected:
The basic argument is that the technology behind o1 doubles down on a reinforcement learning paradigm, which puts us closer to the world where we have to get the value specification exactly right in order to avert catastrophic outcomes.
RLHF is just barely RL.
- Andrej Karpathy
Additionally, this technology takes us further from interpretability. If you ask GPT4 to produce a chain-of-thought (with prompts such as "reason step-by-step to arrive at an answer"), you know that in some sense, the natural-language reasoning you see in the output is how it arrived at the answer.[1] This is not true of systems like o1. The o1 training rewards [...]
The original text contained 1 footnote which was omitted from this narration.
---
First published:
November 11th, 2024
Source:
https://www.lesswrong.com/posts/BEFbC8sLkur7DGCYB/o1-is-a-bad-idea
---
Narrated by TYPE III AUDIO.
…
continue reading
The basic argument is that the technology behind o1 doubles down on a reinforcement learning paradigm, which puts us closer to the world where we have to get the value specification exactly right in order to avert catastrophic outcomes.
RLHF is just barely RL.
- Andrej Karpathy
Additionally, this technology takes us further from interpretability. If you ask GPT4 to produce a chain-of-thought (with prompts such as "reason step-by-step to arrive at an answer"), you know that in some sense, the natural-language reasoning you see in the output is how it arrived at the answer.[1] This is not true of systems like o1. The o1 training rewards [...]
The original text contained 1 footnote which was omitted from this narration.
---
First published:
November 11th, 2024
Source:
https://www.lesswrong.com/posts/BEFbC8sLkur7DGCYB/o1-is-a-bad-idea
---
Narrated by TYPE III AUDIO.
496 episodi
Condividi
