))
“The Most Forbidden Technique” by Zvi
Manage episode 471358300 series 3364758
Contenuto fornito da LessWrong. Tutti i contenuti dei podcast, inclusi episodi, grafica e descrizioni dei podcast, vengono caricati e forniti direttamente da LessWrong o dal partner della piattaforma podcast. Se ritieni che qualcuno stia utilizzando la tua opera protetta da copyright senza la tua autorizzazione, puoi seguire la procedura descritta qui https://it.player.fm/legal.
The Most Forbidden Technique is training an AI using interpretability techniques.
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.
You train on [X]. Only [X]. Never [M], never [T].
Why? Because [T] is how you figure out when the model is misbehaving.
If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.
Those bits of optimization pressure from [T] are precious. Use them wisely.
Table of Contents
Outline:
(00:57) New Paper Warns Against the Most Forbidden Technique
(06:52) Reward Hacking Is The Default
(09:25) Using CoT to Detect Reward Hacking Is Most Forbidden Technique
(11:49) Not Using the Most Forbidden Technique Is Harder Than It Looks
(14:10) It's You, It's Also the Incentives
(17:41) The Most Forbidden Technique Quickly Backfires
(18:58) Focus Only On What Matters
(19:33) Is There a Better Way?
(21:34) What Might We Do Next?
The original text contained 6 images which were described by AI.
---
First published:
March 12th, 2025
Source:
https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-forbidden-technique
---
Narrated by TYPE III AUDIO.
---
…
continue reading
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.
You train on [X]. Only [X]. Never [M], never [T].
Why? Because [T] is how you figure out when the model is misbehaving.
If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.
Those bits of optimization pressure from [T] are precious. Use them wisely.
Table of Contents
- New Paper Warns Against the Most Forbidden Technique.
- Reward Hacking Is The Default.
- Using [...]
Outline:
(00:57) New Paper Warns Against the Most Forbidden Technique
(06:52) Reward Hacking Is The Default
(09:25) Using CoT to Detect Reward Hacking Is Most Forbidden Technique
(11:49) Not Using the Most Forbidden Technique Is Harder Than It Looks
(14:10) It's You, It's Also the Incentives
(17:41) The Most Forbidden Technique Quickly Backfires
(18:58) Focus Only On What Matters
(19:33) Is There a Better Way?
(21:34) What Might We Do Next?
The original text contained 6 images which were described by AI.
---
First published:
March 12th, 2025
Source:
https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-forbidden-technique
---
Narrated by TYPE III AUDIO.
---
712 episodi
Condividi
