AI’s Dunning-Kruger Problem | feat. @JordanHarrod

Published 2023-09-04
Half of Americans TRUST what AI says to them, but they should NOT. 🙅‍♂️ Feat. ‪@JordanHarrod‬ !
📒 Watch an ad-free, director's cut version of this video on Nebula: go.nebula.tv/tracedominguez (Joining directly supports the channel!)

⭐ ⭐ ⭐ S U G G E S T ⭐ ⭐ ⭐
Want me to make a video about a burning question you have? A cool story you found? Think of me as your nerdy best friend. I'll do the research and you get the benefits!
Suggest a video here: noteforms.com/forms/think-of-me-as-your-nerdy-best…

🔉🔉🔉 A U D I O 🔉🔉🔉
ENGLISH
This video has been dubbed using an artificial voice. You can change the audio track language in the Settings menu. aloud.area120.google.com/
SPANISH
Este video ha sido doblado al español con voz artificial. Puede cambiar el idioma de la pista de audio en el menú Configuración. aloud.area120.google.com/
PORTUGESE
Este vídeo foi dublado para o português usando uma voz artificial. Você pode alterar o idioma do áudio no menu Configurações. aloud.area120.google.com/

💁‍♂️💁‍♀️💁 D E T A I L 💁💁‍♀️💁‍♂️
All three below are TRUE:
1. #ChatGPT and other ChatBot AI is the hot technology rn.
2. These #LargeLanguageModels don't know anything.
3. Humans are terrible lie detectors.

These three truthfacts make AI's overwhelming confident "speech" a kryptonite to human thinking.
The #Dunning-KrugerEffect is a psychological tool to understand cognitive bias. It says when people learn a little about a complex topic, they are over confident in their understanding. #LargeLanguageModels present this exact behavior to audiences. Not only can LLMs like ChatGPT not fact check their replies, they also have no way of knowing what they don't know!

On top of that, most humans don't know how AI chatbots (like ChatGPT) work, meaning our own Dunning-Kruger bias comes into play as well. Tech like ChatGPT cannot fact check itself! Here's what you need to know.

This Dunning-Kruger Effect is AI's Biggest problem, and ours too.

🎙️🎙️🎙️ P O D C A S T 🎙️🎙️🎙️
I've got a BRAND NEW PODCAST called That's Absurd Please Elaborate. In each episode, my buddy Julian Huguet and I take silly questions from the audience and answer them with serious research.

Some of my favorites:
- How long would it take to make a LEGO replica of the sun?
- What's the pointiest thing?
- Why aren't humans born with adult sized hands and feet?
- Where is the internet?

We release episodes every other Thursday!
🌌 Nebula: go.nebula.tv/thatsabsurdshow
Or if you prefer to listen OFF Nebula, go here:
🍎 Apple Podcasts: podcasts.apple.com/us/podcast/thats-absurd-please-…
🍏 Spotify: open.spotify.com/show/6JOkmJBsVOuODHarxzmotZ?si=bb…
Or check Pod Link, they've got all the apps: pod.link/1680094699

🔊🔊🔊 A U D I O 🔊🔊🔊
This video has been dubbed using an artificial voice via aloud.area120.google.com/ to increase accessibility. You can change the audio track language in the Settings menu.



🧲🧲🧲 F O L L O W 🧲🧲🧲
Threads 🧵:: www.threads.net/@tracedominguez
BlueSky 🌌:: bsky.app/profile/tracedominguez.com
Instagram 📷:: instagram.com/tracedominguez
Twitter ❎:: twitter.com/tracedominguez
Patreon 🙏:: patreon.com/tracedominguez

📒📒📒 C H A P T E R S 📒📒📒
00:00 There are no AIs that can Fact Check
00:44 ChatGPT’s Dunning-Kruger Problem
1:50 What is the Dunning-Kruger Effect?
6:33 How does this apply to AI & ChatGPT?
7:41 Before you ask ChatGPT to write things, you should know
9:40 ChatGPT cannot Fact Check
11:18 The main takeaway from this video…
13:03 “I dream of a better world”
14:20 I have a podcast!

📕📕📕 L E A R N 📕📕📕
Cornell University's Preprint: Extracting Training Data from AI Large Language Models (LLMs)
arxiv.org/pdf/2012.07805.pdf

Humans Are Pretty Lousy Lie Detectors
www.scientificamerican.com/article/humans-are-pret…

Unskilled and Unaware of It: How Difficulties in Recognizing One's Own
Incompetence Lead to Inflated Self-Assessments (actual PDF)
www.avaresearch.com/files/UnskilledAndUnawareOfIt.…

ChatGPT is Dumber Than You Think
www.theatlantic.com/technology/archive/2022/12/cha…

Google Warns Staff about AI ChatBots
www.reuters.com/technology/google-one-ais-biggest-…

Studies show ChatGPT cheating is on the rise among students
www.businessinsider.com/teachers-caught-students-c…

Vintage Wallpaper image by Getty Images
Music by Epidemic Sound

Thanks for supporting my sc

All Comments (21)
  • @michaelsander2878
    My AIM away message used to be "Dropping the kids off at the pool" because poop jokes are hilarious.
  • @likebot.
    So... AI is no different than my inlaws. "I read it on the internet" (by which they mean FB). Ah, yes, you can dream up a thing, post it to social media and it metamorphs into fact to be quoted smugly in my face complete with a bobbling head. That was meant to sound as ranty as it sounds.
  • @ifur
    I just use it to get things started instead of staring at a blank page i just correct it and voila I have a good paper.
  • @jeffrick
    There's a second AI Dunning-Kruger Problem: It takes a good amount of competence to realize when AI Hype is unrealistic.
  • @takeraparterer
    her: you better not be fearmongering with clickbait titles me:
  • @jamesnewman9547
    It has been trained on more code than any human ever will... And yet it makes mistakes that are so bad that it makes my head hurt. It's not a fun way to code. It is literally autocomplete of next statistically likely token. That is just language, not fact or reasoning. Great video, thank you so much for spreading information!
  • @MrMash-mh9dy
    Thank you, Trace, this is a video that needs to be made really. Too many people conflate the AI of movies with the ones we actually have. When Watson beat Garry Kasparov in a six-game set, the debate raged on how humans would be replaced but few talked about how he won a match against it and made it draw three times in those matches. Now in 2023, it's ChatGPT but the same problem exists with it in a way. Watson couldn't predict human unpredictability perfectly and neither can ChatGPT. What sounds best mathematically, or what move has the highest probability of success is not always compatible with human intelligence and norms or even reality, itself.
  • @mikefirth9654
    When I tested chat AI with a python program and interactively corrected it, the result though brief was good. When I asked it about the differences between the North Shore and the south shore of Crystal Lake, it gave me an answer based on being north and south of downtown Crystal Lake which is actually a few miles from the lake itself. In other words it didn't catch the idea of shore applying to a body of water.
  • @gipanze1
    Once I asked a question to AI and I sourced the answer it gave me back to a comment that I made on a forum discussing the same topic. Chatgpt stated my words with confidence when I myself didn't have any idea what I was talking about
  • @SanctorumDivinity
    Another powerful example is how bad it is at math, it's well worth a discussion point
  • @GaviLazan
    Very minor correction: Woody is the one who says that Buzz isn't "flying, he's falling with style". Buzz, on the other hand, is convinced he is flying.
  • @bakerfx4968
    Holy crap you just unlocked so many memories by just mentioning smarterchild lol
  • @Angela10
    Oh my goodness the conversations I had with SmarterChild. The abuse that poor bot took from my friends. 🤦🏻‍♀️
  • The solution is quite easy. Like what they already does for images and videos, but for text. I can't believe that nobody who has a working computer has thought about this.. 🤔
  • @srvictorbatista
    Tio Lu, term um otimo curso sobre redublagem de videos. Seria bom dar uma olhada.
  • @OsirisMalkovich
    AI might become an unobtrusive and useful part of most of our technology, eventually, but in the words of Chief Wiggum, "This is gonna get worse before it gets better."
  • @lucidmoses
    I look at it as, ChatGPT tells stories and are as truthful as your typical movie.
  • @shao2307
    Just treat AI like the rest of the world treats American businessman - with a huge pinch of salt.