> But don't you think that it's pretty likely that this, or something phrased slightly differently, appeared in the training data?
I don't think so, but I could be wrong. It's definitely not "likely", see the math below.
I base that on the fact that people seemed to spend quite a bit of time trying to find the phrase "the confetti has left the cannon" that GPT-4 phrased. It seems Google search has no records of it before then?
I've seen many other examples where GPT-4 can translate sentences between using different types of idioms, and I just can't picture all these weird examples already being present on the Internet?
Do you think GPT-4 is a stochastic parrot that just has a large database of responses?
If so, how would we test that claim? What logical and reasoning problems can we give it where it fails to answer, but a human doesn't?
My understanding is that even with an extremely limited vocabulary of 32 words, you quickly run out of atoms in the universe (10^80) if you string more than 50 words together. If your vocabulary instead is 10k words, you reach 10^80 combinations after 20 words.
By training the LLMs on "fill in the missing word", they were forced to evolve ever more sophisticated algorithms.
If you look at the performance over the last 5 years of increasingly larger LLMs, there was a hockey-stick jump in performance 1-2 years ago. My hunch is that is when they started evolving structures to generate better responses by using logic and reasoning instead of lookup tables.
> I base that on the fact that people seemed to spend quite a bit of time trying to find the phrase "the confetti has left the cannon" that GPT-4 phrased. It seems Google search has no records of it before then?
Could it be that the expression in some form has been used in languages other than English?
If that is the case, it would downgrade the achievement from "super impressive" to just "impressive".
I spent some time trying to find it in other languages, but couldn't. Doesn't prove much of course, hopefully native speakers can weigh in on this.
I did find this though:
'TIL that a young stripper named Shelly Bauman lost her leg in a freak confetti cannon accident. She sued and used the money from the settlement to open Seattle's first gay bar, which she named "Shelly's Leg."'
I don't think so, but I could be wrong. It's definitely not "likely", see the math below.
I base that on the fact that people seemed to spend quite a bit of time trying to find the phrase "the confetti has left the cannon" that GPT-4 phrased. It seems Google search has no records of it before then?
I've seen many other examples where GPT-4 can translate sentences between using different types of idioms, and I just can't picture all these weird examples already being present on the Internet?
Do you think GPT-4 is a stochastic parrot that just has a large database of responses?
If so, how would we test that claim? What logical and reasoning problems can we give it where it fails to answer, but a human doesn't?
My understanding is that even with an extremely limited vocabulary of 32 words, you quickly run out of atoms in the universe (10^80) if you string more than 50 words together. If your vocabulary instead is 10k words, you reach 10^80 combinations after 20 words.
By training the LLMs on "fill in the missing word", they were forced to evolve ever more sophisticated algorithms.
If you look at the performance over the last 5 years of increasingly larger LLMs, there was a hockey-stick jump in performance 1-2 years ago. My hunch is that is when they started evolving structures to generate better responses by using logic and reasoning instead of lookup tables.