1. No it's not lol. If the model was only trained on that much data, it wouldn't be anywhere near as good in french. 1.8% is only enough here because it trained on other languages as well.
GPT-3 is also fluent in languages with less training data.
3. LLMs trained on code score noticeably higher on reasoning benchmarks
1.8% does look like a small number but imagine (i know its hard in this day and age with 4TB finger nail usb sticks) a physical library holding good old fashioned paper made artifacts and what does 1.8% of that looks like?
It’s also fluent (though not at a native level) in, say, Finnish, and can write reasonable poems and whatever, even though Finnish text probably comprises something like 0.01% of the corpus.
Norwegian as well. Though given Norwegian is also a Germanic language like English, Finnish is a lot more impressive. I managed to get it trip up on one single gendered article in Norwegian, while asking about Sweden, in a way very similar to how a human switching between Norwegian and Swedish might (it would be valid Swedish, and was followed by a Swedish word that's also valid in Norwegian but with another gender), but for the most part it writes Norwegian better than a lot of native speakers I've known would write (but not better than how they'd speak).
EDIT: I got it to "translate" a test sentence from Norwegian to "Riksmål", an old-fashioned artificial Norwegian dialect that seeks to stay closer to very conservative old Norwegian (basically a lot closer to Danish) and it managed that just fine too, and there's very little material that you'd be able to tell explicitly is Riksmål as most of the time it'd just be described as Norwegian unless the writer is trying to make a point.
Since that worked, I tried Nynorsk ("New Norwegian") which is used by something like 20% of the population, and it did fine.
So I tried a local dialect from Eastern Norway, that is predominantly spoken and only occasionally written and it got even that pretty much how I'd have said it myself, after a lecture on how it varies greatly within the region (which is true)
The differences there are subtle, and frankly a lot of native speakers would get one or more of these wrong.
EDIT2: Last try at stumping it with Norwegian: "AKP-dialect". AKP was a Maoist political party of a few hundred members whose members were largely intellectuals. Many of them adopted a "faux workers dialect". This had no formal name, so it takes some understanding of Norwegian 1970's and 80's politics to even guess what it is. It understood what I meant, but didn't understood what spesifikk language-features I was referring to. Then again, if I asked random Norwegians about that most would need an explanation too. When I mentioned a couple of people who wrote like that it produced a plausible version (but clearly more inspired by the writing style of one of them - Jon Michelet)
Google’s transformer implementation for translate similarly understands the meaning and relation of words, which is what allows it to translate between any pair of languages. They spoke about this years ago.
Somewhat offtopic but what kind of French, do you think, is GPT's speech pattern? They sound translated in thinking and in verbiage in my language(which isn't French).
I am a French native and GPT has a really good level in my experience, albeit not as good as in English — which I speak quite well, having spent many years in the US. But it’s not as good, maybe 80-90% there.
I can't vouch for French, but it's fluent in Dutch as well (my native language). And you can ask it to adopt different styles and dialects even. Anything it can generate in English it can generate in Dutch as well and without any obvious grammar mistakes. So, it's not merely translating but actually able to generate answers in Dutch from training content that was never translated into Dutch.
It's why:
1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus.
2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English.
3. LLMs trained on code demonstrably reason better.
It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level.