"This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (not results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models."
Finally, a concise explanation why this generation of chatbots –while absolutely superior to previous art (where applicable)— still throws me back to discovering markov chains*, 'cat'-ing countless amount of text into a 'corpora.txt' to build models of various personalities.
The idea that a larger corpora == better was quickly established, but I hit a ceiling quite soon on early RPi's.
It was possible to seed a start word for text generation, a Baby prompt we might call it?
Nice feature was being able to make a mini chain (model) of the user input sentence, then merge that statistics into the bots chain. The verbatim record of what had been said was obscured but some words and their relations —the experience— was kept. So I imagined, at least.
replacing gpt(x) w/ opt_gpt(x) speeds it up ~1.5-2x at first blush (really crude measurement)
The Bad:
The very first run took 43s on the initial compile! Gross! We had a mode for progress bar somewhere, but it would be cool if it was on by default. It would also be neat if there was a better set of utils around preheating.
Speed, but, honestly, if I am being critical, this example is both beautiful and extremely spartan. I think on an example so thoroughly bare, it is hard to see any benefits here.
The mathematical modeling of most NLP tasks is Markovian. (More specifically, they tend to be HMMs, Hidden Markov Models). One of the popular "solutions" for Markovian processes is the Metropolis algorithm which you might find relevant. Note in statistical learning the mathematical formulation generally operates as part of the loss function, the actual "model" is usually a neural network (in the context of deep or variational learning).
Computers are (with a couple caveats) deterministic, so I think the better analogy would be to say that it's the same as saying that "computers are just state machines".
But this is a bit nitpicky. The core idea is of course conveyed either way, that yes, this very abstract model does describe it while also completely missing that the magic is in how it's done.
If I can continue the nitpickery, plenty of computer algorithms are deterministic but also pseudorandom--including Markov chains. Only when the randomness is outside your control are things truly non-deterministic. A simulation with many possible outcomes is still deterministic if you get to choose the "seed".
In other words, statistical permutations are deterministic if you control all the input variables (in terms of computer science; physics is another story).
Sorry, I just didn't like the comparison between saying GPT is a markov chain and computers are a markov chain. Really learned something from the actual colab. Thank you, appreciated. Nobody likes a sad karpathy.
Yes, but much more efficiently stored and looked up than just a massive table. The way its very large space is encoded in a small amount of memory is what makes GPT so interesting.
A markov chain maps states to states.
You move from word "I" to word "will" or "can" or "want"
The transition probabilities could be stored in a huge table.
You have a word and the probability of each other word. Draw a number and hop. [1,2]
Alternatively you could ask a neural net for the next state. Pass it the current token or the last 4000 token.
Even in the case of a full size GPT, one could equivalently write out a lookup table for every possible permutation of tokens in the context window and the resulting output distribution. The table would of course be impossibly large.
Sort of, but only if you consider that the states are massive. It’s not just “what’s the next word after this one.” It’s, “what’s the next word after this one, considering the other several hundred other words that came first.”
So, the input sequence is an extremely high-dimensional state value in a gigantic Markov chain.
To me, that sounds like not a Markov chain. Yes if you consider the Markov chain over entire sequences, but that’s not what people mean with Markovian.
I have a hunch that a lot of the deep relationships between words in languages can be inferred by using something like binary regular expressions. Defining states by a context length have a close analogue to this technique to speed up applying regular expressions over large data: https://swtch.com/~rsc/regexp/regexp4.html. Now imagine if context lengths could be grouped together to form higher types, that could then be used in a regular expression. For example, currently 3 bits are grouped to form the context window, and in the realm of regular expressions, one could say this forms the "3 bit alphabet". Now a higher level alphabet could be built upon the "3 bit alphabet", maybe a "9 bit alphabet" from a grouping of 3 symbols from the "3 bit alphabet". In other words, imagine if you had a transition diagram, but instead of each state being a context window that relates to direct data, you had something like a parsing graph for a programming language. Now imagine you had a statistical method to gradually build these nested parsing structures. In a way convolutions and regular expressions are similar, they are a way of partitioning data to eventually signify different values to different parts, except convolutions have better statistical properties.
The problem with using a regex as you describe is that GPT isn't working at the level of words, its working at a level of meaning and intent. You can have it translate rather meaningfully between metaphors that exist in completely different contexts.
Somewhere in the 90+ levels of interconnected neurons, the literal meanings are abstracted and re-encoded. A regex isn't going to accomplish that.
edit: you might be able to match this little tiny GPT, but not one of the billions-of-parameters varieties.
LLMs work on tokens, which, for the most part, are words (longer words are generally split up into multiple tokens). Using concepts like "meaning" and "intent" muddies the waters unnecessarily.
1. No it's not lol. If the model was only trained on that much data, it wouldn't be anywhere near as good in french. 1.8% is only enough here because it trained on other languages as well.
GPT-3 is also fluent in languages with less training data.
3. LLMs trained on code score noticeably higher on reasoning benchmarks
1.8% does look like a small number but imagine (i know its hard in this day and age with 4TB finger nail usb sticks) a physical library holding good old fashioned paper made artifacts and what does 1.8% of that looks like?
It’s also fluent (though not at a native level) in, say, Finnish, and can write reasonable poems and whatever, even though Finnish text probably comprises something like 0.01% of the corpus.
Norwegian as well. Though given Norwegian is also a Germanic language like English, Finnish is a lot more impressive. I managed to get it trip up on one single gendered article in Norwegian, while asking about Sweden, in a way very similar to how a human switching between Norwegian and Swedish might (it would be valid Swedish, and was followed by a Swedish word that's also valid in Norwegian but with another gender), but for the most part it writes Norwegian better than a lot of native speakers I've known would write (but not better than how they'd speak).
EDIT: I got it to "translate" a test sentence from Norwegian to "Riksmål", an old-fashioned artificial Norwegian dialect that seeks to stay closer to very conservative old Norwegian (basically a lot closer to Danish) and it managed that just fine too, and there's very little material that you'd be able to tell explicitly is Riksmål as most of the time it'd just be described as Norwegian unless the writer is trying to make a point.
Since that worked, I tried Nynorsk ("New Norwegian") which is used by something like 20% of the population, and it did fine.
So I tried a local dialect from Eastern Norway, that is predominantly spoken and only occasionally written and it got even that pretty much how I'd have said it myself, after a lecture on how it varies greatly within the region (which is true)
The differences there are subtle, and frankly a lot of native speakers would get one or more of these wrong.
EDIT2: Last try at stumping it with Norwegian: "AKP-dialect". AKP was a Maoist political party of a few hundred members whose members were largely intellectuals. Many of them adopted a "faux workers dialect". This had no formal name, so it takes some understanding of Norwegian 1970's and 80's politics to even guess what it is. It understood what I meant, but didn't understood what spesifikk language-features I was referring to. Then again, if I asked random Norwegians about that most would need an explanation too. When I mentioned a couple of people who wrote like that it produced a plausible version (but clearly more inspired by the writing style of one of them - Jon Michelet)
Google’s transformer implementation for translate similarly understands the meaning and relation of words, which is what allows it to translate between any pair of languages. They spoke about this years ago.
Somewhat offtopic but what kind of French, do you think, is GPT's speech pattern? They sound translated in thinking and in verbiage in my language(which isn't French).
I am a French native and GPT has a really good level in my experience, albeit not as good as in English — which I speak quite well, having spent many years in the US. But it’s not as good, maybe 80-90% there.
I can't vouch for French, but it's fluent in Dutch as well (my native language). And you can ask it to adopt different styles and dialects even. Anything it can generate in English it can generate in Dutch as well and without any obvious grammar mistakes. So, it's not merely translating but actually able to generate answers in Dutch from training content that was never translated into Dutch.
We read words. To translate some words into other words of different literal meanings while maintaining non-literal metaphorical meaning implies the capacity to "understand" the words at that level. That it represents data as tokens is irrelevant to the kinds of processing it obviously performs.
Pretending GPT is just using a percent chance that some token should show up next is absurd. It's almost a hundred layers of neural pathways.
In the grand scheme of things GPT isn't learning "meaning" in any different sense than Latent Semantic Analysis (LSA) learns meaning (which is just using "percent chance that some token should show up next" using covariance).
For those without a background in NLP, LSA is literally SVD/PCA (which is ultimately just based on the covariance matrix) on the term frequency matrix for a corpus of text. However LSA is well known to learn certain meanings such as "pet" by projecting words like "dog" and "cat" onto the same principle component. Semantics is learned by exploring the "loadings" in PCA terminology.
Now LSA is just a linear transformation, but the idea is that same as GPT that uses a wildly more sophisticated non-linear transformation: the assumption is that words (or tokens in the case of GPT) are caused by some latent variables (in the case of LSA, for GPT high dimensional embedding) and most importantly the patterns of word/token usage allow us to infer what those latent variables are.
So it's not absurd and it's how statistical NLP has always worked, only now we have much, much more sophisticated ways to model that latent space. LSA, HMM, and Transformers all assume that the patterns of text usage can be used to model an unobserved latent space that generates the text, which is what we traditionally call "meaning".
Ultimately the limits of GPT-N (unless there are none and it does become AGI) will be valuable in showing the limitations of this fundamental model of semantics.
> 1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus.
I personally think 1.8% is a lot. If I lived in France for 1.8% of my life, and aggressively consumed only French for that time, I would probably speak French too: I learned English faster than that.
> 2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English.
Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages.
> 3. LLMs trained on code demonstrably reason better.
What? No. Nonsense. I haven't seen any demonstration of reasoning. LLM's are more like the cargo-cult tribes watching things they don't understand and mimicking those things; Sure, they built an airport, but it doesn't work, and that makes all the difference.
> It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level.
What's obvious is that there's a relationship between words/tokens and ideas. What is not obvious is how far that relationship goes.
It's not lol. If the model was only trained on that content, it wouldn't be anywhere near as good at french. That's a fact. And French is one thing but they're are languages it's fluent in with less data.
>Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages.
First of all, what you're saying is straight up nonsensical. Knowing English doesn't make you even close to competent on the vast majority of European languages.
Secondly, it has nothing to do with European languages. It's excellent following instructions in Mandarin (0.1%) and Korean(0.02%), languages that had much less than 1.8% data.
>What? No. Nonsense. I haven't seen any demonstration of reasoning.
Many benchmarks that test reasoning. LLMs trained on code perform better here. Simple as that.
>What is not obvious is how far that relationship goes.
I really think you need to research a bit more about how statistical language models work.
The reason this models are able to learn other languages so quickly is that they're using the majority language to do much of the heavy lifting in defining the overall latent language space.
Once this is established, adding new languages requires much less information since you only have to learn to encode and slightly tweak the latent presentation you learn.
Again the view of statistical NLP, from LSA to GPT-N is that the words we write are generated from transformation of a latent space.
For LSA this transformation is linear and the space is low dimensional, and the model only learn to encode words to the space.
For GPT-N this transformation is non-linear, the latent space is very high dimensional and the model learns both the encoder and a decoder allowing it to generate language from the latent space.
If you have enough language to model this latent space well, then learning additional languages requires very little training data since the space itself is well modeled, you just need to learn the details of the encoder/decoder. That is you've learned the abstract point in high dimensional space that represents the English word "cat", you just need to learn that "猫" points to the same place.
Of course these models aren't just representing words/tokens without context to the latent space (like word vector models had done in the past) but an entire sequence of words.
Interestingly enough even with word vectors you can learn word vectors in language A and also language B and then manually align the latent spaces in order to get simple translation for close to free.
So given the training data size of GPT-3+ 0.02% is a lot of language.
Jan Leike's point is a bit more subtle there. He's not surprised it can learn translation with a relatively small number of training examples, but surprised that it seems to know some instruction with no examples.
And just a few Tweets later speculates it's likely because there are examples in the training data, and the success of the model to follow instructions in a non-English language from "nothing" is proportional to the presence of those examples in the training data [0].
I don't think he would disagree with anything in my statement.
> > Almost every language in Europe shares a lot of words and idioms, and English took every one it could, so training on English is training it on those other languages
> Knowing English doesn't make you even close to competent on the vast majority of European languages.
I cannot fathom the mind that thinks these two statements occurred in the same conversation.
When I said it operates on a level of meaning, I refer to its ability to perform transformations like this:
H: translate the following sentence to instead use olympic curling related metaphors ( do not explicitly mention curling, and do not use similes in forming the translation ): man:"I've been spinning my tires all day without getting anywhere" woman:"running out of gas?" man:"nah, I'm gassed up, geared up, oiled and revving. I just can't seem to get in the right lane, you know?" woman:"anything knocking?" man:"nah, I'm tuned just fine. I can't seem to get my GPS to connect, you know?" woman:"take a few laps without a destination, see if you can figure out the lay of the land" man:"you know, that might help. thanks barbara"
R: man: "I've been sweeping the ice all day without hitting the target."
. woman: "Losing momentum?"
. man: "Nah, I've got the energy, the technique, and the focus. I just can't seem to find the right path, you know?"
. woman: "Anything coming in your way?"
. man: "Nah, I'm on point with my aim. I just can't seem to get my strategy to align, you know?"
. woman: "Maybe glide a few times without aiming specifically, see if you can figure out the ice conditions."
. man: "You know, that might help. Thanks, Barbara."
I don't think you can perform this kind of translation by merely comparing tokens statistically tending to be near one another. I understand that this is how the data was encoded and the output judged. However, the neural net between those two points is free to encode whatever interconnecting values it happens to as its training drives it through to whatever multidimensional local maximum it manages to settle upon.
In short, I expect that it has to translate to and from an abstract meaning in order to manipulate metaphors in such a manner.
Most human language is metaphorical, as discussed in Lakoff & Johnson's "Metaphors We Live By". We have a core of experiential terms, but near the whole of the rest of our language is metaphoric in nature. The better a model gets at "understanding" English, the closer to a metaphor engine it must become, needing to be capable of applying arbitrary metaphors and partial metaphors in order to properly manipulate the language.
I'm not saying your position is wrong, but it reminds me of the contrasting viewpoints given in Hofstadter's "I am a Strange Loop" as he discussed his "careenium" and "simm balls" analogy for the rise of complex epiphenomena arising from simple rule bound systems.
That's amazing and all, but it's also speculation, yes? It's how you imagine, or guess, that things are happening. It's not like you, or anyone else, knows that this kind of thing is really happening.
Moreover, there isn't any clear reason why you need to speculate all that, in order to explain the observed behaviour of Large Language Models (LLMs). You say you don't think LLMs "can perform this kind of translation by merely comparing tokens statistically tending to be near one another". And yet, that's what we know that LLMs do, because that's what they are made to do [edit: they do what you describe after the "by merely"]. Why do we need to imagine some other, so far unobserved, process at play? What is the motivation? What is the justification?
Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another".
It's clear what you are saying, but, why? Why are you saying that? Why should we assume that the well-known and understood design and functionality of LLMs is not enough to explain their behaviour?
Is it because we haven't observed such behaviour before and some people are surprised by it? Well, we didn't have as large language models before, so obviously we didn't have a chance to observe their behaviour first-hand, but we also didn't have cars that could go 100 miles an hour, until we did, and making a faster car is no reason to be surprised, or wonder "how does it do it if it's just a car?".
>Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another".
The implication I gather from those dismissing this technology as "merely comparing tokens statistically tending to be near one another" is that they think the whole of the LLM is nothing but a trick, a sleight of hand where it generates output token by token Markov style after consuming enough text, just using some percent chance of tokens following on after another.
In my example, it generated the metaphor "sweeping the ice" as a replacement for "spinning my tires". Both are using in context to imply a level of busywork associated with the respective contexts. To generate such a transformation requires understanding this inner meaning and associating it. It requires a level of creativity to determine that "sweeping the ice" in curling is an appropriate replacement.
You cannot rearrange "spinning my tires" into "sweeping the ice" without having derived and translated the underlying meaning. There is nothing to connect the two sets of tokens except the metaphoric intent.
>Why should we assume that the well-known and understood design and functionality of LLMs is not enough to explain their behaviour?
Obviously LLMs are capable of this as it is an LLM that is producing it.
I am not denying that the inputs and outputs are tokens marked up with distance information, nor that the process of generating tokens is just a bunch of vector multiplication to generate neuron values from weights across the layers of the network. I am suggesting that with sufficient training, layers and parameters, that the neural net must be forced into a configuration that allows it to perform this meaning/intent based processing as gradient descent pushes and pulls its the network along the twelve thousand or so dimensions GPT-3 uses. Undoubtedly GPT-4 and soon 5 use even more.
LLMs have repeatedly been found to have developed novel and unintended capabilities as their complexity has increased.
Why do you think that LLMs shouldn't be capable of manipulating on the level of meanings and intents rather than just rearranging tokens?
>> You cannot rearrange "spinning my tires" into "sweeping the ice" without having derived and translated the underlying meaning. There is nothing to connect the two sets of tokens except the metaphoric intent.
Here, you are again explaining what, but you are not explaining why. You are saying what you think, but I still don't know why you think that. Why is it that you >> cannot rearrange "spinning my tires" into "sweeping the ice" << etc? Why do you say this?
For example, I can see that both are verb phrases with a similar structure. Language models excel in reproducing structure and we know they can translate between structures, also, as they do when translating between languages. So why can't they do it in this case without "underlying meaning"?
>> Why do you think that LLMs shouldn't be capable of manipulating on the level of meanings and intents rather than just rearranging tokens?
I didn't say I think that. I asked you:
>> Why do you think LLM's can't "perform this kind of translation by merely comparing tokens statistically tending to be near one another".
And I still don't know why you say that. For example, what you say here doesn't help me understand:
>> I am suggesting that with sufficient training, layers and parameters, that the neural net must be forced into a configuration that allows it to perform this meaning/intent based processing as gradient descent pushes and pulls its the network along the twelve thousand or so dimensions GPT-3 uses.
That's a description of the what, not the how. But it's not a description of any "whats" we know anyone coded into a language model in the first place. It's a description of a mechanism that is claimed to have appeared spontaneously.
That is a very big claim and I don't see why I have to accept it just because someone said that's what's going on.
I want to know how things work. You're not explaining how things work. You're explaining how you think they might work. But I don't even know why you think they work that way.
It maps a vector of eight thousand tokens to some number, likely thousands, of "neurons", multiplying each by a weight associated by each connection. All of these are then combined to form the next layer of values by multiplying and combining all of the values to all of the neurons in the next layer. Since all neurons connect to each other at each layer, many complex interactions can be created between them. These multiplications will go through about a hundred layers until being mapped back to a single token.
That token will be inserted at the end of the prompt and it will do this again and again until some stop token is reached, or the eight thousand token prompt is full.
---
As for what I think is happening, I think that information about meaning and methods for extracting it are being encoded across those billions of weights and thousand and thousands of neurons such that these multiplications will transform an initial vector into an abstract numeric form that across the whole span of any set of neurons might represent various abstract concepts and their interrelationships, such that they can then be used to select the next token in the series.
---
As for why, I expect this happens because abstract representation is how the model compresses data as gradient descent drives it twelve thousand different directions just a bit at a time as it trains on each document, meandering through the abstract space of potential neural net configurations to arrive at a local maximum that is sufficiently powerful enough to manipulate human language.
First of all, thanks for your patience. I'm still not convinced, but I appreciate your effort to explain how you think. Now.
For brevity, I'll refer to the three sections (separated by "---" in your comment) as A, B, and C.
Section A is a description of how a neural net trains a language model. I have some quibbles ("neurons"...) but it's talking about known and observable facts. So I will call it a "factual" description.
I note you did not include the word "meaning" in that factual description and that's because you restricted yourself to facts. You could not give a factual description with the word "meaning" in it (in the sense of "the meaning of...").
Section B and section C are again speculation and interpretation that are not based on known or observed facts, or even observable facts. So I still do not know why you think all this is going on inside a language model.
To clarify, in sections B and C you are at the very least putting an interpretation on the facts in section A. But I have no idea why you even need an interpretation, and why you choose that particular interpretation.
What do language models have to do with meaning, and why do we absolutely need to explain their behaviour by talking about meaning? Where is this information in your comment? I cannot see it.
This is literally the accepted term for the nodes wherein sums are produced of the set of weighted products which connect such nodes together. If you know a better term, or would prefer a better term that "neuron" to describe these points in a "neural network", by all means speak your mind, but don't pretend some offense at it.
>and that's because you restricted yourself to facts.
I restricted myself to writing from a point of view concordant with your own because that is what one does when explaining a position well, not because I agree with your point of view.
>You could not give a factual description with the word "meaning" in it (in the sense of "the meaning of...").
If we're going to play a game of semantics, than so be it.
If your argument is simply that symbols and numbers can't contain meaning because they are symbols and numbers, then you've worked your way into a tautological box that I, nor anyone else, will be able to free you from.
It is known that memories in biological neuron-based networks are not locally stored or unique to some neuron ( there is no "grandma" neuron in your head to keep track of dear old grannie ), but that instead memories are spread across the set of neurons there available.
This sort of representation in known as a "holographic" representation, which I expect our neural net will also utilize. Information will not be stored in any specific neurons in the trained model, but will be spread across the whole of them.
But even if true, that is merely data, you will no doubt point out. Where is the "meaning"?
As you want to delve philosophically into it, we must question the very nature of the word meaning here. Our model lacks any form of experiential qualia, and so cannot associate words with experienced reality, which is the normal basis for human concepts of meaning, which then underlie our development out of concrete language and into metaphoric.
This leaves meaning within the model as only being the derived relationship between words.
Words, on their own and bereft of qualia, can present our model with structure, grammar, punctuation. Words will have certain uses within these structures and grammars, they will have certain places. Words will modify words, phrases will modify larger sentences. Trues can be asserted and falsehoods. Truth and falseness itself can be demonstrated through sentences making claims about other sentences.
Would the model know what a red ball is? Of course not. But it would be aware of how the term ball is used, what sort of type it is given, the qualifications and usages of words of that type. The modifiers it is reasonable to use on it, etc.
And incredibly, with no basis in anything other than being slightly corrected for each incorrect token generated, our training harness manages to discover a mathematical neural net model of these relationships between words such that it can manipulate them as well or better than many people.
The use of backpropagation and gradient descent allows us to search the space of all possible models starting from a randomized model, and find a model that does this by nudging our model to be a little more capable at it over and over again.
However, English is not just a fixed mathematical set of symbols that can be rotely manipulated.
Our language is metaphoric from its very foundations. Our hearts soar, sink and swell in ways that have nothing to do with the reality, but only our perceptions of ourselves.
To manipulate English as it does, the traversal of the model-space must advance inexorably on a model capable of handling such non-literal language.
The model contains meanings for car related actions, for curling related actions, it contains the ability to translate between them when the only connection is a meaning that is implied by the words but not either of their literal meanings.
What is a meaning here? It is a relationship between the words. The relationship between the car words and curling words only exists as the relationship to a common implied meaning that can be transformed under the context of other words.
Where is the meaning? We don't even know for certain where the meaning is inside ourselves. I expect it to be similarly holographically stored across the set of neurons and provide the appropriate context to twist the swirl of mathematics that occurs between input vector and output.
>So I still do not know why you think all this is going on inside a language model.
Because it is the simplest explanation of what it is doing. Call it "latent spaces", "embeddings" and the assume the compression I discuss is via "manifold hypothesis".
>What do language models have to do with meaning, and why do we absolutely need to explain their behaviour by talking about meaning? Where is this information in your comment? I cannot see it.
So, where is the meaning? It is everywhere in the model once trained. It creates a holographically stored mapping of reality as presented through the corpus of human thought in the form of writing based solely on being iteratively nudged through model-space.
An amazing feat by the engineers that dreamed it up.
If you are unwilling to call this mapping model of reality built only of tokens/symbols/words "meaning", then I have no argument for you, except to again conclude that you've locked yourself in a tautological box.
If your argument is merely "ha ha, you're merely conjecturing this", then yes, of course I am. Why? Because it behooves someone to hypothesize, test, and study things which interest them.
Do you have some competing idea of how it works? It would behoove you to share it with the thread.
As it is, I only see you making demands for proof of things that you are well aware that no human has yet completely modeled, even after decades of studying such phenomena. What you gain from this incessant nay-saying, I do not understand.
>> As it is, I only see you making demands for proof of things that you are well aware that no human has yet completely modeled, even after decades of studying such phenomena.
What you gain from this incessant nay-saying, I do not understand.
I just wanted to comment on this. I've just finished a PhD (in AI, no less) and I've learned a few things about "nay-sayers", which in research are more formally critics and reviewers. What I've learned is that the "nay-sayers" are your friends. They're your friends even if they don't know it, even if their incessant nagging is driving you mad. They're your friends, because they give you the opportunity to see your work as it really is, not as the product of an unrecognised genious, but as the clumsy fumblings in the dark of a clueless and confused limited human mind. And they're your friends because they help you figure out where it is that you're wrong, and, ultimately, to improve your work.
It took me a long time to see things that way, it took many "dark tea-times of the soul" to figure out where I was wrong, when my critics were right, and how to improve my work to address the criticisms. And, you know the funny thing? Once I did that, all the frustration I felt at the "nay-saying" evaporated. The critics continued "nay-saying", but I just don't care, because I have proved to myself, and to others besides, that the criticisms have been addressed. The critics now have to find another angle of attack, and I'm disappointed as long as they can't, because that means my own understanding will stagnate and my work will not progress. Critics are worth their weight in gold.
But all that works much better in an academic environment where the critics and that "nay-sayers" are experts in their field. On the internet anyone can say anything and everybody pretends to know everything all the time, so all that we're left with is frustration. So I apologise for the frustration I discern in your use of "nay-sayers". This kind of conversation doesn't work well on the internet.
Part of why I comment is specifically to attract naysayers. I can learn from them. I've never been one to mind being wrong. I will argue vehemently for a position both because it seems correct to me, but also because it exposes my ideas to be torn down, so when I rebuild them they can be better.
I looked into things a number of people just in this thread brought up. It was a very rewarding discussion for me. I feel like my understanding is better than before it.
I became frustrated with you because your stance seemed a combination of "you can't prove it" and "it works like it does because it is what it is". You weren't expanding on your positions. It is possible that you are assuming context others in the thread do not share. It seems likely you think many here are reading into or seeing things that aren't actually present. Being tricked by the model, essentially. It is possible. But it also seemed possible that you were simply enjoying telling everyone they are wrong for fun.
With your refusal to expand on anything, it all looks much the same from this side.
I would like to understand what perspective you were arguing from, because even if I disagree with it in the end, knowing your perspective will allow me to understand where my own may be weak or flawed, and either shore it up or discard and rebuild it.
It is very frustrating to have a discussion with someone who seems to only contradict, but never explain their contradictions. They may be right. You may be right. But it was frustrating to talk to you because you didn't seem to want to say where or how people were wrong.
"It's not a brain". "It's works because it's a language model".
Okay, it's not a brain and it's just a language model. How does it perform what appears to be, whether it is or not, obvious translation of subtext in sentences between unrelated literal domains?
When I specifically asked if your position was simply that you would not attribute "meaning" to its representations, you declined to respond.
That's not an unfair position to take. An LLM has no experiential data in it. It has only the words. Philosophically, one could argue that even if it were "proven intelligent", it would still have no knowledge or contain actual meaning because it has only the words to go by. That would be a reasonable position, but you wouldn't confirm if that was what perspective you were taking.
I genuinely would enjoy knowing how you think translations on the level of the one I presented happened. Another poster discussed "latent spaces" where terms can grow closer even if not literally related. Articles I've read indicate any number of surprising results from LLms, assertions that LLMs can or cannot reason, various levels of assertion on whether LLMs can "understand" things. It appears like there are many perspectives active in this field all at once, and that they are quickly out of date as better and better LLMs are trained.
If you would be so kind, please add context to when you correct others. Pointers we can follow in order to understand your perspective.
For much of the thread, it felt like you were arguing that deep blue can't actually play chess because it has no concept of games or people or chess pieces or winning or losing, rather it only performs statistical analyses of numerically represented move sets and just outputs a move based on that statistical analysis. How can it play chess, after all, if it doesn't know what chess is?
>This kind of conversation doesn't work well on the internet
Possibly. I appreciate your having it nonetheless, as I have no one I know that cares to discuss any of this. So I have to make due with discussing it in online forums :)
Thanks for your comment! I gave my word I'd let you have the last word so I'm
not going to continue arguing. I'm just happy because I'm getting the feeling
you are not frustrated at our conversation anymore. I hope we can continue it
at some point and that I'll do a better job explaining things to you.
Especially when it comes to your big question about how can language models do
the kind of thing you illustrate with the translation example.
This conversation was interesting. I pasted it into ChatGPT (GPT-4) and asked it some questions. Here's one of its replies:
ChatGPT:
To better explain Knome's point to YeGoblynQueenne, I would provide a more concrete example and focus on the mechanisms behind LLMs that give rise to the appearance of meaning.
Large language models like GPT-3 are trained on vast amounts of text data, learning to predict the next word in a sentence based on context. This process enables the model to capture patterns, relationships, and regularities in the data, which can be seen as a form of "meaning" or understanding of language. The model's neural network, with its numerous layers, neurons, and connections, essentially forms a high-dimensional space that maps the relationships between words and their contexts.
To make Knome's argument more concrete, let's consider a specific example: the model's ability to translate the metaphor "spinning my tires" into "sweeping the ice." While it's true that the model doesn't have a direct understanding of the real-world experiences or objects the words represent, it has learned the contextual relationships between words and their usage in various situations. The model can associate "spinning my tires" with the concept of being stuck or not making progress, and "sweeping the ice" with a similar concept in the context of curling. This association is not a direct result of knowing the literal meanings of the words but rather of having learned the patterns and relationships in the data.
In essence, the "meaning" Knome refers to in LLMs is not the same as the meaning humans typically ascribe to language, which is derived from direct experiences and real-world knowledge. Instead, it is a form of abstract, statistical understanding that emerges from the relationships between words, their contexts, and the patterns in the data. The model essentially learns a high-dimensional representation of these relationships, which can be thought of as an approximation of meaning within the context of the model's training data.
Thus, when discussing meaning in the context of LLMs, it is essential to recognize that the term refers to an approximation of linguistic relationships and patterns rather than a direct understanding of real-world experiences. This approximation allows the model to manipulate language in a way that seems meaningful to humans, even though it lacks direct experiential knowledge of the world.
In summary, Knome's argument can be made more concrete and rigorous by focusing on the mechanisms behind LLMs, such as the high-dimensional representation of linguistic relationships, and by clarifying that the "meaning" in LLMs is an approximation derived from patterns and associations in the data rather than a direct understanding of real-world experiences.
>> If your argument is simply that symbols and numbers can't contain meaning because they are symbols and numbers, then you've worked your way into a tautological box that I, nor anyone else, will be able to free you from.
Too many assumptions.
>> Do you have some competing idea of how it works? It would behoove you to share it with the thread.
Yes, I do. It's a language model. It is trained to model the text in a corpus. There's no mystery there and there's no reason to assume it's doing anything else than what it was designed to do. It's a machine. People made it. It does what it was made to do. Other explanations of its behaviour are fantasies.
And computers aren't brains. Brains may be computers, or they may not, but you can't look at brains and draw conclusions about computers. For the same reason that you can't look at planes and draw conclusions about birds.
>> The use of backpropagation and gradient descent allows us to search the space of all possible models starting from a randomized model, and find a model that does this by nudging our model to be a little more capable at it over and over again.
If you haven't already, you should try implementing backpropagation, or even just gradient descent/hill climb, and observe its behaviour for a while. I suspect that, if you haven't done that, the experience will clear a lot of the rosy clouds about what backpropagation and gradient optimisation can really do.
Anyway this is getting a little too long-winded for me and I don't have time to continue the conversation. I must respectfully bow out. Last word is yours and thanks for the interaction.
> the experience will clear a lot of the rosy clouds about what backpropagation and gradient optimisation can really do.
I'm not seeing any "rosy clouds" in his description. Gradient descent is a search in model space for one that best models "meaning" relations in the data, among other things. We all know that GPT is not conscious, but it doesn't need to be. It seems obvious that it is recognizing abstract patterns and relations in the data that match our metaphorical comprehension of words and phrases, because it was designed and trained to do that, on human language that is filled with metaphorical relations. This is incredibly sophisticated mechanical pattern recognition with many layers.
> Other explanations of its behaviour are fantasies.
GPT is a type of deep neural network and the explanations I've seen in this thread have been along those lines.
>> Gradient descent is a search in model space for one that best models "meaning" relations in the data, among other things.
Not at all. Gradient optimisation "searches" for a set of parameter values that optimise (maximise or minimise) the gradient of a function. In machine learning, the function is some measure of error over a set of training data, that is to be minimised. There isn't anything in gradient optimisation that has to do with "meaning" at all.
What I think you're saying is that by training on a lot of text, a language model somehow learns to minimise its error on the meaning of the text. If you squint a bit, you'll see there's an obvious gap, there, between "text" and "meaning", and it seems to me you're trying to close that gap by alluding to how humans close it. But that's just comparing apples and oranges. As Yann LeCun pointed out recently (there was a video of a discussion with Andrew Ng posted on HN a few days ago) for a human to "train" on all the text that large language models have trained on, the human would have to read several books a day for some thousands of years. So whatever humans can do to learn how to bridge the gap between text and meaning, that's not how language models do it, and drawing an analogy between the two obscures more than it elucidates.
>> This is incredibly sophisticated mechanical pattern recognition with many layers.
It's incredibly complex. I don't know what "sophisticated" means. Is a jumbo jet "sophisticated"? I'd say it's a very complex machine, but that doesn't mean it has abilities other than the ones it was manufactured to have. Complexity is not magick, you know. We have maths for it and all. Those are even foundational maths of computer science. If we started imagining "meaning" everytime complexity cropped up in computer systems, we'd be up to our knees in it.
> There isn't anything in gradient optimisation that has to do with "meaning" at all.
That seems like a disingenuous interpretation of what I was saying. It should have been obvious and implied that I was referring to a loss function that was designed and engineered to choose a model that optimally captures meaning relations in the data (among other things). Gradient descent is directed by its loss function and that is not some random arbitrary function. For GPT it was specifically engineered for the purpose of selecting a model that captures meaning etc. These models are also trained with reinforcement learning which is a topic I don't know much about yet.
> it seems to me you're trying to close that gap by alluding to how humans close it.
I am talking about deep neural networks, not humans. GPT is a type of deep neural network. We do know something about how they work.
> It's incredibly complex. I don't know what "sophisticated" means.
I don't have time or patience for this pseudo-intellectual nitpicking. Just look it up in a dictionary. Basic English.
"Sophisticated (adjective) - (of a machine, system, or technique) developed to a high degree of complexity. E.g.
"highly sophisticated computer systems""
>> That seems like a disingenuous interpretation of what I was saying.
I hope not. It was not at all obvious to me what you were trying to say. It really doesn't help communication to use metaphors and analogies when discussing technical matters, especially when there is simple and commonly accepted terminology. Trust me, I've learned that the hard way (i.e. repeated rejections of my research articles with very frustrated comments by reviewers).
Besides, what you say is not right. Language models are not trained to optimise "a loss function that was designed and engineered to choose a model that optimally captures meaning relations in the data". Language models of the GPT line, specifically, are trained to maximise the following probability:
P(tᵢ|tᵢ₋ₖ, ..., tᵢ₋₁)
Where tᵢ is some token in the training corpus and tᵢ₋ₖ, ..., tᵢ₋₁ are the tokens in a "context window" of length k. That is actually the pre-training objective. The same probability is maximised in the supervised fine-tuning phase, but where tᵢ is replaced by a label and tᵢ₋ₖ, ..., tᵢ₋₁ is a sequence of tokens representing an instance with the given label.
You can find this information in the first GPT paper, here:
See equations (1) and (4) on page 3 (with more careful notation than mine).
Note that the above probabilities do not have anything to do with "meaning". They represent the ability of the model to, as people say, "predict the next token in a sequence". That is the only thing that GPT language models are designed and engineered to do.
You can search the entire GPT article above and you will not find any mention of "meaning". Nobody knows how to represent meaning and how to train a neural network on examples of it. It's much easier to train a neural net on text, of which there are no end of examples. That's what OpenAI has done.
>> I don't have time or patience for this pseudo-intellectual nitpicking.
Sorry, I meant that "sophisticated" is a loaded term. It's the kind of thing I hear often associated with socialites and gourmets. I find that it has no place in conversations about technology. It's the kind of expression people use when they try too hard to convince of the value of some technological artifact whose value should be clear without such exaggerations. So for example, you can talk about the complexity of large language models without having to say anything about their "sophistication", and that works well because one is measurable while the other is only a matter of opinion.
It's in the tile of the paper. "Improving Language Understanding"
So they call it "understanding". No problem. Understanding natural language requires building a representation of the meaning encoded in language. Otherwise what exactly do you think they are trying to "understand" about natural language here? GPT is not a spell-checker. It has a semantic model of language.
The stated goal is to optimize for "natural language understanding tasks", to use the exact phrasing in the article.
The article refers to "semantic similarity" which is precisely about modelling the meaning of language, by definition of the term.
Here's a definition of "semantic similarity"
"In the context of Natural Language Processing (NLP), "semantic similarity" refers to the measure of how closely related the meanings of two linguistic units, such as words, phrases, sentences, or documents, are to each other. The goal of assessing semantic similarity is to quantify the degree to which the meanings of these linguistic units overlap or share common conceptual features.
Semantic similarity plays a crucial role in various NLP tasks, including information retrieval, text summarization, document clustering, machine translation, and sentiment analysis. By understanding and quantifying the semantic relationships between linguistic units, NLP systems can provide more accurate results and better handle the complexities of natural language."
Yes, the word "meaning" is not explicitly used in the article, but it is implied. Ironically you are nitpicking over semantics that GPT can already comprehend, since it contains a model of the meaning of the word "meaning".
> They represent the ability of the model to, as people say, "predict the next token in a sequence". That is the only thing that GPT language models are designed and engineered to do.
GPT language models are designed and engineered to perform natural language understanding tasks. This is discussed in the paper that you linked to.
In order to predict or generate the next token, it uses a very super sophisticated semantic model of natural language and the previous tokens. You could also argue that a human is just "predicting the next token" when they generate a sequence of words while doing the tasks that GPT is designed to do. That misses a lot of important details. The devil is in those details.
>> It's in the tile of the paper. "Improving Language Understanding"
I have to confess I'm a little bit disappointed. I pointed you to the
OpenAI paper that has the technical details of how their models are trained,
and you chose to concentrate on the title, and the claims made about the
abilities of the model, rather than the way it actually works. The technical
details about the model's training make it clear that it is a language model,
trained to maximise the probability of a sequence of tokens, but you keep
believing that it can "understand"? Or represent meaning? I'm not even sure
what you believe.
The point is that you shouldn't believe. The OpenAI paper is not gospel. It's
a technical report. You should even be able to implement a system like the one
they describe, though at smaller scale. Maybe that will help you understand
how it really works? I don't know.
I think you don't want to know how things work. You just want to believe. And I
can't deal with faith. Faith makes no sense when you have a technical specification.
Sorry.
OK now I see your problem. You currently lack a fundamental understanding of neural networks and machine learning in general, and it seems that you didn't actually read the entire paper.
The loss function is involved in the training of a neural network (transformer).
THE LOSS FUNCTION IS NOT THE MODEL ITSELF.
The loss function is not a neural network. The loss function is not the transformer model. It is only part of the training step.
This is a major blind spot in your current technical understanding of this paper and machine learning in general.
The loss function (in GPT-3) is optimizing a 100 layer neural network with 175 billion parameters, and that is where the semantic relations emerge, in the model itself. The neural network is inferring these relations from the training data. Obviously there is no "meaning" defined in the loss function, and it makes absolutely no sense to look for it there. The structure to be learned is in the training data. Semantics are not explicitly coded into the model. The paper does not give you all of the technical details. The authors assume you have the pre-requisite knowledge of NLP, deep learning, and the transformer architecture.
"Semantic similarity" is not gospel. This is a standard NLP term and it is discussed in the paper, as it is one of the training objectives.
This emergent properties of transformers and other deep learning models is a fascinating thing to witness and to study. You should go back to basics in machine learning to properly understand how the loss function is only directing the training of a model, but is not the model itself, which can be incomprehensibly vast as in the case of GPT-3. You could not fit a full technical description of the actual trained GPT-3 model into any paper. The emergent properties of neural networks is a new field of research and we don't yet have a complete understanding of all of these properties or how to investigate them.
> Nobody knows how to represent meaning and how to train a neural network on examples of it.
We represent meaning with language models, and GPT has been trained on examples of it (semantic similarity). It learns how to represent meaning through the training process.
The meaning relations are an emergent property of the trained neural network, not explicitly programmed. The meaning relations are automatically learned by the machine. This is the magic of machine learning and neural networks. (I don't mean literal magic. These large language models are simply too large and opaque. You can't easily identify the specific neural activation pattern that encodes meaning of any given concept within a 100 layer deep 175 billion parameter neural network.)
"In essence, the "meaning" referred to in LLMs is not the same as the meaning humans typically ascribe to language, which is derived from direct experiences and real-world knowledge. Instead, it is a form of abstract, statistical understanding that emerges from the relationships between words, their contexts, and the patterns in the data. The model essentially learns a high-dimensional representation of these relationships, which can be thought of as an approximation of meaning within the context of the model's training data."
This might help get you on the right track.
Question:
Do neural networks have emergent properties?
Answer (from GPT-4):
Yes, neural networks can exhibit emergent properties. Emergent properties are characteristics or behaviors that arise from the collective interactions of the individual components in a system, which are not directly attributable to any single component. In the case of neural networks, these individual components are the neurons (or nodes) and their connections (or weights).
As neural networks learn from data through a training process, they adjust the weights between neurons, which can lead to complex patterns of activation across the network. The network's ability to recognize patterns, make predictions, and perform tasks can be seen as an emergent property that arises from the collective behavior of its interconnected neurons.
For example, in the case of language models like GPT-3, the ability to generate coherent and contextually relevant text can be considered an emergent property. This capability is not explicitly programmed into the model but arises from the learned relationships between words and phrases through the training process.
It's important to note that while neural networks can exhibit emergent properties, these properties might not always be easily explainable or interpretable, as they result from the complex interactions within the network.
Look, this is the internet. I'm sure you know what crazy things happen on the internet. In view of that, are you sure it's such a great idea going around telling people that you are "`stalking'" them, quotes or no quotes?
Could you think for a moment how some people might take your comment, including myself? Is that really how you want to interact with people online?
> And yet, that's what we know that LLMs do, because that's what they are made to do [edit: they do what you describe after the "by merely"]. Why do we need to imagine some other, so far unobserved, process at play? What is the motivation? What is the justification?
We also know how neurons work, interconnect, and communicate.
There is very clearly emergent behavior going on, and while neuroscientists have yielded this point for decades about the brain (any brain, even tadpoles), the AI guys still are stubbornly holding on to "we know exactly how the system works, because we know how the individual bits work".
>> There is very clearly emergent behavior going on, and while neuroscientists have yielded this point for decades about the brain (any brain, even tadpoles), the AI guys still are stubbornly holding on to "we know exactly how the system works, because we know how the individual bits work".
That's because the "AI guys" are working with computers, and not with tadpoles, or even neurons. And certainly not brains.
There is no difference unless you believe in some kind of supernatural influence in biological organisms.
The "brains are not computers" point has been moot for decades since it would require brains to be able to perform actions that cannot be mathematically described i.e. break the laws of nature. Which is to say "If a brain can do it, a Turing machine can simulate it" which pretty much everyone in the field agrees with.
You can stand on a dualistic platform and hold that the brain has supernatural abilities, but in mainstream discussions of the idea has been dead for a long time.
You can argue that computers aren't close to simulating a brain, sure, but that doesn't really get you anywhere, since the difference just becomes a matter of practical scale rather than something that escapes theory. Or even just a factor of efficiency, a synthetic consciousness might be able to run on current hardware if the right program is found. The brain might be grossly inefficient for consciousness, we don't know.
What does all that have to do with anything? We know how computers work and we can make computers. We can't make brains. That's the difference between brains and computers.
Who every said anything about supernatural influence in biological organisms? And what biological organisms? We're talking about computers. Are you sure you're not the one who is seeing ghosts in the machine, when you're talking about "emerging behaviour"?
It's possible to understand how something works at a low level, but not at a high level. Imagine an electronic machine designed by someone else, with millions of components, completely undocumented. You might understand how the resistors and transistors and capacitors and whatnot each work, but not understand how the machine itself does what it does. You can see that you give it some input, and get some output, but if someone asked you to point at the paths in the machine that cause that to happen, you would not know. You would need a long time to study it before you could understand.
We know how computers work because every part of a computer, every bit of software that runs on it, was at some point designed by a human. There's no single human that understands everything in a computer at once, but if you ask a good developer why some bit of software is not working, or what they would expect it to do given certain inputs, they can generally figure things out.
With AI, we're creating systems to design other systems automatically. We understand the process works, but the systems being designed are complex enough that we don't really understand much about them, except that they work. They are just massive arrays of numbers that feed into equations and then spit out some output. They're not like systems designed by humans, which are usually intended to be understandable by other humans -- they're weird and naturally incomprehensible. The only way to know what ChatGPT will output, at this point, is to actually run it, even if you were to take the randomness part out of it.
Emergent behavior isn't pseudoscience, it's a real thing. Sometimes when a system gets complex enough, it starts to behave in surprising ways. Maybe we can make a brain, using a computer, and this is how you do it.
What I think you're trying to say is that because we can't trace the behaviour of ChatGPT or whatever, because they are black boxes, we might as well assume they work like a brain because we don't really know they're not; and that makes no sense at all. If you don't understand how something works, you wait until you can understand how it works. You don't just start coming up with fanciful theories that have no basis in fact.
If you want to say that a machine works like a brain, then you should explain how it does that. If you can't do that, then there's no reason for anyone to believe it, and there's no obvious reason for you to assume it. So at least explain why you think that ChatGPT works like a brain, if that's what you really think.
ChatGPT is a deep neural network. We know that such neural networks construct a hierarchical model of the data. The fact that it is recognizing abstract "meaning" relations in that model doesn't seem particularly controversial to me. We build deep neural networks specifically for pattern recognition. It is similar to a brain in some aspects but obviously not the same.
I asked GPT-4: "Does the neural network in GPT-4 build a hierarchical model of the data? Does it derive metaphorical "meaning" from the symbolic data?"
Answer:
As a transformer-based neural network, GPT-4 does build a hierarchical model of the data to some extent. Although the hierarchy in transformer models like GPT-4 is not as explicitly structured as it is in convolutional neural networks (CNNs) used for image processing, the model does learn to encode complex patterns and relationships in the input text data through its multiple layers and self-attention mechanism.
The transformer architecture, which GPT-4 is based on, is designed to capture long-range dependencies and relationships in the input data. The self-attention mechanism allows the model to weigh the importance of different parts of the input in relation to each other, which helps it learn and capture semantic and syntactic structures in the data.
Regarding deriving metaphorical "meaning" from symbolic data, GPT-4 can identify and understand metaphors, idiomatic expressions, and abstract concepts to some extent, as it has been trained on a diverse range of text data. However, it is crucial to note that GPT-4 does not "understand" meaning in the same way humans do. It is exceptionally good at identifying patterns and generating text that resembles human-like understanding, but its "understanding" of meaning is more akin to a sophisticated pattern-matching process rather than true comprehension.
In summary, GPT-4 does build a hierarchical model of data to some extent and can generate text that exhibits an understanding of metaphorical meaning. However, this understanding is based on the model's ability to recognize and generate patterns in the data rather than genuine comprehension of the underlying meaning.
You seem to be tripping up on an imaginary difference between computers and brains. There really isn't one and 99% of the field agrees that brains are just computers.
The only out you would have is to claim that there is something supernatural that separates the fundamental operations of a brain from those of computation, i.e. given an computer the size of the universe, with billions of CPUS and GPUS to simulate each neuron of a brain, that you still would not be able to simulate one. That there is something supernatural that even infinite regular computation could never capture.
If you don't take that stance, then the only line you are drawing between brains and computers is their compute performance for a given task (even if that task is "be conscious").
This is (perhaps) similar to the difference between regular and quantum computers. There is nothing a quantum computer can do that a regular computer can't, the difference is just in their relative performance for given tasks.
>> You seem to be tripping up on an imaginary difference between computers and brains.
No, you're mixing up computers with computational devices. A "computer" is what I have in front of me now, as I write these words. My brain is not a "computer" it's maybe a computational device, maybe something else entirely. We have no idea how brains really work. And there is no consensus like the one you try to claim in any field. I mean "99% of the field?". That sounds more like an advert for toothpaste. "99% of dentists recommend Whitex!!".
>> The only out you would have
>> If you don't take that stance
Get off your high horse and stop god-moding the conversation. You are trying very hard to invent a hill for me to die on based on something you'd really have liked me to say, just so you could pretend to win the conversation. The only person who has said anything about anything supernatural in this conversation is you. If you can't imagine why else I might disagree with you, that's your lack of perspective and nothing to do with me, or what I think.
> There really isn't one and 99% of the field agrees that brains are just computers.
Again, this is blatantly false. There is no such consensus on the computability of consciousness. Where do you get that 99% from?
> That there is something supernatural that even infinite regular computation could never capture.
You are asserting the universe is equivalent to a Turing machine (digital physics hypothesis). There is no consensus on that. It's an open research topic.
The idea that the universe can be simulated on a Turing machine to a sufficient accuracy is not the same as the digital physics hypothesis, and it's currently the vastly dominant position amongst physicists.
If our current understanding of physics is accurate, then yes the universe can be sampled by a probabilistic Turing machine to any required precision. For this to be false would require a new and fundamental discovery in physics.
Wether or not that means consciousness is computable is an ontologically disconnected argument.
"Wether or not that means consciousness is computable is an ontologically disconnected argument."
How is that ontologically disconnected? You forgot to provide an explanation. You wrote something interesting and I would like to know why you think that.
My understanding is that consciousness and physics being ontologically disconnected means that there is no direct causal relationship between the two.
Ah, that's not what I mean by ontologically disconnected. I just mean that since consciousness is not defined, we don't know for sure that it's a purely physical process. I certainly think so, but the argument I was making can't prove it is or isn't, so it's not an argument that can touch consciousness by itself, because you need an additional assumption to bridge from known physical processes into consciousness that isn't in the argument I presented. In other words even after accepting my argument a skeptic could hold the position that it's ontologically disconnected, so at the level of my argument it remains so.
To be even clearer, I just meant that my argument is not sufficient to prove that consciousness can be simulated because I didn't make a good argument for consciousness to be in the category of physical processes in that comment, hence, an ontological disconnection.
I personally have the position that they are in the same category, I just meant that my comment above wasn't sufficient to substantiate that claim.
[ this comment was written in response to a user who appears to have deleted their comment before I could finish putting my response together. I'm not sure why they deleted it, it seemed reasonable in the face of the ongoing discussion in this thread. I'm going to leave my response here, partly so they can see it if so interested, and mostly just because I already spent the time to write it :) ]
I fail to see any distinction between your description of "mapping terms to latent spaces" and my assertion of GPT "working at a level of meaning and intent".
I fail to see any distinction between "It's using complex non-linear transformations to map observed text to an extremely high-dimensional latent space, and in addition learning to both encode text to this representation and decode it from this representation back to the source language" and my assertion that "the literal meanings are abstracted and re-encoded".
I feel like you're disagreeing with my stated positions because I am expressing them in a lay anthropomorphic language to express its functioning, but that we're otherwise agreeing?
I had initially been responding to the user dvt, who said
>LLMs work on tokens, which, for the most part, are words (longer words are generally split up into multiple tokens). Using concepts like "meaning" and "intent" muddies the waters unnecessarily.
this seemed like an incorrect characterization of the process as I understood it. he doesn't acknowledge the existence of what you refer to as latent spaces, and seems fixated on GPT as merely moving around tokens (though I may be reading his position incorrectly, of course). I was arguing a claim of GPT's ability to represent concept distinct from literal meanings of terms, and for "understanding" how to transform these abstract meanings into different contexts.
As far as I can tell, I don't really disagree with what you're saying at all.
> Pretending GPT is just using a percent chance that some token should show up next is absurd. It's almost a hundred layers of neural pathways.
That is literally what it's doing though. "Pretending a CPU is just adding 0s and 1s is absurd. It's made out of billions and billions of transistors."
A multi-layer neural net is not limited to vague statistical combinations.
You can represent XOR in a neural network. A NAND. If you can represent NAND. You can represent arbitrary logic. Across billions of weights and across nearly a hundred layers of possible decision points. Meaning can be spread across the neurons in ways that allow each to be part of many many different representations. This is only in GPT-3. GPT-4 and soon 5 no doubt exceed these considerably.
> A multi-layer neural net is not limited to vague statistical combinations.
It, absolutely, unequivocally, 100% is only limited to statistical inference. That's how it works. The problem of linear separability (which is like 60 years old, by the way) has nothing to do with logic, but it's actually a byproduct of (a) the fact that neural networks use linear activation functions, and (b) XOR is not separable by a single hyperplane. Hence, you need multiple layers.
No idea how you get to "arbitrary logic" from this.
Exactly. Pretending our own brains are much more sophisticated seems increasingly a hard to defend position. Apparently, this is good enough for a lot of stuff we think we are good at already.
> LLMs work on tokens, which, for the most part, are words (longer words are generally split up into multiple tokens).
Although this is mostly true, there are exceptions to the rule. Look at the ByT5 family[1] of models for instance. These models operate on byte level representations of text and are inherently token free. I read a recent paper[2] that found that replacing token based text encoders in text to image models with these byte level models improves the spelling abilities of the txt2img models (which are otherwise notoriously bad at spelling in all human languages).
I feel like if meaning itself was simply contained or even encoded in the words we write, the world would be a very different place to us. How could we ever find ourselves, as we do, arguing about the meaning of a text? Even then driest prose can sometimes lead to different interpretations.
Is there "meaning" somehow inside the novel on my desk? Can you extract it and manipulate it? If I had no body, environment, temporality, what even could the book mean to me? Even more, if nobody could teach me the meanings of the words themselves in the book, what could it ever be to me but a collection of certain patterns?
but it is exactly human-kinds ability to put meaning to words that make us unique. That give us language and communication, artistry and music. Words mean nothing without meaning. It's not encoded, it's inferred. There's nouns, adjectives, verbs, etc and some of them are tied to the same word. That's very important. The chain of words derives it's meaning in context so having a system that operates on meaning is, dare I say, meaningful. It's the whole point.
I'm thinking in the other direction. Of course context reveals (or as in your philosophical exercise, defines) some of the meaning of the word. But how much of the meaning is context-dependent depends on the word and the context itself where it's being used.
For example, consider a word that means two different things in two different contexts that would otherwise be very far apart in whatever latent "meaning space" they might exist in. A naive model might end up averaging across both contexts, resulting in a representation of that word that represents neither meaning of the word, being located vaguely in between both regions of space. Alternatively, this could pull "groups" of words that otherwise are not related into adjacent regions of space, creating the appearance of semantic similarity when there shouldn't be any. In this example, the physical surrounding word context is actually misleading, and addition information is needed in order to make the learned embeddings more realistic.
In any case, it should be obvious to people trained in statistical practice that word models do not encode "meaning" per se, but rather an estimate of a model of meaning, which is not a complete model, but it does a pretty good job most of the time. We as the experts should be careful to stick to language along these lines, so as not to promulgate fanciful ideas about AI. I am not even going to bother with the "it's not really intelligent!" angle here. For all I care, the model is fully sentient. But the fact is that the token embeddings do not encode "meaning", it encodes "an estimate of model of meaning". They are our best effort attempt at encoding meaning parsimoniously, but that is not the same thing as meaning itself. Map is not the territory, all models are wrong, etc.
This is word salad. I don't even know what you mean by "meaning," but an embedding is simply an array filled with a bunch of floating point numbers[1] and, importantly, these embeddings are trained, so "doctor" will have an embedding similar to "hospital" because it's seen in close proximity in the training data.
The length of the number array is referred to as dimensions for math reasons. That's half the statement right there.
> importantly, these embeddings are trained, so "doctor" will have an embedding similar to "hospital" because it's seen in close proximity in the training data
It is trained to put together words with similar... meaning. So the embedding is a representation of... meaning.
> It is trained to put together words with similar... meaning. So the embedding is a representation of... meaning.
This might be waxing philosophical, but my position is that the meaning of a word is intrinsic, not extrinsic. The meaning of the word "ball" has nothing to do with the fact that it's next to the word "net" on a page (which is what LLMs do). But rather, "ball" is the round rubbery thing I kick around in my back yard.
Locality is just how the embeddings are created, and at that only one way. “Ball” will also be closer to “round” and “net” will be closer to “string”. So this is the creation process, but you end up with something that sure does look a lot like meaning. You can do math with these vectors, like addition. king - man + woman = queen is the classic Word2Vec example.
> but you end up with something that sure does look a lot like meaning
I suggest you read some Quine and Kripke; the argument that word embeddings encode meaning (particularly via locality) is an incredibly philosophically-naive position. People have been debating what language is for a very long time.
Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago?
Maybe the thing that will land is to say that embeddings are an attempt at representing meaning.
There’s lots of other stuff you could represent about words. You could represent popularity by giving each word a number between 0 and 1 representing what portion of the training text was that word. That would tell you if a text might be harder for someone new to the language to understand. You could represent the spelling of the word as a list of integers. You could represent the pronunciation difficulty with a score based on ambiguous and infrequent n-grams. Or, you could try to represent the concept that the word is pointing to so you can follow synonyms or categorize text by topic. This would be an attempt at representing the word’s meaning.
I’ve been explaining as if you didn’t understand what this was, but if you disagree that’s another thing.
>> Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago?
From the conversation so far it's clear to me that the OP has at least some idea of how word embeddings work and they described as "word salad" the expression that related them to "meaning", i.e. they quibbled about the "... representation of meaning", not the "high dimensional representation... " part.
I hope the OP can give a reference to the Quine and Kripke source they refer to.
As far as I remember it, the point about arithmetic with word verctors (man, king, woman, queen and all that) is a motivational example for Word2Vec by Tomas Mikolov, which wikpedia tells me was published in 2013. I can't find the reference right now, but in 2014 I was taking a Master's in all that jazz and I remember one of my tutors having the paper on his desk, and commenting that the claim that word embeddings model meaning was "well, that's what he says". So there has been debate and disagreement about the ability of word embeddings to represent meaning for at least as long as Word2Vec has existed, and as far as I know the idea of word embeddings existed earlier than that (I was taught the concept unconnected to word2vec in my Master's).
The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion.
So I, for one, want to see that Quine and Kripke stuff the OP is referring to.
> The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion.
That critical discussion was 10 years ago. Embeddings are vector representations of meaning. They’re not perfect, but they can be used for enough meaning-dependent tasks that this statement shouldn’t be controversial.
All this AI/ML stuff is so exhausting because the philosophers come out of the woodwork. They don’t prove anything or disprove anything, they just kinda whine.
When I was a flight instructor and said the airplane wanted to float down the runway, no one tried to point out that airplanes don’t have desires. I can say a server knows another server is offline and that’s fine too. But say “reason”, “think”, or “meaning” with something AI related and it’s like all work has to stop until we stop using that precious word.
I’ve tried to engage with that stuff less, but got tricked cause I thought this person just literally didn’t understand.
>But say “reason”, “think”, or “meaning” with something AI related and it’s like all work has to stop until we stop using that precious word.
The unwillingness to admit to AI capabilities because they are expressed in terms of human capabilities is what I deem an argument that "boats can't swim".
True, in that what it does is different, but irrelevant in almost every facet that most people discussing it would care about. The boat, if anything, "swims" better than humans in basically every way. The argument against its use is only that swimming doesn't refer to boats.
(I admit I might have picked this argument and phrase somewhere, but I did recently a cursory glance back through old comments, and think I might have coined it based on having then recently learned that Russian uses the same term for swimming and sailing, which I had found interesting at the time. But perhaps not, my first use appears almost a decade ago, and my mind is ever forgetful)
Ah, fantastic and thank you. I had suspected I picked it up somewhere, I just couldn't find any proof of it. My previous failure to find the origin was likely because I was searching for "boats" rather than "submarines".
I hope you're not calling me a "philosopher". I identify as a coder.
And what I want to know is where is the code for "meaning". I don't care why it's the code for "meaning", but if you point me to the code for "word embeddings" and you say "that's the code for 'meaning", then I'm going to wonder whether, if I ask for the code for "quicksort" you'll point me to the code for "bubblesort".
So, no, what we call things that do things is important, otherwise we don't know what things do what, and what things are being done.
And lest we forget:
However, in AI, our programs to a great degree are problems
rather than solutions. If a researcher tries to write an
"understanding" program, it isn't because he has thought of a better
way of implementing this well-understood task, but because he
thinks he can come closer to writing the implementation. If he
calls the main loop of his program "UNDERSTAND', he i s (until
proven innocent) merely begging the question. He may mislead a lot
of people, most prominently himself, and enrage a lot of others.
What he should do instead is refer to this main loop as
"G0034", and see if he can convince himself or anyone else that
G0034 implements some part of understanding. Or he could give i t
a name that reveals its intrinsic properties, like NODE-NET-
INTERSECTION-FINDER, it being the substance of his theory that
finding intersections in networks of nodes constitutes
understanding. If Quillian <1969> had called his program the
"Teachable Language Node Net Intersection Finder", he would have
saved us some reading. (Except for those of us fanatic about
finding the part on teachability.)
In AI, there are lots of people who say lots of things. They keep saying them, even when people point out that they are just things that they say. They continue to say them even long after the other people get tired and give up trying to make sense of what is being said. That doesn't mean that the "discussion" is over, it's just that there is no real way to stop people saying things if that's what they really want to do.
I can point at code that sorts words by use pretty easily. And I can show you how the embeddings correlate words with similar meanings. Surely that answers the question?
Remember that the real claim is that embeddings represent meaning. In a flawed but working way. You could say they are a map, not the territory, so arguing about the true meaning of meaning isn't relevant.
> I'm going to wonder whether, if I ask for the code for "quicksort" you'll point me to the code for "bubblesort"
Embeddings are meaning (noun) like a dictionary, not meaning (verb) like a qualia.
The code sorts. It doesn't matter what algorithm because the claim was just that it gives you sorted data. Just like a claim that embeddings represent things is not a claim about methods or qualia.
I was going to say something similar. The meanings aren’t code, they’re data. So you can definitely pull up the vector for “meaning” or anything else
The claim (that I’m making at least) is not that these systems can understand meaning. It’s that they can store a representation of meaning and do something with it. I think there’s a way lower bar for this. (I also think the bar was cleared quite a while ago and we should consider it settled.)
I could write down the meaning of a few words on some pieces of paper. (I guess if someone disagrees with that then talking about vectors is pointless). I could stuff each of those papers into a different dog toy and train a dog to get them on command. Then I think you could truthfully say something like “this dog can retrieve the meaning of the word ‘independence’, ‘hammer’ or ‘sing’ for you”. It wouldn’t be true to say the dog understands any of those words.
So in other words, you're saying what you said above:
>> Maybe the thing that will land is to say that embeddings are an attempt at representing meaning.
And I don't disagree with that, neither with what you say in this comment.
Or at least, I think I agree. What I would have said is that you can write down words on a paper, and the words are a representation of meaning, but the paper doesn't understand meaning, nor do the words. You need a human, with a human understanding of words, and language, and what those things mean, in order to decode the meaning from the words. In other ah words, the words on the paper are a representation of meaning, but only for a human. For a cat, say, they don't represent anything.
Wat's more, just because a language model is trained on word embeddings, doesn't mean it encodes meaning. It's certainly an attempt to do that, but just because someone made an attempt doesn't mean they've done it.
>Or at least, I think I agree. What I would have said is that you can write down words on a paper, and the words are a representation of meaning, but the paper doesn't understand meaning, nor do the words. You need a human, with a human understanding of words, and language, and what those things mean, in order to decode the meaning from the words. In other ah words, the words on the paper are a representation of meaning, but only for a human. For a cat, say, they don't represent anything.
If you change your perspective on language to mean any arbitrary capture of useful information (such that it can be used in the future), then you can see that the boundary between words and the world is not the heart of the issue. For example, your perception of the world works in a similar manner in that your sensory organs cannot comprehend the world, almost like your sensory organs interpret the world using their own language. Or maybe if that example is not very intuitive, then how about imagining an alien species that has sensory organs that act on linguistic structures. In some way, the aliens will figure out a coherent structure of their own, even though they cannot "experience" the world through senses like ours. What "intelligence" is doesn't seem to be bound by how "close" someone is to reality, and "close" might not even be the right word, since different perceptions can have different capabilities. I think the tricky part about "intelligence" is that there is always some "meaning" captured, it is just alien to those who do not share the same interpretive capacity. A cat could extract meaningful information from words on a paper, but certainly not in the same way we do.
Now if we want to make an AI that acts and thinks like us, then understanding our own machinery (the relationship between the world and language) is certainly important. But I think the bitter lesson rears its head here, and I believe that something that is truly worthy of being called AGI will be able to thrive given any set of arbitrary senses, even linguistic ones. In other words, I do not think embodiment will naturally lead to AGI, rather that embodiment is a necessity if we want make AI in our image. And making an AI in our image is the fastest way to get an AI to do useful work for us.
>> If you change your perspective on language to mean any arbitrary capture of useful information (such that it can be used in the future), then you can see that the boundary between words and the world is not the heart of the issue.
I don't think that it is. What I think is that language encodes meaning, that can be decoded only by an entity that knows how to decode meaning from language; which is a bit of a tautology, but that's the point, you can't expect an entity without the ability to decode meaning from language to understand what language means. Or at least I don't expect that.
Here's an analogy, and I shouldn't be making it because it's about cryptography and I'm not an expert there. Suppose I sent you an encrypted message and you knew how to decrypt it- I encoded it with your public key and you decrypted it with your private key, or whatever. If you could do that, then you could read my message and know what it says.
If you couldn't decrypt my message, then you could stare at it for as long as you liked, you could make copies of it, you could make variations of it, you could even learn a model of the structure of encrypted messages like mine, and be able to produce many more of those messages, but you would still not know what those encrypted messages say. Because they're encrypted and you can't read them, you can only read their encoding.
That's what I'm driving at. I think that's how language works, in practice. Not that it's some form of encryption, but that it only makes sense to humans, so only humans can decode meaning from language. With language modelling, we're reproducing the encoded message, but we haven't yet found out how to equip the machines modelling language with the ability to decode meaning from it, so for all intents and purposes it might just as well be encrypted.
I'm not arguing for embodiment, either. I'm perfectly fine with the idea of a "brain in a jar". But I agree that if we want our AI's to behave like humans, they will have to have some experience in the world.
If I can make convincing messages based on the structure and swapping around pieces of data, then it sounds like I got past the encryption well enough to have a partial understanding of the plaintext.
>> If I can make convincing messages based on the structure and swapping around pieces of data, then it sounds like I got past the encryption well enough to have a partial understanding of the plaintext.
I think that no, because you can manipulate the structure of an encrypted string and swap around pieces of it etc, without having to decrypt it.
As to making convincing messages, as you say, the entity making the messages and swapping around the data is a language model, but the entity reading the generated messages is a human. It is the human that finds the messages "convincing" as you say. We have no doubts that humans can decode meaning from text, even if we have no idea how we do it, yet. The question is whether language models can do the same thing. And what I say above is that there is no indication that they can, because all they do can be done by language generation, without any decoding of meaning needed.
I split the idea into “attempt” when they didn’t seem to understand what was being talked about at all. After that, you could discuss if the attempt was successful. I think that because these systems can solve problems that involve the meanings of words, we can say there must be some meaning information encoded in the data.
> You need a human, with a human understanding of words, and language, and what those things mean, in order to decode the meaning from the words.
By this definition I think you’ll always have doubt about these systems.
Not necessarily. I'm describing the current situation, as I understand it. There's nothing stopping a future system from having the same abilities as a human to decode meaning from words, or to do anything else. But I for one would like to know exactly how such a system would work, and why it would have the capabilities it would have. I don't think that's a particularly high bar to clear, for technology and science. The whole point of both is to figure out how things work, and how to make them work like we want.
Embeddings can go beyond a contextual definition of meaning.
You can use (transformer) LLMs to get a "word sense" aka its predicted probability distribution, which also induces a network of associations (not via context, but via whatever the LLM has learned as fitting words). This gives a structural definition of meaning the ye olde sense (v. Humboldt and so forth).
This of course starts with the assumption that LLMs can encode meaning, albeit not in a simplistic linear fashion as Word2Vec or GloVe did
More fundamentally, until someone has told you that "ball" can mean "a round thing" or pointed to a ball and said that "ball" can mean that, "ball" does not have that meaning to you.
The meaning is purely extrinsic. There's nothing about a ball that makes it named that, as shown by the fact that a ball is called other things in other languages.
The notion that the meaning is intrinsic is hogwash.
A word without someone ascribing it meaning means nothing.
A word has meaning because we give it to it, and we give it that meaning based on context.
Some of that context is remote in time and space (we learned the word ball a long time ago), some is near (that it occurs near "back yard" and "round rubbery thing" makes it more likely it's that thing we use to play games rather than a testicle or a party where people dance), but it is wholly dependent on context.
Put it alongside words in a different language, for example, and it might have a whole other set of potential meanings.
If I understand correctly, that's what the OP, and others, are claiming, that it's possible to figure out how to represent meaning by putting words next to each other, if you look for long enough, at many enough words sitting next to other words. So far this remains a matter of debate.
The objection I though you were making, with the "ball" etc, is that all these words that sit next to other words according to what the words mean, must mean something in isolation. For example, even though "ball" can be used in different contexts, it can only mean so many things, and it would never mean, say, "a cooking utensil made of aluminum where I fry my eggs each morning" no matter what other words you put it next to.
And, as Young et al (1976) have demonstrated, it is possible to move words out of their expected context for great fun and profit, but it's not clear that a word can always be moved in any context, and still make sense. I would even go so far as to say that can probably not be done, at least not without a drastic reconfiguration of all of English (where "ball" is used).
So the question is not only "what other words do we find the word 'ball' close to" but also "why do we find 'ball' next to those other words?". The latter question can't be answered just by looking at what words hang out with what other words, unless we already know what words mean on their own.
And that is, at least for me, the objection about LLMs "meaning" and "understanding" anything. No matter what text an LLM is generating, the entity that is decoding the meaning of the generating text is always a human. The LLM can't do that on its own. Because there is no mechanism that it is equipped with that could ever do that.
P.S. "words hanging out with other words" are what are known as "token collocations" in technical jargon. They are an idea as old as linguistics itself, a basal concept whose modern implementations in NLP systems are just, well, modern implementations. We still don't know how humans decode meaning from token collocations even though it's such an ancient idea. And it might even be wrong, at this point.
_________________
Bibliography
1. Angus Young, Malcolm Young, and Bon Scott (1976) Big Balls. In Harry Vanda, George Young (prod), Dirty Deeds Done Dirt Cheap, Albert productions - Atlantic Records, Sydney, Australia, side 2, track 2. URL: https://youtu.be/4WwJ6OVSwkM
> If I understand correctly, that's what the OP, and others, are claiming, that it's possible to figure out how to represent meaning by putting words next to each other, if you look for long enough, at many enough words sitting next to other words. So far this remains a matter of debate.
The fact that we can have conversations with GPT where these models are able to correctly e.g. write working code and symbolically evaluate code is sufficient proof of that assertion that it is possible to figure this out that I don't think there is any reasonable basis for debating this any more. Claiming it's not possible is just demonstrably wrong.
> And, as Young et al (1976) have demonstrated, it is possible to move words out of their expected context for great fun and profit, but it's not clear that a word can always be moved in any context, and still make sense. I would even go so far as to say that can probably not be done, at least not without a drastic reconfiguration of all of English (where "ball" is used).
I don't see what this has to do with anything. Yes, we can shift around, alter their meanings, use them in unusual contexts and still make sense of it, and yes of course they won't always make sense in every context. How is that relevant?
> The objection I though you were making, with the "ball" etc, is that all these words that sit next to other words according to what the words mean, must mean something in isolation. For example, even though "ball" can be used in different contexts, it can only mean so many things, and it would never mean, say, "a cooking utensil made of aluminum where I fry my eggs each morning" no matter what other words you put it next to.
If I use it to mean "a cooking utensil", then it means a cooking utensil when I communicate with others that have that shared context. It has no meaning separate from context.
> So the question is not only "what other words do we find the word 'ball' close to" but also "why do we find 'ball' next to those other words?". The latter question can't be answered just by looking at what words hang out with what other words, unless we already know what words mean on their own.
"What words mean on their own" makes no sense. No word has a meaning separate from context. Without context there's nothing to assign them meaning. And that context is interactions. For GPT purely words. For us, words and other sensory input. In either case, without context assigning attributes to the word "ball" it is nothing more than a meaningless a sequence of letters. What does "JMw3rfd" mean? What's its intrinsic meaning? Until I tell you that this is my new word for "ball" it has none. Once I do, it has an extrinsic meaning. It's one that isn't widespread and will soon be forgotten, but the meaning exists the moment we label it, and only once we label it, and it is purely extrinsic.
> And that is, at least for me, the objection about LLMs "meaning" and "understanding" anything. No matter what text an LLM is generating, the entity that is decoding the meaning of the generating text is always a human. The LLM can't do that on its own. Because there is no mechanism that it is equipped with that could ever do that.
We don't know enough about human reasoning, and so how close how LLMs are to how human reasoning works to be able to even begin to determine whether this is true or false.
>> The fact that we can have conversations with GPT where these models are able to correctly e.g. write working code and symbolically evaluate code is sufficient proof of that assertion that it is possible to figure this out that I don't think there is any reasonable basis for debating this any more.
Then there's no reason to continue this conversation.
I propose a ban on all arguments involving LLMs that boil down to “your usage of this word doesn’t match my personal connotations for that word”. It’s fine to ask someone to be clarify their definitions, but every other discussion of these models becomes a pointless debate about semantics. See, for example, below.
Has this been rigorously considered or is it just vibes? Seems preemptive to presume that unveiling more structure wouldn't give understanding meaning and intent in AI systems a leg up.
1. GPT-3 can speak native-level french despite the language only being 1.8% of it's training corpus.
2. GPT-3 can follow instruction in other languages just fine despite the instruct tuning data being almost entirely in English.
3. LLMs trained on code demonstrably reason better.
It's fairly obvious pre-trained LLMs aren't really operating at just the word/token level.
Part of the breakthrough with LLMs has precisely been about encoding nothing at all about language and letting the transformer figure it out from scratch.
We don't know what is necessary for learning language. Whatever structure we would add at our current level of understanding would far more likely hamper progress than anything. It would also be extremely surface level which LLMs go far beyond.
Meaning and intent can only be applied if structure can be captured. I'm not proposing building some god-machine that has a library of all possible regular expressions. I'm merely saying that it could be helpful to nudge something like a neural network towards looking for these types of structures, such that they can be a basis for probabilistic encoding. It may be possible that the meaning that gets abstracted thereafter might actually be more concise and generalized.
Part of the breakthrough with LLMs has precisely been about encoding nothing at all about language and letting the transformer figure it out from scratch.
We don't know what is necessary for learning language.
My hunch is that all probabilistic methods eventually degrade and they cannot capture "rules". Otherwise you would end up with a classical computer. I think what we are really looking for is a way to learn "rules" such that they remain stable over a certain number of iterations. Hopefully enough iterations to do useful work. Once you have a way to do this, you can extend the number of iterations by further imprinting on the system. The simplest example of this is if you have an oscillating signal whose amplitude is slowly decaying. A certain number of cycles will have an amplitude above a certain threshold that allows for useful work.
Would an LLM predicting the next bit rather than the next token have any advantages over GPT? In some ways it seems more general - for example it can predict an image or even executable machine code. I'm guessing that it's more difficult to train however?
GPT’s vocab size is north of tens of thousands. Included in this vocab are the 256 unique byte values (2^8). This means GPT can already generate/interpret all byte sequences.
The vocab size is larger than 256 (or in your example 2) because some sequences of bytes occur much more frequently than others. This allows the model to generalize more efficiently.
By analogy consider that we think of “words” as atomic concepts much more than we think of the individual lines that make up the letters in a word - even though the former can be thought of being composed of the latter.
I've seen you pop up and recommend Hopf algebras and your paper in several deep learning threads here on HN. I am interested in your work, but find the topic very opaque and unintuitive (despite a background in physics and CS). Do you have any recommended educational resources for a gentle introduction to this?
If you're studying the basics and want to sorta grok the general concepts by repetition, I had gpt-4 write a technical paper about a fantasy magic system based upon Hopf algebras.
Pretty cool stuff and helped me grok some of the more difficult back and forths from a tutoring session w/ GPT-4.
Also the particle appendix is really cool and I wish we had the whole elemental table. Lots of self-consistency in what the algo wrote and I'm pretty impressed, though of course I'm sure it's maybe best to treat it as a leaky bucket of information while learning it.
Join the discord is a charmingly opaque answer to the question of where to find learning resources for an opaque topic. I'd guess OP was looking for something like youtube tutorials or short "programmer's guide to..." type resources.
Obviously an early line of research won't have anywhere near the depth, but as an example the geometric algebra community has done a great job of this. Lots of 10 minute videos to 2 hour lectures. Steve Brunton's videos are another example "learning ML math" resource that has helped me a great deal.
The thing that really boosted the GA community (I assume you mean bivector) engagement was the discord channel which was started by yours truly. You can check out the references in my paper.
Then good luck building community around Hopf algebras, you certainly know the roadmap better than I. The internet is an amazing place to spread word on better mathematical tools.
Just to let you know, scam alarms are going off in my head. You comment offering what seems to be an alternative view/opinion, and promoting your own work and research papers. When asked for more details the only thing you do is ask people to join your Discord. Although given the context, I have a hard time seeing how it would actually be a scam. Just be aware of how this appears to others.
I did not get much info at all from joining the Discord and had to learn it on my own, but the discord has some interesting info.
That said, Adam tends to follow a very precise and measurable interaction selection path with many people regarding some of his work and the discord, so I also just see it as how he interacts with others.
But yes, agreed, I wish I had a human-understandable resource that a mortal of my own kind of flesh could more easily grok. Even the 'for absolute beginner' online tutorials are a swamp of symbology and prior assumptions, and it doesn't even seem to be that terribly difficult of a concept on the whole, at least for the basics of it.
Just my 2c, lots of good and mixed for us all in this crazy road we call life! <3 :D :)))) :D
I argue that Hopf algebras, which are spicy tensors, provide a better autodiff and are a better model for SSMs, convnets, transformers, and diffusion models.
How do we use them in place of transformers? Would they allow something like the power of ChatGPT but with interpretability or modularity? Would they make it easier to create systems that behave like hybrid architectures or automatically combine different types of models (algebras?)
It sounds so promising. But I have to assume that if it really is then this can be translated to code that can run on Colab or something, at least by some genius. Until it is I have to be a little skeptical and I'm not going to be able to access it anyway.
Because it would show that all the models are just slightly different Hopf algebras. And yes to interpretability and modularity. It would provide guidance for building new architectures.
"This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (not results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models."