Thanks for sharing your hunch. As you say most people fall into camps; LLMs are just next token predictors or fully-fledged conscious beings. Such a disruptive novel technology is never going to fit into any neat descriptions. As we grapple to understand them, hunches and open debates are exactly what we need.
Very interesting, thank you for this essay. I just published a long form yesterday where I argue that LLMs are essentially a fossilized slice of human thought output.
Therefore I think “intelligence gateways to our collective intelligence” would be a better descriptor that “AI”.
I agree that "collective intelligence" is a better descriptor than "artificial intelligence." It all came from us, and it pulls away the need for analysis about sentience pretty handily, at least for now. True AI would be an entirely different beast.
I challenge the notion that if we humans built something, then we must know what it is. Setting Internet kooks aside who think firing up CERN's HLC will create a black hole that eats Switzerland, the Manhattan Project had top thinkers like Edward Teller who questioned if testing an atomic bomb would incinerate the atmosphere of the entire earth.
We build a lot of things not knowing the true outcomes. It's even more unpredictable now, because the adoption of something like AI by millions of people creates additional layers of complex, unpredictable phenomena and outcomes. As much as I distrust Mark Zuckerberg, I don't think he created the news feed and added post-sharing features with the awareness it would be used by Myanmar for targeting a genocide against Rohingyas.
The uncertainty is core to the design of these AIs. Designing an effective neural net (the building blocks of LLMs) requires a balance between specialization and generalization tuning. In the 1990s, neural net developers realized overtraining was a problem: systems became too rigid and brittle in their responses.
In the opposite direction, using fewer nodes in the net lead to generalization, which is generally good inference. It is what enables an AI to interpret between strings of tokens in completely unrelated documents let alone fields of knowledge. The trouble is that we can over-index for it. which leads to more hallucinations -- which is effectively bad generalization.
Point is, these systems are designed to produce what appear to be rigid results but under uncertain circumstances. The uncertainty is the point... it's not something to be locked down with explicit certainty under a microscope.
Now Dario is one of the AI leads I trust most. Unlike the others, I don't feel as strongly that he falls trap to the old Upton Sinclair quote: "It is difficult to get a man to understand something, when his salary depends on his not understanding it."
But Mechanistic Interpretability, or mechinterp, is one of the biggest scientific research boondoggles in AI today. It's overly reductive premise is "AI is like brains, therefore we can use electron microscopes on them to understand them like brains".
There are multiple fail levels here:
1) Silicon chips shuttling bits around do not work like brains. This is a physicist's "assume a spherical cow" joke.
2) Even if 1) were true, brain function -- like LLM function -- is a systemic response. Putting a lone pine tree under an electron microscope doesn't tell you how the forest is evolving. Similarly, Finite Element Analysis will not explain the human brain (I know, I tried in the 90s in grad school).
3) To even create the conditions for reductionist, direct causality in mechinterp, many of the successful learnings have required forcing what are called "monosemantic" neural net models: models where a single neural net node maps to a single data feature for explainability. These are models I called "overtrained" above, lacking generalization. Effectively, they are recreating the "grandmother cell" fallacy in neuroscience (https://en.wikipedia.org/wiki/Grandmother_cell), and their research is so far removed from the designs of LLMs we actually want to use it is effectively inapplicable.
As for consciousness discussions, I'm baffled why the natural world gets completely thrown under the bus and we seem to only have these discussions once text spits out of silicon. It feels like when Saudia Arabia granted citizenship and rights to the robot Sofia in 2017, before women in the country had a legal right to drive and robots had more rights than they did.
Thanks for sharing your hunch. As you say most people fall into camps; LLMs are just next token predictors or fully-fledged conscious beings. Such a disruptive novel technology is never going to fit into any neat descriptions. As we grapple to understand them, hunches and open debates are exactly what we need.
Very interesting, thank you for this essay. I just published a long form yesterday where I argue that LLMs are essentially a fossilized slice of human thought output.
Therefore I think “intelligence gateways to our collective intelligence” would be a better descriptor that “AI”.
Would love your thoughts on that.
I agree that "collective intelligence" is a better descriptor than "artificial intelligence." It all came from us, and it pulls away the need for analysis about sentience pretty handily, at least for now. True AI would be an entirely different beast.
I challenge the notion that if we humans built something, then we must know what it is. Setting Internet kooks aside who think firing up CERN's HLC will create a black hole that eats Switzerland, the Manhattan Project had top thinkers like Edward Teller who questioned if testing an atomic bomb would incinerate the atmosphere of the entire earth.
We build a lot of things not knowing the true outcomes. It's even more unpredictable now, because the adoption of something like AI by millions of people creates additional layers of complex, unpredictable phenomena and outcomes. As much as I distrust Mark Zuckerberg, I don't think he created the news feed and added post-sharing features with the awareness it would be used by Myanmar for targeting a genocide against Rohingyas.
The uncertainty is core to the design of these AIs. Designing an effective neural net (the building blocks of LLMs) requires a balance between specialization and generalization tuning. In the 1990s, neural net developers realized overtraining was a problem: systems became too rigid and brittle in their responses.
In the opposite direction, using fewer nodes in the net lead to generalization, which is generally good inference. It is what enables an AI to interpret between strings of tokens in completely unrelated documents let alone fields of knowledge. The trouble is that we can over-index for it. which leads to more hallucinations -- which is effectively bad generalization.
Point is, these systems are designed to produce what appear to be rigid results but under uncertain circumstances. The uncertainty is the point... it's not something to be locked down with explicit certainty under a microscope.
Now Dario is one of the AI leads I trust most. Unlike the others, I don't feel as strongly that he falls trap to the old Upton Sinclair quote: "It is difficult to get a man to understand something, when his salary depends on his not understanding it."
But Mechanistic Interpretability, or mechinterp, is one of the biggest scientific research boondoggles in AI today. It's overly reductive premise is "AI is like brains, therefore we can use electron microscopes on them to understand them like brains".
There are multiple fail levels here:
1) Silicon chips shuttling bits around do not work like brains. This is a physicist's "assume a spherical cow" joke.
2) Even if 1) were true, brain function -- like LLM function -- is a systemic response. Putting a lone pine tree under an electron microscope doesn't tell you how the forest is evolving. Similarly, Finite Element Analysis will not explain the human brain (I know, I tried in the 90s in grad school).
3) To even create the conditions for reductionist, direct causality in mechinterp, many of the successful learnings have required forcing what are called "monosemantic" neural net models: models where a single neural net node maps to a single data feature for explainability. These are models I called "overtrained" above, lacking generalization. Effectively, they are recreating the "grandmother cell" fallacy in neuroscience (https://en.wikipedia.org/wiki/Grandmother_cell), and their research is so far removed from the designs of LLMs we actually want to use it is effectively inapplicable.
As for consciousness discussions, I'm baffled why the natural world gets completely thrown under the bus and we seem to only have these discussions once text spits out of silicon. It feels like when Saudia Arabia granted citizenship and rights to the robot Sofia in 2017, before women in the country had a legal right to drive and robots had more rights than they did.