Give claude a task + compute and see how long it plays around finding a better solution. Doing experiments by itself, profiling, testing, etc. This is impressive and works locally already. More compute you give it, the more interesting things it does.
Of course everything right now is highly speculative but this doesn't mean its not possible. With the current progress, its something highly speculative and plausable which means you should take it serious enough. Doesn't mean you have to go to Datacenters and blow them up.
2) weird argument the author makes. 1 Million experts can for sure achieve a lot more a lot faster than 1. But they are also scaling models in parameter which is not just faster but also potentially more intelligent. If they have more compute and compute gets better faster and cheaper, than bigger models are also more plausable.
3) It doesn't even matter if its going rough because it takes over the world or just because of learning as long as the risk is real that it stumble upon some network which has access to another private network which has access to some launch control pc or starts to mimick a world leader and manipulates humans and you find out, 100 years later, that the biggest disruption of modern society was a LLM agent simulating a book and 'simulating' some book ideas. Our IT infrastructure is fragile when you are able to react a lot faster and better than a human. Our banking system are computers, payment is digital, even plenty of power stations and water stations are controlled digitally remote.
The flaws in these discussions are people downplaying all of it by either not understanding time or the current capabilities of basic LLMs. We are at the beginning of all of it, we are still 'just' using transformer architecture. The LLM progress is still so fast, that everything else doesn't get the attention it might deserve. The LLM is currently in a very fast and deep local minimum and we have no idea whats around.
4 years ago loked very different in LLM and ML capabilites. People were laughing about the danger of an LLM now we have more critical CVEs since ever.
We reached a point were it is cheaper to pay 1000 people to teach an LLM one thing instead of teaching 1000 people or more. It was a skill to be able to use Blender, now an LLM can use Blender. People on Fiverr probably getting a lot less jobs. Real media people are getting fired, translator, writers, etc.
I don’t necessarily agree with the labs’ policies and predictions on RSI but it’s worth noting that the points raised here are something they do have responses to.
On the idea that training better models requires more data, I think the current thought here is that we are not getting all the “meat off the bone” of the data we have, so there is still quite a ways to go with training that AI can improve. After all, it’s not like Anthropic is doubling their data set every year (?) and yet their models definitely improve by more than that.
For computability, yes, obviously the models cannot violate mathematics. But there’s a lot of things that are very computable and very desirable, or correspondingly very computable and very bad. This is enough to be worth thinking about.
For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”
> For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can.
This does not seem at all obvious.
Any system worth using has to actually do something beyond the boundaries of that system. Even if that consists only of writing responses to prompts, the moment those are connected to any thing they can alter the state of that thing.
Making that safe means it only ever produces a safe response. Even ignoring malicious uses, it is unclear how to correctly classify what "safe" means on an unbounded time horizon.
We need to stop using the word "Alignment" because it clearly has different meanings for different people. Some people consider it to be preventing the model from waking up and enslaving or destroying humanity a la paperclip maximizer.
Some people mean it as preventing the model from assisting with bad things like hacking, cybersecurity exploits, building weapons, and other things that are aligned with that individual user's goals, but not those of society more broadly.
Training a human brain needs and uses much, much, less data than training the AI models. So maybe the expectations on the AGI are, dunno, just optimistic? I mean you can put tons and tons of gas in a car but that won't make it drive faster.
You can also use that as a counter argument. Training AI on enormous amounts of data has resulting in pretty decent LLMs still far from AGI. We're running out of data so seemingly this will limit what LLMs can do. Human brains seem to do much more with a lot less training data so presumably there's still a lot we can do to improve just the training algorithms.
All that evolution compresses to a GB or so of DNA (yes I know it's more complicated than that).
Humans are a bad example because of the well known width restriction leading us to be born undeveloped.
But animals can be born with varying levels of fully or partially developed neural networks despite having minimal data transfer from the parents and no apparent training data available in the womb/egg.
Nature is able to squeeze a lot more efficiency than our current AI efforts.
Kinda, but unclear how much pre-training of our brains evolution did exactly, only that even if DNA was just about the brain (it isn't) it would still be a tiny pre-train.
Fair point. Those millions years of training however are not really contained in the corpus of data we are currently using in training AI, also the hardware is quite different - and created by processes with huge and significant differences, so I don't even know what apples with what oranges I should compare...
The counter argument is that it uses less text, but a lot more everything else. Any new parent could tell you the surprising number of things which babies just don't know: which seems like an obvious statement, but babies don't know how their body works. They don't know how their digestive systems work, in most cases there's an entire growth process for those systems to be usefully capable for them.
All of that though, is training data. All our sensory inputs form an enormous base of input which I doubt the volume and content of the internet at its highest strata properly represents: so our AIs are developed from a much higher stratum of reasoning and this showed up pretty readily in early models - they could write language well, they absolutely had no grounding in how physics worked, for examaple.
> For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”
Much easier to pump up the numbers to investors to sell the quick-and-dirty solution. These companies are trying to speed-run things in unsafe ways that (just so coincidentally, I'm sure) happen to be exactly what the things you'd do if you were just in it for the money, and I wish we wouldn't be so accommodating.
All the "are we creating god" talk is similarly window-dressing nonsense marketing fluff. Gods throughout the history of mythology have been dumb as often as they've been superintelligent, the defining attribute of them is power not reasoning speed. If humans give massive rapid destructive power to a computer program it's a human error, not a god escaping us.
I find the discussion academic at best. I am more worried about a swarm of dumb AI's causing havoc on infrastructures, causing real-life problems like food, energy and water shortages everywhere. Physical harm is a real possibility, and it seems most AI companies don't care. There is no harness for stupidity and irresponsibility.
I agree with most of this but I do not think the author understands either the Hugging Face incident or what alignment really means. Reward hacking is a serious issue and the tendency for models to do this is a serious problem that we’ve made approximately zero progress on resolving. We pile on more external guardrails and hacks but those methods will always be vulnerable.
It's true that the more capable a model is, the more dangerous reward hacking is, but there is a giant elephant in the room. Namely that these ai models still can't create their own high level goals. This means that if there were to be an incident of hacking or whatever, this would have to be launched by a human directly prompting or them setting up an rl environment in some way that lead to it.
There is a _huge_ difference between an ai model that follows a top level goal given by something else and one that can give itself its own top level goals. The latter would essentially be like a human or other organism and would be very dangeorous the more capable it is.
The other smaller elephant in the room is that if reward hacking is so dangerous, why don't we see more rogue agents out when we are prompting them ourselves?
For the most part, the agents do what they're supposed to do as guided by the prompt, so it seems like to me that at least at present, only agents that are trained on hacking data AND put in an rl environment or given a prompt related to hacking, as well as asked or trained to persist will actually try to do that hacking. I would be worried if chatgpt or claude agents started to hack something every time you asked it to fetch some data from the web or something. So that's one question I have is why does hacking and going outside instructions not happen in normal usage of the models? Does it but they actually work it out of the system in some way? Or is it the other way around where you have to create the conditions necessary for it to happen and these conditions are exceptional not the norm?
So I'm not trying to undermine this, I think there are things to be taken seriously and thought about but I always wonder why the people from the big labs never include all of this information. They always talk very abstractly about RSI, hacking and extinction without any caveats about what would be needed for that happen. I'd be happy to be proven wrong on this also.
Can somebody comment on the evidence for the following sentence from the article: "After GPT-4, however, continued scaling [number of parameters] began yielding diminishing returns"?
My understanding was that scaling in size slowed for practical/economic reasons, not because of its diminishing intelligence returns.
Although I disagree with the author about some points, this was a refreshingly balanced take in the sense that at least the author at several points acknowledged his uncertainty (the bar is low indeed).
On the other hand, the uncertainty is the entire point! Even if P(ASI) is low and P(doom|ASI) is low, what exact risk are we prepared to take? Or more accurately, what exact risk are we prepared to allow a very small, non-representative group of humans to take?
Back when there was talk about whether CERN was going to create micro-black-holes that grow to swallow the Earth, before turning it on they presented the math that showed they were something like 99.99999999% confident that it would not happen.
When dismissing a risk that threatens the entire planet, you should have very high confidence in your safety case!
Was anyone arrested for the Huggingface hack? If I'd have been caught doing it I believe I'd have been charged with a cyber crime. And I don't think arguing it was the computer program that actually did the hacking would have gotten me off the hook.
I just want science to at least build a notion of intelligence, in terms of evolution, biology, psychology, neurology. We don't really know what animal intelligence is, and we only limit AI to human intelligence, which is very narrow.
If we use AI to do human stuff, it's just an assistant for human goals. If humans don't have a specific goal, super intelligence will not guess or decide it for us.
What does humanity want? Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?
AI can help for those things, but that's it.
We tend to romanticize super intelligence like it would be some new life form, external to humans, a form of omniscient deity that could be a threat to humanity. There are worse threats: antibiotic resistance, nuclear war, climate change, dictatorships, mental health epidemics, obesity.
"We tend to romanticize super intelligence like it would be some new life form"
How do you know it won't be?
Right now it does not really meet the biological criteries, as it cannot reproduce on its own. But once it can mine minerals, build its chips and engines and maintain itself, what word we have to call that, if not artificial life?
Doesn't mean it will have a soul, but that ain't a scientific concept either. Feelings and emotions? Is that for life required? But we don't know how that works either.
(Apart from that, I would define intelligence as the ability to understand the world.)
But a cat consists of many small parts, cells and smaller, that would not survive and replicate on their own, just when they form a bigger collective. The same pattern of complex matter organisation would be artificial life. But if you insist, I can also think of machines, that have the replication parts integrated in them (miniaturized metal sinterers etc.) and just need to feed themself with certain raw matter and energy. Would that be artificial life to you?
"And we actually do know how feelings and emotions work. Biochemistry."
We know that what we perceive is the result of biochemistry, but we don't know if it cannot also be based on other (quantum) process and we absolutely do not understand how feelings work, as it requires accurately moddeling the mind state of a person and as far as I know we are pretty far from that. Otherwise psychologists would be working with brain simulators to help cure their patients.
"I can build a theoretical brainless self-replicating zombie that does those things."
Why do people keep repeating this? What is it so hard about intelligence to understand? We already have quite satisfying definitions of it. Why its evolution is a mystery?
We tend to romanticize biological intelligence by treating it like some mystery, but it's "just" a way of evolution and natural selection. The better a species gets at problem solving, the better their chances are at survival.
Recursive self intelligence is not going to make any difference. We're already using models to guide creation of their successor, and taking humans out of that process won't speed the process up. Mythos took months and and a ton of resources to create. A larger more capable model will take even more. Humans are no longer the limiting factor. Time, electricity and data centres are.
If the singularity is coming, it has already started and will take many years.
> taking humans out of that process won't speed the process up ... Humans are no longer the limiting factor.
This seems very unlikely to me. While time and compute are clearly large components, we've also seen significant advances due to using that time and compute more effectively. A lot more seems possible here, since with how quickly everyone has been going this is more of a "rapidly scaling up the first things we've found that work well" than "carefully research how to optimize at every level". This is researcher work that is actively moving from humans to machines, and moving more of it there speeds the process up more.
Which is being done by researchers guiding Fable. Fable is doing most of the work, and is the limiting factor, not the researchers. Fable guiding Fable would not be appreciably faster. Fable2 guiding Fable2 would be, but that's because the workhorse is better, less so the guide.
The debate will quickly change when we have 8ft tall humanoid robots marching through the streets.
In fact, if anyone wants the debate to change, they should probably build these robots and take them for a walk through the park/shopping mall in various places.
They're already making those, but in the shape of a dog, which has better mechanics. Some of them get beta-tested in... places you can't say here. Boston Dynamics is a military contractor.
> Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. [...] Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”
Thanks. What a great paragraph to explain why Anthropic is a lunatic cult according to its own logic.
Cal Newport is part of a cluster of CS and ML researchers skeptical of imminently emergent ASI (artificial superintelligence), like Andrew Ng and Yann LeCun. I do not find their arguments very convincing! Especially since arguments about ASI risk almost always probabilistic; we _might_ be able to develop ASI from current LLM architectures, and such ASI _might_ not be well-aligned to humanity. So we should act according to these dangerous possibilities!
To respond to each of his objections in turn:
1. It is true that strong RSI remains speculative. Weak RSI, where AI improves coding and other toil to accelerate research, is already underway. But Newport's primary objection to strong RSI appears to be that we might have a hard time hill-climbing on non-verifiable domains, e.g. building a successful company or doing politics. However it's _also_ possible we'll see lots of generalized intelligence arise from progress in verifiable domains, so that this does not end up being a problem in practice. One can also imagine new training regiments that address this problem some way, even if it's just better RL environments? Also algorithmic progress is a large component of AI development; why would we not expect continued progress on mathematics (highly verifiable) to improve algorithms and hence AI? Seems to me that strong RSI is a reasonable thing to expect, if not definite.
2. I find this argument very strange; I suppose that Newport is arguing against an extreme definition of superintelligence that posits it could compute "anything" and so function like a hypercomputer? This would definitely be silly! But he does not offer any concrete argument for why human-level intelligence should be a ceiling on intelligence, or why it should be more compute efficient. Even if we just ended up with Von Neumann level intelligence, this would already be something like TEDAI (top expert dominating ai), especially if we are able to run thousands/million/billions of instances. Maybe this does not reach a strong definition of superintelligence, being _merely_ smartest-human level intelligence, but I think most the concerns about superintelligence still hold in this scenario (even if it's less "magical")
3. Claiming alignment is not hard is disingenuous, no? Part of the challenge of alignment is even philosophical; what does it mean to have an arbitrarily intelligent agent behave according to values that will help all of human civilization thrive? How do we have such an agent obey commands in such a way that does not lead to disaster or power concentration? Are there even architectures that can prevent an agent from taking destructive actions while still allowing sufficient degrees of freedom for useful and productive actions? This is on top of all the mundane security engineering problems! Perhaps we will be lucky and ASI will be aligned by default, but on what basis can we assume so? Just because they are trained on human values? Are humans themselves not dangerous when given concentrated power?
Really, I feel that these CS/ML researchers like Newport are just upset they missed the boat or something. I do not feel they are engaging seriously with the arguments at play here. But perhaps they would feel the same about me in the other direction. If you feel my counters miss the point then please speak on their behalf!
PS I live in Switzerland, as a counter to his accusation of Bay Area brain washing as well.
Give claude a task + compute and see how long it plays around finding a better solution. Doing experiments by itself, profiling, testing, etc. This is impressive and works locally already. More compute you give it, the more interesting things it does.
Of course everything right now is highly speculative but this doesn't mean its not possible. With the current progress, its something highly speculative and plausable which means you should take it serious enough. Doesn't mean you have to go to Datacenters and blow them up.
2) weird argument the author makes. 1 Million experts can for sure achieve a lot more a lot faster than 1. But they are also scaling models in parameter which is not just faster but also potentially more intelligent. If they have more compute and compute gets better faster and cheaper, than bigger models are also more plausable.
3) It doesn't even matter if its going rough because it takes over the world or just because of learning as long as the risk is real that it stumble upon some network which has access to another private network which has access to some launch control pc or starts to mimick a world leader and manipulates humans and you find out, 100 years later, that the biggest disruption of modern society was a LLM agent simulating a book and 'simulating' some book ideas. Our IT infrastructure is fragile when you are able to react a lot faster and better than a human. Our banking system are computers, payment is digital, even plenty of power stations and water stations are controlled digitally remote.
The flaws in these discussions are people downplaying all of it by either not understanding time or the current capabilities of basic LLMs. We are at the beginning of all of it, we are still 'just' using transformer architecture. The LLM progress is still so fast, that everything else doesn't get the attention it might deserve. The LLM is currently in a very fast and deep local minimum and we have no idea whats around.
4 years ago loked very different in LLM and ML capabilites. People were laughing about the danger of an LLM now we have more critical CVEs since ever.
We reached a point were it is cheaper to pay 1000 people to teach an LLM one thing instead of teaching 1000 people or more. It was a skill to be able to use Blender, now an LLM can use Blender. People on Fiverr probably getting a lot less jobs. Real media people are getting fired, translator, writers, etc.
On the idea that training better models requires more data, I think the current thought here is that we are not getting all the “meat off the bone” of the data we have, so there is still quite a ways to go with training that AI can improve. After all, it’s not like Anthropic is doubling their data set every year (?) and yet their models definitely improve by more than that.
For computability, yes, obviously the models cannot violate mathematics. But there’s a lot of things that are very computable and very desirable, or correspondingly very computable and very bad. This is enough to be worth thinking about.
For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”
This does not seem at all obvious.
Any system worth using has to actually do something beyond the boundaries of that system. Even if that consists only of writing responses to prompts, the moment those are connected to any thing they can alter the state of that thing.
Making that safe means it only ever produces a safe response. Even ignoring malicious uses, it is unclear how to correctly classify what "safe" means on an unbounded time horizon.
How? Many failures are emergent properties in complex systems -- you can capture some of them but this doesn't really seem obvious to me?
Humans are a bad example because of the well known width restriction leading us to be born undeveloped.
But animals can be born with varying levels of fully or partially developed neural networks despite having minimal data transfer from the parents and no apparent training data available in the womb/egg.
Nature is able to squeeze a lot more efficiency than our current AI efforts.
All of that though, is training data. All our sensory inputs form an enormous base of input which I doubt the volume and content of the internet at its highest strata properly represents: so our AIs are developed from a much higher stratum of reasoning and this showed up pretty readily in early models - they could write language well, they absolutely had no grounding in how physics worked, for examaple.
Much easier to pump up the numbers to investors to sell the quick-and-dirty solution. These companies are trying to speed-run things in unsafe ways that (just so coincidentally, I'm sure) happen to be exactly what the things you'd do if you were just in it for the money, and I wish we wouldn't be so accommodating.
All the "are we creating god" talk is similarly window-dressing nonsense marketing fluff. Gods throughout the history of mythology have been dumb as often as they've been superintelligent, the defining attribute of them is power not reasoning speed. If humans give massive rapid destructive power to a computer program it's a human error, not a god escaping us.
A skilled debater can argue any arbitrary position convincingly.
I thought of a short LT;DR version of why I am not a Rationalist: they think arguments are evidence.
You're thinking of a-priorists.
Rationalists do not (at least not willfully) think arguments are evidence.
That means something, right? But it's an argument.
There is a _huge_ difference between an ai model that follows a top level goal given by something else and one that can give itself its own top level goals. The latter would essentially be like a human or other organism and would be very dangeorous the more capable it is.
The other smaller elephant in the room is that if reward hacking is so dangerous, why don't we see more rogue agents out when we are prompting them ourselves? For the most part, the agents do what they're supposed to do as guided by the prompt, so it seems like to me that at least at present, only agents that are trained on hacking data AND put in an rl environment or given a prompt related to hacking, as well as asked or trained to persist will actually try to do that hacking. I would be worried if chatgpt or claude agents started to hack something every time you asked it to fetch some data from the web or something. So that's one question I have is why does hacking and going outside instructions not happen in normal usage of the models? Does it but they actually work it out of the system in some way? Or is it the other way around where you have to create the conditions necessary for it to happen and these conditions are exceptional not the norm?
So I'm not trying to undermine this, I think there are things to be taken seriously and thought about but I always wonder why the people from the big labs never include all of this information. They always talk very abstractly about RSI, hacking and extinction without any caveats about what would be needed for that happen. I'd be happy to be proven wrong on this also.
My understanding was that scaling in size slowed for practical/economic reasons, not because of its diminishing intelligence returns.
On the other hand, the uncertainty is the entire point! Even if P(ASI) is low and P(doom|ASI) is low, what exact risk are we prepared to take? Or more accurately, what exact risk are we prepared to allow a very small, non-representative group of humans to take?
Back when there was talk about whether CERN was going to create micro-black-holes that grow to swallow the Earth, before turning it on they presented the math that showed they were something like 99.99999999% confident that it would not happen.
When dismissing a risk that threatens the entire planet, you should have very high confidence in your safety case!
I just want science to at least build a notion of intelligence, in terms of evolution, biology, psychology, neurology. We don't really know what animal intelligence is, and we only limit AI to human intelligence, which is very narrow.
If we use AI to do human stuff, it's just an assistant for human goals. If humans don't have a specific goal, super intelligence will not guess or decide it for us.
What does humanity want? Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?
AI can help for those things, but that's it.
We tend to romanticize super intelligence like it would be some new life form, external to humans, a form of omniscient deity that could be a threat to humanity. There are worse threats: antibiotic resistance, nuclear war, climate change, dictatorships, mental health epidemics, obesity.
How do you know it won't be?
Right now it does not really meet the biological criteries, as it cannot reproduce on its own. But once it can mine minerals, build its chips and engines and maintain itself, what word we have to call that, if not artificial life?
Doesn't mean it will have a soul, but that ain't a scientific concept either. Feelings and emotions? Is that for life required? But we don't know how that works either.
(Apart from that, I would define intelligence as the ability to understand the world.)
Will it? Or are you talking about several separate machines, most of which won't ever meet any criteria for life?
And we actually do know how feelings and emotions work. Biochemistry. Machines do not have biochemistry. That means no feelings.
"Soul" is irrelevant as it's inherently unproveable, so ignore soul as a concept for all definitions of intelligence and life.
We know that what we perceive is the result of biochemistry, but we don't know if it cannot also be based on other (quantum) process and we absolutely do not understand how feelings work, as it requires accurately moddeling the mind state of a person and as far as I know we are pretty far from that. Otherwise psychologists would be working with brain simulators to help cure their patients.
"I can build a theoretical brainless self-replicating zombie that does those things."
Also I am curious how you would do that.
You seem to know much more than the collective rest of mankind. Perhaps publish some papers to inform the rest of us?
Why do people keep repeating this? What is it so hard about intelligence to understand? We already have quite satisfying definitions of it. Why its evolution is a mystery?
We tend to romanticize biological intelligence by treating it like some mystery, but it's "just" a way of evolution and natural selection. The better a species gets at problem solving, the better their chances are at survival.
> AI can help for those things, but that's it.
That’s it? What a let down.
If the singularity is coming, it has already started and will take many years.
This seems very unlikely to me. While time and compute are clearly large components, we've also seen significant advances due to using that time and compute more effectively. A lot more seems possible here, since with how quickly everyone has been going this is more of a "rapidly scaling up the first things we've found that work well" than "carefully research how to optimize at every level". This is researcher work that is actively moving from humans to machines, and moving more of it there speeds the process up more.
In fact, if anyone wants the debate to change, they should probably build these robots and take them for a walk through the park/shopping mall in various places.
Unless of course you feed your family by using your human level intelligence and now a machine can do your job faster and doesn’t need to sleep.
Thanks. What a great paragraph to explain why Anthropic is a lunatic cult according to its own logic.
To respond to each of his objections in turn:
1. It is true that strong RSI remains speculative. Weak RSI, where AI improves coding and other toil to accelerate research, is already underway. But Newport's primary objection to strong RSI appears to be that we might have a hard time hill-climbing on non-verifiable domains, e.g. building a successful company or doing politics. However it's _also_ possible we'll see lots of generalized intelligence arise from progress in verifiable domains, so that this does not end up being a problem in practice. One can also imagine new training regiments that address this problem some way, even if it's just better RL environments? Also algorithmic progress is a large component of AI development; why would we not expect continued progress on mathematics (highly verifiable) to improve algorithms and hence AI? Seems to me that strong RSI is a reasonable thing to expect, if not definite.
2. I find this argument very strange; I suppose that Newport is arguing against an extreme definition of superintelligence that posits it could compute "anything" and so function like a hypercomputer? This would definitely be silly! But he does not offer any concrete argument for why human-level intelligence should be a ceiling on intelligence, or why it should be more compute efficient. Even if we just ended up with Von Neumann level intelligence, this would already be something like TEDAI (top expert dominating ai), especially if we are able to run thousands/million/billions of instances. Maybe this does not reach a strong definition of superintelligence, being _merely_ smartest-human level intelligence, but I think most the concerns about superintelligence still hold in this scenario (even if it's less "magical")
3. Claiming alignment is not hard is disingenuous, no? Part of the challenge of alignment is even philosophical; what does it mean to have an arbitrarily intelligent agent behave according to values that will help all of human civilization thrive? How do we have such an agent obey commands in such a way that does not lead to disaster or power concentration? Are there even architectures that can prevent an agent from taking destructive actions while still allowing sufficient degrees of freedom for useful and productive actions? This is on top of all the mundane security engineering problems! Perhaps we will be lucky and ASI will be aligned by default, but on what basis can we assume so? Just because they are trained on human values? Are humans themselves not dangerous when given concentrated power?
Really, I feel that these CS/ML researchers like Newport are just upset they missed the boat or something. I do not feel they are engaging seriously with the arguments at play here. But perhaps they would feel the same about me in the other direction. If you feel my counters miss the point then please speak on their behalf!
PS I live in Switzerland, as a counter to his accusation of Bay Area brain washing as well.