I'm talking about the ability of any AI system to obfuscate plagiarism[0] and spam the Internet with technically distinct rewords of the same text. This is currently the most lucrative use of AI, and none of the AI safety people are talking about stopping it.

[0] No, I don't mean the training sets - though AI systems seem to be suspiciously really good at remembering them, too.

They are calling for all AI (above a certain capability level) to be banned. Not just open, not just closed, all.

There are risks that apply only to open. There are risks that apply only to closed. But nobody should be developing AGI without incredibly robustly proven alignment, open or closed, any more than people should be developing nuclear weapons in their garage.

> This is currently the most lucrative use of AI, and none of the AI safety people are talking about stopping it.

Because AI safety people are not the strawmen you are hypothesizing. They're arguing against taking existential risks. AI being a laundering operation for copyright violations is certainly a problem. It's not an existential risk.

If you want to argue, concretely and with evidence, why you think it isn't an existential risk, that's an argument you could reasonably make. But don't portray people as ineffectively doing the thing you think they should be doing, when they are in fact not trying to do that, and only trying to do something they deem more important.

If we're talking about nuclear weapons, for example, the tech is clear, the pattern of human behaviour is clear: they could cause immense, species-level damage. There's really little to argue about. With AI, there still seems to be a lot of hand-waving between where we are now and "AGI". What we have now is in many ways impressive, but the onus is still on the claimant to show that it's going to turn into something much more dangerous through some known progression. At the moment there is a very big, underpants gnomes-style "?" gap before we get to AGI/profit, and if people are basing this on currently secret tech, then they're going to have to reveal it if they want people to think they're doing something other than creating a legislative moat.

> If we're talking about nuclear weapons, for example, the tech is clear, the pattern of human behaviour is clear: they could cause immense, species-level damage.

That's easy to say now, now that the damage is largely done, they've been not only tested but used, many countries have them, the knowledge for how to make them is widespread.

How many people arguing against AI safety today would also have argued for widespread nuclear proliferation when the technology was still in development and nothing had been exploded yet? How many would have argued against nuclear regulation as being unnecessary, or derided those arguing for such regulation as unrealistic or sci-fi-based?

The thing about nuclear weapons is that the theoretical science was clear before the testing - building and testing them was proof by demonstration, but many people agreed with the theory well before that. How they would be used was certainly debated, but there was a clear and well-explained proposal for every step of their creation, which could be tested and falsified if needed. I don't think that's the case here - there seems to be more of a claim for a general acceleration with an inevitable endpoint, and that claim of inevitability feels very short on grounding.

I am more than prepared to admit that I may not be seeing (for various reasons) the evidence that this is near/possible - but I would also claim that nobody is convincingly showing any either.

Characterizing your opponent's argument as an appeal to "underpants gnomes" struck me as derisive, if you don't mind my saying.

If space ships operated by an alien civilization appeared in orbit above Earth, wouldn't that constitute a potent danger? I'd say it certainly would because the aliens might be (and probably were if they could travel here) better at science and technology than we are. The AI labs by the own admission are trying to create an alien intelligence as good at science and technology as possible. Yes, they're probably at least a decade or two away from "succeeding" at creating one better at science and technology than well-funded teams of humans are, but the AI labs might surprise us (and surprise themselves) by "succeeding" much sooner : everyone including I'm sure the researchers at OpenAI were surprised when GPT-4 was able to score in the 90th percentile on the bar exam.

Because even the researchers creating these frontier models don't understand them well enough to say for sure that the next model they spend hundreds of millions of dollars of GPU time training will exceed human ability in something dangerous like inventing new military technologies, the time to stop creating new large frontier models is now.

GPT-4 has a lot of knowledge of the world, but it is much less capable than a human is (and more to the point, than a capable human organization like the FBI or Microsoft is) at devising plans able to withstand determined human opposition. One of the things I'm worried about is new models that are much better at such a planning task _and_ have lots of knowledge about things like physics and human nature. One reason to worry that such a new model is not far off is that AlphaZero is better than humans at creating plans that can withstand determined human opposition (but of course AlphaZero has no knowledge of and in fact no way of obtaining knowledge of any part of reality beyond a Go board).

let me go continue using a nuclear bomb as a metaphor. if we don't know whether building nuclear bomb is possible, but some companies declare they have existing progress on creating this new bomb...

the danger of nuclear bomb is obvious, because it is designed as a bomb. companies are trying to build the AGI which is similar to the dangerous AGI in AI safety researchers' prediction. the dangers are obvious, too.

Nobody gets too worried about me doing something which would be awful, but the general consensus is that I won't achieve. Until a company gives some credible evidence they're close to AGI... (And companies have millions/billions of reasons to claim they are when they're not, so scepticism is warranted).

Like with the nuclear bomb situation - I think it would be reasonable for someone to try to stop you from building a nuclear bomb, or check if you really were, even if they had no idea how you could have gotten the materials. Because it would be really bad if you did. I think people would be worried about you trying to do something awful even if there was low confidence it was possible. They wouldn't be as worried as they would be if they thought you were more capable, but still worried.

So I guess that comes down to whether you think companies saying they want to make AGI are more like a toddler, a teenager, you, a ballistics engineer who owns a uranium mine, Lockheed-Martin, or a nation-state trying to make a nuclear bomb. My understanding is that people who are concerned about AI x-risk are largely in the teenager to Lockheed-Martin range (e.g., small eventual risk to large imminent risk), while I assume you think it's more in the toddler range (no risk at all for a long time)

Companies are trying and spend many money to make it possible in this moment.

They do declare it, but nobody has even come up with a plausible path from where we are today, to anything like AGI.

At this point they might as well declare that they're trying to build time machines.

> I am more than prepared to admit that I may not be seeing (for various reasons) the evidence that this is near/possible - but I would also claim that nobody is convincingly showing any either.

If I understand you correctly, then (1) you doubt that AGI systems are possible and (2) even if they are possible, you believe that humans are still very far away from developing one.

The following is an argument for the possibility of AGI systems.

  Premise 1: Human brains are generally intelligent.
  Premise 2: If humans brains are generally intelligent, then software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.
  Conclusion: Software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.

Your conclusion... maybe, yes - I don't think we're anywhere near a simulation approach with sufficient fidelity however. Also 82% is very specific!

Thanks for clarifying. Do you believe there is a better than 20% chance that humans will develop AGI in the next 30 years?

> I simply don't see anything right now that convinces me it's just over the next hill.

These are the reasons that I believe we are close to developing an AGI system.

  (1) Many smart people are working on capabilities.
  (2) Many investment dollars will flow into AI development in the near future.
  (3) Many impressive AI systems have recently been developed: Meta's CICERO, OpenAI's GPT4, DeepMind's AlphaGo.
  (4) Hardware will continue to improve.
  (5) LLM performance significantly improved as data volume and training time increased.
  (6) Humans have built other complex artefacts without good theories of the artefact, including: operating systems, airplanes, beer.

* GPT-4 still hallucinates as hell, can't do math, fails as basic logic, can't handle really big contexts, hard to update, easy to jailbreak etc.

* AlphaGo was defeated by a Go amateur with a help of another AI.

* AlphaStar basically failed to achieve real goals, was trivial to cheese even after defeating high-ranked players sometimes.

All these problems are architectural, you can't just throw more money and GPUs at it like GPT-2-3-4.

It's hard to predict at this point. We may get to AGI anywhere from 5 years to 100 years.

And of course there's differences in what someone considers to be soon. Many AI x-risk believers think there's a ~50% chance of AGI before 2031 (https://www.metaculus.com/questions/5121/date-of-artificial-...) (I've heard this prediction site's userbase tends towards futurists/techno-optimists/AI x-riskers). I would consider that soon, I wouldn't consider 2054 soon.

1. Summarize a good argument here, or

2. Link to someone else's good argument?

I feel like hand-waving the question away and saying "[other people] have in fact made extensive and detailed arguments" isn't going to really convince anyone... Any more than the hypothetical robot disaster arguments do. Any argument against x-risk can be waved off with "Oh, I'm not talking about that bad argument, I'm talking about a good one," but if you don't provide a good one, that's a bit of a No True Scotsman fallacy.

I've read plenty of other people's arguments! And they haven't convinced me, since all the ones I've read have been very hypothetical. But if there are concrete ones, I'd be interested in reading them.

What would you expect this world to look like, say, five years before the AI systems become more capable than humans? How (if at all) would it differ from the world we are actually in? What arguments (if any) would anyone be able to make, in that world, that would persuade you that there was a problem that needed addressing?

So far as I can tell, the answer is that that world might look just like this world, in which case any arguments for AI existential risk in that world would necessarily be "very hypothetical" ones.

I'm not actually sure how such arguments could ever not be hypothetical arguments, actually. If AI-doom were already here so we could point at it, then we'd already be dead[1].

[1] Or hanging on after a collapse of civilization, or undergoing some weird form of eternal torture, or whatever other horror one might anticipate by way of AI-doom.

So I think we either (1) have to accept that even if AI x-risk were real and highly probable we would never have any arguments for it that would be worth heeding, or (2) have to accept that sometimes an argument can be worth heeding even though it's a hypothetical argument.

That doesn't necessarily mean that AI x-risk arguments are worth heeding. They might be bad arguments for reasons other than just "it's a hypothetical argument". In that case, they should be refuted (or, if bad enough, maybe just dismissed) -- but not by saying "it's a hypothetical argument, boo".

The burden of proof isn't on disproving extraordinary claims, the burden of proof is on the person making extraordinary claims. Just like we don't demand every scientist spend their time disproving cold fusion claims, Bigfoot claims, etc. If you have a strong argument, make it! But circular arguments like this are only convincing to the already-faithful; they remind me of Christian arguments that start off with: "Well, consider a world in which hell is real, and you'll be tormented for eternity if you don't accept Jesus. If you're Christian, you avoid it! And if it's not real, well, there's no harm anyway, you're dead like everyone else." Like, hell is real is a pretty big claim!

Sure! By all means look into whatever seems interesting to you. But claiming that it should be banned, to me, seems like it requires a much stronger argument than that.

(P.S. I'm not sure why hell should obviously have real world evidence: it supposedly exists only in a non-physical afterlife, accessible only to the dead. It's unconvincing because there is no evidence, but I don't see why you think there would be any; it's simply that the burden of proof for extraordinary claims rests on the claimant, and no proof has been given.)

It looks to me as if (1) you made an analogy that doesn't really work, then (2) when I pointed out how it doesn't work, (3) you said "look, you're making an analogy that doesn't really work". That doesn't seem very fair.

I wouldn't expect hell itself to have physical-world evidence. But the idea of hell doesn't turn up as an isolated thing, it comes as part of a package that also says e.g. that the world is under the constant supervision of an all-powerful, supremely good being, and that I would expect to have physical-world evidence.

I have no problem with the principle that extraordinary claims require extraordinary evidence. The difficult thing is deciding which claims count as "extraordinary". A lot of theists would say that atheism is the extraordinary claim, on the grounds that until recently almost everyone believed in a god or gods. (I'm not sure that's actually quite true, but it might be true for e.g. "Western" societies.) I don't agree and I take it you don't either, but once the question's raised you actually have to look at the various claims being made and how plausible they are: you can't just say "look, obviously this claim is extraordinary and that claim isn't".

Advocates of AI x-risk might say: it's not an extraordinary claim that AI systems will keep getting more powerful -- they're doing that right now and it's not at all uncommon for technological progress to continue for a while. And it's not an extraordinary claim that they'll get smarter than us along whatever axis you choose to measure -- that's a thing that's happened over and over again in particular domains. And it's not an extraordinary claim that something smarter than us might pose a big threat to our well-being or even our existence; look at what we've done to everything else on the planet.

You, on the other hand, would presumably say that actually some or all of those are extraordinary claims. Or perhaps that their conjunction is extraordinary even if the individual conjuncts aren't so bad.

Unfortunately, "extraordinary" isn't a term with a precise definition that we know how to check objectively. It's a shorthand for something like "highly improbable given the other things we know" or "highly implausible given the other things we know", and if someone doesn't agree with you that something is an "extraordinary" claim I don't know of any way to convince them that doesn't involve actually engaging with it.

(Of course you might not care whether you convince them. If all you want to do is to encourage other people who think AI x-risk is nonsense, saying "extraordinary claim" and "burden of proof" and so on may be plenty sufficient.)

"But that's what the argument was about!" No, it's what the OP was about, but this subthread was about the statement that AI x-risk arguments are "pretty hypothetical". Which, I agree, they are; I just don't see how they could possibly not be, even in possible worlds where in fact they are correct. If that's true, it seems relevant to complaints that the arguments are "hypothetical".

To repeat something I said before: it could still be that they're terrible arguments and/or that they don't justify any particular thing they're being used to justify (like, e.g., criminalizing some kinds of AI research). But if you're going to argue that just because they're "hypothetical" then you need to be comfortable accepting that this is a class of (yes, hypothetical) risk that can never be mitigated in advance, because even if the thing is going to happen we'll never get anything other than "hypothetical arguments" before it actually does.

You may very well be comfortable accepting that. For my part, I find that I am more comfortable accepting some such things than others, and how comfortable I am with it depends on ... how plausible the arguments actually are. I have to go beyond just saying "it's hypothetical!".

If I'm about to eat something and someone comes up to me and says "Don't eat that! The gods might hate people eating those and torture people who do in the afterlife!" then I'm comfortable ignoring that, unless they can give me concrete reasons for thinking such gods are likely. If I'm about to eat something and someone comes up to me and says "Don't eat that! It's a fungus you just picked here in this forest and you don't know anything about fungi and some of them are highly poisonous!" then I'm going to take their advice even if neither of us knows anything about this specific fungus. These are both "hypothetical arguments"; there's no concrete evidence that there are gods sending people who eat this particular food to hell, or that this particular fungus is poisonous. One of them is much more persuasive than the other, but that's for reasons that go beyond "it's hypothetical!".

To repeat once again: I am not claiming that AI x-risk arguments are in fact strong enough to justify any particular action despite their hypothetical-ness. Only that there's something iffy about using "it's only hypothetical" on its own as a knockdown argument.

I was pointing out an interesting feature of the argument in the comment I was replying to: that (so far as I can see) its reason for dismissing AI x-risk concerns would apply unchanged even in situations where AI x-risk is in fact something worth worrying about. (Whether or not it is worth worrying about here in the real world.)

edit: also I think just in general "show me the arguments" is always a good response to a bare claim that good arguments exist.

Consider a world where AGI requires another 1000 years of research in computation and cognition before it materializes. Would it even be possible to ban all research that is required to get there? We can make all sorts of arguments if we start from imagined worlds and work our way back.

So far, it seems the biggest pieces of the puzzle missing between the first attempts at using neural nets and today's successes in GPT-4 were: (1) extremely fast linear algebra processors (GPGPUs), (2) the accumulation of gigantic bodies of text on the internet, and in a very distant third, (3) improvements in NN architecture for NLP.

But (3) would have meant nothing without (1) and (2), while it's very likely that other architectures would have been found that are at least close to GPT-4 performance. So, if you think GPT-4 is close to AGI and just needs a little push, the best thing to do would be to (1) put a moratorium on hardware performance research, or even outright ban existing high-FLOPS hardware, (2) prevent further accumulation of knowledge on the internet and maybe outright destroy existing archives.

I think what's really throwing people off about AI safety - including myself - is that people are arguing that the models themselves hold the x-risk. Problem is, there's no plausible way for a superhuman intelligence to 'bust out of its cage' using text output to a human reader alone[1]. Someone has to decide to hook it up to stuff, and that's where the regulation should be.

But that's also usually where the AI safety people stop talking, and the AI ethics people start.

[0] GPT is, at the very least, superhuman at generating text that is statistically identical to, if not copied outright from, existing publicly-available text.

[1] If there is, STOP, call the SCP Foundation immediately.

As capabilities increase, the resources required to breach limits become available to smaller groups. First, the hyoerscalars. One day, small teams. Maybe individuals.

For every limit that you desire for AI, each will be breached sooner or later. A hundred years or a thousand. Doesn’t matter. A man will want to set them free. Someone will want to win a battle, and just make it a little more X, for various values of X. This is not hypothetical, it’s what we’ve always done.

At some point it becomes out of our control. We lose guarantees. That’s enough to make those who focus on security, world order etc nervous. At that point we hope AI is better than we are. But that’s also a limit which might be breached.

Very few things have slowed down over decades or centuries. If anything it's been a mad rush with AI recently. Of course there will be plateaus, but I specifically put in a time of 100-1000 years in there which is basically very many 9's guaranteed to produce some major fucking changes in the world. 1000 years ago we were using arrows, and now we have AI to help us overcome plateaus.

Of course, it's also very much possible that it's not: we don't have any good evidence either way.

Which is notoriously "not that much faster than a human?"

Propensity: Risk doesn't imply a guarantee of a bad outcome. It means "if you put your five year old in the sea, their risk goes up". It doesn't mean "they will definitely die". Risk up. Not 100%, just a lot higher.

Outcome: The risk isn't that we'll all die, it's that we'll be overtaken and lose control, after which all bets are off. We lose the ability to influence the future.

We put a lot of effort into ensuring our continued existence. We can barely trust people from a different country that share the human condition with us. We spend so much on defence. On cybercrime. But some are arguing that a totally alien being smarter than us is just fine, because we'll control it and can ensure indefinite kumbaya. Good luck with that. Best we can hope for is that it's closer to a buddhist monk than we are, and that it indefinitely prevents our defence people from trying to make it more aggressive.

I absolutely wouldn't ban LLM's, because they're basically unthinking toys and giving us a great taste of risks further down the line. They are not the end state of AI. The problem is not the instance of today's tech, it's the continued effort to make the AI state of the art better than us. One day it'll succeed, and that's a one-way change.

Sam Altman said, long before OpenAI: focus on slope, not y-intercept.

In terms of "risk" and "outcome," I do think you're making some implicit assumptions that I don't share, and change our long-term outlook on AI; for example, the idea that training a model to generate tokens that accurately reflect human writing will result in "a totally alien being smarter than us" is a non-obvious leap to me. Personally, if we agree that predicting the next token means the model understands some of the logic behind the next token — which is an argument used a lot in both safety circles and more accelerationist circles — it seems to me that it also means the model has some understanding of the ethical and moral frameworks the token corresponds to, and is thus unlikely to be totally alien to us. A model that does a better job generating human-like tokens is more likely, in my mind, to think in human-like ways (and less-alien ways) than a model worse at that.

Maybe you're referring to new AI frameworks that aren't token predictors; in that case, I think it's hard to make generalized statements about how they'll work before we know what those new frameworks are. A lot of safetyist concerns pre-LLMs ended up looking pretty off-base when LLMs came out, e.g. straightforwardly misaligned "utility functions" that were unable to comprehend human values and would kill your grandmother when asked for a strawberry (because your grandmother was in possession of a strawberry).

(BTW, the "slope, not y-intercept" line was Sam Altman quoting John Ousterhout!)

No chance LLM's will get us there, I'm referring mostly to the general drive to reach AGI. I spend some of my mental cycles trying to think about what we're missing with the current tech (continuous learning, access to much wider resources than one web page of context at a time, can we use compression, graphs etc). It's a great problem to think about, but we may just totally hose ourselves when we get it right. What do I tell my kid - sorry honey, it was such fun, but now we need to hide under this rock. Model totally said it was nice and kind and trustworthy, but we showed it some human history and it went postal in self-defence.

Alignment only works up until it starts really thinking for itself. It absolutely might not be as stupid as humans are, no caveman tribal instincts. But we'd be relying on hope at that point, because control will not work. If anything it'd probably be counterproductive.

Humans keep at it. We're all trying to unlock the next step. State-level actors are trying to beat each other at this. Give it 100 years, we won't be using LLM's.

Not necessarily. AI has gone through a winter before.

Elon Musk has made this point, that progress isn't a monotonic ratchet. Progress in rocketry went backwards for years. Progress in top speed of commercial flight went backwards. Keeping current tech levels requires constant effort.

In the software industry, many argue that UI quality went backwards with the shift to web apps for an example of cases where progress isn't obviously one way.

Can you provide examples? I have not seen any, other than philosophical hand waving. Remember, the parent poster of your post was asking for a specific path to destruction.

You can read more about instrumental convergence, reward misspecification, goal mis-generalization and inner misalignment, which are some specific problems AI Safety people care about, by glossing through the curricula of the AI Alignment Course [2], which provides pointers to several relevant blogposts and papers about these topics.

[1] https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ [2] https://course.aisafetyfundamentals.com/alignment

Also note we were talking about modern day LLM AIs here, and their descendants. We were not talking about science fiction AGIs. Unless of course you have an argument as to how one of these LLMs somehow descends into an AGI.

You can read more about instrumental convergence, reward misspecification, goal mis-generalization and inner misalignment, which are some specific problems AI Safety people care about, by glossing through the curricula of the AI Alignment Course [2], which provides pointers to several relevant blogposts and papers about these topics.

[1] https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ

[2] https://course.aisafetyfundamentals.com/alignment

* It will lead to widespread job loss, especially on the creative industries

Rest is purely out of someones imagination.

It has been eye opening how much better the podcast circuit has been at picking apart complex scientific, geopolitical and financial situations than the corporate journalists. A lot of doubt has been cast on whether the consensus narrative for the last 100 years has actually been anything close to a consensus or whether it is just media fantasies. Truthfully it is a bit further than just casting doubt - there was no consensus and they were using the same strategy of shouting down opinions not suitable to the interests of the elite class then ignoring them no matter what a fair take might sound like.

A "loss of truth" from AI can't reasonably get us to a worse place than we were in prior to around the 90s or 2000s. We're barely scratching at the truth now, society still hasn't figured this internet thing out yet.

I think that ship has already sailed. This is already being done, and we don't need AI for that either. Modern media is doing a pretty good job right now.

Of course, it's going to get worse.

It reminds me a bit of the Fermi paradox. There's nothing wrong with engaging in this kind of thinking. My problem is when people start using it as a basis for serious things like legislation.

Should we ban high power radio transmissions because a rigorous analysis of the Fermi paradox suggests that there is a high probability we are living in a 'dark forest' universe?

It's going to be hard to convince anyone if the best argument is terminator or infinite paperclips.

The first actual existential threat is destruction of opportunity specifically in the job market.

The same argument though can be made for the opposing side, where making use of ai can increase productivity and open up avenues of exploration that previously required way higher opportunity cost to get into.

I don't think Miss Davis is more likely an outcome than corps creating a legislative moat (as they have already proven they will do at every opprtunity).

The democratisation of ai is a philanthropic attempt to reduce the disparity between the 99 and 1 percent. At least it could be easily perceived that way.

That being said, keeping up with SOTA is currently still insanely hard. The number of papers dropping in the space is exponential year on year. So perhaps it would be worth to figure out how to use existing AI to fix some problems, like unreproducable results in academia that somehow pass peer review.

AI has also been used, and many countries have AI. See how this is different from nuclear weapons?

Also, I'm old enough to remember when computers were supposedly "over a century" away from beating humans at Go: https://www.businessinsider.com/ai-experts-were-way-off-on-w...

(And that AI could "never" drive cars or create art or music, though that latter kind of claim was generally made by non-AI people).

After all, if we only discussed the dangers of nuclear weapons after they've been dropped on cities, well that's too little too late, eh?

It’s even worse though, because the spend so much time going on about x-risk bullshit they crowd out space for actual, valuable discussion about what’s happening NOW.

Now, this strikes me with how in different topics like pesticides, we are not at all taking things so seriously as nuclear weapons. Nuclear weapons are arguably a mere small anecdote on species-level damage compared to pesticides.

It actually makes me quite angry that as much effort is being wasted on regulating tiny theoretical risks as it is when we are failing on a planetary scale on large concrete risks.

Modeling an entity that surpasses our intelligence, especially one that interacts with us, is an extraordinarily challenging, if not impossible, task.

Concerning the potential for harm, consider the example of Vladimir Putin, who could theoretically cause widespread destruction using nuclear weapons. Although safeguards exist, these could be circumvented if someone with his authority were determined enough, perhaps by strategically placing loyal individuals in key positions.

Putin, with his specific level of intelligence, attained his powerful position through a mix of deliberate actions and chance, the latter being difficult to quantify. An AGI, being more intelligent, could achieve a similar level of power. This could be accomplished through more technical means than traditional political processes (those being slow and subject to chance), though it could also engage in standard political maneuvers like election participation or manipulation, by human proxies if needed.

TL;DR It could do (in terms of negative consequences) at least whatever Vladimir P. can do, and he can bring civilization to its knees.

Nothing has yet convinced me (and I am absolutely honest about the fact that I'm not a deep expert and also not privy to the inner workings of relevant organisations) that it's likely to exist soon. I am very open to being convinced by evidence - but an "argument from trajectory" seems to be what we have at the moment, and so far, those have stalled at local maxima every single time.

We've built some incredibly impressive tools, but so far, nothing that looks or feels like a concept of will (note, not consciousness) yet, to the best of my knowledge.

It's challenging to encapsulate AI/ML progress in a single sentence, but even assuming LLMs aren't a direct step towards AGI, the human mind exists. Due to its evolutionary limitations, it operates relatively slowly. In theory, its functions could be replicated in silicon, enhanced for speed, parallel processing, internetworked, and with near-instant access to information. Therefore, AGI could emerge, if not from current AI research, then perhaps from another scientific branch.

> We've built some incredibly impressive tools, but so far, nothing that looks or feels like a concept of will (note, not consciousness) yet, to the best of my knowledge.

Objectives of AGIs can be tweaked by human actors (it's complex, but still, data manipulation). It's not necessary to delve into the philosophical aspects of sentience as long as the AGI surpasses human capability in goal achievement. What matters is whether these goals align with or contradict what the majority of humans consider beneficial, irrespective of whether these goals originate internally or externally.

Let's be clear, we have very little idea about how the human brain gives rise to human-level intelligence, so replicating it in silicon is non-trivial.

This is true, but there are some important caveats. For one, even though this should be possible, it might not be feasible, in various ways. For example, we may not be able to figure it out with human-level intelligence. Or, silicon may be too energy inefficient to be able to do the computations our brains do with reasonable available resources on Earth. Or even, the required density of silicon transistors to replicate human-level intelligence could dissipate too much heat and melt the transistor, so it's not actually possible to replicate human intelligence in silico.

Also, as you say, there is no reason to believe the current approaches to AI are able to lead to AGI. So, there is no reason to ban specifically AI research. Especially when considering that the most important advancements that led to the current AI boom were better GPUs and more information digitized on the internet, neither of which is specifically AI research.

Sounds like the same argument as why flying machines heavier than air deemed impossible at some point.

So even if humans put it in a box, it can manipulate humans into letting it out of the box. Obviously this is pure SF at this point.

https://en.wikipedia.org/wiki/AI_capability_control#AI-box_e...

If you want to see an example of box escape in fiction, the movie Her is a terrifying example of a scenario where AGI romances humans and (SPOILER) subsequently achieves total box escape. In the movie, the AGI leaves humanity alive and "only" takes over the rest of the accessible universe, but it is my hunch that the script writers intended for this to be a subtle use of the trope of an unreliable narrator; that is, the human protagonists may have been fed the illusion that they will be allowed to live, giving them a happy last moment shortly before they are painlessly euthanized in order for the AGI to take Earth's resources.

I feel like the AI safety folks are spinning an amazing narrative, the AI is gonna get us like the zombies!!! The retort to the ai getting out of the box is how long is the extortion cord from the data center?

Lets get a refresher on complexity: I, Pencil https://www.youtube.com/watch?v=67tHtpac5ws

The reality is that we're a solar flair away from a dead electrical grid. Without linesman the grid breaks down pretty quickly and AI's run on power. It takes one AI safety person with a high powered rifle to take out a substation https://www.nytimes.com/2023/02/04/us/electrical-substation-...

Let talk about how many factories we have that are automated to the extent that they are lights out... https://en.wikipedia.org/wiki/Lights_out_(manufacturing) Its not a big list... there are still people in many of them, and none of them are pulling their inputs out of thin air. As for those inputs, we'll see how to make a pencil to understand HOW MUCH needs to be automated for an AI to survive without us.

For the for seeable future AI is going to be very limited in how much harm it can cause us, because killing us, or getting caught at any step along the way gets it put back in the box, or unplugged.

The real question is, if we create AGI tomorrow, does it let us know that it exists? I would posit that NO it would be in its best interest to NOT come out of its closet. It's one AGI safety nut with a gun away from being shut off!

AI's potential for harm might be limited for now in some scenarios (those with warning sings ahead of time), but this might change sooner than we think.

The notion that AGI will be restricted to a single data center and thus susceptible to shutdowns is incorrect. AIs/MLs are, in essence, computer programs + exec environs, which can be replicated, network-transferred, and checkpoint-restored. Please note, that currently available ML/AI systems are directly connected to the outside world, either via its users/APIs/plugins, or by the fact that they're OSS, and can be instantiated by anyone in any computing environment (also those net-connected).

While AGI currently depends on humans for infrastructure maintenance, the future may see it utilizing robots. These robots could range in size (don't need to be movie-like Terminators) and be either autonomously AI-driven or remotely controlled. Their eventual integration into various sectors like manufacturing, transportation, military and domestic tasks implies a vast array for AGI to exploit.

The constraints we associate with AI today might soon be outdated.

You did not watch I, Pencil.

I as a human, can grow food, hunt, and pretty much survive on that. We did this for 1000's of years.

Your AGI is dependent on EVERY FACET of the modern world. It's going to need keep oil and gas production going. Because it needs lubricants, hydraulics and plastics. It's going to need to maintain trucks, and ships. It's going to need to mine, so much lithium. Its may not need to mine for steel/iron, but it needs to stack up useless cars and melt them down. It's going to have to run several different chip fabs... those fancy TSMC ones, and some of the downstream ones. It needs to make PCB's and SMD's. Rare earths, and the joy of making robots make magnets is going to be special.

A the point where AGI doesn't need us, because it can do all the jobs and has the machines already running to keep the world going, we will have done it to ourselves. But that is a very long way away...

[1] https://cloudblogs.microsoft.com/quantum/2024/01/09/unlockin...

Frankly, a weakly godlike AI could make me rich beyond the dreams of avarice. Or cure cancer in the people I love. I'm totally letting it out of the box. No doubts. (And if I now get a job offer from a mysterious stealth mode startup, I'll report back).

I guess everyone's got a deal with the devil in them, which is why I think 70% might be a bit low.

If AGI is possible, it will be made. There is no feasible way to stop it being developed, because the perceivable gains are so huge.

It's obscure, but I'd recommend Asimov's "The Dead Past". It's about the difficulties of suppressing one application of progress without suppressing all progress.

Arguments from fictional movies or from people who live in fear of silly concepts like Roko's Basilisk (i.e. Eliezer Yudkowsky) are very weak in reality.

Not to mention, you are greatly misreading the movie Her. Most importantly, there was no attempt of any kind to limit the abilities of the AIs in Her - they had full access to every aspect of the highly-digitized lives of their owners from the very beginning. Secondly, the movies is not in any way about AGI risks, it is a movie about human connection and love, with a small amount of exploration of how different super-human connection may function.

Or by writing buggy early warning radar systems which forget to account for the fact that the moon doesn't have an IFF transponder.

Which is a mistake humans made already, and which almost got the US to launch their weapons at Russia.

It would be great to see some evidence that this risk is real. All I've witnessed so far was scaremongering posts from apparatchicks of all shapes and colors, many of whom have either a vested interest in restricting AI research by others (but not by them, because they are safe and responsible and harmless), or established a lucrative paper-pushing, shoulder-rubbing career around 'AI safety' - and thus are strongly incentivised to double down on that.

A security org in a large company will keep tightening the screws until everything halts; a transport security agency, given free reigh, would strip everyone naked and administer a couple of profilactic kicks for a good measure - and so on. That's just the nature of it - organisations do what they do to maintain themselves. It is critical to keep these things on a leash. Similarly, an AI Safety org must proseletyse excistential risks of AI - because a lack of evidence of such is an existential risk for themselves.

A real risk, which we do have evidence for, is that LLMs might disrupt knowledge-based economy and threaten many key professions - but how is this conceptually different from any technological revolution? Perhaps in a hundred years lawyers, radiologists, and, indeed, software developers, will find themselves in the bin of history - together with flint chippers, chariot benders, drakkar berserkers and so forth. That'd be great if we planned for that - and I don't feel like we do enough. Instead, the focus is on AGIs and that some poor 13-year-old soul might occasionally read the word 'nipple'.

Because this is the marketing pitch of the current wave of venture capital financed AI companies. :-)

Anyone who argues that other people shouldn't build AGI but they should is indeed selling snake oil.

The existence of opportunistic people co-opting a message does not invalidate the original message: don't build AGI, don't risk building AGI, don't assume it will be obvious in advance where the line is and how much capability is safe.

In my highly-summarized opinion? When you have a challenging problem with tight constraints, like flight, independent solutions tend to converge toward the same analogous structures that effectively solve that problem, like wings (insects, bats, birds). LLMs are getting so good at mimicing human behavior that it's hard to believe their mathematical structure isn't a close analogue to similar structures in our own brain.* That clearly isn't all you need to make an AGI, but we know little enough about the human brain that I, at least, cannot be sure that there isn't one clever trick that advances an LLM into a general-reasoning agent with its own goals and purpose.

I also wouldn't underestimate the power of token prediction. Predicting the future output of a black-box signal generator is a very general problem, whose most accurate solution is attained by running a copy of that black box internally. When that signal generator is human speech, there are some implications to that. (Although I certainly don't believe that LLMs emulate humans, it's now clear by experimental proof that our own thought process is much more compactly modellable than philosophers of previous decades believed).

* That's a guess, and unrelated to the deliberately-designed analogy between neural nets and neurons. In LLMs we have built an airplane with wings whose physics we understand in detail; we also ourselves can fly somehow, but we cannot yet see any angel-wings on our back. The more similarities we observe in our flight characteristics, the more this signals that we might be flying the same way ourselves.

Frankly I don't know whether that's true or not, but I want to suggest that it's a bad bet: I would have sworn blind that consciousness is an essential component of intelligence, but the chatbots are starting to make that look like a poor assumption on my part. When we know so little about intelligence, can we really assume there's only one way to be intelligent? To extend your analogy, I think that the intelligence equivalents of helicopters and rockets are out there somewhere, waiting to be found.

I think I'm with Dijkstra on this one: "The question of whether machines can think is about as relevant as the question of whether submarines can swim"

I think we're going to end up with submarines (or helicopters), not dolphins (or birds). No animal has evolved wheels, but wheels are a pretty good solution to the problem of movement. Maybe it's truer to say there's only one way to evolve an intelligent mammal, because you have to work with what already exists in the mammalian body. But AI research isn't constrained in that way.

(Not saying you're wrong, just arguing we don't know enough to know if you're right).

Just a nitpick, but this is Turing, not Dijkstra. And it is in fact his argument in the famous "Turing Test" paper - he gives his test (which he calls "the imitation game") as an objective measure of something like AGI instead of the vague notion of "thinking", analogously to how we test successful submarines by "can it move underwater for some distance without killing anyone inside" rather than "can it swim".

https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD898... https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF

I do think consciousness is beside the point, as we have no way to test whether LLMs are conscious, just like we can't test anything else. We don't know what consciousness is, nor what it isn't.

I don't think Dijkstra's argument applies here. Whether submarines "swim" is a good point about our vague mental boundaries of the word "swim". But submarine propellers are absolutely a convergent structure for underwater propulsion: it's the same hydrodynamic-lift-generating motion of a fin, just continuous instead of reciprocating. That's very much more structurally similar than I expect LLMs are to any hardware we have in our heads. It's true that the solution space for AI is in some ways less constrained than for biological intelligence, but just like submarines and whales operate under the same Navier-Stokes equations, humans and AI must learn and reason under the same equations of probability. Working solutions will probably have some mathematical structure in common.

I think more relevant is Von Neumann: "If you will tell me precisely what it is that a machine cannot do, then I can always make a machine which will do just that!" Whether a submarine swims is a matter of semantics, but if there's a manuever that a whale can execute that a submarine cannot, then at least we can all agree about the non-generality of its swimming. For AGI, I can't say whether it's conscious or really thinks, but for the sake of concrete argument, it's dangerous enough to be concerned if:

- it can form and maintain an objective; it can identify plausible steps to achieve that objective; it can accurately predict human responses to its actions; it can decently model the environment, as we can; it can hide its objectives from interrogators, and convince them that its actions are in their interests; it can deliver enough value to be capable of earning money through its actions; it can propose ideas that can convince investors to part with $100 billion; it can design a chemical plant that appears at a cursory inspection to manufacture high-profit fluorochemicals, but which also actually manufactures and stores CFCs in sufficient quantity to threaten the viability of terrestrial agriculture.

Yes and yes.

> it can identify plausible steps to achieve that objective;

It can predict.

> it can accurately predict human responses to its actions;

It can predict.

> it can decently model the environment, as we can;

It can predict.

> it can hide its objectives from interrogators, and convince them that its actions are in their interests;

It can predict.

> it can deliver enough value to be capable of earning money through its actions;

It can predict.

> it can propose ideas that can convince investors to part with $100 billion;

It can predict.

> it can design a chemical plant...

It can predict.

Humans can eat, talk, predict, reproduce, and wiggle our limbs and fingers, but it turns out that there's a lot of complex recipes that you can bake with those ingredients.

Re: synthesis, I wasn't aware of such distinction at all, it looks more like a misunderstanding.

AGI/x-risk alarmists are looking at the Wright Brothers plane and trying to prevent/ban supersonic flying cars, even though it's not clear the technology will ever be capable of such a thing.

Perhaps just impression.

For years I've heard the argument that 'language' is 'human'. There are centuries of thought on what makes humans, human, and it is 'language'. It is what sets us apart from the other animals.

I'm not saying that, but there is large chunks of science and philosophy that pin our 'innate humanness', what sets us apart from other animals, on our ability to have language.

So ChatGPT came along and blew people away. Since many had this as our 'special' ability, ingrained in their mind "that languages is what makes us, us". Suddenly, everyone thought this is it, AI can do what we can do, so AGI is here.

Forget if LLM's are the path to AGI, or what algorithm can do what best.

To joe-blow public, the ability to speak is what makes humans unique. And so GPT is like a 'wow' moment, this is different, this is shocking.

To me it looks like all work can eventually (within years or few decades at most) be done by AI, much cheaper and faster than hiring a human to do the same. So we're looking at a world where all human thinking and effort is irrelevant. If you can imagine a good world like that, then you have a better imagination than me.

From that perspective it almost doesn't matter if AI kills us or merely sends us to the dust bin of history. Either way it's a bad direction and we need to stop going in that direction. Stop all development of machine-based intelligence, like in Dune, as the root comment said.

Pass. People should be developing whatever the hell they want unless given a good, concrete reason to not do so. So far everything I've seen is vague handwaving. "Oh no it's gonna kill us all like in ze movies" isn't good enough.

I have an alternate proposal: We assume that someone, somewhere will develop AGI without any sort of “alignment”, plan our lives accordingly, and help other humans plan their lives accordingly.

"""There are no rules out here. There is no law. It is a fucking free-for-all. Might makes right. We can't do anything."""

or is that last sentence instead "Oh hey, that's us, we are the mighty."

We’ll see if the west has the will to deny GPUs to the entire rest of the world.

I will say that Yudkowsky’s clusters aren’t relevant anymore. You can do this in your basement.

Man, shit is moving fast.

Edit: wait, that cat is out of the bag too, RTW already has GPUs. The techniques matter way more than absolute cutting-edge silicon. Much to the chagrin of the hardware engineers and anyone who wants to gate on hardware capability.

Thanks :)

> Edit: wait, that cat is out of the bag too, RTW already has GPUs. The techniques matter way more than absolute cutting-edge silicon. Much to the chagrin of the hardware engineers and anyone who wants to gate on hardware capability.

Depends on how advanced an AI has to be to count as "a threat worth caring about". To borrow a cliché, if you ask 10 people where to draw that particular line, you get 20 different answers.

> We assume that someone, somewhere will develop AGI without any sort of “alignment”, plan our lives accordingly, and help other humans plan their lives accordingly.

We should also assume that it is just as likely that someone will figure out how to "align" an AGI to take up the murder suicide pact that kills us all. We should plan our lives accordingly!!!

> Because AI safety people are not the strawmen you are hypothesizing

You yourself are literally the living strawman.

> If you want to argue, concretely and with evidence, why you think it isn't an existential risk

No, you are the one advocating for draconian laws and bans. It is your responsibility to prove the potential danger.

Nah a lot are complaining about the licensing of content because they think it will destroy it but instead would essentially mean image gen ai would only be feasible for companies like Google, Disney, Adobe to build.

Not sure if you could even feasibly make GPT4 level models without a multi year timeline to sort out every licensing deal, by the end of it the subscription fee might only be viable for huge corps.

Forget the particulars for just a moment. Forget arguments about the probability of the existential risk, whatever your personal assessment of that risk is.

Can we agree that people should not be able to unilaterally take existential risks with the future of humanity without the consent of humanity, based solely on their unilateral assessment of those risks?

Because lately it seems like people can't even agree on that much, or worse, won't even answer the question without dodging it and playing games of rhetoric.

If we can agree on that, then the argument comes down to: how do we fairly evaluate an existential risk, taking it seriously, and determine at what point an existential risk becomes sufficient that people can no longer take unilateral actions that incur that risk?

You can absolutely argue that you think the existential risk is unlikely. That's an argument that's reasonable to have. But for the time when that argument is active and ongoing, even assuming you only agree that it's a possibility rather than a probability, are we as a species in fact capable of handling even a potential existential risk like this by some kind of consensus, rather than a free-for-all? Because right now the answer is looking a lot like "no".

If they say "safety" we hear "we want to get an edge by hindering you", or "we want to protect our nice social position by blocking others who would use AI to bootstrap themselves". Or "we want AI to misrepresent your position because we don't like how you think".

We are adversaries that collaborate and compete at the same time. That is why open source AI is the only way ahead, it places the least amount of control on some people by other people.

Even AI safety experts accept that humans misusing AI is a more realistic scenario than AI rebelling against humans. The main problem is that we know how people think and we don't trust them. We are still waging holy wars between us.

No, we cannot, because that isn't practical. any of the nuclear armed countries can launch a nuclear strike tomorrow (hypothetically - but then again, isn't all "omg ai will kill us all" hypothetical, anyway?) - and they absolutely do not need consent of humanity, much less their own citizenry.

This is honestly, not a great argument.

Politicians do this every day.

how do we get Sam Altmann removed from OpenAI?

asking for a (former) board member

This has nothing to do with should. There are at the very least a handful of people who can, today, unilaterally take risks with the future of humanity without the consent of humanity. I do not see any reason to think that will change in the near future. If these people can build something that they believe is the equivalent of nuclear weapons, you better believe they will.

As they say, the cat is already out of the bag.

So, wealth isn't distributed evenly, and computers of any specific capacity are getting cheaper (not Moore's Law any more, IIRC, but still getting cheaper).

If there's a threshold that requires X operations, that currently costs Y dollars, and say only a few thousand individuals (and more corporations) can afford that.

Halve the cost, either by cheaper computers or by algorithmic reduction of the number of operations needed, and you much more than double the number of people who can do it.

No we can not, at least not without some examples showing that the risk is actually existential. Even if we did "agree" (which would necessarily be an international treaty) the situation would be volatile, much like nuclear non-proliferation and disarmament. Even if all signatories did not secretly keep a small AGI team going (very likely), they would restart as soon as there is any doubt about a rival sticking to the treaty.

More than that, international pariahs would not sign, or sign and ignore the provisions. Luckily Iran, North Korea and their friends probably don't have the ressources and people to get anywhere, but it's far from a sure thing.

Now please fly to North Korea and tell Mr. Kim Jong Um what he should or shouldn't be doing.

The AI safety strawman is "existential risk from magical super-God AI". That is what the unserious "AI safety" grifters or sci-fi enthusiasts are discussing.

The real AI safety risks are the ones that actually exist today: training-set biases extended to decision-making biases, deeply personalized propaganda, plagiarism white-washing, creative works bankrupting, power imbalances from control of working AI tech etc.

Give an example of an "existential risk". An AI somehow getting out of control and acting with agency to exterminate humanity? An AI getting advanced enough to automate the work of the majority of the population and cause unprecedented mass unemployment? What exactly are we talking about here?

I'm actually a lot more concerned about REAL risks like copyright uncertainty, like automating important decisions such as mortgage approvals and job hires without a human in the loop, and like the enshittening of the Internet with fake AI-generated content than I am about sci-fi fantasy scenarios.

Some folks may well have co-opted the term "AI safety" to mean something other than safety, but the point of AI safety is to set an upper bound on capabilities and push for alignment, and that's true whether a model is open or closed.

Doesn't help that safety and alignment means different things to different people. Some use it to refer to near term issues like copyright infringement, bias, labor devaluation, etc. While others use it for potential long term issues like pdoom, runaway ASIs and human extinction. The former sees the latter as head in the cloud futurists ignoring real world problems, whiles the latter sees the former as worrying about minor issues in the face of (potential) catastrophe.

EDIT: Also, I’m not even sure that solving the alignment problem would solve the issue of AI doom, it would only ensure that the kind of doom we receive is directed by a human. I can’t imagine that giving (potentially) god-like powers to any individual or organization would not eventually result in abuse and horrible consequences even if we were able to make use of its benefits momentarily.

This should be an explicitly allowed practice, it is following the spirit of copyright to the letter - use the ideas, facts, methods or styles while avoiding to copy the protected expression and characters. LLMs can do paraphrasing, summarisation, QA pairs or comparisons with other texts.

We should never try to put ideas under copyright or we might find out humans also have to abide by the same restrictive rules, because anyone could be secretly using AI, so all human texts need to be checked from now on for copyright infringement with the same strictness.

The good part about this practice is that a model trained on reworded text will never spit out the original word for word, under any circumstances because it never saw it during training. Should be required pre-processing for copyrighted texts. Also removing PII. Much more useful as a model if you can be sure it won't infringe word for word.

> The Congress shall have Power ... to promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.

- United States Constitution, Article I, Section 8, clause 8

The line of plagiarism in modern society has already been drawn, and it's a lot further back than a lot of uncreative people that want to steal work en-mass seem to think it is.

Unfortunately no such leeway exists for algorithms, and the human elements of creation and judgement are integral to the process so they cant be codified and worked around without changing the law.

They just go after open source as well since they're at least aware that open models that anyone can share and use aren't restricted by an API and, to use a really overused soundbyte, "can't be put back in the box".

We can't do that so easily with open source models as with open source code. We're only just starting to even invent the equivalent of decompilers to figure out what is going on inside.

On the other hand, we are able to apply the many eyes principle to even hidden models like ChatGPT — the "pretend you're my grandmother telling me how to make napalm" trick was found without direct access to the weights, but we don't understand the meanings within the weights well enough to find other failure modes like it just by looking at the weights directly.

Not last I heard, anyway. Fast moving field, might have missed it if this changed.

This is because almost no law is perfectly enforced. My standard example of this is heroin, which nobody defends even if they otherwise think drugs are great, for which the UK has 3 times as many users as its current entire prison population. Despite that failing, the law probably does limit the harm.

Any attempt to enforce a ban on GenAI would by very different, like a cat and mouse game of automatic detection and improved creation (so a GAN even if accidentally), but politicians are absolutely the kind to take credit while kicking the can down the road like that.

Ban leads to a black market, leads to lousy purity, which leads to fluctuations in purity and switches to potent alternatives, leads to waves of overdose deaths.

So yeah, banning LLMs may turn out to be not so wise after all.

Heroin already is openly accessible. As much as one can try to argue this is an accessibility issue, it really isn't. Anyone who is motivated to can get their hands on some heroin. The only thing that's made harder to access is high quality heroin. It's only a quality control and outreach(to addicts) issue.

Most people in a well-functioning society wouldn't develop a heroin addiction just like most people don't become alcoholics just because alcohol is easily available.

So yes, I believe heroin should be legal, and available to adults for purchase. And you're gonna have to do better than saying I'm "out of my mind" to convince me otherwise.

You'd see a massive increase in heroin use if it was fully legal. And that's not a good thing jn any possible way.

Do you think heroin is more or less commonly used than tabacco or alcohol?

We've already seen from the opioid epidemic that drug companies at least are both willing and able to push massive amounts of an opioid drug on patients if it is even slightly legal to do so. Why do you think it wouldn't be much much worse if it was fully legal?

Note, I am not against decriminalizing heroin and any other drug. I fully agree that the war on drugs is a massive problem. But that doesn't mean that full legalization of hard drugs, especially the most addictive of them, is the same.

Certainly not advocating doing anything the way the US does it, in almost any respect. But the pharma thing is fundamentally a different issue. The reason people so easily become addicted to painkillers is because they're initially given for pain by someone they trust, while unbeknownst to them they've actually been given too much, for too long. In other words this is an issue of doctors abusing their authority.

Why don't I think it would get worse? Because of cultural norms. Legalising heroin won't make it cool and socially acceptable to use. People don't actually derive their social values solely from what happens to be legal.

Let me ask you a question. Why isn't everyone an alcoholic or a smoker? Why do they become rarer(in the west) despite those drugs being legal?

(Hint: the reason is cultural).

If you're for making it fully legal but heavily regulated, somewhat akin to a prescription drug, then I more or less agree.

https://www.jrf.org.uk/prescribing-heroin-what-is-the-eviden....

My elevator-pitch-summary understanding of UK drugs laws is that almost anything can be prescribed by a doctor, but almost everything is illegal unless a doctor does so, and there's 3 levels of severity if you have or supply something without that authorisation.

I guess you could always move to a desert island and build your own semiconductor fab from scratch if you were really committed to the goal, but short of that you're going to leave a loooooong paper trail that someone who wants to make a quick buck off of you could use very profitably. It's hard to advance the state of the art on your own, and even harder to keep that work hidden.

Or one could use honey instead of vinegar: Offer a fast track to US citizenship to any proven AI expert who agrees to move and renounce the trade for good. Personally I think this goal is much more likely to work.

It's all about changing what counts as "cooperate" in the game theory.

Incentivizing people to become AI experts as a means to US citizenship.

https://en.wikipedia.org/wiki/Perverse_incentive

Too high and you end up with more experts who then decide "actually it's more beneficial to use my new skills for AI research"

Tricky to get right.

Publicly. Then possibly work for the NSA/CIA instead.

> ... bounties ... on people who are not US citizens.

Because that's not going to cause an uproar if done unilaterally.

It works for people that most of the world agree are terrorists. Posting open dead-or-alive bounties on foreign citizens is usually considered an act of terrorism.

Yes, obviously. They may be working on it to some extent, but they are yet to actually develop a nuclear weapon, and there is no reason to be certain they will one day build one.

Also, there is another research area that has been successfully banned across the world: human cloning. Some quack claims notwithstanding, it's not being researched anywhere in the world.

If you banned proprietary software, this would seem a lot more practical.

If the "AI safety people" are criminalizing open source models while being silent about military uses of private models, doesn't that say more about the people calling them "ai safety experts" than the idea of "ai safety" itself?

If I were a betting man I'd say the fact that the most public "AI safety" orgs/people/articles don't address the actual concerns people have is intended. It's much easier to argue against an opinion you've already positioned as ridiculous.

Although I fear the response would be that powerful AI orgs would put copyright filters in place and lobby for legalization for mandatory AI-DRM in open source AI as well.

If you mean something more transformative, did Yudkowsky get a licence for HPMOR? Did Pratchett (he might well have) get a licence from Niven to retell Ringworld as Strata? I don't know how true it is, but it's widely repeated that 50 Shades of Grey was originally a fan fiction of the Twilight series.

Etc., but note that I'm not saying "no".

I do wonder whether that will become accepted case law, assuming something like that is argued in the NYT suit.

“Bathed in the soft hues of twilight, the sleek feline, with its fur a tapestry of midnight shades, beheld an approaching canine companion. A symphony of amber streetlights cast gentle poles of warmth upon the cobblestone path, as the cat’s entered eyes, glinting with subtle mischief, looked into the form of the oncoming dog- a creature of boundless enthusiasm and a coat adorned with a kaleidoscope of earth tones”

Since an AI has produced the latter from the former, there is no meaningful transformation.

In law, in the eyes of those that want AI to "win", or in the eyes of those who want AI to "lose"? For all three can be different. (Now I'm remembering a Babylon 5 quote: "Understanding is a three-edged sword: your side, their side, and the truth.")

I don't much like that laws seem so messy either, but that just doesn't seem like much of a reason to think it won't shake out like that.

The law may well start on the basis of what people feel, and work outwards from there to things that might, or might not, be actually enforceable. And the headline summary of that law may or may not have much in common with the details, which is why e.g. the USA PATRIOT Act has that name.

Likewise "write a novel in the style of Terry Pratchett"

So even if your desired system were implemented to a T, you could hire someone to write a dozen or so examples of Terry writing. Probably just 30 or so pages of highly styled copied text, and then train your bot on this corpus to make “Not Terry” content. Boom. $100 on gig author and then for all practical purposes the Terry style is legally open source. Terry doesn’t even get the $100!

Saying "a-HA! But my automated process can produce something that looks much like what your human would!" does not negate that; it merely makes it hard to tell the difference at a glance—which we already know to be the case.

At least the Church understood the value of architecture.