There is a wide gulf between catastrophe and killing us all, and its not just that these people are weird, they seem to be complete clowns, tossing out % predictions about incalculable future states like spock in star trek. Total clown show.
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
Anti-intellectual? Because they don't believe claims that are backed up only by untested hypotheticals?
If we're going to have whistleblowers emerge declaring a 10% chance we all die because of AI, we're going to need more than vibes. Real experts can explain their position plainly. That has not happened here.
I read Bostrom's Superintelligence: Paths, Dangers, Strategies in 2014. It was interesting, but not terribly persuasive. Does If Anyone Builds It, Everyone Dies contain arguments that would be new to me after reading Superintelligence?
I would personally expect so, but your mileage may vary. Controlling for the fact that you didn't find the former persuasive, my first thought would be that the latter might help. Among other things, it is much more specific and detailed in various ways, not least of which being informed by an additional 11 years of history. The former was not written in a world of pervasive AI, or with current AI architecture in mind; the latter has that context.
You said "Real experts can explain their position plainly.". The experts have explained their position plainly. If you disagree, you should be able to explain your position plainly; "feels like vibes to me" is not an explanation, it's a feeling.
If you want to argue that there's something invalidating the arguments, by all means do. But it's really tiring to get many variations of "nobody's made the case for it!" or "there's no evidence for it!" or "vibes!".
My position is that there are too many off-ramps between us and the proposed inevitability. And the argument is not symmetric. The null hypothesis is that the world continues to function. The burden of proof is on the authors of the book to show that human destruction is an inevitability as the result of this technology, and they have failed to show that.
This is cited in multiple reviews of the book in reputable publications. The book simply fails to compel belief.
That doesn't explain the many former AI researchers (including Yudkowsky) who quit the field because they came to believe it is a potent danger: they didn't choose to become AI researchers because they started with an assumption that AI research is too dangerous to be allowed to continue.
The writers for Star Trek spent a lot more time thinking about interstellar travel than I do, but I'm not going to go to them for advice about building a rocket ship.
The Rationalists are a philosophical/pseudo-religious organization in the same way the Daoists are. Just spending a lot time thinking about something doesn't make them right, it just makes them internally consistent with their chosen axioms. That is useful: if you agree with the axiomatic beliefs of such a movement, you can probably defer a lot of reasoning about the conclusions of those beliefs to the authority figures within that movement. But it doesn't change the probability of the axioms themselves being true, any moreso than the Pope saying so affects the probability of God existing.
AI Is Important And Potentially Dangerous is an axiomatic belief in the rationalist circles that are relevant to AI safety, in the same way that God Exists is axiomatic to Catholicism.
That's a fascinating assumption, and I wonder how you got to it, because it doesn't match anything I've ever seen. Most groups look more homogeneous from the outside than the inside. You are assuming their belief is religious and axiomatic, rather than very carefully reasoned about for decades, iterated on, discussed, debated, tested, and refined.
> Most groups look more homogeneous from the outside than the inside.
The converse is also true: Most groups look more heterogeneous from the inside than the outside. Again, to use the Christianity example: it takes a fair bit of inside baseball to figure out how various different sects and denominations are any different, but if you ask one of their priests, they'll be sure to tell you how Baptists and Episcopalians couldn't be more different.
It's a symptom of normalizing to an Overton window. From inside a church the belief that Jesus was just a guy who talked to God might seem an extreme fringe belief to someone who believes that Jesus is fully God, but from the outside we can lump them together as axiomatically holding that God Exists, because we have an Overton window ranging from "God doesn't exist" to "Allah talks to me everyday, and she gave me my promotion".
"Most groups look more homogeneous from the outside than the inside."
But that doesn't mean that they really are all that different in the end. It reminds me a of joke as told by Emo Philips that I think is quite apropos:
==-==-=
A man was walking across a bridge when he saw someone about to jump. He ran over and said, "Don't do it! There's so much to live for!"
"Like what?" the person asked.
"Well, are you religious?"
"Yes."
"Me too! Are you Christian or Jewish?"
"Christian."
"Me too! Protestant or Catholic?"
"Protestant."
"Me too! What denomination?"
"Baptist."
"Me too! Northern Baptist or Southern Baptist?"
"Northern Baptist."
"Me too! Northern Baptist Conservative or Northern Baptist Liberal?"
It's good game theory. You're much more likely to be able to bully a Northern Baptist Conservative Council of 1879 members into joining your cult than anybody else.
That not exactly a withering criticism of the rationalists because most people who have any opinion about AI believe that AI is important and potentially dangerous.
In a circus troupe, clowns are a critical role, and are an important, if silly, job. In the AI jobpocalypse, we should all be so lucky to have any job.
Really? Forgotten more than we know? So what can they tell us about how this technology works then? It seems a lot of it, besides the actual training engineering parts, is handwaving about "black boxes" and "divine benevolence" (a quote from Noam Shazeer).
