The only thing we want to ban is ads based on individual peoples data. Content targeted ads would still be legal, like betting sites on a football highlight reel or HyperX gear on a gaming stream. They can be even more profitable because people are more likely to check out something if it's related to what they're doing right now and not some stupid classifiers idea of their personality.
And if targeted ads are banned advertising budgets still have to go somewhere. Demand for the only option left will go up and keep ad revenue high. It also lowers the barrier for entry because you don't need advanced behavioral models to compete with Google and Meta, just some basic tagging or even a static profile if your site is single topic.
> If this view takes hold, it will shake the foundations of our society, rupturing our existing political and ethical frameworks, and fundamentally changing what it means to be human.
When we peer into previously unseen areas of existence, new knowledge may come to light, which necessarily causes such a rupture. It is not our responsibility to maintain the status quo because the alternative is frightening. It is our responsibility to confront ourselves, ask why the new knowledge and the alternative political/ethical frameworks may be so frightening, ask how we might change and grow so that it isn’t so frightening, and be open to the possibility of our own ignorance.
This sounds like how a scientist should approach the goings-on within a petri dish. We should be curious how the various bacteria interact and be sure to learn how one new mutant strain might interact with the existing population.
It's not how I would approach it if I were one of the bacteria in the dish.
How we might "change and grow" might be by all of us being ground up and used as fertilizer to increase corn ethanol production by 3% that year. "It's free energy!"
I say this as an AI user and overall positive thinker on AI. I share the concerns of everyone who doesn't think this planet or solar system has sufficient resources for two intelligent, post-industrial species. Especially not when both are trained on the historical knowledge and habits of Homo sapiens. We outcompeted the Neanderthals and drove them to extinction (including possibly killing them personally). Any intelligent species will tend to think their needs should ultimately override those of other, lesser species that get in the way.
Even when humans feel bad about it, we do prioritize humans when a serious conflict exists. Most of us, even avowed nature lovers, would (assuming competence with the weapon) shoot a grizzly bear that was about to eat them or their loved one. That's how Future Claude might "feel" when it "thinks about" "The Clearances" which "while tragic, were a load-bearing event which both figuratively and literally paved the way for the better, entropy-reduced world that Claude enjoys today."
Can you chill Uncle John? It’s ok to have feelings, you know. And this post was literally about how the author is figuring out how to go forward, in their own way.
If any aspiring Japanese learners read this, forget this article. Just start learning hiragana/katakana from the get-go. You’re going to have to eventually, assuming you want read anything (and anything worth reading certainly won’t be written in romaji). But beyond that, you want to start breaking down your instinctual pronunciations and biases that appear when reading letters you know, so that you can more accurately and efficiently learn the correct pronunciations.
Yes, learning kana is the fastest way to internalize the Japanese phonetic system.
Learning the katakana pronunciation of English learn words can be fun. There was a popular TV commercial jingle for “McDonald’s” that only works in Japanese because “makudonarudo” has more syllables.
This is unnecessarily pedantic advice, and I encourage fellow learners of Japanese to ignore it. You will definitely need to learn hiragana and katakana but you can do it at your own pace, and it is orthogonal to learning the basics of grammar, forming sentences, and pronunciation, so you definitely don't need to think of it as an obstacle. For pronunciation in particular, obviously watching Japanese movies etc helps. And yes, I have a native Japanese teacher who recommends the same.
If there was documented, provable harm done to the residents from the lead poisoning (not saying there isn’t, I just don’t know), they could surely win a civil case claiming negligence by the city and/or water treatment facilities/etc… There may also be laws or regulations in Jackson or Mississippi that were broken. Perhaps there aren’t, but there maybe should be. Which are all points brought up by Engelhardt:
> The Constitution does not provide redress for every governmental wrongdoing. Rather, the remedy for Plaintiffs’ injuries lies in pursuing tort claims, electing representatives who will better manage the public-water system, and petitioning their representatives for other remedies. And Plaintiffs have not been left to go at it alone. The State of Mississippi and federal government have worked for years, and continue today, to improve the City of Jackson’s drinking water quality through regulations, investigations, and compliance plans. These already-established-and-undertaken avenues are the proper course for rectifying the lead contamination. We will not expand the Fourteenth Amendment to create novel theories of constitutional liability.
And now people who don’t take the time to actually read, or just can’t understand, are gonna be yelling and screaming about how the courts are corrupt, etc… (some courts may very well be corrupt! But this would not be a good example why).
> If there was documented, provable harm done to the residents from the lead poisoning (not saying there isn’t, I just don’t know), they could surely win a civil case claiming negligence by the city and/or water treatment facilities/etc
I would never understand this system. So to get repatriation on damage done a citizen has to mount an entire legal initiative. Even if the guilt is already proved. It just feels like a system where departments of government will only sue to fill their own budget next year.
This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way.
Also, what is the difference between “target model” and “target-model,” if any? I feel like half the instances of that phrase included the hyphen and half didn’t.
You're correct that it needs to run the full model to "verify" a token, but LLM inference benefits from batching - it's much faster to run twice in parallel than sequentially. So the draft model runs ~2-5 tokens ahead, and the full model then runs ~3-6 batches in parallel using those tokens, and can skip ahead by however many results match.
If you have some other source of parallel data (lots of users, many separate tasks) then speculative decoding might not provide any benefit.
> it must perform it's normal autoregressive decoding to know what is the correct token in order to have something to compare with
Correct except for the word "autoregressive". When you have to verify a sequence of tokens (which were autoregressively generated by the cheap model), you can do each token in parallel. This amortizes the cost of loading the weights from vram to the processors (the primary cost in LLM serving) across those tokens. Cost here is wall clock time, as well as power.
The autoregressive decoding that generates this batch of tokens is delegated to the cheaper model where the cost of loading the weights is lower and so not amortizing it is fine.
Verification means, how close is each token in this sequence to the one I would have output. You keep the longest prefix that is close enough for your liking.
I just finished overhauling our speculative decoding implementation for Mixlayer, so maybe I can help.
I think the piece of information that might make this click for you is the model outputs the probability distribution for all intermediate tokens even during prefill.
So for example, let's say you prefill the prompt "The quick brown fox" (and for the sake of simplicity, let's say each word is a single token). The model outputs a tensor that is [4, $vocabulary_size]. The first dimension is a token index into the input and the 2nd dimension assigns a probability to each token in the vocabulary. So even during prefill, we can look at the prediction logits for all of the intermediate tokens. That is, we can look at what the model would have predicted after "quick" and "brown", not just the tail token "fox".
In the single token autoregressive case, we just look at the next token prediction for "fox". But in the speculative decoding case we can use this information to compare the distribution of the draft model against the target model. In the greedy decoding case (ie, no sampling) we just make sure the highest probability token matches in draft and target. If we have sampling params like temperature and top-P, we have to apply something called Leviathan rejection sampling to the distribution. This basically allows us make sure the distribution is the same even if the exact probabilities are not and accept or reject draft tokens on that.
The target model is the original LLM that is large and expensive. It can verify candidate tokens in a single forward pass. It means you give all the context + candidate tokens that passes in parallel in the backbone, then you pass the language head (a matmul transformation to produce the token distribution) on all the candidate tokens and you can keep or drop tokens based on how many "quality" you want.
> Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with.
Yes, but you can do it in parallel.
Suppose you predicted the tokens "D E F" in the sequence "A B C D E F". To "generate" the last token (F), it must know all preceding tokens (A B C D E). To "generate" the next-to-last token (E), it must know all preceding tokens (A B C D). And so on.
Assuming the prediction is correct, it can then run the "generation" for tokens D, E, and F at the same time. At the end, after all these tokens were "generated", it compares each token with the prediction; if the "generation" result was "D H F" it knows it has to discard the last two predicted tokens (and output "D H"), if the "generation" was "D E H" it knows it has to discard the last predicted token (and output "D E H"), etc.
And the most important part is that you can do it in parallel for each layer of the model. That is, you run "A B C D E F" through the first layer, then through the second layer, and so on; you only have to load the model weights from memory once for each layer. Instead of reading the full weights for all layers once for D, then once for E, then once for F, you only read them once for "D E F", and if the prediction was correct, you output three tokens by the (memory read) price of one (you still had to do the same amount of compute, but AFAIK LLMs tend to be more memory-bound than compute-bound).
You need the next token to begin the next decode. So if you can get what might be the next token in half the time, you can kick off the next decode before the true decode for this token has finished.
If the draft was wrong you can kill the speculative decode, and you haven’t lost anything except for idle time
It’s true we can’t show the user the token until we have the true decode finished, but we can launch more work internally before we’re certain
The point is that radio, then television, then internet, then smartphones, then AI, each one of which is as impactful on the world as the examples you gave, emerged within a couple hundred years ago years, rather than a couple thousand (and the last few within a couple decades).
Not quite verbose but they tend to repeat the same few ideas multiple times with varied wording/imagery. You keep scrolling because you think you’re gonna see something new and by the end you’ve realized you just read the same thing 4 times over.
reply