The killer was usually the downstream processing required to clean out off-flavors and colors, which was required to get even a passable product (these are just protein powder fillers at this point, not whole cuts), but ballooned CapEx and cut yield such that cost went back up to like, A5 Wagyu levels.

I’ve sat in the awkward tasting sessions the author describes, and they’re all the same. The attendees muster up a “that’s pretty good” - but we all know nobody would ever choose to buy this unless they were forced to.

Now that I’ve been out of it for a while, it’s pretty obvious to me what the problems are. Cost, obviously. Taste and texture simply aren’t there. And “the knowing” is real. I never got over imagining the whole factory process involved in the production. I’m sure vegans would say the same about animal farming, though. So my conclusion was, the only way this technology becomes relevant is if people are forced to use it. And that’s either because you’re cruelty-free by choice (small fraction of the population), or because the cost of meat suddenly skyrockets, which today, is not looking that likely. Keep in mind as well that meat costs would likely be driven by feed costs in such a scenario, which also impacts cultivated meat. So it’s ultimately a process efficiency play, and you’re competing against eons of evolution.

But stating it plainly like this would make the contradiction too obvious.

Science article in question: https://www.science.org/doi/abs/10.1126/science.aec6129

Also, the most interesting result here is that the CNN-based feature encoder significantly outperformed a vision transformer encoder backbone…

Waste avoidance.

Bullsh*t is more than sufficient to convince an AI-gulled target audience.

> the classes of firms that can accept the use of fully autonomous LLMs are few, by my count just three:

1. those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc.

2. those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc.

3. those that can accept or already do by nature the costs of rigorous specification and validation: chip design, drug discovery, and other domains where failure on deployment is an existential concern.

the first two classes are price sensitive and arguably don't need the jump in reasoning quality you see going from cheap to frontier models. most of these firms will be best served by open models running on cheap hardware, perhaps even locally at the site of use. for the first and third classes, the type of fuzzy combinatorial search that has produced headline results in mathematics and security research seems more sensitive to agentic swarm width than reasoning capacity

…

This is just so on point. And for the third class (which I would extend to things like materials research as well), specification and validation are already by FAR the larger costs, so automating search and simulation is really not a massive game changer for the broader business.

Test code looks like a mess, even more than the usual LLM code. takes a while to let it go. Test report looks beautiful though.

Like you, I've found that LLMs can improve test coverage by decreasing the amount of developer time spent writing tests. But generally, it takes a lot of manual work to set up the initial testing framework, and even then, a lot of vigilance to ensure that what is actually tested corresponds to the description of the test.

But that's digression. The short of it is I've seen enough to convince me these tools may as well be magic and a whole lot of tasks that break down into "produce media content of some sort" that has a well-defined goal and definition of correctness will be permanently sped up by automation. This includes a lot of software writing. At the same time, I shared the skepticism of estimates of economic impact and irrevocably changing the larger world. I'm a lot closer to the business side of the house these days, working with customers and prospective customers to identify use cases, reference architectures, pain points, feature requests, and bring this back to the development teams to attempt using real-world experience like this to inform how we design products. It's not product management as I'm focused more often on the nitty gritty technical details, not high-level user experience or roadmaps. But it gives me a great avenue into seeing what causes organizations to actually buy and/or adopt new software products, and the rate at which they can do that.

And frankly, it isn't moving the needle much. They have the same budgets they always had, so they're not buying more, and our business is growing, but no faster or better than it grew before agentic coding became a thing. I always wonder because it seems the glowing success stories on Hacker News come in one of three varieties. It's the solo indie dev, usually targeting mobile app stores, who churns out dozens of roughly "will compile and doesn't immediately crash at runtime" apps in the time it used to take to complete one. It's the hobbyist, making software only they and maybe their immediate friends will ever use. Or it's startups, whose monetization model isn't monetization at all; it's just having something shiny to show investors in order to convince those with loose money to give enough to you personally that you can build up a nest egg whether or not your product ultimately ends up ever having a single paying customer.

In my own business, a multi-decade, mature but not hyperscale company selling overwhelmingly self-hosted enterprise open source software, I can see the impacts on output. We have the same major products with the same release cadence. Each point release averages more new features than they used to, but also more regressions. It's overall a mixed bag. Non-technical product management staff is able to contribute code. We have a ton of new internal tools that nobody uses but they're there now. On the customer side, those that hinge decisions on wanting features that didn't exist yet are benefiting from getting those. Those that already had the features they want are losing from the greater rate at which regressions get through. The net business impact seems to be things have definitely changed qualitatively, but in purely financial quantitative terms, things are about the same as they were before. More code being committed to various git forges, but same headcount, same revenue, same margins, and same market cap.

If the models get good enough to solve all of the low-skill problems, then we only need to keep around the people who are highly skilled. That means a fewer number of software engineers, and a difficult path to becoming someone who is highly skilled.

Agreed about the barrier to entry raising though, we're already seeing that in the glut of CS grads who can't actually get a programming job right now. Personally I suspect that this is a cultural issue more than an economic one, though. Companies need to alter their expectation about entry-level engineers and develop a culture of mentorship/apprenticeship so that advanced analysis, architectural, code review, and AI management skills are all passed down successfully.

Programmer / engineer compensation in the US market at least has been looking bimodal for at least 12 years now and the first one looks even more devastated in terms of labor than the big tech companies based upon (lack of) job postings and from browsing my connections on LinkedIn.

The counter argument is of course maybe you don't need to understand our kind of intelligence to create a different kind and that could well be true but then how do you determine if a system is intelligent.

Unless the new system is intelligent enough to reason with us on our level in a way we can "see" is intelligent it becomes a philosophical argument.

We also have a natural inclination towards anthropomorphising systems that mirror us, this is already a problem with LLM's and people overestimating their capabilities or forming actual emotional attachment.

Then there are those of us who know more about how they work who in theory should be more immune to that and aren't.

I added some stuff to my agent.md to make it sound less human and to communicate more like the machine be it is because I find the faked emotion extremely jarring.

It can't be sorry, it's a set of numbers, it sits in the linguistic uncanny valley.

A priori I'm not sure why you would think being a PM at a FAANG, deciding what color the login button should be, is any different.

When it comes to judgment calls for technical decisions, a lot of interesting innovations appear to come from rejecting conventions / averages in favor of a different set of constraints as a trade-off because we challenge the assumptions we make about the demands being asked of a solution / product. I'm thinking in the constellation of the apocryphal Steve Jobs quote about rejecting asking horse riders what they want because if we asked them they'd ask for a more reliable horse.

> the history of artificial intelligence research is littered with examples of humans confidently declaring that task X requires general intelligence, then getting humiliated by a neural network doing task X better than humans a few years later

True, but the history of AI research is also littered with AI researchers confidently predicting X job will be replaced by AI and being completely wrong because they don't actually understand what those jobs actually are. See Geoff Hinton predicting that Radiologists would be obsolete by ~2020, or predictions that truck drivers would all be replaced by self-driving tech.

We will never prove AI is intelligence.

We'll only prove humans are not.

But humans still manage to wrangle these, sometimes seemingly effortlessly, through a process which we call by shorthand "taste". This is a largely vibes-based heuristic that combines expertise with life experience and cultural training -- intuition, more or less.

This is likely not possible to automate either -- aspects of it may be automatable for a given expert, in small pieces in narrow subsets of their particular domains of interest, but even those likely will require some manual intervention.

This is in part because it is, to a large degree, a black box, even to the expert deploying it. With some self-awareness and strong language skills we can articulate approximations of the judgements that go into taste. But even those will fall short, as even the most self-aware individual will fail to notice certain judgements and dependencies.

In practice many of these are not even explicitly articulable. Humans are idiosyncratic and messy and dynamic, and the suggestion that we can build a machine that approximates this in a way that pleases our sensibilities and doesn't require supervision is kind of ludicrous, even in light of recent developments.

I have my own theory about why it's impossible to remove the human from the loop:

1. Any task emerges from a need, from a human context. We need the human to pay and assume the risks and costs of using the model. So intent emerges from context.

2. While the task is being worked on, constant interaction with the context is needed, for action, for feedback, and for steering.

3. At the end of a task, consequences accumulate in the context, they don't fly to the model provider. The cost, risk, liability, gains and losses remain there.

So the LLM is great except for the start, middle and end of a task. Contexts are humans, teams, projects, and they are maximally distributed, you can't copy a context, it is indexical and relational, just as you can't copy my phone number or eat for me.

It feels like today, taste in design is similar to where software engineering was about a year ago.

Though it is not hard to imagine that any and all communications being recorded for AI consumption in the future.

The Next one is the relative lack of prompt feedback (expect the blowup in finite time like Navier-Stokes ;) [there is not much feedback even for humans at middle management positions].

The cost [tokens] might become prohibitive unless LLMs improve further [not a guarantee].

You can make llms perform judgement, and maybe that will get you some progress. But ultimately the value is going to come from engaging with other humans.

LLMs are great at helping with aspects of market research and that’s about it. Aka it’s a good deep search engine. It’s not going to decide what features solve certain customer pain points. It’s certainly not going to prioritize and coordinate between competing stakeholders.

That's like 1/4 of Codex users burning through their free resets just to build shinier todo apps.

It's a shame that we've had decades of shitty customer service from companies that have the most responsibility and resources to do it, and people in tech have shrugged their shoulders saying "it's unreasonable to expect people to scale up their customer service! They gotta make money!" Now AI customer service has arrived and things are worse, not better. Now we can't even get in contact with anyone because an agent redirects us to documentation instead of directing us to a human being.

The future, when a company steals your money, is making viral posts online trying to get enough public shame going that you will maybe get some of that money back.

But, when properly used AI can reduce the customer service load. It can handle a lot of simple questions and status updates. There just needs to be a human available when there's any request that escalates past that simple case.

The problem with this angle is that it is still absolutely terrible at doing call center/customer service work, and the profitability story is that the price is going to go up rather than go down.

For repetitive physical labor in a controlled environment I'm slightly more bullish, but if you control the environment, you mostly don't need AI. You just use traditional deterministic methods, and send a person in when things get stuck or things are by nature irregular.

> 1. those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc.

Those who can accept failure cheaply can't necessarily detect failure cheaply. A ton of insane attempts will have to be picked through carefully to find the candidates for success, because the lack of a thought process makes AI bad in random, inhuman ways. This is basically a version of 3) that wishes away tests. It will be (and is) certainly helpful to replace interns and aid in rapid prototyping, but not because failure can be accepted, but because those are things that are tightly supervised. According to the world thus far, that is resulting in anything from -15% to +25% productivity gains. I'm not seeing it as a game changer simply because if it was, I'd expect to have seen a lot more useful, original software products by now and I haven't. I've just seen old ones get buggier or rewritten in Rust.

I'm only buying 3): when you just want a machine to randomly enumerate through a search space looking for things that make the carefully constructed tests pass. That's a very good thing, though. But as you say, it's not a game changer because you still have to write the tests.

but

> 5. navier-stokes and statements in pure mathematics like it are the absolute best case scenario for agentic work against rigorous specification. the theorem statement itself is already a rigorous specification. it has undergone decades of auditing by the mathematical community and its rendering in lean is a straightforward translation defined in terms of battle-tested mathematical objects from mathlib. the verifier, the lean theorem prover, has been extensively audited and specifically designed to avoid the types of unsoundness that would make it vulnerable to reward hacks. even lean and theorem provers like it are not invulnerable: soundness bugs have allowed LLMs to launder bogus proofs through the proof kernel before and it is not improbable that more such bugs exist. this is the rosiest setup; the vast majority of human knowledge work does not look like this. i'll comment below on the few areas of knowledge work that do resemble pure mathematics in this respect.

This is the real deep point, and one I've been repeating since I heard Navier-Stokes was a fraud.

This is exactly where I expected that LLMs would do well, and they are not.

It shows that I have a basic misunderstanding of LLMs, and that misunderstanding is causing me to think that they have more potential than they have actually shown.

Maybe the nature of the architecture, where it picks out features, intrinsically limits its ability to search a solution space?

Maybe the fact that they modally predict what someone might say, and nobody has said a thing as of yet (when many people were knowledgeable enough to have, if it is correct), means that the LLM is not going to say it either?

Maybe the fact that it consumes all information and blends it in a structured way, instead of synthesizing an entire space from a relatively very small amount of input like a human does, means that it won't ever accidentally synthesize something that can't be pieced together from things that have already been said? Is its accuracy its flaw, where a human's "mistaken" synthesis might ultimately correct everyone's understanding?

Really not beating the charge of being a stochastic parrot. It might just be that we were underestimating stochastic parrots; if a million monkeys on a million typewriters were all getting treats when they satisfied a trainer who wanted to see a new work of Shakespeare; they could look at his published work, and they could watch each other type and when each other got treats; whenever they successfully spelled a word or put words into an intelligible phrase, that was made into a keyboard key for a group of sentence monkeys, and the successes of the sentence monkeys were made into keys for the paragraph monkeys, etc... could you get something that passed for mediocre, drunken Shakespeare in a thousand years? Or maybe even 10?

I think they’ve made a shrewd move in focusing on search integration and everyday users (Gemini app) vs software power users. They have their corner and nobody is really competing with them, plus it feeds directly into their existing revenue stream.

Maybe everybody is wrong about how valuable these companies will be, but atm it looks very dumb to not be competitive in coding.

Occam's Razor is overwhelmingly that they just don't have the organisational capability to capture this market. If they did then they absolutely would have.

If OpenAI or Anthropic go away tomorrow, people find a new model and move on. If Google goes away tomorrow, people’s lives would be severely disrupted, and in the case of Gmail / Drive / auth access, even temporarily collapse.

This is an interesting framing, but when you put it this way, it makes it quite clear that intelligence without actionability is not so valuable. And capital is what creates action.

I can have the smartest doctor in my pocket, but if they can’t administer treatment, I’ll still die. I can have a world class scientist to talk to as long as I want, but they can’t build me a breakeven fusion reactor. A pocket lawyer… slightly more useful, but they still can’t take a case to court for me.

Talk and ideas are cheap. Always have been. Now they’re even cheaper. Implementation takes resources, and the more you have, the more useful intelligence is to you. Capital will be just fine.

This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level. You trust that those layers work. At some point in the past you couldn’t just trust that. At some point in the near future, you will just trust that the software code works. Instead you’ll just debate the spec (the prompt). Your thinking can move one level up the system hierarchy. Everyone’s can. We’ve done it before, we’ll do it again, and this won’t even be the last time it happens. Let go of the anxiety and let it happen.

AI systems are not the same. You can't guarantee deterministic output.

Also, code is still the best language we have to convey our intentions as engineers into function. Natural language is too imprecise and ambiguous.

Understanding the code allows us to understand the intention behind the code and identify bugs, plan and architect better. Inspecting prompts gives you a blinkered view of the system meaning you're more likely to make incorrect assumptions leading to serious bugs.

When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators.

Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.

There are always limits.

Engineering human types are often guilty of saying something is low cost but then not doing it themselves. Great, "understand everything".

Trust us trust. You trust an entity. What the author is describing is a new kind of transaction. It's not that AI causes people to lie suddenly (although if you trusted the wrong people this might happen to you, but YOU were wrong then) but now you have a lot of work being done more at the edge of uncertainty and recommendations. Ideas are not ideas anymore, they are working implementations.

So instead of submitting discussions or requests for features, you literally submit implemented features because the cost is lower. This is good. You just have to realize that you are starting the discussion with a working implementation. If you continually try to act like this is not the case, without someone literally lying to you about it, this is more about just accepting a now workflow.

And if I can't say that for a coworker, well, that's performance feedback.

While my coworkers are very stable personalities of a consistent work drive, sense of pragmatism, things that they find more and less interesting, career aspirations, etc. they're not all the exact same, and I depend on that as I fit tasks and ownership to people.

Compare a single session. It’ll be extremely consistent.

Unlike AI, those layers were traditionally compiled using deterministic tools. And generally came with a suite of other tools to trace and debug them. I don't trust AI in the same way at all.

Just because something wasn't required at all levels or 100% it doesn't mean it wasn't required or it wasn't important though.

I trusted that the machine code worked because it was a mostly mechanical transformation, with little to zero non-determinism involved, and the design decisions for the high level code were still mine and understood. Not the same with an LLM, even when see just as an added level of abstraction.

Given some LLM produced code, I also trust the compiler will produce working machine code matching it. Doesn't mean I trust the LLM code in the same way.

I figure what is happening is (human) software engineering as a discipline, at least on large scale projects, is rapidly and irreversibly mutating to software project management. But instead of managing teams of people, you're managing teams of agents. Where I disagree with your comment is in the assertion that everyone "can move one level up the system hierarchy." Some folks can't, so they'll stagnate. Welcome to the Peter Principle World.

If you're developing an agentiv system to build software for you in an automated way using a fleet of agents, you're still an engineer of a technical system even if that system is operating on somewhat fuzzy logic.

Management involves complete delegation and instead the task is to find a capable person for the task and then mostly let them just give feedback so you know whether you hired the right person or whether you need someone else. When you're working with a fleet of agents, you're not assembling a team of heterogeneous workers with different skills, you're hiring the same guy over and over again, hoping that he can do any job and yy same guy I also mean that the guy does not learn anything new over time.

Meaning traditional management skills count for nothing here.

The difference of course being that you had 20+ developers who did understand it. With AI generated code no one has that understanding. The AI sure doesn't

> Some folks can't, so they'll stagnate.

And we have no answer for those people. Just "fuck you the world changed and you're being left behind"? Where exactly does that lead?

But I think what’s being overlooked in the race to claim absolute credit is that both sides ultimately relied on a LLM (and one of OpenAI’s at that). Either a human researcher made a breakthrough discovery with the help of Codex, or the latest GPT model made a breakthrough with the help of human training data, or a little of both… either way it is undeniable that LLMs have quickly become an integral part of R&D workflows and are accelerating research.

This would be a major win for any normal company. You could even build a bigger collaboration with this guy, give him a big budget and push for extensions to this preliminary result, and in return do a write up on how he uses your model in his workflow. Huge PR win. What this says to me is that their valuation is so astronomical that they feel the only way to justify it is to demonstrate a fully autonomous discovery bot… which it simply is not.

The frontier labs are likely betting on lay observers (read: investors) confusing headline-grabbing results in abstract mathematics with phenomenal profitability in more grounded endeavours. There is an implicit fallacy that "If our models can solve mathematics they can do everything else."

2. Even their pursuit of this problem was itself unprofitable - $15M in compute to solve a problem with a $1M prize. Not that that was the point, but still.

It would actually be a really interesting study, if they would ever be willing to be transparent about this, how the result differs with and without his conversations in the training set. How quickly it arrives at the result, whether it takes the same approach, etc.