>An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.
This is not really true in practice because of multi-threading and out-of-order execution. Mathematically equivalent orderings of operations are not equivalent when dealing with floating point values, so most practical LLM implementations end up being non-deterministic.
"An LLM is just as deterministic as any other computer program" is not refuted by pointing out hardware limitations that would affect any other computer program implemented at similar scale (weather forecasts, or even just computing the average of a large stream of sensor readings).
I'm not really trying to 'refute' the original statement. Certainly, an LLM is just doing some calculations that can be done deterministically in principle. However, I think it's worth pointing out that there are practical barriers to doing those particular calculations both deterministically and efficiently. People who worry about LLM output not being reproducible aren't necessarily misunderstanding what an LLM is doing; they are responding to a real feature of most practical LLM implementations.
If you wanted to and had enough engineering effort to spare, you could run an LLM deterministically at relatively small impacts to performance.
One approach is to make sure you run things in the same order. Another is to change your operations so that more of them become associative or even commutative.
See eg the paper 'A Lattice-Based Approach to Deterministic Parallelism' for some interesting ideas on the latter.
"Astra's hypothesis for why this particular message was previously unsolved is that “TRUPPENVERSCHIEBUNG” was used as the key starting on December 9, 1918 - whereas, as noted above, this message was transmitted earlier, on November 27, 1918. The reason for this discrepancy is unknown."
So it used a known key. It didn't come up with a key from thin air. The only gotcha is that apparently this key was used two weeks earlier than it was documented (maybe the operator was using the wrong page from the codebook?).
The cerebellum isn't just for motor skills. It's more of a place where very commonly used patterns get a bunch of dedicated wetware so the computation is superfast and precise. In an (undoubtedly very limited) analogy a bunch of FPGAs to complement the general processing. It does make a lot of sense to use this functionality for motor skills, though.
It's spent on 'fast, effortless' recall. Maybe, long-term memory, or memories associated with activities like balance, singing, and anything that requires precise steady control.
A keeper of balance and rhythm, if you want to think of it that way, and anything that is too much of a burden to creativity: a mind is a bunch of ideas (thoughts) collected into a moment in the mind, all the ones needed for controlling things like balance and rhythm are best handled in an autonomously looped fashion (think Cybernetics: control loops that guide an agent).
The article mentions the forebrain controls creativity, reasoning, higher-level thinking. And the hindbrain controls muscles of the throat, tongue, etc: so speech, singing. We all reach for some 'flow' state: perhaps it's this state where the 80% and 20% are more in harmony within the mind. Where what we create with our minds is appreciated, and the outer loop that critiques ourselves lays quiet. 'Flow' is achieved.
Everything becomes an autonomous loop: which is the study of Cybernetics, and (coincidentally?) the birthplace of AI.
If we follow the pattern of simulating biology, I would expect some ratio of compute could be determined 'optimal' for the new Dream-RSI technique recently: https://www.dream-rsi.com/ to eventually move towards '80%' of compute for autonomously controlled systems, and 20% towards 'something else'. The ratio won't be there in the beginning because an AI system has different autonomous needs.
Through testing and iteration, we'll arrive at some ratio as technology matures: my guess is it will match the one we find in our own heads: 80% for autonomous functions, 20% for creativity.
Keep in mind, dreaming and recursive self-improvement as it is now, is a big unknown: nobody knows what ratio is best, but honestly it's less important because we're all looking at 'Quality' of results right now.
But as the technology matures and moves into robotics and more, and more complexity is required to 'do work': I expect 80% to be accurate.
I can't quite agree with your framing of loops, balance, and rhythm. A lot of the output patterns produced by the cerebellum are singular actions: tying your shoe laces, for instance. Progressing from learning how to do that to struggling whenever you try to do it differently than you always do (start halfway, reverse directions, verbally describing what you're doing, etc.) is more indicative of the optimizations that are encoded in the cerebellum, imho.
I can certainly imagine AI architectures evolving similar dedicated circuitry to perform common computations. As in us, it might be particularly useful for applications where some form of motor control is required, such as in self-driving and robotics.
It's a bit harder to come up with areas beyond that where it makes sense for general AI. Math and physics spring to mind, but in general the tendency to snap to a certain handling of an input pattern might be way too limiting in terms of creativity; it might become narrow-minded and primitive in those areas. OTOH, again, that might be fantastic for narrow AI applications.
Pipes and cycles is a good way to think about it. Our brain has cycles and we have pipes that move information around: we're processing information to persist.
Some of the information we're aware of, some if it were not, and sometimes we want to be unaware, of certain things, when we're aware (bored).
Recently we learned there are 2 separate lines of development: 2 separate progenitor stem cells that lead to a fully developed front and hindbrain.
Cybernetics is about autonomous loops: there is a loop that keeps us alive. The loop iterates, has rhythm, cycles. There is information that has to be 'brought into a moment' to control your body and mind. And we need two systems to keep our stuff held together and give us a sense of self.
Whatever you think of a given moment: the electric order required to operate your body seems to break down to about an 80/20 ratio of processing power, brain material (a dual system in itself, with two halves) if you consider neuron count.
An 80/20 split to properly persist us and provide a 'now' experience that grows over time. We use the same system to imagine and predict.
Also: consider recall speed, and speed of information transmission itself. What is required for the brain to respond to input, or just persist the body? It's not only important to move information quickly, you just move the right information.
When writing? When riding a bike? Singing, sleeping, etc.
For any moment, like the singular moments you mentioned, we wrestle with boredom and recreation. And we balance time, work, etc. because we have basic needs that have to be met first. Your heart beat must be adequate, you need a certain amount of breath, sleep, food, etc.
It would be a lot of mental burden: try manual breathing and going about your day for as long as possible.
The ratio of neurons is about 80/20 for some reason, and it must be a good one for us.
There is a lot of looping and control at play however 'aware' we are.
The cerebellum piping, the control, requires a lot of matter/piping, and the ratio is interesting. I'm considering a whole robotic thing: servos, and brains, and all the control that's required to balance input and output to 'do something', the keepalive loop for a robot that thinks and moves like us:
Bringing it back though, there's an outer loop that allows us to persist. For us it's two systems.
Our 'keepalive' loop two systems, 80/20 split.
is an outer control loop for robotics, like you're mentioning. Those pipes and cycles from the start of this message. If the goal is to create something that resembles a living creature that moves and responds to environment, and does work to persist:
I expect to see a repeated pattern again, the 80/20 ratio. 80 percent required for managing autonomous loops, and 20 percent to do 'creative thinking' or something else, simply because of the new 'dreaming RSI' news: new insights into automated improvement, another autonomous loop.
As we add more loops...the ratio might matter, depending on what you want to do. But if it's lifelike robotics, I expect to see the 80/20 split. But obviously we would just iteratively work out the optimal ratio based on economy.
Good! My dream is an alliance with Canada and Australia, a truce with China, the dismantling of Russia (Siberia goes to China), and the isolation of USA.
China has a reasonable claim to Outer Manchuria, which includes Vladivostok; the treaties giving it up were signed under duress. But that's a far cry from all of Siberia. Siberia has not been treated well by the Russian or Soviet empires, but many non-Han territories have also been treated poorly by Chinese Empire.
The word concentric reminded me of sort of an inverse idea in vinyl. Equally cool IMO: concentric grooves. Rush did it. Kate Bush did it. Probably more recent examples. It works like this: drop the needle at a different starting point and the whole record plays through a different groove. Cool, eh?
Compliments of product managers, engineers, etc making millions and millions of dollars to produce a much worse end user experience, if it wasn’t painful enough
reply