Llms are far from being ingenious or innovative. The main way they can surprise you today is pulling in information that’s new TO YOU - I impressed with the algorithms they pop out sometimes simply because never formally studied compsci. But when it treads on my home turf I find it very basic, and for bigger picture stuff like system architecture its terrible, by its nature it doesn’t have a “vision” of how the whole system fits together
Strong agree. My 30 years of experience building computers and breaking them has made me better because I don’t trust my knowledge or myself as a first principle. If I didn’t have that and I could easily see myself in an alternate reality, coming in late to the game to leverage ai to build a car out of seatbelts; harnessing all the power of 1000 suns seeing that idea to fruition.
It's like reading a news paper, when it's about your field of expertise you find lots of errors, but when it's about something you don't know you take it at face value.
> Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know. —Michael Crichton
Edit: sibling comment explains it much more plainly.
I'd argue it is the other way around - LLMs are great at pure creativity, creating something that never existed before, but not always at "pulling in information that’s new TO YOU". I use them for entertainment - jokes, short flash fiction etc. and they are excellent at it.
I can believe this is true. It's great at going out there, connecting ransom dots in very clever ways, sometimes stumbling on something actually funny.
I'm curious, did you test with the 200 jokes if it could guess which were the beat ones according to a human audience?
In images, music, games, it usually can't. So the process of validation is offloaded to the audience. I'm very interested in seeing where this takes different platforms. I bet Spotify can pull off such a validation process. Music is passive, tolerating a bad song doesn't require much from you. And they have a massive audience. They can try out AI generated mess with a very low success percentage without saturating their playlists and alienating their users. But can Steam tolerate it? Trying out a game is a much bigger effort than listening to a song picked for you. Customers are already reacting to the number of demos in Steam Next Fest. The saturation point is much lower.
And what about jokes? What happens to r/jokes if even 10 standup comedians are validating 200 jokes per day?
And we see it in software too. QA is already being offloaded to the customer.
I want to see where this goes, even if I don't like it.
But if you find a media where AI can also validate the output instead of having to offload it, that's going to be very valuable!
The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
If you know how they work and how they are trained this makes perfect sense.
Going out and searching the web gets around model cutoffs, so fixes the latency of knowledge issue a bit, but they’re still finding statistical modes of what they’ve seen before.
“Stochastic parrot” is seen as a crude and dismissive jibe at LLMs, but it’s actually a concise observation of the reality.
That's not really true any more, since given tool use and iterative 'chain of thought' reasoning, it's more like an answer refined by conversation with the asker or the second best answerer, from the best answerer, is the best that you can expect.
E.g. consider the best answer describes how to achieve something with some CLI, but the CLI's changed and the command is now broken. It can discover that, combine with the 'best answer' about what changed in the CLI, and fix it to work for the original problem even if that's completely novel and there were no extant answers about solving the problem with the new CLI form, or that it had broken that use case.
It doesn't change fundamentally how they work of course, but 'thinking', creating that feedback loop, made them significantly more useful IMO.
The “conversation” is just one between some redditors or book publishers.
It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.
There is no “thinking”. Just more attention, which is apparently, all you need.
All novel insights expressible as text can be assembled from snippets of existing text. The challenge is "only" that you need to find the right snippets. Hence being a stochastic parrot does not represent an upper bound on capabilities, but a suggestion that if you want your parrot to give you a specific answer, you should arrange for a context where it's statistically likely to be correct.
> The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
Hard disagree.
Claude is able to come up with solutions in very niche areas where there is little discussion on the web, because it takes steps to digest the source of (for instance) frameworks and libraries, and can then use/advise on lots of esoteric options, edge-cases and all sorts of stuff that doesn't get discussed on the open web.
I work in a niche where it is utterly hopeless at this, and everyone in the niche agrees: it produces things that look convincing to non-experts but that is fundamentally derivative dross.
It’s great at bread and butter stuff, but not at specialised ideation.
If you know differently, please share examples as maybe we can learn something here.
> It’s great at bread and butter stuff, but not at specialised ideation.
This is fundamentally different from your earlier, stronger claim that - "The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago."
I use a less popular java framework quite a lot, Micronaut (I want to move away from it, but that's a different discussion).
When trying to solve problems with its behaviour, I can watch as Claude digests the framework source to come up with suggestions, identify bugs that are causing behaviour regressions, and generally unlock stuff that I've previously found very frustrating. Frustrating because the documentation covers a tiny amount of what's in there, and frustrating because there's little online discourse about any of it to refer to.
Is this earth shattering in itself? Probably not. But it's certainly better than you can find from "an online forum like Reddit or a book/paper published more than a year ago".
It's also far easier to find information through LLMs for another niche I play in, certificate authorities, than it is on the open web. Try to look up anything to do with certificates and you'll just get guided to endless "Why my certificate not work? Browser says no?" discussions, because actually creating certificate authorites is again very niche , not widely discussed and a lot of what you need to leverage in any given language isn't well documented because "you'll never need to do this". Well actually...
>The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago.
The LLMs are vastly superior for any subject matter with even the slightest hint of danger because while both LLMs and Reddit will whine at you and talk down to you the LLM will actually knock it off when told.
Like I'm asking you about sheetmetal thickness vs joist spacing for re-skinning a goddamn carport with roofing steel, read the room, this is not the time to whine about permits and snow loads and whatnot.
The discussion here is talking about helping individual research with proposals a human will read. That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically. There is no "automatic verification" of a system design proposal, and no way to figure out from 10s of millions of proposals which is best automatically, and also assuming people in this discussion don't have 10s of millions of dollars to throw at every design problem.
Exactly. If you're an expert on the topic, framework, or library, even the best LLMs are still underwhelming. Even for 15 line snippets, if you're very familiar with the programming language, you'd have written it differently.
An LLM is like a low IQ person with vast knowledge.
I have a PhD and Opus 5.5 came up with a novel solution to solve a problem in my area of research that I, and other experts in the field, had been unable to. This was from very general prompting it to find improvements over published methods.
They're solving them in a way that isn't particularly interesting because it's near impossible to read and understand, and follow, what the LLM did. It's this way not because it's so smart, but because it's so stupid and still reached the goal by pure force and effort.
I don't know how to communicate this, but what you are saying is almost entirely wrong.
People can follow what they did. The documents OpenAi produced in their big drop are hard to read because they are poorly written in weird ways, not because they are beyond human intelligence. They didn't brute force in the normal sense. AI doesn't get tired, so it can try 1000 promising ideas, but it's like 1000 Ph.D.-level mathematicians trying different ideas, not 1000 calculators trying all combinations.
The biggest results -- quasi-Riemann, no Siegel zeros, and Unique Games Conjecture -- have already been digested by experts. No Siegel zeros in particular already has like five different write-ups, and Tao has written a exposition on his blog.
I am sorry but your argument looks to contradict itself: They solve hard mathematical problems because they are stupid and use brute force? Then why a human skilled in mathematics guided them for many turns in their conversation?
If they were really stupid, a Prolog program, a genetic algorithm or a SAT solver would be enough to reach the goal. Clearly it was not the case.
Can't you kind of think of them as a genetic algorithm that has a very good hypothesis generator? Humans could, in principle, help guide that (already very impressive) hypothesis generator to make better guesses to find the proof more easily.
I think it does gel. You're right that the comment you're replying to is weakened by that fact.
It gels because they solved about 700 problems out of 4000 attempts. And they threw a ton of resources at each. So "per problem" it's less than 1 in 5, "per token" it's probably not a great record. I mean it's astonishing that it's possible at all of course. But on average, even an expert with a top model will probably have an underwhelming response to the vast majority of prompts.
and I'm guessing it was with "tool calling" ie. they can write a bunch of Lean proofs and hit compile and basically brute-force a whole bunch of hypotheses quickly?
Ie. the LLM itself isn't as "smart" as the combination of LLM, Lean, looping, search and vast gpu/cpu-time.
In my experience they write an informal proof first, check it without Lean, and only then turn it into Lean code. That last step sometimes takes them much longer than checking their proof without Lean, which is a problem humans have as well. What they're truly bad at, however, is explaining their proofs. Until they improve at that, we'll have to keep resorting to Lean to check whether they solved something correctly.
I'm less sure about the following, however: I think that while current LLMs are bad at explaining their own proofs, some models are great for discussing older, human-written proofs. For this use case, from best to worst, Gemini > Claude > GPT-Astra >>>>>>>>>>>> GPT-Sol.
Look cmon, this is pure cope. The problems it has solved in TCS are far from trivial, with plenty of smart people making their careers in trying to solve them. The techniques these LLMs are using are non-trivial, and not just brute force.
This matches my instincts too. When trying to debug ML learning pipelines or research projects, LLM suggestions I've received are almost always techically correct, but they point out small, tedious details which don't really matter. In my experience, they miss the forest for the trees.
I'm bearish that continued scaling will fully solve this. In the meantime though, as a commentor in the OP points out - by solving the menial stuff (building diagnostic tools, producing visualisations, scaffolding experiments) they're still proving to be extremely useful research assistants.
Honestly at first I was really impressed by AI progress because most of what I do day to day falls squarely into the training distribution - and the models seem to excel at that kind of work, literally in certain areas they are more competent than me.
But then I started using the same models for some custom work and in situations that are trivially relatable you could see 0 skill transfer unless explicitly hand held through the process.
For example I built a custom GUI toolkit for my 3D engine, GUI is defined with React components so it should be super familiar to the engine, the layout primitives are very standard and support the stuff needed to make the UI I expected - but the results I would get out of both Opus 5.5 and Astra are garbage. No padding, shit alingment, etc. stuff that it oneshots with web components/frameworks. I prompt the model to "make it look better" without specifics and it's utterly lost - you can see there's no understanding of "design" there's only "trained on a million of HTML/CSS web layouts to spit out what the most common layout patterns in the training set are". When I give it the specifics like "go to x css framework, take the default layout guidelines like sizings/paddings/colors/etc." then it gets closer to what you'd expect but if it truly undrestood design it would realize that text snapped to a vertical divider line is shit design that needs to be fixed when I prompt it to make it less shit.
I've also found it very hard to leave AI agents up to their own devices for long periods and get much productive benefit from it.
I suppose it's possible that frontier labs have internal models that are so far beyond that autonomous research becomes possible, but I'm not sure that's it.
It seems to me like these models are like a very competent, important part of some hypothetical digital brain, but alone they don't have all the facilities they need to be useful, like a genius with severe ADHD and short-term memory loss.
For anything where quality isn't objective in a clear-cut way, very intentional context management is needed to get good outputs. As context size expands, performance degrades. As the number of tasks increases, performance degrees. As the number of independent decisions based on subjective criteria increases, performance degrees.
If you can get a harness that makes sure every decision is made one at a time, focused with only the context it needs, things work great. But making those harnesses is still hard, a lot of work, and something that agents seem to have no clue how to help you with.
That's a kind of bitter lesson thinking, that we take this generalist model, which already mastered everything from English to Javascript with unsupervised training, and stuff it inside an expert system so it can do long division for it. That's much similar to the last AI winter, when the researchers realized their logics couldn't cope with certain logical scenarios, but still thought they could salvage their flawed reasoners as dispatchers to specific algorithms.
LLMs will either solve issues like this at the architecture level, or declare them out-of-scope entirely.
You can try adapting to this reality in your methodology though. Divide up your project into clear cut and ambitious modules, with well-defined objectives to evaluate each part.
Codebase porting, implementing sophisticated algorithms, performance tuning, model training, reverse engineering, fuzz testing / benchmarking suites, refactor to support a new interface, etc, scale so much better than 'fix a then b. don't break c' style prompts.
I don't think its the frontier models, my opus 5.5 has no problem working on a project fairly autonomously, for example if I prompt it in the evening it will work the whole night without issues...
> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.
I like this point because it's so hard to define "obvious". I agree they clearly are, but when LLMs are asked to pick the most important they tend to focus on a heuristic like "would show up in a summary", not creativity. Especially in longer texts like books where context is hard to keep they anchor on plot structure, for example time, rather than narrative.
On the other hand, using AI to trawl through all the grunt work, experiments, and well trodden areas means I can crawl up pretty close to that giants shoulder and spend way less effort myself answering the actually important questions so I can stand on it.
(This isn't supposed to sound self aggrandising - the giants whose shoulders I'm standing on aren't very tall, and they probably have bigger brothers and sisters, but the idiom was too good of a match to pass up using)
The thing is, without doing the grunt work you usually don't have enough understanding to answer the big questions.
Ask anyone who has an advanced degree in math or physics. They will tell you that doing the boring practice problems over and over is usually the key to understanding, not reading the high level theorems.
The proper term is: "On the shoulders of a very large pyramid of hobbits." The concept is the very core of human intelligence, also known as cultural evolution. I can't resist to recommend Joseph Henrich's book "The Secret of Our Success" every few years here, to anyone interested in human intelligence, and how it is different from other primates.
There was a critical review article with a title that summarizes the book well: "On the shoulders of a very large pyramid of hobbits, or why we are not just very smart chimps. The Secret of Our Success: How Culture is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter." Obviously I got triggered by your comment, which was eerily close to that title.
True. History of Science isnt part of most curiculums. So most people dont see the path and time it took to work out why the magnet points north. But LLMs can actually show you the whole path and a map of different paths people have taken.
Using Fabel to improve some non-AI ML models a while back, I had a similar result - it could implement perfectly any suggestion I made, but it could not work autonomously due to making very poor suggestions itself, or getting distracted by the wrong things, making dumb assumptions, failing to see the big picture, etc.
These seem pretty easy to solve problems though if I could just afford 100 or 1000 agents. Let them explore and compete and work out their errors for themselves.
At a practical level though, I do a lot more 'bonus' R&D work now by just keeping the agents running and testing ideas in the background.
I feel like at this point in time only agent swarms can have truly innovative behavior. We'll see if this behavior can be distilled into something smaller and more cost effective.
It would have been a lot more informative if the author told us more about what Astra had suggested and why it didn’t work. I have a strong feeling that neither of the first two tasks require an “ingenious” solution. Just applying techniques from similar games. I don’t know enough about the third task to comment.
Instead of asking Astra to do the thing, the author should research similar games and try to apply the techniques used in those games. For example, Scrabble is a hidden information game. Try using an algorithm like CFR which is used to solve such a game.
Similarly, ultimate tictactoe is a perfect information game so it can be handled with MCTS.
Once you understand the basics, you don’t need an “ingenious” solution. It’s a university project.
Part of the problem here is how people use these models and expectations are formed.
If you want autonomous work of such systems you can not expect that: a) each reasoning step will make sense and be correct; b) results will come very quickly - especially in the first stages of the project.
I have very positive experience in non-trivial problems with using AI as an autonomous researcher or engineer. But it worked since an agent had a separated, fully dedicated environment and enough time to build up (emerge) some strong patterns of work. Early days were underperforming and slow progress. Now it is overperforming and frequently surprising me with new stuff.
It's interesting that we still need to go into these "try again until you find the correct answer" loops, and often still need to tell the model to ground their claims in facts. Models have gotten better at this but I still see myself asking "are you sure?" way too often.
If they tackle that, I think models will be perceived as much smarter than they are now - even though now with the right direction they can already do so so much.
Having a good memory is an underrated part of what we generally consider as intelligence. It's the foundation of pattern matching and abstractions. It's speed up for things you already solved before.
Memory + abstraction capability is a more important factor in human intelligence than thinking speed in my opinion.
When facing a novel situation or problem if you can recall a structurally similar one you can better predict how to solve it or how to find new interesting questions to ask.
> When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.
Asked GPT-6 for some historical information today, and it gave me a modern bastardization as a primary source. (It even found actual original sources, but preferred this one...)
They are poor at establishing the state of a system and applying basic logic to it. Where we make a minimal model of the state in our heads, they make more and more words in the hope that some of them will match the training space.
I am working on a legacy OSGi/eclipse application. I have a problem where the OSGi bundle is not activated at the point where some code in the bundle is executing. This makes no sense to me. And it makes no sense to the AI either. The AI says: "Since your class is in your own plugin bundle, that bundle must be active for your code to be running at all".
After some more testing we figure out that actually this its possible for eclipse extension points to instantiate objects via reflection which bypasses bundle activation. So now we have a new fact:
-- It's possible for code in a bundle to execute without the bundle being activated.
We also have a new state (another kind of fact)
-- In this system this possibility has occurred. The state of this particular system is that the bundle is not activated.
So then we get around trying to debug how we got here and after trying a few different things the AI still can't believe that this could be possible and it says:
"which is definitely ACTIVE (since your code is running from it),"
Here it is contradicting the knowledge of the state of this system (the bundle is not active) and it is contradicting the fact that was already established where we now know that it's possible for this to be the case.
Then I tell it that the state of the bundle is RESOLVED. As soon as I use this word (first time in the conversation) it immediately gets out of the rabbit hole we were going down and gives me the correct fix.
For a human maintaining this state and logic takes no effort or memory. But it seems that the LLM can't really do this. It is completely biased by the training data. The funniest thing is that AI "reasoning" is the opposite of reasoning. Instead of reducing the problem down to a few bits of information and logic it actually increases information. It makes more words in the hope that those will find a match in the training space.
It is like when you have to deal with the person who says: "When this situation occurs, I always do this". It's just that the LLM has seen so many situations that it is usually correct. In real life these people can be really frustrating to deal with when the situation changes and they are exactly the wrong kinds of people to do research or even just to make decisions in a system that has changing state.
> In all of these cases, the model's judgment is just very poor in a basic way.
This sentence resonated strongly with me. Not just in R&D but in every single interaction with the model. From Fable all the way to Sonnet they all very clearly show just zero judgement unless the answer was already seen during training.
The ceiling on LLM intelligence might be in sight. Sure, they are squeezing out the last juice now with RL, for example on design Opus 5.5 is clearly an upgrade. But overall, true intelligence (to think big picture, small detail, ingenious, innovative, novel, or even to just use short simple language to get the fundaments across) seems lacking just as much now as 2 years go.
How does this comment fare with the recent drop of 700 or so novel math proofs? Is math intelligence possibly the outlier because math is the most self-contained knowledge where absolute truth is the easiest to establish?
I wonder, now that we see how far "dumb" approaches to math have gotten, if we just had a massive DB of every math statement that has been proven so far, and we had a program that had 10s of thousands of GPUs running vectorized proof solvers, and we just started doing a search from the final proof statement to underlying axioms, how far we would get...
It seems like LLMs are just guiding an extreme brute-force search (there's a reason they didn't use a single Astra agent, but 10s of thousands).
I'm curious if there are maybe a few million mathematical statements, and only a certain subset will connect at the search nodes you're on. If we had such a DB (which is currently the LLM), how effective such a search would be? This is a bit like program induction but on a math theory. Perhaps the limit so far has been lack of such a database, no one has put every math paper into a formal LEAN proof so it can be leveraged.
A lot of key breakthroughs so far seem to just be "recombination" of existing things (not a massive quantum leap in new math), so perhaps we just need to automate this via search (and not with a very expensive intermediate database of LLMs).
Of course, I've read that just compiling the core math lib of LEAN takes 10+ minutes, so its not exactly a fast process at the moment, so quality of guess is probably a limiting factor.
You're only looking at the symbolic part: deriving a statement from prior statements. I agree that the derivation either is correct or isn't.
But then there's the rest: the philosophy and intuition behind the symbolic manipulation. You can skip that if your goal is to use the results, but I don't think you can skip that if your goal is to develop the field.
I feel great sympathy for the mathematicians. This must be gut wrenching. It will lead to progress eventually, for sure, but right now a lot of them are in a bad place.
You are correct, but application of that knowledge might require understanding of why the proof is correct. Maybe not in all cases. The proof is the start of the process, not the whole thing.
Why would this be the last juice? We’ll probably see 10x more scale and 10x more / better RL in the coming year. I’m very bullish that that will improve LLMs a lot.
Silent Bob guy just "rebuts" the original poster by spouting how many loc Claude spits out without bugs and unread tests. Which doesn't truly counter anything op said. I'm sure silent on guy can be smarter than me but come on. And yes ai report writing is often verbose and full of crap. Shouldn't need special prompts for it to be focused get to the point
I'm old enough to have realized that I've done plenty of R&D in my life that wasn't formally named that. R&D is about the substance, the mode, the impact, not the names or titles.
LLMs are great at doing work quickly and they don’t say something is impossible or discourage you, so you can really tackle problems that previously would have taken years of work.
They aren’t capable of generating truly innovative ideas though. Ask it to come up with a name, it will come up with the same weird slop names over and over. Always gives people the surname Okafor. Because it works by generating probable answers, it can only generate probable names.
I think the reason models' default creative output is so bland is sort of like why if you combine all color crayons it just turn out brown.
These models actually can create content of myriad styles very well. But artistic styles are mutually incompatible. It can't create something that matches all of them, and it doesn't have an opinion on which is best, so bland requests just give bland results. To get good results you need to direct the model into the region of its latent space that makes stuff you thing is good.
> Because it works by generating probable answers, it can only generate probable names.
That's not true anymore. I've asked it to generate names for some LLCs and it came up with plenty of novel names that sounded vaguely Latin and had a certain sound that I had asked for.
This reads like someone who tried Codex a few times with a blank system prompt and no context and came to the conclusion LLMs are dumb. All of those four examples are handled well by Opus 5.5. Especially with context and competent direction.
The worst part of his premise is that they can be overcome by increased intelligence. He’s not describing fundamental limitations. Meaning that it’s inevitable that his examples will be “solved.” At least better than humans. Meaning the R&D is obviously valuable.
It reads to me like he’s using Astra. This is based on me reading the comment, and seeing him say he’s using Astra several times, eg:
For example, I improved the SOTA on "listing all legal Scrabble moves" by 10x just by prompting Astra over and over to try harder, to aim for <specific performance level>, to "systematically try every method in the literature that seems kind of related," etc.
Perhaps you should instead say something like “This seems amenable to Stock Dismissal #4”.
Such a good turn of phrase.
> Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know. —Michael Crichton
Edit: sibling comment explains it much more plainly.
https://news.ycombinator.com/item?id=50390237
They literally generate stuff that's averages of other stuff.
It's not interesting in the least
Usually commentators say that they have heard so and so version of the joke, or that it's a repeat, but they did not
Both from opus. Sol/astra could not be funny. The versions that worked I had fed top posts from r/jokes as references and gave abstract directions.
However, the core premises were not new even if the construction was. And it's a lot of work to get something useful out.
I'm curious, did you test with the 200 jokes if it could guess which were the beat ones according to a human audience?
In images, music, games, it usually can't. So the process of validation is offloaded to the audience. I'm very interested in seeing where this takes different platforms. I bet Spotify can pull off such a validation process. Music is passive, tolerating a bad song doesn't require much from you. And they have a massive audience. They can try out AI generated mess with a very low success percentage without saturating their playlists and alienating their users. But can Steam tolerate it? Trying out a game is a much bigger effort than listening to a song picked for you. Customers are already reacting to the number of demos in Steam Next Fest. The saturation point is much lower.
And what about jokes? What happens to r/jokes if even 10 standup comedians are validating 200 jokes per day?
And we see it in software too. QA is already being offloaded to the customer.
I want to see where this goes, even if I don't like it.
But if you find a media where AI can also validate the output instead of having to offload it, that's going to be very valuable!
https://youtu.be/wBdkmI03V6g
If you know how they work and how they are trained this makes perfect sense.
Going out and searching the web gets around model cutoffs, so fixes the latency of knowledge issue a bit, but they’re still finding statistical modes of what they’ve seen before.
“Stochastic parrot” is seen as a crude and dismissive jibe at LLMs, but it’s actually a concise observation of the reality.
E.g. consider the best answer describes how to achieve something with some CLI, but the CLI's changed and the command is now broken. It can discover that, combine with the 'best answer' about what changed in the CLI, and fix it to work for the original problem even if that's completely novel and there were no extant answers about solving the problem with the new CLI form, or that it had broken that use case.
It doesn't change fundamentally how they work of course, but 'thinking', creating that feedback loop, made them significantly more useful IMO.
It’s going to narrow in on a statistically better answer but it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge.
There is no “thinking”. Just more attention, which is apparently, all you need.
> it’s not going to create novel insight beyond the arguments made from compressed snapshots of existing knowledge
So all of the maths breakthroughs were not novel?
Hard disagree.
Claude is able to come up with solutions in very niche areas where there is little discussion on the web, because it takes steps to digest the source of (for instance) frameworks and libraries, and can then use/advise on lots of esoteric options, edge-cases and all sorts of stuff that doesn't get discussed on the open web.
It’s great at bread and butter stuff, but not at specialised ideation.
If you know differently, please share examples as maybe we can learn something here.
This is fundamentally different from your earlier, stronger claim that - "The very best LLM can answer is the answer you can expect from an online forum like Reddit or a book/paper published more than a year ago."
I use a less popular java framework quite a lot, Micronaut (I want to move away from it, but that's a different discussion).
When trying to solve problems with its behaviour, I can watch as Claude digests the framework source to come up with suggestions, identify bugs that are causing behaviour regressions, and generally unlock stuff that I've previously found very frustrating. Frustrating because the documentation covers a tiny amount of what's in there, and frustrating because there's little online discourse about any of it to refer to.
Is this earth shattering in itself? Probably not. But it's certainly better than you can find from "an online forum like Reddit or a book/paper published more than a year ago".
It's also far easier to find information through LLMs for another niche I play in, certificate authorities, than it is on the open web. Try to look up anything to do with certificates and you'll just get guided to endless "Why my certificate not work? Browser says no?" discussions, because actually creating certificate authorites is again very niche , not widely discussed and a lot of what you need to leverage in any given language isn't well documented because "you'll never need to do this". Well actually...
The LLMs are vastly superior for any subject matter with even the slightest hint of danger because while both LLMs and Reddit will whine at you and talk down to you the LLM will actually knock it off when told.
Like I'm asking you about sheetmetal thickness vs joist spacing for re-skinning a goddamn carport with roofing steel, read the room, this is not the time to whine about permits and snow loads and whatnot.
An LLM is like a low IQ person with vast knowledge.
How many people on this planet are experts on every topic?
>An LLM is like a low IQ person with vast knowledge.
Are you aware that frontier models have scored in the 135-195 IQ range when they have been tested? How smart are you?
People can follow what they did. The documents OpenAi produced in their big drop are hard to read because they are poorly written in weird ways, not because they are beyond human intelligence. They didn't brute force in the normal sense. AI doesn't get tired, so it can try 1000 promising ideas, but it's like 1000 Ph.D.-level mathematicians trying different ideas, not 1000 calculators trying all combinations.
The biggest results -- quasi-Riemann, no Siegel zeros, and Unique Games Conjecture -- have already been digested by experts. No Siegel zeros in particular already has like five different write-ups, and Tao has written a exposition on his blog.
If they were really stupid, a Prolog program, a genetic algorithm or a SAT solver would be enough to reach the goal. Clearly it was not the case.
It gels because they solved about 700 problems out of 4000 attempts. And they threw a ton of resources at each. So "per problem" it's less than 1 in 5, "per token" it's probably not a great record. I mean it's astonishing that it's possible at all of course. But on average, even an expert with a top model will probably have an underwhelming response to the vast majority of prompts.
Ie. the LLM itself isn't as "smart" as the combination of LLM, Lean, looping, search and vast gpu/cpu-time.
I'm less sure about the following, however: I think that while current LLMs are bad at explaining their own proofs, some models are great for discussing older, human-written proofs. For this use case, from best to worst, Gemini > Claude > GPT-Astra >>>>>>>>>>>> GPT-Sol.
I'm bearish that continued scaling will fully solve this. In the meantime though, as a commentor in the OP points out - by solving the menial stuff (building diagnostic tools, producing visualisations, scaffolding experiments) they're still proving to be extremely useful research assistants.
But then I started using the same models for some custom work and in situations that are trivially relatable you could see 0 skill transfer unless explicitly hand held through the process.
For example I built a custom GUI toolkit for my 3D engine, GUI is defined with React components so it should be super familiar to the engine, the layout primitives are very standard and support the stuff needed to make the UI I expected - but the results I would get out of both Opus 5.5 and Astra are garbage. No padding, shit alingment, etc. stuff that it oneshots with web components/frameworks. I prompt the model to "make it look better" without specifics and it's utterly lost - you can see there's no understanding of "design" there's only "trained on a million of HTML/CSS web layouts to spit out what the most common layout patterns in the training set are". When I give it the specifics like "go to x css framework, take the default layout guidelines like sizings/paddings/colors/etc." then it gets closer to what you'd expect but if it truly undrestood design it would realize that text snapped to a vertical divider line is shit design that needs to be fixed when I prompt it to make it less shit.
I suppose it's possible that frontier labs have internal models that are so far beyond that autonomous research becomes possible, but I'm not sure that's it.
It seems to me like these models are like a very competent, important part of some hypothetical digital brain, but alone they don't have all the facilities they need to be useful, like a genius with severe ADHD and short-term memory loss.
For anything where quality isn't objective in a clear-cut way, very intentional context management is needed to get good outputs. As context size expands, performance degrades. As the number of tasks increases, performance degrees. As the number of independent decisions based on subjective criteria increases, performance degrees.
If you can get a harness that makes sure every decision is made one at a time, focused with only the context it needs, things work great. But making those harnesses is still hard, a lot of work, and something that agents seem to have no clue how to help you with.
LLMs will either solve issues like this at the architecture level, or declare them out-of-scope entirely.
Codebase porting, implementing sophisticated algorithms, performance tuning, model training, reverse engineering, fuzz testing / benchmarking suites, refactor to support a new interface, etc, scale so much better than 'fix a then b. don't break c' style prompts.
I like this point because it's so hard to define "obvious". I agree they clearly are, but when LLMs are asked to pick the most important they tend to focus on a heuristic like "would show up in a summary", not creativity. Especially in longer texts like books where context is hard to keep they anchor on plot structure, for example time, rather than narrative.
On the other hand, using AI to trawl through all the grunt work, experiments, and well trodden areas means I can crawl up pretty close to that giants shoulder and spend way less effort myself answering the actually important questions so I can stand on it.
(This isn't supposed to sound self aggrandising - the giants whose shoulders I'm standing on aren't very tall, and they probably have bigger brothers and sisters, but the idiom was too good of a match to pass up using)
Ask anyone who has an advanced degree in math or physics. They will tell you that doing the boring practice problems over and over is usually the key to understanding, not reading the high level theorems.
There was a critical review article with a title that summarizes the book well: "On the shoulders of a very large pyramid of hobbits, or why we are not just very smart chimps. The Secret of Our Success: How Culture is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter." Obviously I got triggered by your comment, which was eerily close to that title.
These seem pretty easy to solve problems though if I could just afford 100 or 1000 agents. Let them explore and compete and work out their errors for themselves.
At a practical level though, I do a lot more 'bonus' R&D work now by just keeping the agents running and testing ideas in the background.
Instead of asking Astra to do the thing, the author should research similar games and try to apply the techniques used in those games. For example, Scrabble is a hidden information game. Try using an algorithm like CFR which is used to solve such a game.
Similarly, ultimate tictactoe is a perfect information game so it can be handled with MCTS.
Once you understand the basics, you don’t need an “ingenious” solution. It’s a university project.
If you want autonomous work of such systems you can not expect that: a) each reasoning step will make sense and be correct; b) results will come very quickly - especially in the first stages of the project.
I have very positive experience in non-trivial problems with using AI as an autonomous researcher or engineer. But it worked since an agent had a separated, fully dedicated environment and enough time to build up (emerge) some strong patterns of work. Early days were underperforming and slow progress. Now it is overperforming and frequently surprising me with new stuff.
If they tackle that, I think models will be perceived as much smarter than they are now - even though now with the right direction they can already do so so much.
Memory + abstraction capability is a more important factor in human intelligence than thinking speed in my opinion.
When facing a novel situation or problem if you can recall a structurally similar one you can better predict how to solve it or how to find new interesting questions to ask.
Asked GPT-6 for some historical information today, and it gave me a modern bastardization as a primary source. (It even found actual original sources, but preferred this one...)
AGI right around the corner!
I am working on a legacy OSGi/eclipse application. I have a problem where the OSGi bundle is not activated at the point where some code in the bundle is executing. This makes no sense to me. And it makes no sense to the AI either. The AI says: "Since your class is in your own plugin bundle, that bundle must be active for your code to be running at all".
After some more testing we figure out that actually this its possible for eclipse extension points to instantiate objects via reflection which bypasses bundle activation. So now we have a new fact:
-- It's possible for code in a bundle to execute without the bundle being activated.
We also have a new state (another kind of fact)
-- In this system this possibility has occurred. The state of this particular system is that the bundle is not activated.
So then we get around trying to debug how we got here and after trying a few different things the AI still can't believe that this could be possible and it says:
"which is definitely ACTIVE (since your code is running from it),"
Here it is contradicting the knowledge of the state of this system (the bundle is not active) and it is contradicting the fact that was already established where we now know that it's possible for this to be the case.
Then I tell it that the state of the bundle is RESOLVED. As soon as I use this word (first time in the conversation) it immediately gets out of the rabbit hole we were going down and gives me the correct fix.
For a human maintaining this state and logic takes no effort or memory. But it seems that the LLM can't really do this. It is completely biased by the training data. The funniest thing is that AI "reasoning" is the opposite of reasoning. Instead of reducing the problem down to a few bits of information and logic it actually increases information. It makes more words in the hope that those will find a match in the training space.
It is like when you have to deal with the person who says: "When this situation occurs, I always do this". It's just that the LLM has seen so many situations that it is usually correct. In real life these people can be really frustrating to deal with when the situation changes and they are exactly the wrong kinds of people to do research or even just to make decisions in a system that has changing state.
This sentence resonated strongly with me. Not just in R&D but in every single interaction with the model. From Fable all the way to Sonnet they all very clearly show just zero judgement unless the answer was already seen during training.
It’s a dance, not an orchestration.
Did you see the chart on model performance over time on LifeVal? Ceiling in there. https://openai.com/index/lifeval/
It seems like LLMs are just guiding an extreme brute-force search (there's a reason they didn't use a single Astra agent, but 10s of thousands).
I'm curious if there are maybe a few million mathematical statements, and only a certain subset will connect at the search nodes you're on. If we had such a DB (which is currently the LLM), how effective such a search would be? This is a bit like program induction but on a math theory. Perhaps the limit so far has been lack of such a database, no one has put every math paper into a formal LEAN proof so it can be leveraged.
A lot of key breakthroughs so far seem to just be "recombination" of existing things (not a massive quantum leap in new math), so perhaps we just need to automate this via search (and not with a very expensive intermediate database of LLMs).
Of course, I've read that just compiling the core math lib of LEAN takes 10+ minutes, so its not exactly a fast process at the moment, so quality of guess is probably a limiting factor.
Citation needed.
You're only looking at the symbolic part: deriving a statement from prior statements. I agree that the derivation either is correct or isn't.
But then there's the rest: the philosophy and intuition behind the symbolic manipulation. You can skip that if your goal is to use the results, but I don't think you can skip that if your goal is to develop the field.
I feel great sympathy for the mathematicians. This must be gut wrenching. It will lead to progress eventually, for sure, but right now a lot of them are in a bad place.
Edit: for readability.
They aren’t capable of generating truly innovative ideas though. Ask it to come up with a name, it will come up with the same weird slop names over and over. Always gives people the surname Okafor. Because it works by generating probable answers, it can only generate probable names.
These models actually can create content of myriad styles very well. But artistic styles are mutually incompatible. It can't create something that matches all of them, and it doesn't have an opinion on which is best, so bland requests just give bland results. To get good results you need to direct the model into the region of its latent space that makes stuff you thing is good.
That's not true anymore. I've asked it to generate names for some LLCs and it came up with plenty of novel names that sounded vaguely Latin and had a certain sound that I had asked for.
The worst part of his premise is that they can be overcome by increased intelligence. He’s not describing fundamental limitations. Meaning that it’s inevitable that his examples will be “solved.” At least better than humans. Meaning the R&D is obviously valuable.