Records¶
llm-code-accuracy¶
https://www.coze.cn/s/C9xkXQ5_YBI/
while LLMs have inherent limitations in handling complex vulnerabilities independently, they can become effective tools for automated vulnerability repair when guided by carefully crafted prompts
// 2. https://arxiv.org/abs/2503.15341
We propose two confidence-based uncertainty measures: Entropy-based and Probability Differential-based methods. When uncertainty is high, UnCert-CoT activates CoT-decoding to generate multiple reasoning paths and selects the final code that exhibits the highest likelihood of correctness.
In contrast, LLM directly generates the code when uncertainty is low.
This uncertainty judgment mechanism allows LLMs to prioritize complex tasks and avoid unnecessary steps in simpler cases, thereby improving overall efficiency and accuracy in code generation.
LLM - Lean¶
https://mathlib-initiative.org/ from https://www.renaissancephilanthropy.org/funds For "research", commercialization (the "rigid demand") is actually quite rare ... right?
======
Ironically interesting to see LLM vibe proving !! https://axiommath.ai/territory/from-seeing-why-to-checking-everything
For humans, we have decent intuitions. We know problems with heavy case analysis are tedious to enumerate. We know problems requiring one clever construction can make one stuck forever or shout aha! But for machines, we're still in the dark. A real theory of machine difficulty—what structural features make problems easy or hard for automated provers – seems like a genuinely interesting research direction.
Right now the workflow is simple: humans pose problems, machines find proofs. But you can imagine something more interactive. Machines sanity-checking human ideas in real time. Humans reviewing machine feedback and steering search toward promising directions. Each side complementing the other.
Axiom is building a world where human intuitions are grounded by machine verification and machine verification in turn inspires human intuitions.
======
Remember how the Fourier transform implements modular addition?
https://x.com/xleaps/status/1627873094814531584
& https://twitter.com/xleaps/status/1627844991824297984
======
The characteristic of combinatorics lies in its diversity. Combinatorics problems come in rich variety, with no "standard tools" and largely relying on very clever observations, so it is often considered in math and informatics competitions one of the standards for measuring a person's "thinking ability".
And with no standard tools, proofs are flexible — even without considering AI, this is quite a challenge for those who write Lean code.
... AlphaProof wastes a great deal of time on trivial details, rather than on the one or two critical, inspired mathematical constructions in combinatorics problems. https://www.zhihu.com/question/662586636/answer/3577050463
Combinatorics problems are relatively flexible, requiring few complex theorems or formulas; what they need is the ability to abstract and transform the problem. This is also why dual IMO and IOI contestants often score higher on combinatorics problems.
Temporary Summary¶
LLM_auto_formalize_prove.m4a
Now, the auto-formalization of the proof is not complete. You need humans to check if the LLM's formalization is correct.
Based on the feedback of this formalized result, the reasoning model ( let's call it Model 2 ) uses search, heuristic methods, or other approaches to make some assumptions about the final possible proof.
Some people say that LLM will just expand its search space to mechanically check for any possible proof of the unsolved problems raised by humans. What if it's an open-ended problem, which is much more difficult than Olympic math or standardized tests? It's in this area that humans' intuitive inspiration -- so-called "Aha moments" as certain shortcuts can be triggered in mathematicians' minds.
But I don't think it's completely impossible for LLMs to achieve this kind of human intelligence. Do you remember the example of the Fourier transform? After 9,000 or more training steps, an LLM achieved 100% accuracy in finding the modular additive operation from the Fourier transform. This finding is correct and counterintuitive to humans. So, this is a form of emergence. Yet emergence is not a free lunch; it comes from high-quality data and training.
In navigating the large-scale complex mathematical map (a graph of connected proofs), LLMs could find a shortcut from one already known theorem to another unsolved, open-ended problem.
Proof auto-formalization: [LLM formalization (1)] is not yet complete, and humans need to check whether its formalization is correct;
[LLM reasoning (2)]: The reasoning of Model (2) is based on feedback from the formalization result, and can make hypotheses about the final possible proof through search or heuristic methods;
Human intuition advantage: Some open-ended problems are harder than Olympiad exams; human intuition can trigger "Aha moments" to find shortcuts;
LLM potential: Taking the Fourier transform for computing modular addition as an example, after extensive training an LLM can produce counterintuitive yet correct derivations;
Training data for theorem proving must ensure quality; it is not impossible for LLMs to reach the human level of "emergent" insight.