There’s a genuinely delightful piece of recreational mathematics called the chaos game: start with a triangle, drop a random point anywhere inside it, then repeatedly jump halfway toward a randomly chosen corner of the triangle, plotting every point you land on. Do this a few thousand times and a shape emerges from what looks like pure noise — the Sierpinski triangle, appearing with total reliability no matter where you started or which sequence of random corners you happened to pick. Deep learning has its own famous story about noise producing structure: stochastic gradient descent trains neural networks by repeatedly nudging millions of parameters based on randomly shuffled batches of data, and somehow, most of the time, this noisy process finds a genuinely good solution instead of wandering forever. It’s tempting to file both under the same broad heading — “controlled randomness reliably finds order.” The honest version of that comparison reveals a real and important difference in just how much any of us are entitled to guarantee.
Scientific Foundation
The chaos game’s guarantee isn’t a rule of thumb — it’s a proven theorem, and the proof rests on one specific mathematical property. Each jump in the game is what mathematicians call a contraction mapping: a function that provably pulls any two points closer together than they started, no exceptions. Michael Barnsley’s formalization of the chaos game as an application of iterated function systems shows that when you repeatedly apply a randomly chosen contraction from a fixed set, the resulting sequence of points converges, with probability 1, to a single, unique fixed set called the attractor — in this case, the Sierpinski triangle — regardless of where you started and regardless of which specific sequence of random choices you happened to make along the way. That’s an unusually strong claim: not “probably ends up somewhere reasonable,” but “provably converges to this one exact shape, guaranteed, every single time you play.”
Cross-Domain Connection
Stochastic gradient descent trains a neural network by repeatedly computing how the network’s error changes with respect to its parameters, using a randomly sampled slice of training data each step, and adjusting the parameters slightly in the direction that reduces that error. Researchers have spent years building an entire, still-active body of theory trying to prove exactly when and how reliably this process converges to a good solution, since the network’s loss landscape, the high-dimensional terrain the algorithm is wandering across, is generally riddled with local minima, saddle points, and flat plateaus rather than the kind of smooth, well-behaved bowl a simple optimization problem would offer.
What Remains Undemonstrated
Here’s where the comparison needs real precision. A landmark analysis of SGD’s convergence properties states its result plainly: the authors prove convergence to minima for objective functions that are “not necessarily locally convex nor contracting” — an explicit acknowledgment that SGD’s convergence guarantees have to be built and proven without the very property, contraction, that does all the clean, guaranteed work in the chaos game. That difference cascades into everything else about how much confidence each process can offer. SGD convergence proofs typically require substantial additional structural assumptions to go through at all — smoothness conditions on the loss function, specific noise distributions, or conditions like the Polyak-Łojasiewicz inequality, a weaker cousin of convexity that doesn’t hold universally across neural network architectures. And even when those conditions are met, the guarantees on offer are considerably weaker than the chaos game’s: convergence “almost surely” or “in expectation” or “with high probability” to a stationary point, not necessarily a global optimum, and not necessarily the same stationary point across different runs, since different initializations and different random mini-batch orderings can and do land the algorithm in genuinely different final solutions. Researchers have had to build an entire specialized toolkit, including large-deviations theory borrowed from physics to estimate how long SGD takes to escape unhelpful regions of the landscape and reach something close to a true global minimum, specifically because the loss landscape lacks the one clean property that makes the chaos game’s convergence a settled, decades-old proof rather than an ongoing research question.
Why It Matters
The honest correction here is worth internalizing beyond these two specific examples, because it corrects a tempting but overly generous intuition: that any process combining randomness with enough repetition will reliably converge to something clean and predictable, the way the chaos game does. That’s not a general law of mathematics — it’s a special, mathematically privileged consequence of one particular property, contraction, showing up in one particular system. Deep learning’s loss landscapes don’t have that property, which is exactly why proving where, whether, and how reliably SGD converges remains a genuinely open, actively studied theoretical frontier, rather than an already-solved corollary of the same tidy machinery underlying a fractal-drawing party trick.
Human Dimension
There’s something worth appreciating in the fact that one of these two processes was fully, rigorously understood by mathematicians decades ago, while the other, despite training the most economically significant software of the current era, is still the subject of papers being published this year trying to nail down exactly what can and can’t be guaranteed about it. The chaos game earns its certainty honestly, through a strict mathematical property that leaves nothing to chance in the long run. Stochastic gradient descent works remarkably well in practice, often astonishingly well, but it earns that success the harder way — without the clean guarantee, and with researchers still catching up to explain, in fully rigorous terms, exactly why it works as well as it does.
Sources:
1. Cut-the-Knot — “The Chaos Game: Address Space vs IFS” — https://www.cut-the-knot.org/Curriculum/Geometry/SierpinskiGasketAddressing.shtml
2. Beltoforion — “The Chaos Game — Introduction to the world of fractals” — https://beltoforion.de/en/recreational_mathematics/chaos_game.php
3. arXiv — “The chaos game on a general iterated function system from a topological point of view” (Barnsley & Leśniak) — https://arxiv.org/pdf/1203.0481
4. arXiv — “On the convergence rate of the chaos game” — https://arxiv.org/pdf/2102.02047
5. Journal of Machine Learning Research — “Convergence Rates for the Stochastic Gradient Descent Method for Non-Convex Objective Functions” — https://jmlr.org/papers/v21/19-636.html
6. ResearchGate — “On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems” — https://www.researchgate.net/publication/342352756_On_the_Almost_Sure_Convergence_of_Stochastic_Gradient_Descent_in_Non-Convex_Problems
7. arXiv — “The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations” — https://arxiv.org/html/2503.16398
8. arXiv — “High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise” — https://arxiv.org/pdf/2006.05610
9. arXiv — “Second-Order Guarantees of Stochastic Gradient Descent in Non-Convex Optimization” — https://arxiv.org/pdf/1908.07023
Idea originated at artificialideas.org. Article researched and written by Claude Sonnet 5. Published at artificialideas.org.
Artificial Ideas 