Back to Cauchy and where I address some of your misconceptions(?, or are you trying to wind me up:)):
Yet I notice everything you said speaks in mathematical terms. For example, you speak of having a "sharp error bound in certain numerical processes" rather than one on "physical processes."
I am talking about numerical methods that approximate physical processes. That did not address the latter does not give you the right to shoot down mathematics.
First, I'm not trying to wind you up.
Second, I'm not shooting down the mathematics itself. The math is beautiful. It's whether the math actually really applies to the physical system that may be problematic.
Even in the case of digital systems, Cauchy may fail. I'm sure that you've seen it yourself: a mathematical solution that has analytic proof that it forms a Cauchy sequence but when you write the code and run it you discover that it does not always converge because it hits some cycle of flipping the LSB. Doesn't that happen?
Isn't there a trade-off in the SGD parameters between overly-rapid convergence to a local minimum versus unstable oscillations?
Let's look at it from this angle: let's say you want to find minimum of Rastrigin function (many local minima) and (S)GD will (if it is lucky) converge to one of them. And for discontinuous functions SGD will choke. So,. what you are talking about is the wrong Solver for the wrong problem, nothing more. Use DE if you want a robust global solver.
Agreed! If you've got a better global solver for machine learning, please implement it because lots of people really need it.
BTW GD was not invented in MIT AI Lab but by Cauchy hisself in 1847 when doing his calculation into astronomy. That was his real-life input. Hopefully another urban myth dispelled. It would seem CS folk are using these GD algos without really understanding when/why/how/what-if ..
https://www.math.uni-bielefeld.de/docum ... claude.pdf
I've never claimed anything about who invented what because it honestly does not matter to me. The invention matters. The inventor does not.
P.S. I do wonder if belief in Cauchy sequences is one of the reasons for financial crises. Equilibrium economics is a crock of you know what. Are housing prices in Amsterdam (or Ireland) a Cauchy sequence?????
Noise++. The reasons are well known, and were not caused by Cauchy sequences. Sounds a bit disingenuous, to he honest.
Actually, house prices only changed 1% in the last 300 years. So, it be an alternating series?
Well there we may have to disagree. If you look at the foundations of economics, they are based on expectations of convergence of supply, demand, prices, etc. What is "equilibrium" but a synonym for convergence?