wilmott.com

How can we teach Cauchy to engineers and computer scientists? They don't teach that in school, Maybe it's because ML and and ANNs don't do Cauchy sequnces.

SGD: It's like getting a flat tire and fixing the problem by driving around the block a few times; maybe it will fix itself.

The utility of Cauchy sequences lies in the fact that in a complete metric space (one where all such sequences are known to converge to a limit), the criterion for convergence depends only on the terms of the sequence itself, as opposed to the definition of convergence, which uses the limit value as well as the terms. This is often exploited in algorithms, both theoretical and applied, where an iterative process can be shown relatively easily to produce a Cauchy sequence, consisting of the iterates, thus fulfilling a logical condition, such as termination.

I repeat for our CS friends and partners.

convergence depends only on the terms of the sequence itself, as opposed to the definition of convergence, which uses the limit value as well as the terms.

And since we work in a complete RKHS space, it's bingo.

Is this statement always true?

any two numbers x and y must satisfy one and only one of the conditions x < y, x = y, or x > y.

Cauchy sequence framework is a good heuristic on the weight training updates in the training process. The updates should be diminishing according to the Cauchy sequence - if not, apart from some stochastic noise, one should adjust the training parameters or something more serious is wrong; and you can actually apply the Cauchy criterion as stopping.

Cauchy sequences are bread-and-butter of real, functional and applied analysis. You want these sequences to converge in the (complete metric) space of interest.

MMD and kernel 2-sample test is a good example.

kats, 
What's your opinion of this? Is it 'better' than DL etc.

https://arxiv.org/abs/2106.08443