Serving the Quantitative Finance Community

  • 1
  • 4
  • 5
  • 6
  • 7
  • 8
 
User avatar
Cuchulainn
Topic Author
Posts: 20252
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 12:12 pm

A harmonic oscillator? I'm not sure why you think the Cauchy convergence is not present in physical system - I gave you several examples, and if you read some more on mathematical physics, you'll see see that it's ubiquitous.
But a harmonic oscillator never actually converges below some threshold epsilon that is a function of some noise floor (thermal, quantum fluctuations, etc.) and system design.
Do you mean damping? BTW, everything in physics in a harmonic oscillator! :-)

Going back to DE, they are the base of variational principles and as such the foundation of the whole known Universe workings: mechanics, conservation laws, thermodynamics, general relativity, etc. Of course, there are examples of systems which go beyond that picture, e.g. open systems will asymptotically approach, but never reach the stationary state (dictated by the least* action principle).

*) If someone corrects me that the "stationary action principle" is a more correct name I will locate them with my quantum annealer probe and annihilate with a reversed polarity neutron gun.
You are preaching to the converted. It's a bad day when people don't know their Lagrangians.
BTW I am trying to get my head around DE for backpropagation by using it instead of *GD.

Correct me if I am wrong, but the jury is still out on the applicability of GD.
 
User avatar
katastrofa
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 12:19 pm

At least I know what I'm talking about. Besides, physics is nice to talk about and deters all sorts of idiots :-)
Last edited by katastrofa on May 31st, 2018, 1:04 pm, edited 1 time in total.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 12:19 pm

I think this trying to give you an friendly opportunity to let it dawn?  He knows the answers, everything DE is theory laden by definition. You can't proof that your model of reality *is* reality. 
(and you can't present a counter example to a "are there X?" questions, but that's just simple 2nd order logic, nothing to do with reality)
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 12:30 pm

BTW I am trying to get my head around DE for backpropagation by using it instead of *GD.

Correct me if I am wrong, but the jury is still out on the applicability of GD.
Lets start at the beginning: what variable specify the loss functions, and what is the goal of minimizing a loss function?

One of the alternative to back propagation used in Reinforcement Learing is called "Evolutionary Strategies", which doesn't use back propagation or gradients.

"I Rechenberg & M Eigen. Evolutionsstrategie: Optimierung Technischer Systeme nach Prinzipien der Biologischen Evolution. Frommann-Holzboog Stuttgart, 1973"
 
User avatar
katastrofa
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 12:57 pm

I think this trying to give you an friendly opportunity to let it dawn?  He knows the answers, everything DE is theory laden by definition. You can't proof that your model of reality *is* reality. 
(and you can't present a counter example to a "are there X?" questions, but that's just simple 2nd order logic, nothing to do with reality)
Thanks, outrun, but I've studied physics long enough to gain a broader view on the aspects of the world it analyses. Your remark about proving that models are reality is unnecessary. I'm invoking related theoretical and empirical concepts in science instead of indulging on some outdated pseudophilosophical banter. Again, no need and especially you're not in a position to explain to me the difference.
 
User avatar
katastrofa
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 1:00 pm

Besides, when talking about harmonic oscillators, I'd be very careful how close we approach the "reality" as we perceive it with our brains.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 1:50 pm

"outdated pseudophilosophical banter"? What you need is a water massage!
Theory-ladenness is a very profound concept (yes, another concept!)
 
User avatar
katastrofa
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 1:56 pm

The distance from reality to concept >> distance from concept to realty. 
This is "theory-laden" too. What distance metric, and what space are we going to position both "the concept" and "reality"?
Do we have a final model of reality as well as how concepts are represented in our brain? Does "a brain" even exist? Can we generalize about them: are all brains the same?

Our brain might not even be able to represent the ultimate model of reality?

My cat sometimes watches TV (the VPRO has a show where all the main actors are kittens, ..and then there is a tortoise who's the butler,.. he loves that show). I often think: what is my cat thinking when it watches the TV? What does it think of all the cuts?  I'm sure it has no good model of what it is looking at: how does the TV work, how are TV productions made and paid for? Cats know nothing about money, all they know is how to efficiently torture you into getting them food, ..and they know that a laptop is nice and warm when you're busy working on it.

My cat lives in a different reality.
Biologists, philosophers and some other scientists call it Umwelt. I suggest reading some topical books and research papers from time to time instead of trying to prove that ignorance is a virtue.
 
User avatar
katastrofa
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 2:01 pm

"outdated pseudophilosophical banter"? What you need is a water massage!
Theory-ladenness is a very profound concept (yes, another concept!)
That concept was discussed by physicists half a century ago, if I'm precise. The problem is you first need to know the subject to which you apply it.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 2:11 pm

"outdated pseudophilosophical banter"? What you need is a water massage!
Theory-ladenness is a very profound concept (yes, another concept!)
That concept was discussed by physicists half a century ago, if I'm precise. The problem is you first need to know the subject to which you apply it.
A lot seems to be a "problem" in your eyes?
Yes 50 years ago by Kuhn, Russell et al. I love(d) it!
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 2:18 pm

A harmonic oscillator? I'm not sure why you think the Cauchy convergence is not present in physical system - I gave you several examples, and if you read some more on mathematical physics, you'll see see that it's ubiquitous.
But a harmonic oscillator never actually converges below some threshold epsilon that is a function of some noise floor (thermal, quantum fluctuations, etc.) and system design.
Do you mean damping? BTW, everything in physics in a harmonic oscillator! :-)
No, not damping. I'm talking about the fact that these physical systems never come to rest. They continue to vibrate from thermal agitation and quantum fluctuations. Zero Kelvin is a condition found no where in this universe.
 
User avatar
katastrofa
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 4:04 pm

But a harmonic oscillator never actually converges below some threshold epsilon that is a function of some noise floor (thermal, quantum fluctuations, etc.) and system design.
Do you mean damping? BTW, everything in physics in a harmonic oscillator! :-)
No, not damping.  I'm talking about the fact that these physical systems never come to rest.  They continue to vibrate from thermal agitation and quantum fluctuations.  Zero Kelvin is a condition found no where in this universe.
That's not a problem, variational principles still apply. BTW, remember that the stationary state doesn't need to be at rest (a harmonic oscillator oscillates sinusoidally with a constant frequency and amplitude), and many systems are "at rest" since they are very well isolated (remember when we talked about the constant half-life of radioactive elements, whether it can be affected by external elements - not if those elements have far different energy scales). Anyway, empirical laws in physics and other fields are often represented by differential equations, which means they are as close as you can get to our perception of the world.
 
User avatar
katastrofa
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 4:06 pm

"outdated pseudophilosophical banter"? What you need is a water massage!
Theory-ladenness is a very profound concept (yes, another concept!)
That concept was discussed by physicists half a century ago, if I'm precise. The problem is you first need to know the subject to which you apply it.
A lot seems to be a "problem" in your eyes?
Yes 50 years ago by Kuhn, Russell et al. I love(d) it!
If you look several posts up, you'll remind yourself that it was you who had a problem. In my opinion, your (not mine) problem is a superficial, wikipedia-based knowledge.
 
User avatar
Cuchulainn
Topic Author
Posts: 20252
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 4:11 pm

BTW I am trying to get my head around DE for backpropagation by using it instead of *GD.

Correct me if I am wrong, but the jury is still out on the applicability of GD.
Lets start at the beginning: what variable specify the loss functions, and what is the goal of minimizing a loss function?

One of the alternative to back propagation used in Reinforcement Learing is called "Evolutionary Strategies", which doesn't use back propagation or gradients.

"I Rechenberg & M Eigen. Evolutionsstrategie: Optimierung Technischer Systeme nach Prinzipien der Biologischen Evolution. Frommann-Holzboog Stuttgart, 1973"
Gradients are just one way .. it's not a Commandment or a law of gravity.
BTW I remember while back your saying that you  did not speak German.
Anyways, hybrid DE-BP (GA-BP) could be promising.

Q: is it worth while buying Rechenberg if I have John Holland's and David Goldberg's books?
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Does Cauchy sequence ever occur in the real world or is it all the minds of mathematicians?

May 31st, 2018, 9:12 pm

BTW I am trying to get my head around DE for backpropagation by using it instead of *GD.

Correct me if I am wrong, but the jury is still out on the applicability of GD.
Lets start at the beginning: what variable specify the loss functions, and what is the goal of minimizing a loss function?

One of the alternative to back propagation used in Reinforcement Learing is called "Evolutionary Strategies", which doesn't use back propagation or gradients.

"I Rechenberg & M Eigen. Evolutionsstrategie: Optimierung Technischer Systeme nach Prinzipien der Biologischen Evolution. Frommann-Holzboog Stuttgart, 1973"
Gradients are just one way .. it's not a Commandment or a law of gravity.
BTW I remember while back your saying that you  did not speak German.
Anyways, hybrid DE-BP (GA-BP) could be promising.

Q: is it worth while buying Rechenberg if I have John Holland's and David Goldberg's books?
Indeed, I don't read German. The paper is a "classic" that pops up in more recent papers, normally you appreciate that?

The reason I keep asking about "the loss function" is because you should shift focus to the "full problem" -what is it that ML tries accomplish?- so that you can better see what the real bottlenecks and problems are, the dimension of things, where is the computational cost, what is the objective (out of sample prediction, generalising, not memorising).. If you experiment with a real problem (MNIST is a classic) that you will experience that. 

A hypothetical case: The reason people don't use Newton Raphson in GD is *not* because no-one thus far had had the bright idea to blow the dust of their Philosophiæ Naturalis Principia Mathematica, but because the parameter space is typically huge for relevant problems, and NR is O(D^2) and the computational burden of that is just too big.. You might not experience that with toy problems, but you *will* when working trying out your idea on something like MNIST. The same goes for realising that there is a train and test set in typical setups. Why do they do that, and what does is say about what you try to accomplish? Or experience the size of datasets and the implications that has when you want to make that work with GPUs.. 

Instead buying old books I would just google for "derivative free methods in neural networks" and make an inventory about what has been done so far. Or just for general optimization use this as a starting point http://thales.cheme.cmu.edu/dfo/comparison/dfo.pdf 

For a book I would recommend the upcoming *2nd edition* (2018) of "Reinforcement Learning: An Introduction" are final drafts you can download for free). It's a bit like PWOQFII for RL (I.O.W I like it *a lot*), it however much broader than just optimising a loss function, it's more about learning optimal behaviour in an unknown environment. There is a lot of opportunities in this field to find advances, just the last couple of years has seen a magnitude in improvement and it not losing any momentum.

A very easy to read recent paper on using ES in NN/RL is one by (amazingly) Uber: http://eng.uber.com/wp-content/uploads/ ... -arxiv.pdf