Serving the Quantitative Finance Community

 
User avatar
JohnLeM
Posts: 379
Joined: September 16th, 2008, 7:15 pm

Re: Universal Approximation theorem

October 15th, 2019, 2:12 pm

Homework for Cuch: this recent paper (jun 2017) is getting many people excited, it proposes SELU (instead of RELU, sigmoid). It works really well, I'm seeing very stable learning with deep networks.

You can go straight to the appendix with the proofs (page 9 ..100) that motivate why it should work that good.

https://arxiv.org/abs/1706.02515
ELU family not optimal for option pricing. You need [$]C^{2}[$] functions.
Are people still excited?
I thought that options are [$]C^{1}[$] functions ? Even less (Bounded variations functions) if you consider exotic ones as Autocalls ?
 
User avatar
FaridMoussaoui
Posts: 327
Joined: June 20th, 2008, 10:05 am
Location: Genève, Genf, Ginevra, Geneva

Re: Universal Approximation theorem

October 15th, 2019, 2:40 pm

Seriously ... Arnulf Jentzen wrote 11 articles during the last three months...I think this is more than Nash in his entire life ! That's exactly what publish or perish leads to.
He is full professor, he is not obliged to do that.
 
User avatar
Cuchulainn
Topic Author
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

October 15th, 2019, 3:47 pm

Seriously ... Arnulf Jentzen wrote 11 articles during the last three months...I think this is more than Nash in his entire life ! That's exactly what publish or perish leads to.
He is full professor, he is not obliged to do that.
So, it's just the writing style that needs tweaking????

Who takes arXiv seriously???
 
User avatar
katastrofa
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Universal Approximation theorem

October 15th, 2019, 3:58 pm

Seriously ... Arnulf Jentzen wrote 11 articles during the last three months...I think this is more than Nash in his entire life ! That's exactly what publish or perish leads to.
He is full professor, he is not obliged to do that.
So what? At my institute they were re-appointed every year (even guys typed for the Nobel Prize), same about the abode of mediocrity called UCL here in London. Today's scientists optimise their work for a number of and top journal publications - bullshit, lies, cliques and review circles. Any good science in it is coincidental.
 
User avatar
Cuchulainn
Topic Author
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

October 16th, 2019, 7:54 am

The mathematical precision in Cybenko 1988 has been superceded/improved on here

http://www2.math.technion.ac.il/~pinkus/papers/acta.pdf

In particular, Theorems 3.1, 4.1, 5.1, 6.2, 6.7,  Proposition 3.3. 

Seems like ML' maths is stuck in the 80s. The mathematical subtleties surrounding activation functions seem to have been missed, causing issues during numerics. Maybe I have missed something.

// Allan Pinkus studied with Samuel Karlin. 
Pinkus' work is known in the ML theory community and Cybenko's result has been built upon. See https://www.brown.edu/research/projects ... tworks.pdf
How come, only Cybenko is touted in the literature? Did my post finally trigger a response??

Pinkus claims that the activation function may NOT be a polynomial. Now, that's kinda breaking news??
Last edited by Cuchulainn on October 16th, 2019, 8:05 am, edited 1 time in total.
 
User avatar
Cuchulainn
Topic Author
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

October 16th, 2019, 7:57 am

oh wait, a function you don't even know? We all know that it doesn't make sense.
That's one possible weltanschauung, that does not produce interesting cases.
It does - it's what Deep RL is about.
Any good examples? Does it always work or just in 'most' cases?
 
User avatar
JohnLeM
Posts: 379
Joined: September 16th, 2008, 7:15 pm

Re: Universal Approximation theorem

October 16th, 2019, 9:00 am

My one-penny guess : take any square-integrable function [$]\phi[$], and call the convolution [$]\varphi = \phi \ast \phi[$] an activation function. Then you can use it in Cybenko Theorem.

That's already a lot of examples. But there exists much more examples. In fact, give me any probability measure over [$]\mathbb{R}^D[$] and I can build for you some kind of Cybenko space with it. I figure out that this is how Kaggle-type contest work : take an activation function, measure error, use it as a benchmark to find a better activation function.
 
User avatar
Cuchulainn
Topic Author
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

October 16th, 2019, 10:01 am

Speak of the devil Sobolev training, train the function and its derivatives. That sound relevant.

http://mcneela.github.io/machine_learni ... works.html

Sounds like a logical step.

Sobolev spaces are the bread and butter of advanced numerical analysis (e.g. FEM).
 
User avatar
JohnLeM
Posts: 379
Joined: September 16th, 2008, 7:15 pm

Re: Universal Approximation theorem

October 16th, 2019, 10:32 am

Speak of the devil Sobolev training, train the function and its derivatives. That sound relevant.

http://mcneela.github.io/machine_learni ... works.html

Sounds like a logical step.

Sobolev spaces are the bread and butter of advanced numerical analysis (e.g. FEM).
Yes, it is very relevant, but it is a closed topic today : we can provide sharp estimations of convergence rate for any Sobolev spaces [$]\mathcal{H}^{s,p}(\mathbb{R}^D), s > D/p[$] (required to get continuity), using radial basis functions. I think that we can do it for any deep feed-forward neural networks as well.

By the way, if this guy is studying the interpolation problem in Sobolev spaces, maybe he should start by looking to a proper definition of what is a Sobolev space.
 
User avatar
ISayMoo
Posts: 2332
Joined: September 30th, 2015, 8:30 pm

Re: Universal Approximation theorem

October 16th, 2019, 12:38 pm

oh wait, a function you don't even know? We all know that it doesn't make sense.
That's one possible weltanschauung, that does not produce interesting cases.
It does - it's what Deep RL is about.
Any good examples? Does it always work or just in 'most' cases?
No, it doesn't always work. What does?

A good example: AlphaZero.
 
User avatar
ISayMoo
Posts: 2332
Joined: September 30th, 2015, 8:30 pm

Re: Universal Approximation theorem

October 16th, 2019, 12:38 pm

How come, only Cybenko is touted in the literature?
It isn't (only). You need to read more literature.
 
User avatar
Cuchulainn
Topic Author
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

October 16th, 2019, 12:57 pm

How come, only Cybenko is touted in the literature?
It isn't (only). You need to read more literature.
Cheap one.

How come, only Cybenko is touted in the MAINSTREAM literature?

I see, Most ML users don't know either. 
Show me a ML article that mentions Pinkus?

I'm expecting you experts to come up with the  goodies. I don't have a boss who will pay me to research all this stuff. 
 
User avatar
ISayMoo
Posts: 2332
Joined: September 30th, 2015, 8:30 pm

Re: Universal Approximation theorem

October 16th, 2019, 2:26 pm

Yes. You aren't paying me for it, either ;-)
 
User avatar
Cuchulainn
Topic Author
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

October 16th, 2019, 3:58 pm

What rocks your boat then as it were on this forum?
 
User avatar
katastrofa
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Universal Approximation theorem

October 16th, 2019, 11:30 pm

I see often cited K Hornik, Multilayer feedforward networks are universal approximators, Neural Networks 2 (1989)
Pinkus is commonly cited too.

Science is not exact, it's always an approximation. That's why tge universal approximation theorem had a profound meaning. It showed that universal approximators, namely families of functions which ca approximate anything, exist. It addressed over two centuries of mathematical studies, which in turn emerged from centuries old practical problems - e.g. drawing a map, solving Navier-Stokes eqn, ... Euler had this problem when he pored over the map of Russia to approximate its territory; where do you think Chebyshev polynomials came from? etc. I think it's fair to say that 18th and 19th century mathematical analysis focused on the theory of approximation, asking if universal approximators existed (Chebyshev asking if his polynomials are such approximators). UAT answered that: for polynomials (Weierstrass), feed-forward NNs (Cybenko) and decision trees [??].

Apart from the above, in the perceptron sitting in my skull the synapses between "mathematical", "precision" and "of a theorem" are inhibitory. Sorry! Try again, pls.