SERVING THE QUANTITATIVE FINANCE COMMUNITY

Search found 380 matches

by JohnLeM
October 18th, 2019, 1:48 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

Indeed, I would be much more general than Cybenko theorem, thus much more general than any feed forward neural network, writing consider [$]\varphi(x,y)[$] any function, don't call it an activation function, because such functions are called kernels, and define the following space [$]\psi(x) \in Sp...
by JohnLeM
October 18th, 2019, 12:29 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

Indeed, I would be much more general than Cybenko theorem, thus much more general than any feed forward neural network, writing consider [$]\varphi(x,y)[$] any function, don't call it an activation function, because such functions are called kernels, and define the following space [$]\psi(x) \in Spa...
by JohnLeM
October 18th, 2019, 12:17 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

If I had to write this wikipedia page, I would start by : consider [$]\varphi[$] any function, and define the following space [$] Span \{  \varphi \left( <y, x> \right), y \in \mathbb{R}^D \}[$] Consider a clever scalar product on this space, and define the closure of this space relatively to this n...
by JohnLeM
October 18th, 2019, 11:40 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

Previously you said you wanted a constant function, phi(x) = 1.
Yes. But after your pertinent remark, I adapted my function to the wikipedia requirements to UAT.
by JohnLeM
October 18th, 2019, 11:39 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

Honestly, do you really think that the Cybenko theorem is correctly stated ? What is important in this Theorem is the following statement

Functions of the form F(x) are Dense in C(I_m).
by JohnLeM
October 18th, 2019, 11:33 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

Could you go back to wikipedia and read the first sentence of the theorem? Well. If you are frightened by a constant function, consider [$]\varphi(x) = \inf (exp(1-|x|), 1)[$]. It is continuous, and fulfill Cybenko assumptions : one point is enough to get infinite accuracy on Cuchullain example. An...
by JohnLeM
October 18th, 2019, 10:39 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

As an activation function in an NN? I don't think so. Here is Cybenko decomposition in wikipedia. [$]\sum_{i=1}^{N}v_i\varphi \left(w_i^T x+b_i\right)  [$] I am not authorized to consider [$]\varphi(x) =1 [$] ? Could you go back to wikipedia and read the first sentence of the theorem? Well. If you ...
by JohnLeM
October 18th, 2019, 8:11 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

As an activation function in an NN? I don't think so.
Here is Cybenko decomposition in wikipedia. [$]\sum_{i=1}^{N}v_i\varphi \left(w_i^T x+b_i\right)  [$]
I am not authorized to consider [$]\varphi(x) =1 [$] ?
by JohnLeM
October 17th, 2019, 9:55 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

@Which activation function should we use to approximate function B?

I'd use two differences of ReLu's, but two sigmoids might be good too.
Why not using a constant function for this particular case ? It is really better : just one point to get infinite accuracy. You can't beat that !
by JohnLeM
October 17th, 2019, 9:26 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

It's enough to read the abstract of Cybenko's paper to answer your questions. Function A is nowhere continuous. B is alright. Function B is not continuous at x = {0,1}. Which activation function should we use to approximate function B? That's a nice question, and it has a nice answer. Take as activ...
by JohnLeM
October 17th, 2019, 3:38 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

To really understand Cybenko and Hornik you really need to have a degree in pure mathematics (lots of Measure Theory and hard Real Analysis). CS or Physics won't cut  it!  Otherwise, how's it possible to visualise [$]L_1[$] functions which are slippery at the best of times? Here's a test of Cybenko...
by JohnLeM
October 17th, 2019, 10:58 am
Forum: Technical Forum
Topic: Why is Bellman Equation solved by backwards?
Replies: 32
Views: 10628

Re: Why is Bellman Equation solved by backwards?

That's my nervous reaction to Cuchulainn and JLM's ramblings about the universal approximation theorem.
*plop* you summoned me ?
by JohnLeM
October 16th, 2019, 10:32 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

Speak of the devil Sobolev training , train the function and its derivatives. That sound relevant. http://mcneela.github.io/machine_learning/2018/02/19/A-Synopsis-Of-DeepMinds-Sobolev-Training-Of-Neural-Networks.html Sounds like a logical step. Sobolev spaces are the bread and butter of advanced nu...
by JohnLeM
October 16th, 2019, 9:00 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

My one-penny guess : take any square-integrable function [$]\phi[$], and call the convolution [$]\varphi = \phi \ast \phi[$] an activation function. Then you can use it in Cybenko Theorem. That's already a lot of examples. But there exists much more examples. In fact, give me any probability measure...
by JohnLeM
October 15th, 2019, 2:12 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Re: Universal Approximation theorem

Homework for Cuch: this recent paper (jun 2017) is getting many people excited, it proposes SELU (instead of RELU, sigmoid). It works really well, I'm seeing very stable learning with deep networks. You can go straight to the appendix with the proofs (page 9 ..100) that motivate why it should work ...
GZIP: On