- October 18th, 2019, 1:48 pm
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

Indeed, I would be much more general than Cybenko theorem, thus much more general than any feed forward neural network, writing consider [$]\varphi(x,y)[$] any function, don't call it an activation function, because such functions are called kernels, and define the following space [$]\psi(x) \in Sp...

- October 18th, 2019, 12:29 pm
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

Indeed, I would be much more general than Cybenko theorem, thus much more general than any feed forward neural network, writing consider [$]\varphi(x,y)[$] any function, don't call it an activation function, because such functions are called kernels, and define the following space [$]\psi(x) \in Spa...

- October 18th, 2019, 12:17 pm
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

If I had to write this wikipedia page, I would start by : consider [$]\varphi[$] any function, and define the following space [$] Span \{ \varphi \left( <y, x> \right), y \in \mathbb{R}^D \}[$] Consider a clever scalar product on this space, and define the closure of this space relatively to this n...

- October 18th, 2019, 11:40 am
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

Yes. But after your pertinent remark, I adapted my function to the wikipedia requirements to UAT.Previously you said you wanted a constant function, phi(x) = 1.

- October 18th, 2019, 11:39 am
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

Honestly, do you really think that the Cybenko theorem is correctly stated ? What is important in this Theorem is the following statement

**Functions of the form F(x) are Dense in C(I_m).**

- October 18th, 2019, 11:33 am
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

Could you go back to wikipedia and read the first sentence of the theorem? Well. If you are frightened by a constant function, consider [$]\varphi(x) = \inf (exp(1-|x|), 1)[$]. It is continuous, and fulfill Cybenko assumptions : one point is enough to get infinite accuracy on Cuchullain example. An...

- October 18th, 2019, 10:39 am
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

As an activation function in an NN? I don't think so. Here is Cybenko decomposition in wikipedia. [$]\sum_{i=1}^{N}v_i\varphi \left(w_i^T x+b_i\right) [$] I am not authorized to consider [$]\varphi(x) =1 [$] ? Could you go back to wikipedia and read the first sentence of the theorem? Well. If you ...

- October 18th, 2019, 8:11 am
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

Here is Cybenko decomposition in wikipedia. [$]\sum_{i=1}^{N}v_i\varphi \left(w_i^T x+b_i\right) [$]As an activation function in an NN? I don't think so.

I am not authorized to consider [$]\varphi(x) =1 [$] ?

- October 17th, 2019, 9:55 pm
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

Why not using a constant function for this particular case ? It is really better : just one point to get infinite accuracy. You can't beat that !@Which activation function should we use to approximate function B?

I'd use two differences of ReLu's, but two sigmoids might be good too.

- October 17th, 2019, 9:26 pm
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

It's enough to read the abstract of Cybenko's paper to answer your questions. Function A is nowhere continuous. B is alright. Function B is not continuous at x = {0,1}. Which activation function should we use to approximate function B? That's a nice question, and it has a nice answer. Take as activ...

- October 17th, 2019, 3:38 pm
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

To really understand Cybenko and Hornik you really need to have a degree in pure mathematics (lots of Measure Theory and hard Real Analysis). CS or Physics won't cut it! Otherwise, how's it possible to visualise [$]L_1[$] functions which are slippery at the best of times? Here's a test of Cybenko...

- October 17th, 2019, 10:58 am
- Forum: Technical Forum
- Topic: Why is Bellman Equation solved by backwards?
- Replies:
**32** - Views:
**10628**

*plop* you summoned me ?That's my nervous reaction to Cuchulainn and JLM's ramblings about the universal approximation theorem.

- October 16th, 2019, 10:32 am
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

Speak of the devil Sobolev training , train the function and its derivatives. That sound relevant. http://mcneela.github.io/machine_learning/2018/02/19/A-Synopsis-Of-DeepMinds-Sobolev-Training-Of-Neural-Networks.html Sounds like a logical step. Sobolev spaces are the bread and butter of advanced nu...

- October 16th, 2019, 9:00 am
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

My one-penny guess : take any square-integrable function [$]\phi[$], and call the convolution [$]\varphi = \phi \ast \phi[$] an activation function. Then you can use it in Cybenko Theorem. That's already a lot of examples. But there exists much more examples. In fact, give me any probability measure...

- October 15th, 2019, 2:12 pm
- Forum: Numerical Methods Forum
- Topic: Universal Approximation theorem
- Replies:
**251** - Views:
**42668**

Homework for Cuch: this recent paper (jun 2017) is getting many people excited, it proposes SELU (instead of RELU, sigmoid). It works really well, I'm seeing very stable learning with deep networks. You can go straight to the appendix with the proofs (page 9 ..100) that motivate why it should work ...

GZIP: On