## Search found 380 matches

October 18th, 2019, 1:48 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

Indeed, I would be much more general than Cybenko theorem, thus much more general than any feed forward neural network, writing consider $\varphi(x,y)$ any function, don't call it an activation function, because such functions are called kernels, and define the following space $\psi(x) \in Sp... October 18th, 2019, 12:29 pm Forum: Numerical Methods Forum Topic: Universal Approximation theorem Replies: 251 Views: 42668 ### Re: Universal Approximation theorem Indeed, I would be much more general than Cybenko theorem, thus much more general than any feed forward neural network, writing consider$\varphi(x,y)$any function, don't call it an activation function, because such functions are called kernels, and define the following space$\psi(x) \in Spa...
October 18th, 2019, 12:17 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

If I had to write this wikipedia page, I would start by : consider $\varphi$ any function, and define the following space $Span \{ \varphi \left( <y, x> \right), y \in \mathbb{R}^D \}$ Consider a clever scalar product on this space, and define the closure of this space relatively to this n...
October 18th, 2019, 11:40 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

Previously you said you wanted a constant function, phi(x) = 1.
Yes. But after your pertinent remark, I adapted my function to the wikipedia requirements to UAT.
October 18th, 2019, 11:39 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

Honestly, do you really think that the Cybenko theorem is correctly stated ? What is important in this Theorem is the following statement

Functions of the form F(x) are Dense in C(I_m).
October 18th, 2019, 11:33 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

Could you go back to wikipedia and read the first sentence of the theorem? Well. If you are frightened by a constant function, consider $\varphi(x) = \inf (exp(1-|x|), 1)$. It is continuous, and fulfill Cybenko assumptions : one point is enough to get infinite accuracy on Cuchullain example. An...
October 18th, 2019, 10:39 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

As an activation function in an NN? I don't think so. Here is Cybenko decomposition in wikipedia. $\sum_{i=1}^{N}v_i\varphi \left(w_i^T x+b_i\right)$ I am not authorized to consider $\varphi(x) =1$ ? Could you go back to wikipedia and read the first sentence of the theorem? Well. If you ...
October 18th, 2019, 8:11 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

As an activation function in an NN? I don't think so.
Here is Cybenko decomposition in wikipedia. $\sum_{i=1}^{N}v_i\varphi \left(w_i^T x+b_i\right)$
I am not authorized to consider $\varphi(x) =1$ ?
October 17th, 2019, 9:55 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

@Which activation function should we use to approximate function B?

I'd use two differences of ReLu's, but two sigmoids might be good too.
Why not using a constant function for this particular case ? It is really better : just one point to get infinite accuracy. You can't beat that !
October 17th, 2019, 9:26 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

It's enough to read the abstract of Cybenko's paper to answer your questions. Function A is nowhere continuous. B is alright. Function B is not continuous at x = {0,1}. Which activation function should we use to approximate function B? That's a nice question, and it has a nice answer. Take as activ...
October 17th, 2019, 3:38 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

To really understand Cybenko and Hornik you really need to have a degree in pure mathematics (lots of Measure Theory and hard Real Analysis). CS or Physics won't cut  it!  Otherwise, how's it possible to visualise $L_1$ functions which are slippery at the best of times? Here's a test of Cybenko...
October 17th, 2019, 10:58 am
Forum: Technical Forum
Topic: Why is Bellman Equation solved by backwards?
Replies: 32
Views: 10628

### Re: Why is Bellman Equation solved by backwards?

That's my nervous reaction to Cuchulainn and JLM's ramblings about the universal approximation theorem.
*plop* you summoned me ?
October 16th, 2019, 10:32 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

Speak of the devil Sobolev training , train the function and its derivatives. That sound relevant. http://mcneela.github.io/machine_learning/2018/02/19/A-Synopsis-Of-DeepMinds-Sobolev-Training-Of-Neural-Networks.html Sounds like a logical step. Sobolev spaces are the bread and butter of advanced nu...
October 16th, 2019, 9:00 am
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

My one-penny guess : take any square-integrable function $\phi$, and call the convolution $\varphi = \phi \ast \phi$ an activation function. Then you can use it in Cybenko Theorem. That's already a lot of examples. But there exists much more examples. In fact, give me any probability measure...
October 15th, 2019, 2:12 pm
Forum: Numerical Methods Forum
Topic: Universal Approximation theorem
Replies: 251
Views: 42668

### Re: Universal Approximation theorem

Homework for Cuch: this recent paper (jun 2017) is getting many people excited, it proposes SELU (instead of RELU, sigmoid). It works really well, I'm seeing very stable learning with deep networks. You can go straight to the appendix with the proofs (page 9 ..100) that motivate why it should work ...

GZIP: On