This thread has lost focus. Some issues

1. NNs are rarely used in finance
2. NNs do not respond well to noisy data (gradient is all over the place).
3. Models get stuck in  local minima.
4. Models are too complex.
5. Finance data != 1024X1024X3 image.

Cuchullain, NNs are nothing else but finite difference schemes...you are firing a bullet right into your own foot writing this

Never say that in ;public.
Flattery will get nowhere with me .. you say that to everyone.

Never say that in ;public.

Flattery will get nowhere with me .. you say that to everyone.

Ok, I am apologizing. Me bad :/
But the conclusion is correct: NNs are just finite difference methods.

Good Heavens! For more than forty years I have been speaking NNs without knowing it.
 
Your conclusions are not only not right, they are not even wrong.

Good Heavens! For more than forty years I have been speaking NNs without knowing it.
 
Your conclusions are not only not right, they are not even wrong.

That's not exactly what I am saying. I am saying that NNs are speaking about finite difference schemes without knowing it ! For instance, what I can show to you is, provided that you use RELUs basis functions for your Neural Networks (more precisely rectifiers, following wikipedia vocabulary), then you will end writing a very classical finite difference scheme.

Good Heavens! For more than forty years I have been speaking NNs without knowing it.
 
Your conclusions are not only not right, they are not even wrong.

That's not exactly what I am saying. I am saying that NNs are speaking about finite difference schemes without knowing it ! For instance, what I can show to you is, provided that you use RELUs basis functions for your Neural Networks (more precisely rectifiers, following wikipedia vocabulary), then you will end writing a very classical finite difference scheme.

Bishop's Pattern Recognition and Machine Learning, chapter 5.4.4. Just because you haven't seen unicorns, it doesn't mean they don't exist.

Good Heavens! For more than forty years I have been speaking NNs without knowing it.
 
Your conclusions are not only not right, they are not even wrong.

That's not exactly what I am saying. I am saying that NNs are speaking about finite difference schemes without knowing it ! For instance, what I can show to you is, provided that you use RELUs basis functions for your Neural Networks (more precisely rectifiers, following wikipedia vocabulary), then you will end writing a very classical finite difference scheme.

I wish I had your faith

Good Heavens! For more than forty years I have been speaking NNs without knowing it.
 
Your conclusions are not only not right, they are not even wrong.

That's not exactly what I am saying. I am saying that NNs are speaking about finite difference schemes without knowing it ! For instance, what I can show to you is, provided that you use RELUs basis functions for your Neural Networks (more precisely rectifiers, following wikipedia vocabulary), then you will end writing a very classical finite difference scheme.

Bishop's Pattern Recognition and Machine Learning, chapter 5.4.4. Just because you haven't seen unicorns, it doesn't mean they don't exist.

Nice book, thanks for the reference. In particular the chapter 6 describes what they call "kernel methods" and the authors tried to clarify the link between neural networks and kernel ones. To me they are the same: both are using kernels or basis functions. Difference schemes can also be defined using kernels. Indeed, there is a quite interesting video (in French) of PL Lions, saying that neural networks are also quite similar to mean-field games. The point is that all these methods are so generals that there exists big bridges between them.

The chapter 5.4.4 describes how to compute second order derivative of the error functions using finite difference methods. Is there any link with unicorns here ? To clarify, I was just writing that one could pretend to use a neural network or deep learning tralala to tackle a PDE problem, and end designing a plain old finite difference scheme, and he would be right. Actually, I am sure that a lot of Fintech do this for marketing reasons

The crucial difference between NNs and kernel functions is that the first are parameterised and the second aren't (NNs can change shape).

I quoted the chapter because it relates backpropagation to finite-difference method, AFAIR. You seemed to have said that ML is unaware of this correspondence.

The crucial difference between NNs and kernel functions is that the first are parameterised and the second aren't (NNs can change shape).

I quoted the chapter because it relates backpropagation to finite-difference method, AFAIR. You seemed to have said that ML is unaware of this correspondence.

I see. Thus Lagrangian Meshfree methods are Neural Networks ones ?

I was just saying that I can design finite difference schemes with NNs approach. But I sincerely hope that this bridge between NNs and finite difference methods has already been noticed before.

I was just saying that I can design finite difference schemes with NNs approach. But I sincerely hope that this bridge between NNs and finite difference methods has already been noticed before.

Remind us again. I missed it. The last time the details were missing AFAIR.

Are you referring to approximating the gradient by divided differences? Or something else?

// I have never seen FD mentioned in the same breath as NN.

I was just saying that I can design finite difference schemes with NNs approach. But I sincerely hope that this bridge between NNs and finite difference methods has already been noticed before.

Remind us again. I missed it. The last time the details were missing AFAIR.

Are you referring to approximating the gradient by divided differences?

// I have never seen FD mentioned in the same breath as NN.

Yes, I am saying that with NNs-type kernels I can design any type of finite difference schemes, in particular coinciding with forward / backward differences. I did not give any examples, lets give it : consider a simple 1D example over [0,1], N points x^n = (n+0.5)/N. Consider the rectifier function \varphi(x) = max(x,0) and the kernel K(x,y) = 1 - \varphi(y-x). Define the NxN matrix K = K(x^n,x^m)_{n,m}. Its inverse is (...,0,-N, 2N,-N,0,...) (Cuchullain, you should here have a tickle in your head). Takes now the derivative : K' = -\varphi'(x^n-x^m)_{n,m}. Do you want to express the derivative of a function with this kernel ? you would find the following NxN matrix operator (...,0, -N,+N,0,...), coinciding with the classical finite forward difference operator. You want to compute the Laplacian of a function associated with this kernel ? This is the very classical finite difference NxN matrix operator (...0,-N^2, 2N^2,-N^2,0,...). You want to move the x^n's because your algorithms is learning something ? it won't change these operators so much.

All right guys, but can your FD scheme play chess?

We're still working on the HOPSCOTCH method.
https://core.ac.uk/download/pdf/82024935.pdf

I don't play chess but did visit Bobby Fisher's last resting place.

All right guys, but can your FD scheme play chess?

No, cuchullain and I are about to launch finite-difference skynet. Bye human kind !

The crucial difference between NNs and kernel functions is that the first are parameterised and the second aren't (NNs can change shape).

I quoted the chapter because it relates backpropagation to finite-difference method, AFAIR. You seemed to have said that ML is unaware of this correspondence.

I see. Thus Lagrangian Meshfree methods are Neural Networks ones ?

I was just saying that I can design finite difference schemes with NNs approach. But I sincerely hope that this bridge between NNs and finite difference methods has already been noticed before.

Backpropagation is simply automatic differentiation done backwards. Is that the area where you see the "bridge between NNs and finite difference method"? Many introductions to backpropagation begin with showing that finite difference method fails in high dimensions.