SERVING THE QUANTITATIVE FINANCE COMMUNITY

  • 1
  • 3
  • 4
  • 5
  • 6
  • 7
  • 34
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: If you are bored with Deep Networks

November 15th, 2017, 10:22 pm

..about quantum computers: 

I was was standing in a line at a coffee bar today. The girl behind the counter was giving an weddingcard invitation to a friend who standing in front of me in line. They were chatting about the card: the dress code was "chic" and the guy say "what's that? Ah! I'll wear my kilt then". Then he asked her about wedding gift, she said she'll let him decide himself.. and then he asked her if she wanted some time on his quantum computer?!
..meanwhile I was standing there waiting for my coffee, mainly killing time waiting for a NN job to converge slooooowly! I could use that quantum computer time!!

It was really weird, .. I googled a bit and found out that there is a quantum computing exhibition tomorrow in that building.
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: If you are bored with Deep Networks

November 15th, 2017, 10:33 pm

Is there an equivalent of local and global minima in the training of actual neurons? I mean the ones in your head!
I don't think so: there is no objective function! A NN isn't really trying to mimic a real brain. There is one cool project that tries to simulate the nervous system of a real worm, and including the worms environment. I bet its a very different thype of NN. 

http://www.openworm.org/
Important Limitations

Please note:

* An accurate representation of the ion channels and their distributions in each neuron has not yet been attempted. Work on a cell model from C. elegans with ion channels can be found here
* An accurate representation of the synapses between the neurons has not yet been attempted. They are simplistic synapses only for the moment.
More accurate models of conductance based neurons and more realistic synapses will be incorporated into c302 first, and then the neuroConstruct model will be updated.
 
User avatar
Cuchulainn
Posts: 61516
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Re: If you are bored with Deep Networks

November 15th, 2017, 11:35 pm

I think it's impossible to construct a NN that has a loss surface like that.
Actually, the properties of NN loss surfaces depend quite a lot on the dataset: see this paper

"It is widely believed that training of deep models using gradient methods works so well because the error surface either has no local minima, or if they exist they need to be close in value to the global minimum. It is known that such results hold under very strong assumptions which are not satisfied by real models. In this paper we present examples showing that for such theorem to be true additional assumptions on the data, initialization schemes and/or the model classes have to be made. We look at the particular case of finite size datasets. We demonstrate that in this scenario one can construct counter-examples (datasets or initialization schemes) when the network does become susceptible to bad local minima over the weight space."

Note that their results also cast a slight shadow of doubt on the results of the paper I linked to today in my previous post. Other nice results were also proven under optimistic and, as admitted by the author, somewhat unrealistic assumptions.
What's the academic background of the author? CS, maths, physics, eco. 
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 12:10 am

Is there an equivalent of local and global minima in the training of actual neurons? I mean the ones in your head!
I seem to recall that babies go through something like four different stages with different motion patterns as they learn to crawl -- the inefficient pre-crawling motion patterns would be local minima.

There's probably lots of local minima in sports like golf or tennis in which an untrained novice naturally develops a partially-effective grip or swing that needs to be "unlearned" to achieve a more effective grip or swing.
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 12:50 am

the high jump!
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 1:34 am

Yes! Good old Fosbury had to show them how to flop. But the flop could also be a local minimum.

I'd also say the human brain's current understanding of NN is beset by local minima, no?
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 7:51 am

Yes! Good old Fosbury had to show them how to flop. But the flop could also be a local minimum.

I'd also say the human brain's current understanding of NN is beset by local minima, no?
There was the AI winter but now we have gradient again!

I bet the arrival of quantum computers will help the high dimensional research a lot and keep the momentum going. Another factor is that we still have a lot to learn about possible ways we can use it. You can change your look on things and suddenly see new application domains. A lot of inspiration from that comes from biology. Things like "attention" have their NN couterpart.

Besides the research we are now also seeing lots of useful application, anyone can make disruptive tools now. A couple of days ago a friend who works at a space agency came over for dinner. They have these hackathons where employees get to do fun projects and his roommate did a cool ml project.

One of the things they do is make custom solar cells depending on the mission. Part of that is to inspect the produced solar cells for cracks before they send them up to space. They take high resolution images with a robot and then a human goes and stares at them for weeks. However that is now done by ML with a much better detection rate. They guy demoed it in a single day and now they are taking it to production. Faster inspection, higher quality of inspection, lower cost, shorter deadlines.
 
User avatar
Paul
Posts: 10415
Joined: July 20th, 2001, 3:28 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 8:26 am

Life is a journey not a destination. (I read that on a mug. Or it might have been on a pair of pants.)

This thread reads like we have a place to get to and we have to get there as quickly as possible. That’s a bit sad.
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 9:55 am

There is always even better things beyond the next destination. The amount of things you experience and learn depends on the speed you move, and the time you have left.

Saw this on a pin button.
 
User avatar
Paul
Posts: 10415
Joined: July 20th, 2001, 3:28 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 10:02 am

"Always be yourself...unless you can be a unicorn"
 
User avatar
ISayMoo
Topic Author
Posts: 2292
Joined: September 30th, 2015, 8:30 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 11:22 am

What's the academic background of the author? CS, maths, physics, eco. 
I don't know. Does it matter? What can you say about the paper itself?
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 11:34 am

What's the academic background of the author? CS, maths, physics, eco. 
I don't know. Does it matter? What can you say about the paper itself?
I was puzzles by that too.
 
User avatar
ISayMoo
Topic Author
Posts: 2292
Joined: September 30th, 2015, 8:30 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 12:01 pm

I am worried by the fact that so many people can't be bothered to search Google to find answers to their questions.

the author

(btw, "Puzzles" would be a great name for a cat)
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: If you are bored with Deep Networks

November 16th, 2017, 12:19 pm

Dr Phil would say "all behavior, no matter how strange, becomes rational once you know the value system that drives someone"
 
User avatar
Cuchulainn
Posts: 61516
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Re: If you are bored with Deep Networks

November 16th, 2017, 2:16 pm

What's the academic background of the author? CS, maths, physics, eco. 
I don't know. Does it matter? What can you say about the paper itself?
Yes, it matters. I have not read the paper, but I have read others (e.g. DL fo 100 dimension PDE..).
Are these articles refereed? Many are written in hope. 
too much googling destroys the brain.
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


Twitter LinkedIn Instagram

JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...


GZIP: On