Serving the Quantitative Finance Community

  • 1
  • 3
  • 4
  • 5
  • 6
  • 7
  • 21
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Universal Approximation theorem

July 25th, 2017, 4:31 pm

In ML 'learning' refers to "changing parameters in order to better achieve some goal" *) like finding {a,b} in y=ax+b that minimizes the LSE for a set of {x,y} pairs.

What you refer to is memory. A very popular branch of NN is "recurrent neural networks" which processes an ordered sequence of inputs and updates some memory state, which in turns affect its output. This is however not called learning if the network doesn't change the parameters that determine how it updates memory based on new inputs. A self driving car might see a car overtaking, updating a memory state which represents that as it happens, but it's not learning to achieve some goal better (to detect overtaking cars,.. given the same frames it will do the exact same memory updates)

*) this is supervised learning, (there is also unsupervised learning like finding clusters in data etc)
 
User avatar
tags
Posts: 3162
Joined: February 21st, 2010, 12:58 pm

Re: Universal Approximation theorem

July 25th, 2017, 6:46 pm

@outrun thanks.

I'm not sure where this thread is heading. Still,* there was that article J.P.Morgan’s massive guide to machine learning and big data jobs in finance some weeks ago with some interesting statements.

E.g. "You won’t need to be a machine learning expert, you will need to be an excellent quant and an excellent programmer"

* My first comma in a while
 
User avatar
Cuchulainn
Topic Author
Posts: 20252
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

July 26th, 2017, 11:52 am

Whether "deep learning" is being used in business or sought by employers is an empirical question. An analysis of resumes on LinkedIn and job postings in various internet sites would provide an answer.

Linkedin would not be my first port of call as a source of objective data. 
4. An army of people will be needed to acquire, clean, and assess the data 
 
User avatar
Cuchulainn
Topic Author
Posts: 20252
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

July 29th, 2017, 7:47 pm

 
User avatar
Cuchulainn
Topic Author
Posts: 20252
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

July 31st, 2017, 3:21 pm

Is there an alternative to gradient descent method? It's not that great, at least not always..
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Universal Approximation theorem

July 31st, 2017, 3:53 pm

What gradient descent algorithm are you talking about, and what problem are you applying it to? This reads to me like "a car isn't that great".
 
User avatar
Cuchulainn
Topic Author
Posts: 20252
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

July 31st, 2017, 4:01 pm

What gradient descent algorithm are you  talking about, and what problem are you applying it to? This reads to me like "a car isn't that great".
Any of them. I'm reading up the links posted here and there, NYT, WSJ and my own experience with numerical analysis. In other areas gradient descent is not the only kid on the block.

Here's a random sample from WSJ

Even the trendy technique of “deep learning,” which uses artificial neural networks to discern complex statistical correlations in huge amounts of data, often comes up short. Some of the best image-recognition systems, for example, can successfully distinguish dog breeds, yet remain capable of major blunders, like mistaking a simple pattern of yellow and black stripes for a school bus. Such systems can neither comprehend what is going on in complex visual scenes (“Who is chasing whom and why?”) nor follow simple instructions (“Read this story and summarize what it means”).

Anyhoo, one question above is: is there an alternative to gradient descent? The answer is very simple: it is Yes or No.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Universal Approximation theorem

July 31st, 2017, 4:12 pm

That pointed out problem had a different cause than gradient descent. People have that too: things like optical illusions, is the dress blue or gold?

For a lot of problems you can't even use gradient descent.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Universal Approximation theorem

July 31st, 2017, 8:16 pm

OpenAI is a new, research platform funded by some famous people (Musk, Thiel,..) to research important open problems in general AI. They wrote a blog about this topic not too long ago (probably where WJ picked it up)?

https://blog.openai.com/adversarial-example-research/

There are also quite a few papers on arxiv. A good search key is "adversarial examples deep learning"
Last edited by outrun on July 31st, 2017, 8:20 pm, edited 1 time in total.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Universal Approximation theorem

July 31st, 2017, 8:17 pm

..and the humans equivalent!

https://en.m.wikipedia.org/wiki/The_dress
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Universal Approximation theorem

July 31st, 2017, 8:39 pm

Cool overview of things we can expect in the next couple of years:
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Universal Approximation theorem

July 31st, 2017, 10:25 pm

What gradient descent algorithm are you  talking about, and what problem are you applying it to? This reads to me like "a car isn't that great".
Any of them. I'm reading up the links posted here and there, NYT, WSJ and my own experience with numerical analysis. In other areas gradient descent is not the only kid on the block.

Here's a random sample from WSJ

Even the trendy technique of “deep learning,” which uses artificial neural networks to discern complex statistical correlations in huge amounts of data, often comes up short. Some of the best image-recognition systems, for example, can successfully distinguish dog breeds, yet remain capable of major blunders, like mistaking a simple pattern of yellow and black stripes for a school bus. Such systems can neither comprehend what is going on in complex visual scenes (“Who is chasing whom and why?”) nor follow simple instructions (“Read this story and summarize what it means”).

Anyhoo, one question above is: is there an alternative to gradient descent? The answer is very simple: it is Yes or No.
Absolutely yes.

This link (https://stats.stackexchange.com/questio ... ropagation ) mentions some of the derivative-free optimization algorithms and cites a comparison article. Another article discussing alternatives of gradient descent specifically in a neural net context is http://courses.mai.liu.se/FU/MAI0083/Re ... ikanfs.pdf .

And as outrun mentioned the example failures mentioned in the article may have nothing to do with gradient descent and more to do with the total number of examples given to the deep learning system and the limits of those specific deep learning instances due to current CPU & RAM performance. How well can a chimpanzee perform on this test? Or a mouse? I'd wager a fair percentage of humans would fail the “Who is chasing whom and why?” test and standardized test data certainly prove that most people do poorly on the “Read this story and summarize what it means” test.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Universal Approximation theorem

July 31st, 2017, 10:40 pm

The reason that people pay attention to these adversarial examples is because of *security*. These example are like computer virus, specifically tuned and tailored through inspecting a specific neural network in order to fool it.

The examples are mostly on classification tasks: is this a stop sign or not? There also some remedies being researched but the main concern is a car that gets hacked into taking a turn, a drone that gets fooled into attacking the wrong people.  These adversarial examples won't pop up under normal conditions, but a nefarious person could potentially hack a neural network into illusions. 

E.g. suppose you succeed into training a NN to classify these images into either a chihuahua or a muffin, and it has 99% accuracy. What will it do when presented with a picture of a Kiwi? Whatever it should do, it will very likely do a very bad job. It has learned to look at subtle textures, number of black dots and their relative positions. But a Kiwi lives in a tangent dimension, al the way up there with the shrew. 
Image

Image
Last edited by outrun on July 31st, 2017, 10:53 pm, edited 1 time in total.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Universal Approximation theorem

July 31st, 2017, 10:44 pm

... I'm currently studying HF bots, see if I can manipulate them into 'positive' behaviour. Those bot's aren't very clever.
 
User avatar
Cuchulainn
Topic Author
Posts: 20252
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Universal Approximation theorem

October 11th, 2017, 9:09 pm

Image