SERVING THE QUANTITATIVE FINANCE COMMUNITY

 
User avatar
ISayMoo
Topic Author
Posts: 2293
Joined: September 30th, 2015, 8:30 pm

Re: If you are bored with Deep Networks

June 10th, 2018, 7:34 pm

Adversarial examples exist also for "old school" classifiers such as SVMs. It's not a problem with deep learning only.
Those methods seems to have some commonality.
Yes, they do. The core concept is classification boundary. If your data has very many dimensions (e.g. 100,000 for relatively small RGB images), any random point in the input space is statistically likely to be close to the classification boundary. Just give it a little push...
 
User avatar
katastrofa
Posts: 8992
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: If you are bored with Deep Networks

June 11th, 2018, 3:46 pm

Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition (as I know it from research in crystallography), these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture (you use the classifier trained on black points to assign the red x)?
Attachments
2452CCB8-CE83-47C5-8395-803985902477.jpeg
Last edited by katastrofa on June 11th, 2018, 10:40 pm, edited 1 time in total.
 
User avatar
katastrofa
Posts: 8992
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: If you are bored with Deep Networks

June 11th, 2018, 3:49 pm

@Cuchulainn: "This probably means that the problem is "ill-posed" in some way."


Sometimes it's easier to use a scotch tape rather than shift a paradigm :-)
Last edited by katastrofa on June 11th, 2018, 10:18 pm, edited 1 time in total.
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: If you are bored with Deep Networks

June 11th, 2018, 6:56 pm

Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition, these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture?
The Voronoi partition would create the classification boundary assuming the populations really are separable. And, yes, it can be a bit random-seeming because the directions of the separating lines (or hyperplanes) that radiate off into empty space are defined entirely by only two data points which makes them extremely sensitive to the locations of those data points.

The more likely condition is an overlap in the populations such that a given region of the space has a non-zero probability of being associated with two or more types. That is, there may be some images of dogs that are indistinguishable from some images of cats.

If the training data points do not cover the full space, there's the fundamental problem that the empty parts really cannot be classified without making extrapolating assumptions about the distributions of the categories (i.e., how do the tails of the distributions extend into the empty space). Moreover, the empty space may correspond to objects not in the set of classified categories (e.g., velociraptors) or non-sense images (left-half cat, right-half dog images).
 
User avatar
Cuchulainn
Posts: 61624
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Re: If you are bored with Deep Networks

June 11th, 2018, 8:07 pm

@Cuchilainn: "This probably means that the problem is "ill-posed" in some way."


Sometimes it's easier to use a scotch tape rather than shift a paradigm :-)
Let's take Goodfellow's adversarial example again, page 261
Problem space: it is a panda even when the image is perturbed by a small amount. That's a fact.  And this is reality to be modelled.

Representation space (algorithm). The algo recognises a panda in the unperturbed case and in the perturbed case it is a funky gibbon. Your algo must also return a panda in both cases.
Conclusion: the algorithm is wrong. I can't think of any other explanation.
//
The analogy in another domain in inventing negative probabilities and other profanities (name-dropping names like Dirac and Feynman to help the cause) to explain away when the binomial breaks down. It's quite possibly fallacious thinking. I suspect that this is becoming the standard 'scientific method', i.e fixing and fudging.
Last edited by Cuchulainn on June 11th, 2018, 8:18 pm, edited 4 times in total.
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: If you are bored with Deep Networks

June 11th, 2018, 8:13 pm

@Cuchilainn: "This probably means that the problem is "ill-posed" in some way."


Sometimes it's easier to use a scotch tape rather than shift a paradigm :-)
Let's take Goodfellow's adversarial example page 261
Problem space: it is a panda even when the image is perturbed by a small amount. That's a fact. .

Representation space (algorithm). The algo recognises a panda in the unperturbed and in the perturbed case it a funky gibbon. Your algo must also return a panda in both cases.
Conclusion: the algorithm is wrong. I can't think of any other explanation.
//
The analogy in another domain in inventing negative probabilities and other profanities (name-dropping names like Dirac and Feynman) to explain away when the binomial breaks down. It's quite possibly fallacious thinking. I suspect that this is becoming the standard 'scientific method', i.e fixing and fudging.
Is the "perturbation" a physically accurate one? And was the net trained with examples perturbed in this way?
 
User avatar
Cuchulainn
Posts: 61624
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Re: If you are bored with Deep Networks

June 11th, 2018, 8:27 pm

Is the "perturbation" a physically accurate one? 

Humans see it as a panda. So yes. It looks awfully like a panda.


And was the net trained with examples perturbed in this way?
I don't do pandas. So I don't know. ISM??

Image
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
 
User avatar
Cuchulainn
Posts: 61624
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Re: If you are bored with Deep Networks

June 11th, 2018, 8:37 pm

Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition, these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture?
Some links
https://www.maths.tcd.ie/~odunlain/papers.html
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
 
User avatar
katastrofa
Posts: 8992
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: If you are bored with Deep Networks

June 11th, 2018, 10:52 pm

@Cuchilainn: "This probably means that the problem is "ill-posed" in some way."


Sometimes it's easier to use a scotch tape rather than shift a paradigm :-)
Let's take Goodfellow's adversarial example again, page 261
Problem space: it is a panda even when the image is perturbed by a small amount. That's a fact.  And this is reality to be modelled.

Representation space (algorithm). The algo recognises a panda in the unperturbed case and in the perturbed case it is a funky gibbon. Your algo must also return a panda in both cases.
Conclusion: the algorithm is wrong. I can't think of any other explanation.
//
The analogy in another domain in inventing negative probabilities and other profanities (name-dropping names like Dirac and Feynman to help the cause) to explain away when the binomial breaks down. It's quite possibly fallacious thinking. I suspect that this is becoming the standard 'scientific method', i.e fixing and fudging.
It would be a problem if a scientist did that (use Scotch tape). It's fine though when it's an engineer. One rolls up the sleeves and get hands dirty in the guts of the algorithm. There are probably several effects that make it fail - they need to be identified and fixed. That can be the way to developing a more robust approach. Just my basic philosophy - you probably know better how it works in both cases.
 
User avatar
ISayMoo
Topic Author
Posts: 2293
Joined: September 30th, 2015, 8:30 pm

Re: If you are bored with Deep Networks

June 12th, 2018, 8:23 am

Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition (as I know it from research in crystallography), these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture (you use the classifier trained on black points to assign the red x)?
I think if this was the whole story, then we wouldn't observe adversarial examples arising from the training set, because then all points would have been comfortably away from the classification boundary. But we do observe them also in the training sets.
 
User avatar
ISayMoo
Topic Author
Posts: 2293
Joined: September 30th, 2015, 8:30 pm

Re: If you are bored with Deep Networks

June 12th, 2018, 8:26 am

Is the "perturbation" a physically accurate one? 

Humans see it as a panda. So yes. It looks awfully like a panda.


And was the net trained with examples perturbed in this way?
I don't do pandas. So I don't know. ISM??

Image
I think the Goodfellow et al paper was before they started doing adversarial training. Adversarial training is one of the best defences, but not a 100% sure one. IMHO to get rid of the problem we need to change the architecture and go beyond CNNs.
People are still working on CNNs because they want to understand *why* they break. Everybody knows they're not enough, the question is what do we do next.

I think Cuch's disgust comes from the difference between watching how a cake is being baked (messy, dirty business with things sticking to your fingers and lots of bowls piling up in the dishwasher) and consuming the finished product.
 
User avatar
Cuchulainn
Posts: 61624
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Re: If you are bored with Deep Networks

June 12th, 2018, 8:58 am

That is probably not a totally 100% accurate interpretation. Analogies are not always helpful.

Goodfellow et al is a book from 2016.

Besides, should we not be talking about finance cases rather than gibbon derriere?
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
 
User avatar
ISayMoo
Topic Author
Posts: 2293
Joined: September 30th, 2015, 8:30 pm

Re: If you are bored with Deep Networks

June 12th, 2018, 10:18 am

The panda/gibbon example comes from a 2014 paper: https://arxiv.org/pdf/1412.6572.pdf
which shows that adversarial training reduces the adversarial error rate considerably (89.4% to 17.9% in one case, see Sec. 6). So I was wrong that they didn't discuss adv, training, but right that it helps to avoid such bad examples.

Another thing to consider is that ImageNet is a particular dataset. Over 10% of its images are dogs. Many categories are so close that it's easy for a human to confuse them too. See also this paper which questions the accuracy of the labelling of the original dataset.

IOW: garbage in -> garbage out.
 
User avatar
katastrofa
Posts: 8992
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: If you are bored with Deep Networks

June 12th, 2018, 5:34 pm

Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition (as I know it from research in crystallography), these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture (you use the classifier trained on black points to assign the red x)?
I think if this was the whole story, then we wouldn't observe adversarial examples arising from the training set, because then all points would have been comfortably away from the classification boundary. But we do observe them also in the training sets.
Maybe you just have too little data to train the classifier properly. If you want to achieve a precision higher than a human eye (vide panda + returbed panda), I'm wondering if enough data exists.
 
User avatar
ISayMoo
Topic Author
Posts: 2293
Joined: September 30th, 2015, 8:30 pm

Re: If you are bored with Deep Networks

June 12th, 2018, 7:07 pm

I wonder about that too! But somehow, humans and other animals cope with that.
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


Twitter LinkedIn Instagram

JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...


GZIP: On