Page 11 of 39

Re: If you are bored with Deep Networks

Posted: June 10th, 2018, 7:34 pm
by ISayMoo
Adversarial examples exist also for "old school" classifiers such as SVMs. It's not a problem with deep learning only.
Those methods seems to have some commonality.
Yes, they do. The core concept is classification boundary. If your data has very many dimensions (e.g. 100,000 for relatively small RGB images), any random point in the input space is statistically likely to be close to the classification boundary. Just give it a little push...

Re: If you are bored with Deep Networks

Posted: June 11th, 2018, 3:46 pm
by katastrofa
Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition (as I know it from research in crystallography), these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture (you use the classifier trained on black points to assign the red x)?

Re: If you are bored with Deep Networks

Posted: June 11th, 2018, 3:49 pm
by katastrofa
@Cuchulainn: "This probably means that the problem is "ill-posed" in some way."


Sometimes it's easier to use a scotch tape rather than shift a paradigm :-)

Re: If you are bored with Deep Networks

Posted: June 11th, 2018, 6:56 pm
by Traden4Alpha
Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition, these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture?
The Voronoi partition would create the classification boundary assuming the populations really are separable. And, yes, it can be a bit random-seeming because the directions of the separating lines (or hyperplanes) that radiate off into empty space are defined entirely by only two data points which makes them extremely sensitive to the locations of those data points.

The more likely condition is an overlap in the populations such that a given region of the space has a non-zero probability of being associated with two or more types. That is, there may be some images of dogs that are indistinguishable from some images of cats.

If the training data points do not cover the full space, there's the fundamental problem that the empty parts really cannot be classified without making extrapolating assumptions about the distributions of the categories (i.e., how do the tails of the distributions extend into the empty space). Moreover, the empty space may correspond to objects not in the set of classified categories (e.g., velociraptors) or non-sense images (left-half cat, right-half dog images).

Re: If you are bored with Deep Networks

Posted: June 11th, 2018, 8:07 pm
by Cuchulainn
@Cuchilainn: "This probably means that the problem is "ill-posed" in some way."


Sometimes it's easier to use a scotch tape rather than shift a paradigm :-)
Let's take Goodfellow's adversarial example again, page 261
Problem space: it is a panda even when the image is perturbed by a small amount. That's a fact.  And this is reality to be modelled.

Representation space (algorithm). The algo recognises a panda in the unperturbed case and in the perturbed case it is a funky gibbon. Your algo must also return a panda in both cases.
Conclusion: the algorithm is wrong. I can't think of any other explanation.
//
The analogy in another domain in inventing negative probabilities and other profanities (name-dropping names like Dirac and Feynman to help the cause) to explain away when the binomial breaks down. It's quite possibly fallacious thinking. I suspect that this is becoming the standard 'scientific method', i.e fixing and fudging.

Re: If you are bored with Deep Networks

Posted: June 11th, 2018, 8:13 pm
by Traden4Alpha
@Cuchilainn: "This probably means that the problem is "ill-posed" in some way."


Sometimes it's easier to use a scotch tape rather than shift a paradigm :-)
Let's take Goodfellow's adversarial example page 261
Problem space: it is a panda even when the image is perturbed by a small amount. That's a fact. .

Representation space (algorithm). The algo recognises a panda in the unperturbed and in the perturbed case it a funky gibbon. Your algo must also return a panda in both cases.
Conclusion: the algorithm is wrong. I can't think of any other explanation.
//
The analogy in another domain in inventing negative probabilities and other profanities (name-dropping names like Dirac and Feynman) to explain away when the binomial breaks down. It's quite possibly fallacious thinking. I suspect that this is becoming the standard 'scientific method', i.e fixing and fudging.
Is the "perturbation" a physically accurate one? And was the net trained with examples perturbed in this way?

Re: If you are bored with Deep Networks

Posted: June 11th, 2018, 8:27 pm
by Cuchulainn
Is the "perturbation" a physically accurate one? 

Humans see it as a panda. So yes. It looks awfully like a panda.


And was the net trained with examples perturbed in this way?
I don't do pandas. So I don't know. ISM??

Image

Re: If you are bored with Deep Networks

Posted: June 11th, 2018, 8:37 pm
by Cuchulainn
Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition, these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture?
Some links
https://www.maths.tcd.ie/~odunlain/papers.html

Re: If you are bored with Deep Networks

Posted: June 11th, 2018, 10:52 pm
by katastrofa
@Cuchilainn: "This probably means that the problem is "ill-posed" in some way."


Sometimes it's easier to use a scotch tape rather than shift a paradigm :-)
Let's take Goodfellow's adversarial example again, page 261
Problem space: it is a panda even when the image is perturbed by a small amount. That's a fact.  And this is reality to be modelled.

Representation space (algorithm). The algo recognises a panda in the unperturbed case and in the perturbed case it is a funky gibbon. Your algo must also return a panda in both cases.
Conclusion: the algorithm is wrong. I can't think of any other explanation.
//
The analogy in another domain in inventing negative probabilities and other profanities (name-dropping names like Dirac and Feynman to help the cause) to explain away when the binomial breaks down. It's quite possibly fallacious thinking. I suspect that this is becoming the standard 'scientific method', i.e fixing and fudging.
It would be a problem if a scientist did that (use Scotch tape). It's fine though when it's an engineer. One rolls up the sleeves and get hands dirty in the guts of the algorithm. There are probably several effects that make it fail - they need to be identified and fixed. That can be the way to developing a more robust approach. Just my basic philosophy - you probably know better how it works in both cases.

Re: If you are bored with Deep Networks

Posted: June 12th, 2018, 8:23 am
by ISayMoo
Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition (as I know it from research in crystallography), these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture (you use the classifier trained on black points to assign the red x)?
I think if this was the whole story, then we wouldn't observe adversarial examples arising from the training set, because then all points would have been comfortably away from the classification boundary. But we do observe them also in the training sets.

Re: If you are bored with Deep Networks

Posted: June 12th, 2018, 8:26 am
by ISayMoo
Is the "perturbation" a physically accurate one? 

Humans see it as a panda. So yes. It looks awfully like a panda.


And was the net trained with examples perturbed in this way?
I don't do pandas. So I don't know. ISM??

Image
I think the Goodfellow et al paper was before they started doing adversarial training. Adversarial training is one of the best defences, but not a 100% sure one. IMHO to get rid of the problem we need to change the architecture and go beyond CNNs.
People are still working on CNNs because they want to understand *why* they break. Everybody knows they're not enough, the question is what do we do next.

I think Cuch's disgust comes from the difference between watching how a cake is being baked (messy, dirty business with things sticking to your fingers and lots of bowls piling up in the dishwasher) and consuming the finished product.

Re: If you are bored with Deep Networks

Posted: June 12th, 2018, 8:58 am
by Cuchulainn
That is probably not a totally 100% accurate interpretation. Analogies are not always helpful.

Goodfellow et al is a book from 2016.

Besides, should we not be talking about finance cases rather than gibbon derriere?

Re: If you are bored with Deep Networks

Posted: June 12th, 2018, 10:18 am
by ISayMoo
The panda/gibbon example comes from a 2014 paper: https://arxiv.org/pdf/1412.6572.pdf
which shows that adversarial training reduces the adversarial error rate considerably (89.4% to 17.9% in one case, see Sec. 6). So I was wrong that they didn't discuss adv, training, but right that it helps to avoid such bad examples.

Another thing to consider is that ImageNet is a particular dataset. Over 10% of its images are dogs. Many categories are so close that it's easy for a human to confuse them too. See also this paper which questions the accuracy of the labelling of the original dataset.

IOW: garbage in -> garbage out.

Re: If you are bored with Deep Networks

Posted: June 12th, 2018, 5:34 pm
by katastrofa
Another crazy question. I'm wondering about the opposite to what you wrote about the proximity of the classification boundary: your input does not cover the full space - the points are probably denser in some regions than others and there are empty areas in this space. Based on my understanding of Voronoi partition (as I know it from research in crystallography), these empty areas will also be assigned to some clusters - practically at random. What if the problem is like in situation 1 from the attached picture (you use the classifier trained on black points to assign the red x)?
I think if this was the whole story, then we wouldn't observe adversarial examples arising from the training set, because then all points would have been comfortably away from the classification boundary. But we do observe them also in the training sets.
Maybe you just have too little data to train the classifier properly. If you want to achieve a precision higher than a human eye (vide panda + returbed panda), I'm wondering if enough data exists.

Re: If you are bored with Deep Networks

Posted: June 12th, 2018, 7:07 pm
by ISayMoo
I wonder about that too! But somehow, humans and other animals cope with that.