SERVING THE QUANTITATIVE FINANCE COMMUNITY

 
User avatar
ISayMoo
Topic Author
Posts: 1117
Joined: September 30th, 2015, 8:30 pm

Hacking ML for fun & profit

January 16th, 2018, 4:02 pm

Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples

Many machine learning models are vulnerable to adversarial examples: inputs that are specially crafted to cause a machine learning model to produce an incorrect output. Adversarial examples that affect one model often affect another model, even if the two models have different architectures or were trained on different training sets, so long as both models were trained to perform the same task. An attacker may therefore train their own substitute model, craft adversarial examples against the substitute, and transfer them to a victim model, with very little information about the victim. Recent work has further developed a technique that uses the victim model as an oracle to label a synthetic training set for the substitute, so the attacker need not even collect a training set to mount the attack. We extend these recent techniques using reservoir sampling to greatly enhance the efficiency of the training procedure for the substitute model. We introduce new transferability attacks between previously unexplored (substitute, victim) pairs of machine learning model classes, most notably SVMs and decision trees. We demonstrate our attacks on two commercial machine learning classification systems from Amazon (96.19% misclassification rate) and Google (88.94%) using only 800 queries of the victim model, thereby showing that existing machine learning approaches are in general vulnerable to systematic black-box attacks regardless of their structure.
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: Hacking ML for fun & profit

January 16th, 2018, 4:15 pm

This is good news, we still have a change to win from the upcoming robot T-3000 invasion by placing adversarial stickers on our front-doors and roofs. 
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: Hacking ML for fun & profit

January 16th, 2018, 4:33 pm

Interesting!

This issue really is deeply unavoidable because the distributions of the measurements of the objects overlap. The best solution is if the ML system can construct new sensors that provide new data that is at least somewhat independent of the old data. Some Chihuahuas look like some blueberry muffins in RGB images but maybe they don't in IR or UV. And if the system operator has physical control of the target class, they can change the target class to differentiate it (e.g, anti-counterfieting of paper currency). But if the adversary has control of the anti-target class, they can change it to make it more similar (e.g., add an IR food coloring to the muffin to make in chihuahua-like in IR).
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: Hacking ML for fun & profit

January 16th, 2018, 4:35 pm

A third solution is to create sensors for detecting adversaries and then locking them out of the system, spoofing them with false query-response data, or ignoring their submitted examples in training.
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: Hacking ML for fun & profit

January 16th, 2018, 4:40 pm

Tortoise: Let me tell you, instead. That will save some time. In the first place, Record Player Omega incorporated a television camera whose purpose it was to scan any record before playing it. This camera was hooked up to a small built-in computer, which would determine exactly the nature of the sounds, by looking at the groove- patterns. 

Achilles: Yes, so far so good. But what could Record Player Omega do with this information? 

Tortoise: By elaborate calculations, its little computer figured out what effects the sounds would have upon its phonograph. If it deduced that the sounds were such that they would cause the machine in its present configuration to break, then it did something very clever. Old Omega contained a device which could disassemble large parts of its phonograph subunit, and rebuild them in new ways, so that it could, in effect, change its own structure. If the sounds were "dangerous", a new configuration was chosen, one to which the sounds would pose no threat, and this new configuration would then be built by the rebuilding subunit, under direction of the little computer. Only after this rebuilding operation would Record Player Omega attempt to play the record.

Achilles: Aha! That must have spelled the end of your tricks. I bet you were a little disappointed.

Tortoise: Curious that you should think so ... I don't suppose that you know Godel's Incompleteness Theorem backwards and forwards, do you?
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: Hacking ML for fun & profit

January 16th, 2018, 4:47 pm

I smell a halting problem!

Methinks the clever adversary could create a record that breaks Omega's analysis & reconstruction system.
 
User avatar
ISayMoo
Topic Author
Posts: 1117
Joined: September 30th, 2015, 8:30 pm

Re: Hacking ML for fun & profit

January 16th, 2018, 5:30 pm

 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: Hacking ML for fun & profit

January 16th, 2018, 5:51 pm

That's what I was referring too indeed!

This adverserial sticker on the Velvet Underground records is really one of toaster to me, if would fool me too. In the past the examples used to be better, unrecognizable for humans, or actually recognized as something completely different.
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: Hacking ML for fun & profit

January 16th, 2018, 5:54 pm

I smell a halting problem!

Methinks the clever adversary could create a record that breaks Omega's analysis & reconstruction system.
It's so much fun reading it again (Gödel Esther Bach "contracrostipunctus"), and you're the crab of course!
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: Hacking ML for fun & profit

January 16th, 2018, 6:19 pm

And you need a really dumb AI that insists an image must have only one object.

Chain the object-classifier AI with an object-counter AI and the sticker trick becomes less effective.
 
User avatar
katastrofa
Posts: 6558
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Hacking ML for fun & profit

January 17th, 2018, 3:56 pm

I can see an interesting sticker with a "bugged" toaster photo in it and a banana. So, what was the definition of an adversarial example? :-)
Have you and your AI bros heard about Grey Thumb group?

I wonder if an NN was less prone to such cracking if you began training it on pictures with decreased resolution, and then use the outcome as an initial state for training on gradually improved resolution. I'm probably wrong, if not crazy trying to imagine the training process, thinking such a procedure could guide the parameters towards a specific minimum, rather than some Grand Unified- Martin-Luter King - Ultimate to Life, The Universe and Everything Global Caaaake Minimum. I don't think adding noise works like I think the above would.
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: Hacking ML for fun & profit

January 17th, 2018, 8:35 pm

Part of the failing of the original NN is the implicit assumption that the image is of a "normal" scene with "normal" objects.

A human looking at the scene with banana + sticker would surely be puzzled by the discrepancies between shadowing around the banana versus shadowing around the would be toaster. One of the most common cues for recognizing faked images is in inconsistencies in lighting and shadows. But Google's system apparently has not been trained to spot fakes.

Thus, another solution is to train the NN to detect scenes that contain potential adversarial inputs.
 
User avatar
outrun
Posts: 4573
Joined: April 29th, 2016, 1:40 pm

Re: Hacking ML for fun & profit

January 17th, 2018, 9:18 pm

..it will turtle all the way down until we have to start to discuss in the EU commission the fabric of what makes a real banana a real banana (addenda to Commission Regulation (EC) No. 2257/94). 

E.g. my kinds have plastic bananahodlers to hold their bananas that just look like bananas themselves. And there is fruits that are like the banana brothers or nephews, not to mention the variability in the 12 main banana lineages that haven't been crossbred much yet! 
And then there is the idea to make glow-in-the-dark bananas, do they still classify as bananas?

Btw a month a go I bought a Surinam bananas and baked it in the frying pan. I was expecting something marvellous and sweet, but it was like a potato! 
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Re: Hacking ML for fun & profit

January 17th, 2018, 11:38 pm

I smell sautéing cryptobananas.

Those Surinam bananas were an adversarial attempt to get you to say "potato" when you see a curved yellow object.
 
User avatar
Cuchulainn
Posts: 57301
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Re: Hacking ML for fun & profit

July 12th, 2018, 4:52 pm

Can bee colonies solve NN problems?
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...


GZIP: On