Serving the Quantitative Finance Community

 
User avatar
Cuchulainn
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Machine learning for fun, to pay your rent or whatever

March 31st, 2018, 11:04 am

I was talking to a Swedish mathematician whose husband is a neuro-scientist and is involved in AI research. She was remarking how unpredictable the reliability of the ML algorithms were.

Just saying.

Is SGD the only show in town? What's the story on TDA?
 
User avatar
TidalFlood
Posts: 33
Joined: January 2nd, 2018, 3:13 pm

Re: Machine learning for fun, to pay your rent or whatever

March 31st, 2018, 12:37 pm

I'm currious - what kind of reliability was she referring to?  For example, predictive reliability?
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

March 31st, 2018, 12:52 pm

I'd also wonder how much of the problem arises from:

1. providing poorly sampled or poorly measured data sets
2. misusing the ML algorithms
3. attempting to teach a machine something that is not true

(Does the observation say more about the unreliability of people than the unreliability of ML?)
 
User avatar
tags
Topic Author
Posts: 3162
Joined: February 21st, 2010, 12:58 pm

Re: Machine learning for fun, to pay your rent or whatever

March 31st, 2018, 12:59 pm

I'd also wonder how much of the problem arises from:

1. providing poorly sampled or poorly measured data sets
2. misusing the ML algorithms
3. attempting to teach a machine something that is not true

(Does the observation say more about the unreliability of people than the unreliability of ML?)
4. Reliability measurement
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

March 31st, 2018, 1:10 pm

I'd also wonder how much of the problem arises from:

1. providing poorly sampled or poorly measured data sets
2. misusing the ML algorithms
3. attempting to teach a machine something that is not true

(Does the observation say more about the unreliability of people than the unreliability of ML?)
4. Reliability measurement
LOL! Very good!

(It's another fine example of the unreliability of people to understand unreliability!)
 
User avatar
TidalFlood
Posts: 33
Joined: January 2nd, 2018, 3:13 pm

Re: Machine learning for fun, to pay your rent or whatever

March 31st, 2018, 9:42 pm

Hyper-determinacy was a concept formalised by a Philosopher at NYU.  Just as Horstein has generalised Kripke's fixed point for truth, it is conceivable that such a fixed point and notion of hyper-determinacy can be made applicable to the concept of reliability or to the item of vocabulary 'reliability'.
 
User avatar
ISayMoo
Posts: 2332
Joined: September 30th, 2015, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

April 1st, 2018, 12:55 pm

I was talking to a Swedish mathematician whose husband is a neuro-scientist and is involved in AI research. She was remarking how unpredictable the reliability of the ML algorithms were.

Just saying.

Is SGD the only show in town? What's the story on TDA?
No, it's not.
k-FAC
Difference Target Propagation
The first one is formally analogous to General Relativity.
 
User avatar
ISayMoo
Posts: 2332
Joined: September 30th, 2015, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

April 1st, 2018, 12:57 pm

I'd also wonder how much of the problem arises from:

1. providing poorly sampled or poorly measured data sets
2. misusing the ML algorithms
3. attempting to teach a machine something that is not true

(Does the observation say more about the unreliability of people than the unreliability of ML?)
Adversarial examples occur even if you train on a large dataset, use the network as designed and attempt to teach it the truth.
There is a formal definition of reliability - expected loss - but it's unusable in practice.
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

April 1st, 2018, 1:23 pm

I'd also wonder how much of the problem arises from:

1. providing poorly sampled or poorly measured data sets
2. misusing the ML algorithms
3. attempting to teach a machine something that is not true

(Does the observation say more about the unreliability of people than the unreliability of ML?)
Adversarial examples occur even if you train on a large dataset, use the network as designed and attempt to teach it the truth.
There is a formal definition of reliability - expected loss - but it's unusable in practice.
Indeed!

But it's not just the adversarial examples that cause problems. There's also spurious correlations caused by under-sampling and biased sampling of most of the dimensions of the world.

One early attempt at ML for recognizing armored vehicles seemed to be working well but then further out-of-sample tests proved it was an absolute failure. The root cause was that most of the test images with armored vehicles had been taken on a cloudy day and that weather condition is what the ML algorithm learned to spot.

So how does the ML researcher ensure they've got good training data (and how many "unreliable" ML projects are the fault of the data not the ML)?
 
User avatar
Cuchulainn
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Machine learning for fun, to pay your rent or whatever

April 1st, 2018, 2:57 pm

So how does the ML researcher ensure they've got good training data (and how many "unreliable" ML projects are the fault of the data not the ML)?

This is a mathematical issue: for which class of data is a given algorithm applicable? But does the 'average' ML researcher have the necessary background? 

Aliter.. assuming the input is what it is, which algorithms will work with it? Ideally, you should know a-priori. You can't bend the laws of mathematics, e.g. taking the derivative of a discrete function is darn difficult.
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

April 1st, 2018, 3:08 pm

So how does the ML researcher ensure they've got good training data (and how many "unreliable" ML projects are the fault of the data not the ML)?

This is a mathematical issue: for which class of data is a given algorithm applicable? But does the 'average' ML researcher have the necessary background?

Aliter.. assuming the input is what it is, which algorithms will work with it? Ideally, you should know a-priori. You can't bend the laws of mathematics, e.g. taking the derivative of a discrete function is darn difficult.
Is it a mathematical issue?

What math (actually what algorithm) can determine that a given body of physical data belongs to a given mathematical class?

And, more importantly, isn't it an issue of science to determine whether a given physical system can be modeled by a particular class of mathematics? That decision is then prerequisite to determine whether a particular subset of data from that physical system is suitable for a particular ML algorithm.
 
User avatar
Cuchulainn
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Machine learning for fun, to pay your rent or whatever

April 1st, 2018, 3:31 pm

So how does the ML researcher ensure they've got good training data (and how many "unreliable" ML projects are the fault of the data not the ML)?

This is a mathematical issue: for which class of data is a given algorithm applicable? But does the 'average' ML researcher have the necessary background?

Aliter.. assuming the input is what it is, which algorithms will work with it? Ideally, you should know a-priori. You can't bend the laws of mathematics, e.g. taking the derivative of a discrete function is darn difficult.
Is it a mathematical issue?

What math (actually what algorithm) can determine that a given body of physical data belongs to a given mathematical class?

And, more importantly, isn't it an issue of science to determine whether a given physical system can be modeled by a particular class of mathematics?  That decision is then prerequisite to determine whether a particular subset of data from that physical system is suitable for a particular ML algorithm.
aka the Scientific Method or is it too early?
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

April 1st, 2018, 4:27 pm

So how does the ML researcher ensure they've got good training data (and how many "unreliable" ML projects are the fault of the data not the ML)?

This is a mathematical issue: for which class of data is a given algorithm applicable? But does the 'average' ML researcher have the necessary background?

Aliter.. assuming the input is what it is, which algorithms will work with it? Ideally, you should know a-priori. You can't bend the laws of mathematics, e.g. taking the derivative of a discrete function is darn difficult.
Is it a mathematical issue?

What math (actually what algorithm) can determine that a given body of physical data belongs to a given mathematical class?

And, more importantly, isn't it an issue of science to determine whether a given physical system can be modeled by a particular class of mathematics?  That decision is then prerequisite to determine whether a particular subset of data from that physical system is suitable for a particular ML algorithm.
aka the Scientific Method or is it too early?
No, it's perfect timing. Yet to your point about math, the Scientific Method doesn't provide much specific guidance on the relationships between the mathematical form of a theory, the mathematical nature of controls and interventions in an experiment, and the proper mathematical interpretation of the data. So one needs both science & math.

BTW, I had a coworker who preferred religion to science because he actually said that he hated how science was always changing it's mind! He preferred to believe a reliable falsehood than the unreliable truth! Perhaps ML is unreliable because our knowledge of the world that is used to collect data and apply ML is unreliable. It's the bathwater, not the baby that's dirty.
 
User avatar
TidalFlood
Posts: 33
Joined: January 2nd, 2018, 3:13 pm

Re: Machine learning for fun, to pay your rent or whatever

April 1st, 2018, 6:26 pm

I was talking to a Swedish mathematician whose husband is a neuro-scientist and is involved in AI research. She was remarking how unpredictable the reliability of the ML algorithms were.

Just saying.

Is SGD the only show in town? What's the story on TDA?
No, it's not.
k-FAC
Difference Target Propagation
The first one is formally analogous to General Relativity.
Interesting.
 
User avatar
ISayMoo
Posts: 2332
Joined: September 30th, 2015, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

April 2nd, 2018, 5:06 am

So how does the ML researcher ensure they've got good training data (and how many "unreliable" ML projects are the fault of the data not the ML)?

This is a mathematical issue: for which class of data is a given algorithm applicable? But does the 'average' ML researcher have the necessary background? 
Does the average quant finance researcher have the necessary background to recognise the limits of their models? They're mostly failed physicists (like me).
Aliter.. assuming the input is what it is, which algorithms will work with it? Ideally, you should know a-priori. You can't bend the laws of mathematics, e.g. taking the derivative of a discrete function is darn difficult.
If you're talking about ML, you're right. If you're talking about AI research, the end goal is to build a system which can do its own research, like animals do.