Serving the Quantitative Finance Community

  • 1
  • 5
  • 6
  • 7
  • 8
  • 9
  • 28
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Machine learning for fun, to pay your rent or whatever

December 22nd, 2017, 12:36 pm

Interesting score function. I wouldn't want to publish code unless I'd win (which has a low probability). I'm not sure if that's possible. T4A and I once ended up not collecting the prize because of the endless extra work they wanted: code, paper, posters, etc etc
 
User avatar
tags
Topic Author
Posts: 3162
Joined: February 21st, 2010, 12:58 pm

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 5:29 pm

Wilmotters being often creative -  Say you are working in the trading and shipping division of a oil major ("any resemblance to real persons ...") . You meet the CIO there who asks how you would used all that machine learning and data stuff to improve trading (physical, paper) and risk-management. What great ideas would you come up with?
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 5:41 pm

Off the top of my head, I'd consider saying something like:
1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.
 
User avatar
tw
Posts: 592
Joined: May 10th, 2002, 3:30 pm

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 6:10 pm

commercial analysis of AIS data? But I guess this is already big business.
Off the top of my head, I'd consider saying something like:
1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.
 
User avatar
Cuchulainn
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 7:57 pm

1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.
This is as old as the hills IMO. I think it's called Operations Research. And the Process industry is quite mature. 
So, what part of the Venn diagram does ML bring to the table? There is nothing really new here?

You meet the CIO
He say "what's so different this time?"
Take a pilot project to test the feasibility of this technology (managers look at pounds, shillings and pence). You need 40K. 

ended up not collecting the prize because of the endless extra work they wanted: code, paper, posters, etc etc
That's probably because you saw it as a piece of coding and not as a software project with several stakeholders and viewpoints. 
Best to pin down the requirements _beforehand_ and in a contract.

It would have been more painful if it had been a real-life, fixed-price software project with regular cashflows.

//
Most industries are conservative ("don't change a winning horse") so you need compelling reasons to convince hard-nosed decision makers to get on board.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 9:05 pm

ended up not collecting the prize because of the endless extra work they wanted: code, paper, posters, etc etc

That's probably because you saw it as a piece of coding and not as a software project with several stakeholders and viewpoints. 
Best to pin down the requirements _beforehand_ and in a contract.

It would have been more painful if it had been a real-life, fixed-price software project. 
We both do lots of commercial project, for many many years already.
This was a prize competition (not a software project, that's obvious from the context, no?) with a clear quantified objective beforehand, ..that bit was al fine. We only started looked at the value of the prize (travel + conference presentation + a little money from what I remember) when we became eligible. With us being quants and all we quickly saw  that *not* exercising the right to collect the prize was optimal for us. :-)
The global ranking across multiple competitions led us becoming part of a small group of researchers who get invited for private competitions or get hired for good rates. That's was was *our* aim.
Last edited by outrun on January 13th, 2018, 9:19 pm, edited 1 time in total.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 9:10 pm

I think any aspect of trading where (probability) models (you have those a lot in valuations and risk) are based on simple assumptions has potential to be enhanced with statistical methods. It would also have to be an area with lots of data, and preferably high dimensional.

You won a ML competition in this field recently didn't you? I would start out by sharing that with your company, give a presentation what the competition was about and how you managed to beat the other participants. .. and make sure you invite the CIO to that presentation! .. and have a 20 minute brainstorm session at the end about potential internal projects inside your company. 
Last edited by outrun on January 13th, 2018, 9:11 pm, edited 1 time in total.
 
User avatar
ppauper
Posts: 11729
Joined: November 15th, 2001, 1:29 pm

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 9:10 pm

no doubt that's the same reason you ended up not collecting the Nobel prize, because of the endless extra work they wanted: code, paper, posters, etc etc
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 9:15 pm

They ask for code and posters with a Nobel prize? That's horrible!
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 10:03 pm

1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.
This is as old as the hills IMO. I think it's called Operations Research. And the Process industry is quite mature. 
So, what part of the Venn diagram does ML bring to the table? There is nothing really new here?
Items #1 and #3 are not OR. Instead, they provide more accurate inputs to OR optimization.

In theory, item #2 can be solved with OR. But that approach requires setting up the problem correctly and providing accurate numerical inputs. In practice, the OR solution involves a bunch of nuisance parameters and the OR equations may leave out crucial structure that makes the solution suboptimal or infeasible in real business situations. Sometimes the best model of the system is the system itself as revealed by data from the system. Moreover, ML does not require arbitrary and false mathematical assumptions about linearity, continuity, the existence of analytic solutions, IID, etc. that make the OR solution tractable but imperfect.

As for process control, it's mature but also subject to unmodeled variations -- discrepancies between how the process is supposed to perform versus how it actually performs. In the past, companies used human experts that tried to learn the quirks of each system and adjust the control logic for better performance. But ML systems can do a much better job of learning from data than people.
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 10:13 pm

1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.
This is as old as the hills IMO. I think it's called Operations Research. And the Process industry is quite mature. 
So, what part of the Venn diagram does ML bring to the table? There is nothing really new here?
Items #1 and #3 are not OR. Instead, they provide more accurate inputs to OR optimization.  

In theory, item #2 can be solved with OR.  But that approach requires setting up the problem correctly and providing accurate numerical inputs.  In practice, the OR solution involves a bunch of nuisance parameters and the OR equations may leave out crucial structure that makes the solution suboptimal or infeasible in real business situations.  Sometimes the best model of the system is the system itself as revealed by data from the system.  Moreover, ML does not require arbitrary and false mathematical assumptions about linearity, continuity, the existence of analytic solutions, IID, etc. that make the OR solution tractable but imperfect.

As for process control, it's mature but also subject to unmodeled variations -- discrepancies between how the process is supposed to perform versus how it actually performs.  In the past, companies used human experts that tried to learn the quirks of each system and adjust the control logic for better performance.  But ML systems can do a much better job of learning from data than people.
Exactly, that's the core of where ML is successful in replacing classical models. You need data + 'elegant' simplistic models and you're good to go.
 
User avatar
tags
Topic Author
Posts: 3162
Joined: February 21st, 2010, 12:58 pm

Re: Machine learning for fun, to pay your rent or whatever

January 13th, 2018, 11:31 pm

Thank you all for your comments.
I was also thinking of implementing an alerting system based on online social networks for refinery/pipeline/tanker events. A bit like DataminR.
And maybe trying fun stuff like using image analysis from traffic webcams to forecast gasoline consumption. I also wonder whether one can do it based on google traffic.
 
User avatar
Cuchulainn
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Machine learning for fun, to pay your rent or whatever

January 15th, 2018, 9:55 am

What is the consensus on this?
  • <QUOTE>DL models are mainly differentiable and hence this limits them because not every problem can be reduced to a differentiable form. Not every interesting AI problem can be reduced to differentiable form. In fact most so called AI-complete problems are very hard to solve in a soft (differentiable) manner.<UNQUOTE>
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Re: Machine learning for fun, to pay your rent or whatever

January 15th, 2018, 1:18 pm

Empirically speaking, it seems DL as successfully solved many AI-complete problems (translation, vision, navigation, etc.)

The other question is whether the human brain uses differentiable or non-differentiable methods.

Maybe it's our expert-created models that can't be reduced to differentiable form even if the underlying problems are solvable by a system that uses differentiable methods.
 
User avatar
Cuchulainn
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Machine learning for fun, to pay your rent or whatever

January 15th, 2018, 2:28 pm

The other question is whether the human brain uses differentiable or non-differentiable methods.

The human brain sucks and it uses spline interpolation to create illusions.
AI is a long way off simulating the brain.

Maybe it's our expert-created models that can't be reduced to differentiable form even if the underlying problems are solvable by a system that uses differentiable methods.

In that case, why use models that use derivatives (me a white belt)? Where is the real bottleneck. i.e. where do (mathematical) assumptions meet reality?


Empirically speaking, it seems DL as successfully solved many AI-complete problems (translation, vision, navigation, etc.)
Sounds reasonable.
Last edited by Cuchulainn on January 15th, 2018, 2:34 pm, edited 2 times in total.