Page 7 of 28

Re: Machine learning for fun, to pay your rent or whatever

Posted: December 22nd, 2017, 12:36 pm
by outrun
Interesting score function. I wouldn't want to publish code unless I'd win (which has a low probability). I'm not sure if that's possible. T4A and I once ended up not collecting the prize because of the endless extra work they wanted: code, paper, posters, etc etc

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 5:29 pm
by tags
Wilmotters being often creative -  Say you are working in the trading and shipping division of a oil major ("any resemblance to real persons ...") . You meet the CIO there who asks how you would used all that machine learning and data stuff to improve trading (physical, paper) and risk-management. What great ideas would you come up with?

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 5:41 pm
by Traden4Alpha
Off the top of my head, I'd consider saying something like:
1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 6:10 pm
by tw
commercial analysis of AIS data? But I guess this is already big business.
Off the top of my head, I'd consider saying something like:
1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 7:57 pm
by Cuchulainn
1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.
This is as old as the hills IMO. I think it's called Operations Research. And the Process industry is quite mature. 
So, what part of the Venn diagram does ML bring to the table? There is nothing really new here?

You meet the CIO
He say "what's so different this time?"
Take a pilot project to test the feasibility of this technology (managers look at pounds, shillings and pence). You need 40K. 

ended up not collecting the prize because of the endless extra work they wanted: code, paper, posters, etc etc
That's probably because you saw it as a piece of coding and not as a software project with several stakeholders and viewpoints. 
Best to pin down the requirements _beforehand_ and in a contract.

It would have been more painful if it had been a real-life, fixed-price software project with regular cashflows.

//
Most industries are conservative ("don't change a winning horse") so you need compelling reasons to convince hard-nosed decision makers to get on board.

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 9:05 pm
by outrun
ended up not collecting the prize because of the endless extra work they wanted: code, paper, posters, etc etc

That's probably because you saw it as a piece of coding and not as a software project with several stakeholders and viewpoints. 
Best to pin down the requirements _beforehand_ and in a contract.

It would have been more painful if it had been a real-life, fixed-price software project. 
We both do lots of commercial project, for many many years already.
This was a prize competition (not a software project, that's obvious from the context, no?) with a clear quantified objective beforehand, ..that bit was al fine. We only started looked at the value of the prize (travel + conference presentation + a little money from what I remember) when we became eligible. With us being quants and all we quickly saw  that *not* exercising the right to collect the prize was optimal for us. :-)
The global ranking across multiple competitions led us becoming part of a small group of researchers who get invited for private competitions or get hired for good rates. That's was was *our* aim.

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 9:10 pm
by outrun
I think any aspect of trading where (probability) models (you have those a lot in valuations and risk) are based on simple assumptions has potential to be enhanced with statistical methods. It would also have to be an area with lots of data, and preferably high dimensional.

You won a ML competition in this field recently didn't you? I would start out by sharing that with your company, give a presentation what the competition was about and how you managed to beat the other participants. .. and make sure you invite the CIO to that presentation! .. and have a 20 minute brainstorm session at the end about potential internal projects inside your company. 

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 9:10 pm
by ppauper
no doubt that's the same reason you ended up not collecting the Nobel prize, because of the endless extra work they wanted: code, paper, posters, etc etc

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 9:15 pm
by outrun
They ask for code and posters with a Nobel prize? That's horrible!

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 10:03 pm
by Traden4Alpha
1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.
This is as old as the hills IMO. I think it's called Operations Research. And the Process industry is quite mature. 
So, what part of the Venn diagram does ML bring to the table? There is nothing really new here?
Items #1 and #3 are not OR. Instead, they provide more accurate inputs to OR optimization.

In theory, item #2 can be solved with OR. But that approach requires setting up the problem correctly and providing accurate numerical inputs. In practice, the OR solution involves a bunch of nuisance parameters and the OR equations may leave out crucial structure that makes the solution suboptimal or infeasible in real business situations. Sometimes the best model of the system is the system itself as revealed by data from the system. Moreover, ML does not require arbitrary and false mathematical assumptions about linearity, continuity, the existence of analytic solutions, IID, etc. that make the OR solution tractable but imperfect.

As for process control, it's mature but also subject to unmodeled variations -- discrepancies between how the process is supposed to perform versus how it actually performs. In the past, companies used human experts that tried to learn the quirks of each system and adjust the control logic for better performance. But ML systems can do a much better job of learning from data than people.

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 10:13 pm
by outrun
1) demand forecasting (of crude & distillates by geography)
2) optimization of replenishment (balancing inventory costs and shipping costs)
3) shipment duration estimation (for managing/scheduling network flows)

There's also a lot production applications of this to chemical engineering processes -- managing the refinery to maximize yield of valuable products at minimum energy cost and maximum asset utilization.
This is as old as the hills IMO. I think it's called Operations Research. And the Process industry is quite mature. 
So, what part of the Venn diagram does ML bring to the table? There is nothing really new here?
Items #1 and #3 are not OR. Instead, they provide more accurate inputs to OR optimization.  

In theory, item #2 can be solved with OR.  But that approach requires setting up the problem correctly and providing accurate numerical inputs.  In practice, the OR solution involves a bunch of nuisance parameters and the OR equations may leave out crucial structure that makes the solution suboptimal or infeasible in real business situations.  Sometimes the best model of the system is the system itself as revealed by data from the system.  Moreover, ML does not require arbitrary and false mathematical assumptions about linearity, continuity, the existence of analytic solutions, IID, etc. that make the OR solution tractable but imperfect.

As for process control, it's mature but also subject to unmodeled variations -- discrepancies between how the process is supposed to perform versus how it actually performs.  In the past, companies used human experts that tried to learn the quirks of each system and adjust the control logic for better performance.  But ML systems can do a much better job of learning from data than people.
Exactly, that's the core of where ML is successful in replacing classical models. You need data + 'elegant' simplistic models and you're good to go.

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 13th, 2018, 11:31 pm
by tags
Thank you all for your comments.
I was also thinking of implementing an alerting system based on online social networks for refinery/pipeline/tanker events. A bit like DataminR.
And maybe trying fun stuff like using image analysis from traffic webcams to forecast gasoline consumption. I also wonder whether one can do it based on google traffic.

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 15th, 2018, 9:55 am
by Cuchulainn
What is the consensus on this?
  • <QUOTE>DL models are mainly differentiable and hence this limits them because not every problem can be reduced to a differentiable form. Not every interesting AI problem can be reduced to differentiable form. In fact most so called AI-complete problems are very hard to solve in a soft (differentiable) manner.<UNQUOTE>

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 15th, 2018, 1:18 pm
by Traden4Alpha
Empirically speaking, it seems DL as successfully solved many AI-complete problems (translation, vision, navigation, etc.)

The other question is whether the human brain uses differentiable or non-differentiable methods.

Maybe it's our expert-created models that can't be reduced to differentiable form even if the underlying problems are solvable by a system that uses differentiable methods.

Re: Machine learning for fun, to pay your rent or whatever

Posted: January 15th, 2018, 2:28 pm
by Cuchulainn
The other question is whether the human brain uses differentiable or non-differentiable methods.

The human brain sucks and it uses spline interpolation to create illusions.
AI is a long way off simulating the brain.

Maybe it's our expert-created models that can't be reduced to differentiable form even if the underlying problems are solvable by a system that uses differentiable methods.

In that case, why use models that use derivatives (me a white belt)? Where is the real bottleneck. i.e. where do (mathematical) assumptions meet reality?


Empirically speaking, it seems DL as successfully solved many AI-complete problems (translation, vision, navigation, etc.)
Sounds reasonable.