SERVING THE QUANTITATIVE FINANCE COMMUNITY

 
User avatar
Cuchulainn
Posts: 60193
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 15th, 2016, 2:00 pm

QuoteOriginally posted by: CuchulainnWhat is a developer's mental model? For example, the customer has written this specWhat's triggered in Grammar-Pattern Programmer's head? How does it pan out?No one?This is a serious question. It's a bit like this but then for computers
Last edited by Cuchulainn on April 14th, 2016, 10:00 pm, edited 1 time in total.
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 15th, 2016, 2:39 pm

QuoteOriginally posted by: CuchulainnHow do committees invent? QuoteA contract research organization had eight people who were to produce a COBOL and an ALGOL compiler. After some initial estimates of difficulty and time, five people were assigned to the COBOL job and three to the ALGOL job. The resulting COBOL compiler ran in five phases, the ALG0L compiler ran in three. QuoteOne could argue that each and every programmer (and user) is using a slightly different version of the interpreter/compiler to reach different conclusions about what the contract means and what the software should do.Is this your finding? Don't think medical systems would accept this..It means the final software system fails and you should reduce the team size until quality improves.Actually, the example that prompted this came from the nuclear power industry in which the specification writer and the programmer interpreted the language around measuring coolant levels in different ways. I forget the details (something to do with reacting to measurements over time) but one could see how each side had a valid but different interpretation.Team size > 1 all but guarantees these of kinds problems in which two or more people think the interpretation of the specification is obvious and everyone says "yes, I understand it" but they fail to realize they don't actually agree on the details.
 
User avatar
fulmerspot
Posts: 511
Joined: July 8th, 2009, 12:44 pm

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 15th, 2016, 2:40 pm

QuoteOriginally posted by: CuchulainnWhat is a developer's mental model? For example, the customer has written this specWhat's triggered in Grammar-Pattern Programmer's head? How does it pan out?3 months rolling contract at Market Rates and an opportunity to improve my database skills, I'd recommend an OLAP cube with reports available to the trader through a pivot report so I could hone those skills too. Just setting up all that would take up the first three months.That's what it triggered in my Eye-On-The-Cash Programmers head.Do I get the Gig?Herr KartoffelKopf
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 15th, 2016, 3:04 pm

QuoteOriginally posted by: CuchulainnQuoteOriginally posted by: CuchulainnWhat is a developer's mental model? For example, the customer has written this specWhat's triggered in Grammar-Pattern Programmer's head? How does it pan out?No one?This is a serious question. It's a bit like this but then for computers My first reaction is that the diagram is incomplete unless everyone is typing blind commands. Where does "old price" come from and who sees it? It's also unclear what the ORMS does. Does it only hold the current price state or does it accumulate a time series on each traded instrument?If the trader submits a long spreadsheet with many price changes, the grammar-pattern programmer would probably sort by date, sort or reorganize by symbol, and then foreach the symbols or rows. If end-of-the-week prices are all that matters, then there's some clever algie vector comparison code that extracts the last price foreach symbol in the spreadsheet. If the trader is entering price deltas, then the algie would use cumulative sums and some tricks to extract to cumulative deltas. If the spec calls for continuous updating, then the algie code and the looper code may be pretty similar.
 
User avatar
Cuchulainn
Posts: 60193
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 16th, 2016, 7:18 am

QuoteOriginally posted by: Traden4AlphaQuoteOriginally posted by: CuchulainnQuoteOriginally posted by: CuchulainnWhat is a developer's mental model? For example, the customer has written this specWhat's triggered in Grammar-Pattern Programmer's head? How does it pan out?No one?This is a serious question. It's a bit like this but then for computers My first reaction is that the diagram is incomplete unless everyone is typing blind commands. Where does "old price" come from and who sees it? It's also unclear what the ORMS does. Does it only hold the current price state or does it accumulate a time series on each traded instrument?If the trader submits a long spreadsheet with many price changes, the grammar-pattern programmer would probably sort by date, sort or reorganize by symbol, and then foreach the symbols or rows. If end-of-the-week prices are all that matters, then there's some clever algie vector comparison code that extracts the last price foreach symbol in the spreadsheet. If the trader is entering price deltas, then the algie would use cumulative sums and some tricks to extract to cumulative deltas. If the spec calls for continuous updating, then the algie code and the looper code may be pretty similar.This diagram is incomplete (of course). But the added value is that all stakeholders are looking at the same page. The answers should help elicit precise requirements.The danger I think is the high-level architecture will be immediately lost using a grammarian approach based on low-level data.
Last edited by Cuchulainn on April 15th, 2016, 10:00 pm, edited 1 time in total.
 
User avatar
Cuchulainn
Posts: 60193
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 16th, 2016, 7:25 am

QuoteOriginally posted by: fulmerspotQuoteOriginally posted by: CuchulainnWhat is a developer's mental model? For example, the customer has written this specWhat's triggered in Grammar-Pattern Programmer's head? How does it pan out?3 months rolling contract at Market Rates and an opportunity to improve my database skills, I'd recommend an OLAP cube with reports available to the trader through a pivot report so I could hone those skills too. Just setting up all that would take up the first three months.That's what it triggered in my Eye-On-The-Cash Programmers head.Do I get the Gig?Herr KartoffelKopfI like this initial feedback. The right mix IMHO. The part I liked most is that MFC was not named. In your proposal will you mention ADO.NET and C#?
Last edited by Cuchulainn on April 15th, 2016, 10:00 pm, edited 1 time in total.
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 16th, 2016, 12:02 pm

QuoteOriginally posted by: CuchulainnQuoteOriginally posted by: Traden4AlphaQuoteOriginally posted by: CuchulainnQuoteOriginally posted by: CuchulainnWhat is a developer's mental model? For example, the customer has written this specWhat's triggered in Grammar-Pattern Programmer's head? How does it pan out?No one?This is a serious question. It's a bit like this but then for computers My first reaction is that the diagram is incomplete unless everyone is typing blind commands. Where does "old price" come from and who sees it? It's also unclear what the ORMS does. Does it only hold the current price state or does it accumulate a time series on each traded instrument?If the trader submits a long spreadsheet with many price changes, the grammar-pattern programmer would probably sort by date, sort or reorganize by symbol, and then foreach the symbols or rows. If end-of-the-week prices are all that matters, then there's some clever algie vector comparison code that extracts the last price foreach symbol in the spreadsheet. If the trader is entering price deltas, then the algie would use cumulative sums and some tricks to extract to cumulative deltas. If the spec calls for continuous updating, then the algie code and the looper code may be pretty similar.This diagram is incomplete (of course). But the added value is that all stakeholders are looking at the same page. The answers should help elicit precise requirements.The danger I think is the high-level architecture will be immediately lost using a grammarian approach based on low-level data.Either view can lead to poor design of the high-level architecture. To me, the grammarian approach is the mathematical approach of thinking about invariant truths in the system that must hold true for all items, all times, all states, all etc. The code then expresses a transform or update that maps a previously valid system state (the set of states on a set of items or times) to the next valid system state. It's a highly abstracted view which certainly has both advantages and disadvantages.To me, the non-grammarian approach can get lost in the weeds of the procedural steps and lose the big picture of what must hold true for all items, all times, all states, all etc. The non-grammarian is too busy writing loops and doing one-thing-at-a-time to see the macrostructures and optimize them. That said, computers at the level of their individual cores are intrinsically procedural machines so the non-grammarian may write code that is more efficient at the lower level even if their architecture is less efficient at the higher level. Of course, if we view the machine at the multicore/CUDA level, then the machine is a grammarian one!In this diagram, it's not clear what must be kept true in this system. For example, it's not clear whether "old price" is ALWAYS the previous price in the market, the previous price seen by the trader, the previous price recorded in the ORMS (i.e., not including the data team's pending updates), or the previous end-of-week price in the price report. Determining what must be always true about "old price" may have a significant effect on the architecture.
 
User avatar
Cuchulainn
Posts: 60193
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 16th, 2016, 3:00 pm

QuoteIn this diagram, it's not clear what must be kept true in this system.Hold yer horses,In fairness, the inkblot was quickly scribbled up on a beer mat, so there's a lot of lacunae. Having said that (and initial feedback from fulmerspot) the system system is probably MIS whose DFD is something like:If we reach agreement on this we can reduce project risk and get use cases for free (MIS is WELL_KNOWN). Then we can design a prototype in 3 months. Of course, more precise interfaces must be discovered.
Last edited by Cuchulainn on April 15th, 2016, 10:00 pm, edited 1 time in total.
 
User avatar
Cuchulainn
Posts: 60193
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 16th, 2016, 3:08 pm

And if we agree on the kind of reference model you can detail design in your favourite lingo and idiom:
 
User avatar
Cuchulainn
Posts: 60193
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 16th, 2016, 3:47 pm

QuoteOriginally posted by: CuchulainnWhat is a developer's mental model? For example, the customer has written this specWhat's triggered in Grammar-Pattern Programmer's head? How does it pan out?We missed out on ORMSXL == Excel?Crucial.
Last edited by Cuchulainn on April 15th, 2016, 10:00 pm, edited 1 time in total.
 
User avatar
Cuchulainn
Posts: 60193
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 17th, 2016, 6:06 pm

Are we in agreement? Have we left something out?
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 17th, 2016, 7:45 pm

QuoteOriginally posted by: CuchulainnQuoteIn this diagram, it's not clear what must be kept true in this system.Hold yer horses,In fairness, the inkblot was quickly scribbled up on a beer mat, so there's a lot of lacunae. Having said that (and initial feedback from fulmerspot) the system system is probably MIS whose DFD is something like:If we reach agreement on this we can reduce project risk and get use cases for free (MIS is WELL_KNOWN). Then we can design a prototype in 3 months. Of course, more precise interfaces must be discovered.All of my comments reflect discussions that would help refine the beer mat diagram rather than being criticisms -- all specs have to start somewhere.From a DFD standpoint, I'm always deeply suspicious of one-way arrows which seem to imply uncontrolled data sources and deadend data sinks. Even that MIS DFD has some. Apparently "Source" can't see any of the transaction data or existing data that their input will affect. The ad hoc report system has no inputs except the merged data (does "source" define an ad hoc report even though they can't see into the system?). At the very least, I'd expect some kind of corrective/editor element that can directly view and change erroneous transactions or existing data.As of the original diagram, the overall one-way loop seems to imply that all the price changes and updates could take place as a once-a-week batch job. That's a very different architecture than if the trader and/or the data team expect to see the most recent "old price" (versus last week's old price) before entering in a "new price". The instant the spec changes from slow batch loops to double-ended arrows, the system potentially starts to have response time issues and resource contention issues.
 
User avatar
Cuchulainn
Posts: 60193
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 18th, 2016, 7:03 am

QuoteAll of my comments reflect discussions that would help refine the beer mat diagram rather than being criticisms -- all specs have to start somewhere.True. But we don't want premature opimisation. Of course, this is an ORMS system which means many of the business processes are known and will be taken care of. Of course, we must read the docs. Avoids discovering requirements in the code (costs $$).QuoteFrom a DFD standpoint, I'm always deeply suspicious of one-way arrows which seem to imply uncontrolled data sources and deadend data sinks. Even that MIS DFD has some. Apparently "Source" can't see any of the transaction data or existing data that their input will affect. The ad hoc report system has no inputs except the merged data (does "source" define an ad hoc report even though they can't see into the system?). At the very least, I'd expect some kind of corrective/editor element that can directly view and change erroneous transactions or existing data.DFD is the data view. We need to define a control view. Data sources/sinks are autonomous external systems.DFDs are only an initial train-spotting mechanism. We need more, later.QuoteAs of the original diagram, the overall one-way loop seems to imply that all the price changes and updates could take place as a once-a-week batch job. That's a very different architecture than if the trader and/or the data team expect to see the most recent "old price" (versus last week's old price) before entering in a "new price". The instant the spec changes from slow batch loops to double-ended arrows, the system potentially starts to have response time issues and resource contention issues. I would not draw many conclusions from the inkblot diagram. it needs to be fleshed out. It was an initial sketch.
Last edited by Cuchulainn on April 17th, 2016, 10:00 pm, edited 1 time in total.
 
User avatar
Traden4Alpha
Posts: 23951
Joined: September 20th, 2002, 8:30 pm

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 18th, 2016, 12:27 pm

QuoteOriginally posted by: CuchulainnQuoteAll of my comments reflect discussions that would help refine the beer mat diagram rather than being criticisms -- all specs have to start somewhere.True. But we don't want premature opimisation. Of course, this is an ORMS system which means many of the business processes are known and will be taken care of. Of course, we must read the docs. Avoids discovering requirements in the code (costs $$).Whether a system can be a batch process or must have continuous updating seems like a fundamental architectural decision, not an optimization. We know a lot of companies that are trying to improve visibility into their operations but are discovering that their batch-process MIS systems simply cannot provide it. It's possible that the current ORMS cannot support the needs of this new system if the ORMS is a batch system and the traders now want a realtime system.QuoteQuoteFrom a DFD standpoint, I'm always deeply suspicious of one-way arrows which seem to imply uncontrolled data sources and deadend data sinks. Even that MIS DFD has some. Apparently "Source" can't see any of the transaction data or existing data that their input will affect. The ad hoc report system has no inputs except the merged data (does "source" define an ad hoc report even though they can't see into the system?). At the very least, I'd expect some kind of corrective/editor element that can directly view and change erroneous transactions or existing data.DFD is the data view. We need to define a control view. Data sources/sinks are autonomous external systems.DFDs are only an initial train-spotting mechanism. We need more, later.Agreed! And by that definition, the beer mat diagram is probably missing most of the data flow volume. I'd wager that traders are greater consumers of data than they are producers of data. They probably review historical trades/positions/prices N>>1 times for each new trade they create making the thickest arrow a missing one from the database to the trader.QuoteQuoteAs of the original diagram, the overall one-way loop seems to imply that all the price changes and updates could take place as a once-a-week batch job. That's a very different architecture than if the trader and/or the data team expect to see the most recent "old price" (versus last week's old price) before entering in a "new price". The instant the spec changes from slow batch loops to double-ended arrows, the system potentially starts to have response time issues and resource contention issues. I would not draw many conclusions from the inkblot diagram. it needs to be fleshed out. It was an initial sketch.Agreed! Yet if one cannot draw any conclusions from the inkblot diagram because it is missing more that 50% of the bare bones of the system, what can one do?Maybe that's the broader point -- any sketch of a system will be missing most of the information needed to define a good architecture.
 
User avatar
Cuchulainn
Posts: 60193
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Grammar-Pattern Programmers: Is Over-Abstraction Delivering Projects Faster Or Slower?

April 18th, 2016, 1:21 pm

QuoteWhether a system can be a batch process or must have continuous updating seems like a fundamental architectural decision, not an optimizationI can't remember any one saying that it was an optimization.I cannot see anywhere that the inkblot diagram is or is not a batch system. It's just that way, maybe. Requirements Elicitation resolves this issue. QuoteAgreed! Yet if one cannot draw any conclusions from the inkblot diagram because it is missing more that 50% of the bare bones of the system, what can one do?Maybe that's the broader point -- any sketch of a system will be missing most of the information needed to define a good architecture. The inkblot is past tense.. enter the system architects.. who speak statecharts, concurrency and so on. Quotebatch-process MIS systems simply cannot provide it. It's possible that the current ORMS cannot support the needs of this new system if the ORMS is a batch system and the traders now want a realtime system.What then?
Last edited by Cuchulainn on April 17th, 2016, 10:00 pm, edited 1 time in total.
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


Twitter LinkedIn Instagram

JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...


GZIP: On