Hi guys,I am trying to implement a real-time updating framework for a risk-management system, where changing input data automatically leads to recalculating all dependent results. So I need a kind of subscription mechanism, but a clever one, as I have to handle enormous amounts of data. I like to think about the mechanism as a "calculation tree" or directed graph, with the nodes representing the results, and the edges representing the functions.Something similar must have been implemented in MS Excel, with the cells being the nodes, but Excel will not fulfill my needs as it is not able to handle large amounts of data, and is not flexible enough.While in principle I want to be able to browse through the complete calculation tree (including all results in the complete depth of the tree), I know that this could mean storing several Terabytes of data. So I need to be able to forget or skip nodes if the computer runs out of memory, and then recalculate them as needed. And not to forget: while programming the (short!) functions, I don't want to be bothered with endless technical subscribe stuff (ideally this should be taken care of automatically in the framework).Do you think it's doable, and if so, how would you attack it? Do you know of any component / library which one could use for this type of things? I have thought about publish/subscribe mechanisms and message brokers, but fear they are going to slow down my calculations.Thx in advance for your responses!Calle

these might helpthis is more a messaging library/framework http://akka.io/ this is a graph database that supports triggers http://www.orientechnologies.com/orientdb/

QuoteI am trying to implement a real-time updating framework for a risk-management system, where changing input data automatically leads to recalculating all dependent results. So I need a kind of subscription mechanism, but a clever one, as I have to handle enormous amounts of data. I like to think about the mechanism as a "calculation tree" or directed graph, with the nodes representing the results, and the edges representing the functions.QuoteDo you think it's doable, and if so, how would you attack it? This could be a description of zillions of systems.What is the budget? If this is bigger than a 1-man production project then this step is needed if you want to avoid nasty surprises going forward.These kinds of systems are well-known in general (telecomm, process control etc.) and the models are well documented. QuoteDo you know of any component / library which one could use for this type of things?This application is too complex for a library IMO. I think it is faster to create your own framework. QuoteSomething similar must have been implemented in MS Excel, with the cells being the nodes, but Excel will not fulfill my needs as it is not able to handle large amounts of data, and is not flexible enough.To reduce risk, a proof-of-concept in Excel might be cost-effective. It wil tell you more abut GO/NOGO conclusions that management needs.SQL/Server.This book might be useful and page 271 of this In short, project resource estimation and a feasibility study is needed unless money/time is not a concern, which BTW never holds

Thanks for your hints!@outrun: Your framework sounds pretty much like what I would need - how much effort (in years) went into your project, and how many programmers were working on it?@Cuchulainn: Our project will have around 10 programmers, and last several years, but the framework is only the start of it. And of course we will specify the requirements more in detail first.

Wow! Big project. IMO s/w projects are full risk, especially if you have to estimate it as a fixed-price. If you have a yearly budget it is a bit easier I suppose, at least in the short term.On development process, I personally like Boehm's Spiral Model Good luck with the project! QuoteI like to think about the mechanism as a "calculation tree" or directed graph, with the nodes representing the results, and the edges representing the functions.This is a common pattern. It is like a Command that puhed down the dependency graph from node to node.Is PBGL a way to model distributed graph? This could be used and then build your stuff on top of it?