Hello all, This topic has been discussed, but not as completely or recently. I'm a twenty year market veteran who has a new role. I want to test proprietary trade ideas and somehow balance simplicity with flexibility. I started writing new code in VBA/Excel with C++ Addins, but I couldn't help but wonder whether there aren't better suited tools out there. Why do I always have to reinvent the wheel? Time is always limited, so an ideal environment would allow me to build something simple quickly, and build larger and more complex systems on that over time.My ultimate dream product would have:1) (Fast) Database functionality 2) Easy access to Market Feeds/Data (initially intraday not needed, but maybe in the future)3) Integrate my own data, or transformations of data feed (equity total return index with dividends reinvested, asset volatility, etc)4) Data structured in some clever usable way5) Perhaps even some financial tools such as everything from datetime to interest rate curve calculations, etc6) Backtesting functionality7) Result Analysis tools8) Ability to automate all of this with reasonable speed. 9) Add functionality for instance have it run analysis, download data daily automatically.10) Ability to export data into excel or elsewhere for closer analysisIt's unlikely I would find all of that. But I set out to look. I've noticed in aviation that oftentimes tiny Cessna's will have more advanced avionics than a large airliner. There are several reasons, but the main one is that avionics for the little "retail" guy don't have to deal with same level of bureaucracy as ones for commercial planes carrying lots of passengers. So I wondered whether there might not be products for retail investors that would surprise an institutional guy like myself.I've spent some time now researching various languages/platforms and have found the following:- Charting packages with trading connection: TradeStation, NinjaTrader, MultiCharts, eSignal - These packages have built in DB with ability to access market data for exchange traded and FOREX instruments. They promise flexibility through scripting languages, and even ability to connect DLL's and have some .NET interfaces. At first this seemed great, but when I looked further, it seemed that their cores structures seemed glued to a long only single asset trading philosophy. You could do a spread trade, but would have to find ways to trick the system. It felt like the benefits of their DB, market feed, and gui would be negated by roadblocks to even simple spread trades. [Am I correct here?]- Reuters/CQG - Market data and execution tools. Seems like they have some simple stuff, or ability to link to something properietary. [Anybody have experience with these?]- Excel - I have pushed Excel 2010 to the limit. A must have/wonderful tool, though it tends to get corrupted when you get too large/complex, and is of course slow.- Matlab - It seems like they have plugins that do everything I've asked for, and I've seen many here using it. - Starting from scratch - SQL/C#/C++, etcFor hardcore backtesting with the benefit on not reinventing the wheel, what have you found? Is Matlab the answer? Does it have limitations? Something I have missed? Or are they all workable; nobody is perfect?Thanks,Markus

QuoteI'm a twenty year market veteran who has a new role. I want to test proprietary trade ideas and somehow balance simplicity with flexibility. I started writing new code in VBA/Excel with C++ Addins, but I couldn't help but wonder whether there aren't better suited tools out there.Have you considered C# and related libs in .NET? It is mich more effective IMO than C++ or Matlab in this context.It also might be an idea to look at Excel DNA library. The forthcoming book by Duffy/Germani might shed some light on this discussion. Here is a blog from a while ago on Comparing C# and C++: which one should I go for in QF?

There is a different approach .. declarative qf programming in Mathematica calling quite fast C++ engines ... with an Excel Link, data base Link technologies, ... You can see it that way: UnRisk extends Mathematica into the universe of financial instrument valuation and risk analytics - but you have all Mathematica math and analytics to aggregate and analyze results ...

^MarkusW:They are all workable. Unless low latency is a concern (e.g. you must hack the kernel and network stack or your alpha diminishes asymptotically), the choice usually boils down to practical reasons such as the language that you are familiar with, cost of equipping all of your boxes with Windows etc.If you are going to clear through a particular broker, I would start by looking at they have to offer, in particular the order execution and data feed APIs that they support, as some brokers require you to use a certain platform or API. In that case, I would definitely write in the language that they have their API bindings in, which immediately limits your choice(s) of language. That should help decide the language of choice for your feed handler, order management etc. As for backtesting, there are two common approaches, both for the reason of reducing polyglotism:- Use the same language as the one above.- Use a language with convenient tools for numerical and statistical analysis, e.g. common choices being R, MATLAB and Python.You asked about MATLAB: the problem with backtesting in MATLAB is that as you go high frequency (tick data), you start running into a wall with limitations of an interpreted language. While it can still run reasonably fast if you write some MEX components, make use of its parallel processing toolbox, you must still at the very least vectorize loops. And there are certain cases in strategy logic where this is a bad idea; you also have to store large arrays for vectorized MATLAB code, which could become a problem.

Thank you for your replies. I have been playing with Addin Express which is a commercial version of Excel DNA. I've also heard from some friends about the great community in R.

QuoteOriginally posted by: MarkusWI have been playing with Addin Express which is a commercial version of Excel DNA.Commercial alternative actually, not the same basis.

QuoteOriginally posted by: HansiQuoteOriginally posted by: MarkusWI have been playing with Addin Express which is a commercial version of Excel DNA.Commercial alternative actually, not the same basis.Is this Addin Express?What would be the compelling reason when native C#, C++/ATL add ins and Excel DNA seem to suit most needs?

QuoteOriginally posted by: CuchulainnQuoteOriginally posted by: HansiQuoteOriginally posted by: MarkusWI have been playing with Addin Express which is a commercial version of Excel DNA.Commercial alternative actually, not the same basis.Is this Addin Express?What would be the compelling reason when native C#, C++/ATL add ins and Excel DNA seem to suit most needs?Easier GUI bindings and WYSIWYG Design interface plus some other stuff.Previously discussed here: http://wilmott.com/messageview.cfm?cati ... adid=87860

Add-In Express is great if you need to make add-ins for various Office apps, especially Outlook, and don't want to use VSTO. It is also good for targeting older Office versions - they essentially have version-independent interop assemblies, a bit like NetOffice, and there are wizards for making add-ins, ribbons and installers.Their .xll-based UDF support is based on an early version of Excel-DNA, though they've always had their own unmanaged .NET loader.Excel-DNA is obviously specialized to Excel, and has more features but has a steeper learning curve to get started (and not much documentation). The fact that Excel-DNA is open source is also important in some environments.Both have awesome support.

QuoteBoth have awesome support. Of course, having awesome documentation as well would increase the population by 265%.

catching up on some wilmott feeds.you may want to check out tradelink.org, which is open source and has ~25k installshere's how it matches against your list1) (Fast) Database functionality processes tick data at up to 1million ticks second https://code.google.com/p/tradelink/wiki/SpeedTests2) Easy access to Market Feeds/Data (initially intraday not needed, but maybe in the future)supports 15 brokers and 3 data feeds out of box http://brokers.tradelink.org3) Integrate my own data, or transformations of data feed (equity total return index with dividends reinvested, asset volatility, etc)it's open source4) Data structured in some clever usable wayvisual model creation to avoid programming for most things : model creation, with single call serialize/deserialize for any type :https://code.google.com/p/tradelink/sou ... del.cseasy model to gui wrapping :https://code.google.com/p/tradelink/sou ... icView.cs5) Perhaps even some financial tools such as everything from datetime to interest rate curve calculations, etcvisual programming includes about 200 signals with paramaterization, plus ability to construct new ones.tradelink is open source so it's popular to integrate with quantlib, R, etc.6) Backtesting functionalityincludes several backtesting applications, which are portfolio realistic and can backtest 200,000-1million ticks second off disk, and 20,000,000 ticks/second from memory. Brute force optimization included.SDK includes a number of different backtesting engines, so you can unbundle the applications into automated versions as you need.7) Result Analysis toolswe include basic stuff like 50 summary stats, individual market result summaries, equity curve. all the indicator, trade data is easily spit to CSV so you can roll into excel or R for custom graphing or summaries.8) Ability to automate all of this with reasonable speed. this pretty much describes the purpose of our project since 2008.9) Add functionality for instance have it run analysis, download data daily automatically.this is popular among advanced users. we could use some more code classes to simplify general tasks of doing this probably, maybe you can help with that.10) Ability to export data into excel or elsewhere for closer analysisyeah as mentioned above this is common