I have been using a flat file database system (that I created using Perl) for quite a long time, but increasingly I am beginning to think that I need to learn something about professional database design. Does anyone know of a good database design book that would focus on time series applications?Thanks,

It depends on what "isolation level" (transaction locking) that you'll be needing. My guess is that, like me, you'll only need good retrieval performance (which makesthe problem much, much easier). Unfortunately, the internals of leading DBs are not the same. If you select Oracle or SQL server, there are many excellent books describing their retrieval algorithms (index schemes - hash, seq, B+, etc.) I'd like to find one myself for "mySQL", which I hear is pretty good (US Census Bureau uses it). If you don't already know SQL, I’d recommend most SQL books for Oracle as a way to learn to learn and use SQL (first order logic). For a more academic treatment, I'd recommend Database and Knowledge -base systems by Jeff Ullman. You can, of course, practice using ACCESS.

Thanks N. I am looking more at database design in the abstract, perhaps a book about data warehousing or something.Regards,

Data warehousing... Then I'd look at books on OLAP and multi-dimensional (Star) database design. Obviously - you've got tons of time-series to store and easily access.MSSQL is a great DB for this, since it comes with all the tools.

If you are looking to deisgn a whole DBMS, look for C. J. Date's books, not only his is text book the standard, its been a standard for years (I believe its currently in its 8th edition). Not too long ago he also started writing about temporal databases...although his doesn't seem to focus on time series as might be required by the financial industry (not 100% sure about this).If you just want to use a database such as mysql, postgresql, oracle, etc., then your book choices will be determined by the database you pick. Postgresql and MySql are free; Oracle/Sybase/DB2/MSSQL/etc. are not free; Kdb (and I believe times ten) are designed specifically for time series' in financial markets.By the way, which kind of functionality are you looking for (what kind of functions do you want it to have, etc.). I've been thinking of writing my own little utility/package to allow standard sql queries on very effecient datastructures (has the potential to be orders of magnitudes faster than mainstream packages).

I second the recommendation on books authored by C.J.Date for understanding the theory behind DB design. Just keep in mind that the logical model presented in the theory may end up as a conceptual layer on top of a rather different physical implementation.The topic of DB implementation may not be as esoteric as some might think. While most relational DB's are implemented as disk-based, the data model is readily adaptable to a memory-resident implementation. Relational DB as a data structure model might be useful, depending upon your requirements. You can buy off-the-shelf implementations of memory-resident DB's, but they cost at least the annual salary of one programmer. If you need only one instance, then just buy it, but if you need one copy for each node in a 100-way cluster then it may be cheaper to build.If you're really going to dig into relational DB implementation, it might help to brush up on some aspects of discrete math if you're not already familiar with it. i.e. MIT OCW Discrete Math coursewareAnother thing to keep in mind is that fully implementing a relational DB may mean implementing a subset of the SQL language. SQL is a declarative language, as contrasted against imperative languages like C++ & VB. In an imperative language, the user of the language needs to specify a series of actions to be carried out in order to achieve the desired result. In contrast, with a declarative language like SQL, the user writes a short query staing the desired end results and any constraints to be applied. They generally do not specify a series of steps to be followed in order to achieve the end results. It is up to the query parsing engine to figure out a suitable sequence of steps in order to achieve the desired result. As such, the DB implementation programmer may have a bigger challenge than if he merely set out to write a C compiler. Fortunately, set theory and other aspects of discrete math are used to frame this problem into a more tractable and digestible form.

Thanks shahbaz and kc11415.I am basically setting out to slowly build my technical skills in database construction. I would like to learn basic SQL and slowly make my flat file tick database more robust. Ultimately, I would like something I can use as a research tool in real-time applications, but I would likely buy another application to do the real-time work (perhaps the M3 Market Data Server if I can afford it. This application used to be free to individual traders a few years ago).Regards,

readily adaptable to a memory-resident implementationso why not use oracle & just pin the tables in the SGA?

Apparently the M3 Market Data Server is no longer available to individuals.Regards,