August 17th, 2009, 1:33 pm
The recent book "A Colossal Failure of Common Sense" talked about Lehman's use of VaR and its problems. But it made me think of another dirty secret in financial firms that gets almost no attention.That is the analysis of the models and their implementation. Risk groups and even more so QA [Quality Assurance] groups are assigned to test models and implementation more like the drunk who looks for his lost keys below the street light because there is light there.They and management want testing to cover a certain number of cases in a routine manner ['regression']and see if they differ from day to day. If they find something changed they tell the quant/programmer but have no concern about whether the new number is correct or not---just something changed. If the quant/programmer tells them it is o.k., that ends it. Many firms have Risk and QA act like automatons and not to understand or question the models, theory or anything deeper than 'are numbers the same.' In the rush to develop, programmers are told to produce develop an 'end application' that only allow testing by pressing keys and seeing the results---one at a time---and not provide drivers that allow unit testing of Risk and QA where thousands of cases can be run and, for example, Greeks can be calculated and 'never should happen' cases can be tested.Ivars Peterson wrote a book 'Fatal Defect: Chasing Killer Computer Bugs' that, while not oriented towards the financial industry, points out how failure to be creative in testing, look for things that could go wrong, try to 'break the system', test things that 'should never happen', and failure to have testers [Risk/QA] with the same education/training and compensation on the level of the developers [quants]. As the book points out death can result from his examples [X-Ray machines, airplane] but in finance millions if not billions can be lost---and careers destroyed. I've seen several cases where Risk/QA people wanted to understand what is being produced and test like Peterson points out, they are replaced because management does not want questions raised.