Serving the Quantitative Finance Community

 
User avatar
Church
Topic Author
Posts: 0
Joined: September 4th, 2007, 10:27 am

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

November 17th, 2015, 11:53 am

Hello,Insurance companies often use large correlation matrices (say 100x100) including lots of correlations (between insurance risks for example) that are set using judgment.The resulting matrices are seldom positive semidefinite (PSD). Applying the standard methods of for example Rebonato and Jackel, Higham and Mishra to find the nearest correlation matrix that is PSD give results that are not good enough to be used for publishing stable results based on this.Do you know a better (or the best) method for making such matrices PSD, including the possibility to use weights and to fix the most important correlations? Thanks
 
User avatar
mtsm
Posts: 78
Joined: July 28th, 2010, 1:40 pm

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

November 17th, 2015, 1:15 pm

I can't tell you how to do it, because I don't know, but I am pretty certain that you need to formulate this as a Bayesian problem.You could extend the methods you are using, possibly using some very simple Monte Carlo scheme and bake in your beliefs about certain correlations. See what I am saying?
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

November 17th, 2015, 2:01 pm

A couple of random thoughts:1. A brute force method is to iterate using the standard method with post corrections: build your subjective matrix; find the nearest PSD correlation matrix; adjust any incorrectly changed correlations; find the nearest PSD correlation matrix; rinse & repeat. The amount of adjustment you do could be weighted by the confidence on the individual elements.2. Is there a way to change the construction process to start with the confidently known correlation values and then ask for estimates of less well known values within intervals computed to be safely PSD? Given a series of known correlation values, one can estimate a default chained correlation (Rij = f({Rik,Rkj})) and let the expert adjust it a bit up or down rather than let the expert enter a value that might be wildly inconsistent with PSD.3. One key invariant in PSD correlation matrices is that the total mass of negative correlations is strictly bounded as a function of the number of dimensions. That is, one cannot have that many large magnitude negative correlations in the matrix and still have PSD. Perhaps a quick-and-dirty sum-of-the-negative correlations would provide a warning that the experts are putting in too much negative correlation. The experts could then allocate negativity to avoid gross violations of PSD.
 
User avatar
Church
Topic Author
Posts: 0
Joined: September 4th, 2007, 10:27 am

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

November 19th, 2015, 6:22 pm

Thanks guys.@ Traden4Alpha, I have tried your method under 1) but it does not converge
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

November 19th, 2015, 6:47 pm

QuoteOriginally posted by: ChurchThanks guys.@ Traden4Alpha, I have tried your method under 1) but it does not convergeHow strongly are you adjusting incorrectly changed correlations?You may have a case in which the experts are confident of impossible correlations (aka the experts are wrong).
 
User avatar
londoner
Posts: 1
Joined: January 28th, 2008, 2:52 am

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

November 23rd, 2015, 5:20 pm

Given a 6x6 non-PSD correlation matrix, do you want to construct a PSD correlation matrix with a sub-matrix (say, 4x4) fixed? If so, I don't think any methods in the literature help you achieve this.
 
User avatar
drmwc1
Posts: 7
Joined: November 23rd, 2015, 5:28 pm

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

November 24th, 2015, 3:02 pm

There is an algorithm that lets you choose a weighting, and gets the nearest correlation matrix allowing for the weighting. The weighting matrix needs to be PSD. Here is the paper: http://eprints.ma.man.ac.uk/232/01/cove ... 006_70.pdf
Last edited by drmwc1 on November 23rd, 2015, 11:00 pm, edited 1 time in total.
 
User avatar
Church
Topic Author
Posts: 0
Joined: September 4th, 2007, 10:27 am

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

December 4th, 2015, 3:10 pm

Thanks for your responses.@ drmwc1: Yes, I have tried that, but it introduces an additional PSD problem (of the weights), it doesn't really work well.@ londoner: no, it more like an 70x70 matrix where for example 50 correlations are fixed.Anyone experience with the method of Qi and Sun in 'A quadratically convergent Newton method for computing the nearest correlation matrix'?
 
User avatar
londoner
Posts: 1
Joined: January 28th, 2008, 2:52 am

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

December 4th, 2015, 3:21 pm

Can you arrange the 50 fixed correlations so that the top left corner of the correlation matrix is positive definite? If you can do that, you can apply optimization algorithms such that the top left corner of the matrix is fixed. That should solve your problem.
 
User avatar
drmwc1
Posts: 7
Joined: November 23rd, 2015, 5:28 pm

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

January 6th, 2016, 2:27 pm

You could just re-order tor get the correlations you want fixed to be in the top left corner; then if necessary reorder again back to the original order.NAG algorithm G02an seems to do the trick in this case: http://www.nag.co.uk/IndustryArticles/n ... mk25.pdfIf G is the original matrix, and A is the top-left bit we want fixed, the algorithm minimises alpha s.t. alpha * (A 0 // 0 I) + (1-alpha)*G is psd.
Last edited by drmwc1 on January 5th, 2016, 11:00 pm, edited 1 time in total.
 
User avatar
Orbit
Posts: 36
Joined: October 14th, 2003, 5:34 pm

Best method for finding nearest positive semidefinite correlation matrix for large matrices with judgment embedded

January 6th, 2016, 3:35 pm

Church:Higham's "near" calculation, as suggested by drmwc1 works for me. (Write it in MATLAB, it can execute in about a heartbeat.)IMHO, there is no reason to be trying to fight with a 100x100 or 70x70 correlation matrix. There is something wrong with your problem setup. I think a corr matrix that big is extremely unstable.