I have a variance covariance matrix which is of a high dimension...4000x4000. I am using R and trying to do a spectral decomposition..however, due to the size of the matrix R reports memory issues and is unable to do it. I was wondering whether there is a way for me to do this.. perhaps by breaking up the matrix into smaller matrices or something..?Any advice would be appreciated.Regards,JQ

I assume you're using eigen(X), where X is your matrix. Does it help when you set symmetric (also works for Hermitian if the matrix is complex) to true, i.e. eigen(X, symmetric = TRUE)? What is your ultimate goal, by the way? If you need this for the PCA, for example, then perhaps there's a faster way to do it -- while in principle one could use the eigenvalue decomposition of a covariance matrix, it might be better to use fast.prcomp() instead (needs data matrix, though):http://finzi.psych.upenn.edu/R/library/ ... rcomp.html

Thanks for the suggestion. I am indeed using eigen(X)..I haven't tried symmetric=TRUE. The R help thingie says:symmetric: if 'TRUE', the matrix is assumed to be symmetric (or Hermitian if complex) and only its lower triangle is used. If 'symmetric' is not specified, the matrix is inspected for symmetry.I suspect that since it automatically inspects for symmetry...explicitly specifying symmetric=TRUE would not make a difference. I shall give it a go though and report back!JQ

My goal is to use the decomposition to generate a multivariate Gaussian random vector with my X matrix as its covariance. Unless I am mistaken, this is the standard way to do this.BTW, I tried the symmetric=TRUE part...however, my computer crashed..so I guess its not gonna work!

I think it's simple to just use Cholesky decomposition for this. Check here for example in R:http://astrostatistics.psu.edu/su07/R/mv.htmlStill, R has plenty of packages which have this functionality (generating multivariate Gaussians) built-in:http://finzi.psych.upenn.edu/R/library/ ... orm.htmlIs there a reason you're not using any of the above?

QuoteOriginally posted by: PolterI think it's simple to just use Cholesky decomposition for this. Check here for example in R:http://astrostatistics.psu.edu/su07/R/mv.htmlStill, R has plenty of packages which have this functionality (generating multivariate Gaussians) built-in:http://finzi.psych.upenn.edu/R/library/ ... orm.htmlIs there a reason you're not using any of the above?I actually tried using mvrnorm, however, it uses the eigen function as well and therefore fails. I didn't think there would be any more R packages dealing with this. I shall check these out. Perhaps one of these does it more efficiently. Many thanks for the links!

Maybe some kind of sector/industry model would be better; certainly more feasible.What are you planning to do with these 4000 time series if you could generate them?

QuoteOriginally posted by: AlanMaybe some kind of sector/industry model would be better; certainly more feasible.What are you planning to do with these 4000 time series if you could generate them?Alan,My application is purely time series. I have fitted a model (with a certain correlation structure) to a time series of length 4,000+. Now to check whether the simulations produced from the fitted model match the statistical characteristics of the actual data I am in need of simulating a sequence of the same length as my actual time series. Hence the need for a spectral decomposition of the variance covariance matrix.JQ.

If you can't find a way to do the full 4000 instrument model, you might consider clustering the 4000 instruments and culling some of the nearest-neighbor instruments. Although you may lose a little statistical sensitivity, the smaller data set should capture the dominant variations and covariations across time and industry sectors.The other alternative is to buy more RAM. Have you done any scaling studies to see how much RAM you need or how large a set of equations you can handle? Can you handle 500x500, 1000x1000, or 2000x2000? And, if so, how much RAM is consumed?

QuoteOriginally posted by: JediQuantQuoteOriginally posted by: AlanMaybe some kind of sector/industry model would be better; certainly more feasible.What are you planning to do with these 4000 time series if you could generate them?Alan,My application is purely time series. I have fitted a model (with a certain correlation structure) to a time series of length 4,000+. Now to check whether the simulations produced from the fitted model match the statistical characteristics of the actual data I am in need of simulating a sequence of the same length as my actual time series. Hence the need for a spectral decomposition of the variance covariance matrix.JQ.I'm confused. Is there a single time series of length 4000, and the correlations are auto-correlations? Or 4000 separate time series, each of length T? If so, what is T?

The first one (rmvnorm from the mvtnorm package) works rather fine for me (could be faster).Here's the output for my case (I also use genPositiveDefMat from clusterGeneration) -- let me know if that helps: