Mean-Variance Optimization

skafetaur · March 18th, 2024, 4:35 pm

Fundamental question on mean-variance optimization, please.

For the historical period 2016 through 2023, I picked 30 day windows and have 300 such windows. At the start of each of those windows, I construct 2 portfolios, both containing the same exact assets (shown down below in the form of a Python dictionary). Then, I look back 252 trading days from the start of each window and compute the mean rolling 30 day window returns and variance. I used that time series as input into mean-variance optimization using cvxpy. The objective function is to maximize (returns - variance). Short sales are allowed in the optimized portfolio (which means weights can be between -1 and +1).

The first of the 2 portfolios is assigned the weights from the optimization, whereas the second portfolio is assigned equal weights. An capital of $10,000 is invested into each portfolio and no further capital is injected ever. The performance of each of the 2 portfolios is observed over the following 30 day period. Specifically, the annualized mean and the annualized standard deviation of the daily returns are calculated for each portfolio. Also, the mark-to-market (MtM) value on the 30th day is computed for each portfolio. I calculate the risk-adjusted returns as mean return divided by standard deviation (not using any risk-free rate here).

Please see below density plots for the MtM on the final day and for the risk adjusted return. From the MtM perspective, the equally weighted portfolio shows much less variation compared to the optimized portfolio. And from a risk adjusted return perspective, there isn't that much of a difference between the 2 portfolios.

What could I be missing here, please? I was expecting that the optimized portfolio would perform much better than the equally weighted portfolio in most cases. Could my optimization be sub-par? Thanks for any hints, pointers and suggestions.

assets_dict = {
'SPY': 'SPDR S&P 500 ETF Trust',
'QQQ': 'Invesco QQQ Trust',
'DIA': 'SPDR Dow Jones Industrial Average ETF Trust',
'IWM': 'Russell 2000 Index',
'UUP': 'Invesco DB U.S. Dollar Index Bullish Fund',
'TLT': 'iShares 20+ Year Treasury Bond ETF',
'DBC': 'Invesco DB Commodity Index Tracking Fund',
'GBTC' : 'Grayscale Bitcoin Trust',
'URTH': 'iShares MSCI World ETF',
'VNQ': 'Vanguard Real Estate ETF'
}

bearish · March 19th, 2024, 9:51 pm

I don’t think you are missing anything, aside from perhaps a bit of intuition. Your optimized portfolio weights are basically random, probably resulting in a poorly diversified portfolio most of the time compared to the equal weighted one. The historical mean return over such a short window will have no real predictive power, although the variance estimate might. I wouldn’t be surprised if you could get a better outcome if you assumed that all your mean returns were equal and just tried to minimize the variance.

skafetaur · March 20th, 2024, 12:29 am

Bearish -- For each 30-day performance window, I am using the covariance matrix from 223 historical monthly returns in optimizing the portfolio. How is that a short window, please? For each window, I look back 252 days and get the rolling 30-day returns, and I get 223 of such rolling 30-day returns. I use those returns in cvxpy.

bearish · March 20th, 2024, 11:57 am

Yes. I hadn’t even got started on your covariance matrix estimation. My main point was that your expected return estimates are going to be extremely noisy. Say your estimated mean is 5% with a standard deviation of 15%. A 95% confidence interval would be -25% to +35%, under the most benign assumption (a stationary Gaussian world). Your covariance matrix is also going to be overfitted, and guess what optimizers do when presented with noisy and overfitted data! My recommendation would be to estimate your mean vector and covariance matrix on your whole population, use those parameters to simulate some histories, and see how well your method performs in a controlled experiment. I’d honestly be interested in seeing the results of that, and i think you’ve done the hard work already.

skafetaur · March 23rd, 2024, 5:58 pm

Thanks bearish.

Either my historical returns data is too small and/or my mean-variance optimization logic is flawed. I share aspects of both below, and would be most grateful if you can provide me with any pointers as to where the blunder is!

(1) On the data I'm using, please see below data. This shows monthly returns of the assets shown and during the period May 2015 through Mar 2020. You'll observe that these returns are over non-overlapping 30-day periods. I did try overlapping 30-day periods earlier, but the optimized portfolio wasn't showing good performance, so I switched to non-overlapping periods.

What I mean by overlapping periods is that my first returns data point would be over day 1 through day 30, my second returns data point would be from day 2 through day 31, my third returns datapoint would be over day 3 through day 32, and so on. On the other hand, what I mean by non-overlapping periods is my first returns would be over day 1 through day 30, the second returns would be day 31 through day 60, and so on. Which one of those is the industry preferred method, would you know?

                             SPY       QQQ       DIA       IWM       UUP  \
Date                                                                          
2015-06-25 00:00:00-04:00  0.004099  0.020861 -0.006802  0.043759  0.012180   
2015-08-07 00:00:00-04:00 -0.009101  0.002818 -0.025999 -0.058361  0.023666   
2015-09-21 00:00:00-04:00 -0.050342 -0.039642 -0.046477 -0.036453 -0.019984   
2015-11-02 00:00:00-05:00  0.070905  0.084501  0.080674  0.022231  0.008796   
2015-12-15 00:00:00-05:00 -0.025477 -0.020068 -0.011746 -0.042721  0.012287   
2016-01-29 00:00:00-05:00 -0.049540 -0.070002 -0.060524 -0.082540  0.012529   
2016-03-14 00:00:00-04:00  0.045323  0.024393  0.052838  0.048174 -0.031323   
2016-04-26 00:00:00-04:00  0.037025  0.019703  0.044931  0.063321 -0.021956   
2016-06-08 00:00:00-04:00  0.016514  0.017704  0.005862  0.035137 -0.011021   
2016-07-21 00:00:00-04:00  0.023660  0.028191  0.030135  0.014662  0.035081   
2016-09-01 00:00:00-04:00  0.005179  0.031455 -0.001241  0.031880 -0.013955   
2016-10-14 00:00:00-04:00 -0.014690  0.005940 -0.013347 -0.020736  0.025475   
2016-11-28 00:00:00-05:00  0.034535  0.011866  0.056366  0.098813  0.031940   
2017-01-11 00:00:00-05:00  0.036097  0.040188  0.047093  0.034477  0.004585   
2017-02-24 00:00:00-05:00  0.042448  0.059627  0.047011  0.017018 -0.006086   
2017-04-07 00:00:00-04:00 -0.002183  0.015235 -0.005517 -0.019760  0.000765   
2017-05-22 00:00:00-04:00  0.018367  0.053269  0.014875  0.010552 -0.042065   
2017-07-05 00:00:00-04:00  0.018512 -0.007880  0.030321  0.033881 -0.005589   
2017-08-16 00:00:00-04:00  0.017177  0.049080  0.028572 -0.025117 -0.026897   
2017-09-28 00:00:00-04:00  0.018840  0.003092  0.018765  0.078201 -0.001650   
2017-11-09 00:00:00-05:00  0.031236  0.064262  0.050562 -0.007779  0.014876   
2017-12-22 00:00:00-05:00  0.041474  0.026086  0.058969  0.048340 -0.010877   
2018-02-07 00:00:00-05:00  0.000598  0.018047  0.006054 -0.021868 -0.027606   
2018-03-22 00:00:00-04:00 -0.010999  0.017815 -0.033561  0.025788 -0.004237   
2018-05-04 00:00:00-04:00  0.008913  0.012715  0.013745  0.015520  0.035319   
2018-06-18 00:00:00-04:00  0.044289  0.072832  0.034178  0.082643  0.026305   
2018-07-31 00:00:00-04:00  0.017248 -0.000283  0.019071 -0.012630  0.000401   
2018-09-12 00:00:00-04:00  0.027690  0.034741  0.026078  0.028637  0.007206   
2018-10-24 00:00:00-04:00 -0.078165 -0.092795 -0.053626 -0.142321  0.019475   
2018-12-07 00:00:00-05:00 -0.006596 -0.023951 -0.003048 -0.013224  0.005848   
2019-01-23 00:00:00-05:00  0.005229  0.007646  0.009206  0.007387 -0.000999   
2019-03-07 00:00:00-05:00  0.044037  0.057231  0.039838  0.048963  0.022353   
2019-04-18 00:00:00-04:00  0.059225  0.095089  0.045022  0.028843  0.002302   
2019-06-03 00:00:00-04:00 -0.053272 -0.092161 -0.061910 -0.059561  0.001913   
2019-07-16 00:00:00-04:00  0.097127  0.137885  0.102903  0.064339  0.006494   
2019-08-27 00:00:00-04:00 -0.043065 -0.045146 -0.053868 -0.066929  0.011006   
2019-10-09 00:00:00-04:00  0.020027  0.017229  0.024280  0.019311  0.014640   
2019-11-20 00:00:00-05:00  0.066948  0.079688  0.058700  0.077346 -0.008879   
2020-01-06 00:00:00-05:00  0.046533  0.068649  0.034710  0.046979 -0.008917   
2020-02-19 00:00:00-05:00  0.045421  0.099369  0.025293  0.018446  0.033807   
2020-04-01 00:00:00-04:00 -0.268199 -0.229055 -0.283964 -0.364178  0.003344

(2) On the optimization aspect, I'm using a plain vanilla logic using cvxpy and presented below is the code. Is this the right approach? I have tried variations -- once allowing for short sales (weights >= -1, weights <= 1) and once disallowing short sales (weights >= 0, weights <=1), but the portfolio performance is terrible in both cases. Also, when I do allow for short sales, the resulting portfolio weights from the optimization are coming to [ 1., 1., 1., -1., -1., 1., -1., 1., -0., -1.]. I find that so weird that cvxpy would tell me to put all my money in some assets and then short equivalent amount of money in some other assets, even while the weights all sum up to 1. Shouldn't cvxpy give me decimal weights? That's why I wonder if my optimization is goofed up.

def asset_allocate(self):
        weights = cp.Variable(self._cov_matrix.shape[0])
        portfolio_returns = cp.sum(self._mean_returns.values @ weights)
        portfolio_variance = cp.quad_form(weights, self._cov_matrix)
        risk_adjusted_returns = portfolio_returns #- portfolio_variance
        objective = cp.Maximize(risk_adjusted_returns)
        constraints = [cp.sum(weights) == 1, weights >= -1, weights <= 1]
        problem = cp.Problem(objective, constraints)
        problem.solve()
        self._optimal_weights = weights.value
        self._equal_weights = [1/self._cov_matrix.shape[0]] * self._cov_matrix.shape[0]
        self._portfolio_attributes = {}
        self._portfolio_attributes['Optimal Weights'] = self._optimal_weights
        self._portfolio_attributes['Equal Weights'] = self._equal_weights

I would greatly appreciate your inputs in to this. I really want to see results from the optimized portfolio. Right now, my equally weighted portfolio is consistently performing better than my mean-variance optimized portfolio over a 30-day horizon.

Mean-Variance Optimization

Mean-Variance Optimization

Re: Mean-Variance Optimization

Re: Mean-Variance Optimization

Re: Mean-Variance Optimization

Re: Mean-Variance Optimization