I want to calibrate a mulitvariate linear regression model for forecasting asset return. Now I am doing the following procedure. Subjectively picking up observable factors, like interest rate, fx rate, equity index. Download the data from 2007 to 2016. Select a rolling window, with size 1 year, 2 years or 3 years and the number of lags in regression. Starting from 2010, the linear model is calibrated with data 1 year, 2 years or 3 years back then do the out of sample forecasting of asset return one step ahead. If the forecasted return is positive, then buy, else sell. Then record the actually return. By doing this I pick up size of rolling window and the number of lags, which gives max annual return.With this method, I think there are some problems:(1) Overfitting of parameters. If the model just coincidently successfully predicts certain trend of the asset in some period, then the annual return is high. But this cannot be repeated in the future.(2) In Andrew Ng?s machine learning video, he provides suggestions on how to evaluate the model by looking at the training errors and testing errors. But his method seems not fit in this scenario, because the linear regression model keeps recalibrating in each step. So I down?t know how to determine which features, interest rate, fx rate, equity index can really help to improve forecasting, or just adding noise.