@ISayMoo Thank you for both of the doucments. I am reading these now.
@katastrofa I totally agree on the insanity. That's the reason for the post
Obviously the PCA is only on the inputs, BEFORE using a machine learning algorithm to train those inputs to a target (totally unrelated to the PCA process). But, you have highlighted the issue that if x8 and x9 have the greatest importance then PCA will allow them to be diluted. Many of my inputs are correlated, hence the need to reduce them with PCA first. When you say 'multivariate regression with factors', do you mean linear (ie a n dimensional plane), as opposed to polynomical or GPR etc? I have used BIC and AIC in the past and they are useful, esp when adding a parameter (or input) at a time.