I have used variations of this as an interview question in recent times. If you grab constant maturity US Treasury yields from, say, 1990-2007, calculate the covariance matrix of the absolute (basis point) changes ordered by maturity, and plot the eigenvectors corresponding to the three largest eigenvalues against the bond maturities, you do indeed get something that closely approximate the level, slope and curvature story. However, if you repeat the exercise using data from the last decade (conveniently missing the most turbulent months of 2008 and 2009), the story changes. Now the first PC has a strong slope component over the first 5 years or so, given Fed promises to keep rates "low for long". Since the second PC needs to be orthogonal to the first, it needs to have a richer structure than a simple slope, and so on. These are empirical regularities, rather than laws of nature.
As for higher order PCs, they could just be noise, or they could pick up on more subtle clustering effects. E.g., the 30 year rate could have a bit of a life of its own, driven by insurance companies and pension funds hedging very long dated liabilities. Your specific choice of rates to include in the analysis also matters, especially for the higher order PCs.