IMSL (visual numerics), dnanalytics, extremeoptimization, Matlab .NET, BlueBit, NMath (Centerspace), or ILNumerics.Net

Well I know- "no advertise". But I guess, ILNumerics.Net is what you should give a try. In your case it will suffer from the absence of specific statistical functions. But most the time you can implement them easily on your own. If you use C#, the syntax is like Matlab.