Serving the Quantitative Finance Community

 
User avatar
stonexu1984
Topic Author
Posts: 0
Joined: March 18th, 2007, 6:20 am

"Out of Memory" when doing Linprog in Matlab

March 4th, 2010, 4:45 am

Hi guys, I am encountering this problem when running linear programming in Matlab, ===============================================================??? Out of memory. Type HELP MEMORY for your options.Error in ==> lipsol at 689 x(perm) = (P(perm,perm)+U(perm,*U(perm,')\b(perm);Error in ==> linprog at 244 [x,fval,lambda,exitflag,output] = lipsol(f,A,B,Aeq,Beq,lb,ub,options,defaultopt,computeLambda);==============================================================I tried on 2 computers and monitor the memory, 1: the initial "Largest Contiguous Free Blocks" is around 500MB, and seems it's not enoughmy AA matrix is 3200*3800, Aeq is 4400*3800. which are not very large matrix, and i think as the error msg says: it brokes when doing lipsol at 689 x(perm) = (P(perm,perm)+U(perm,*U(perm,')\b(perm);not sure wut is that, and if i elminate AA inequality constraints, it works fine.2. the initial "Largest Contiguous Free Blocks" is around 1GB, after the first sucessful running, the "Largest Contiguous Free Blocks" reduces to 500MB and then it never can make the 2nd run. which i think is odd, since the space should be freed after the running, shouldnt it? Like what i did in the previous test, the memory reduces in the running and then comes back.any suggestion?
 
User avatar
Hansi
Posts: 41
Joined: January 25th, 2010, 11:47 am

"Out of Memory" when doing Linprog in Matlab

March 4th, 2010, 12:05 pm

(3200*3800 + 4400*3800)*8/(1024^2) = 220.336914 MB + overhead is needed for just the AA and Aeq constraints. Matlab often uses up 2x the needed memory for data because of overhead. So adding on everything to the workspace you may not have enough memory to run this. As for the memory freeing up Matlab is a bit crappy with memory management and will reserve large blocks of memory for itself even after clearing that memory of data, restarting will clear it up or you can explicitly instruct Matlab to clear it.My recommendation is going with R instead of Matlab because it has much better memory management. Or if you are deadset on Matlab, increase the RAM and/or minimize the data set in your workspace.
 
User avatar
stonexu1984
Topic Author
Posts: 0
Joined: March 18th, 2007, 6:20 am

"Out of Memory" when doing Linprog in Matlab

March 4th, 2010, 1:39 pm

Thanks how do i instruct Matlab to clear the memory? i think matlab just couldn't find a enough big contiguous free block to store the data. Actually my data contains large amount of 0s, most of the subsets are zeros(x,y). I tried to use sparse function, it helps in reducing the size of the dataset, but it couldn't help when it goes in the linprog process, unless i modify the lipsol function-which causes the problem. Physical Memory (RAM): In Use: 912 MB (3902c000) Free: 1133 MB (46d5f000) Total: 2045 MB (7fd8b000) Page File (Swap space): In Use: 1020 MB (3fc02000) Free: 2917 MB (b65e7000) Total: 3937 MB (f61e9000) Virtual Memory (Address Space): In Use: 617 MB (26949000) Free: 1430 MB (59697000) Total: 2047 MB (7ffe0000) Largest Contiguous Free Blocks: 1. [at 34727000] 595 MB (25339000) 2. [at 202b0000] 274 MB (11270000) 3. [at 629c9000] 104 MB ( 68a7000) 4. [at 5d36e000] 86 MB ( 5652000) 5. [at 6d346000] 42 MB ( 2a2a000) 6. [at 69276000] 38 MB ( 264a000) 7. [at 5b8b5000] 23 MB ( 17db000) 8. [at 32f60000] 22 MB ( 1620000) 9. [at 71ac7000] 21 MB ( 1539000) 10. [at 6bf65000] 19 MB ( 13db000) ======= ========== 1228 MB (4cc25000)QuoteOriginally posted by: Hansi(3200*3800 + 4400*3800)*8/(1024^2) = 220.336914 MB + overhead is needed for just the AA and Aeq constraints. Matlab often uses up 2x the needed memory for data because of overhead. So adding on everything to the workspace you may not have enough memory to run this. As for the memory freeing up Matlab is a bit crappy with memory management and will reserve large blocks of memory for itself even after clearing that memory of data, restarting will clear it up or you can explicitly instruct Matlab to clear it.My recommendation is going with R instead of Matlab because it has much better memory management. Or if you are deadset on Matlab, increase the RAM and/or minimize the data set in your workspace.
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

"Out of Memory" when doing Linprog in Matlab

March 4th, 2010, 1:44 pm

To increase the contiguous free blocks, you might look at http://www.mathworks.com/support/tech-n ... 106.html#3
 
User avatar
stonexu1984
Topic Author
Posts: 0
Joined: March 18th, 2007, 6:20 am

"Out of Memory" when doing Linprog in Matlab

March 4th, 2010, 2:10 pm

and it's odd i am getting different answers by sparsing Aeq Beq, why?
 
User avatar
Hansi
Posts: 41
Joined: January 25th, 2010, 11:47 am

"Out of Memory" when doing Linprog in Matlab

March 4th, 2010, 3:51 pm

QuoteOriginally posted by: stonexu1984and it's odd i am getting different answers by sparsing Aeq Beq, why?Yes, must be that the linprog function does not have accurate implementation of sparse matrix usage. I know that the ipop function in R handles sparse matrices correctly for these kind of problem but I've never used a sparse matrix in optimization in Matlab.