SERVING THE QUANTITATIVE FINANCE COMMUNITY

 
User avatar
Alan
Posts: 9804
Joined: December 19th, 2001, 4:01 am
Location: California
Contact:

Parallel RNG and distributed MC

July 25th, 2012, 6:57 pm

QuoteOriginally posted by: outrunYes, thing are moving forward!I've picked the top one, the amazon Linux instance, 64 bit. I think this will be easiest to use, even for windows users. E.g. installing all the compiler tools is done with 3 lines of 3 words and it gives you very powerfull tools: more up to date and all for free. Also, the code you can run locally and on Amazon should be exacty the same, so you can do all the preparation locally.I've picked the "micro" instance for now, which is for free 750 hour / month in the first year,.. Great to learn it all for free.Well, "you" can do all the preparations/compiling/debugging locally, but not me, right? (now that you have switched to a Unix environment).The problem is that I was trying to follow along. So, now I can't.Or maybe I am missing something how about a Windows user with Visual Studio Express will use any of what you do fromthis point onward.If the answer is "you can't -- you will need to do all compiles remotely at amazon under Linux", I'm not sure that's workable for people like me. Debugging locally would seem necessary for this kind of thing.So, I think it is worth pausing here to explain how this is all going to work from the perspective of somebody whodoesn't have a local Linux box.
Last edited by Alan on July 24th, 2012, 10:00 pm, edited 1 time in total.
 
User avatar
Alan
Posts: 9804
Joined: December 19th, 2001, 4:01 am
Location: California
Contact:

Parallel RNG and distributed MC

July 25th, 2012, 7:40 pm

Thanks! -- that clarifies it conceptually for me, and sounds like maybe I (and anybody else following along) won't have to become a unix guru to follow your lead.
 
User avatar
Polter
Posts: 2526
Joined: April 29th, 2008, 4:55 pm

Parallel RNG and distributed MC

July 25th, 2012, 10:05 pm

Locally created executable probably won't be optimal and may not even run (dependencies) -- recall that optimizing compilers (C, C++, Fortran) build a machine-specific code (not just x86 vs x86_64, but more CPU-features-used specific, like SSE4, AVX, etc.). You could get around this via cross compilation, but I don't think that's any easier than just compiling two versions (local and remote) separately. From the end-user point of view it shouldn't matter anyway, it's pretty much fire and forget -- either press (CTRL) F5 in VS or type "make"/"run" (or whatever) and press Enter :-)
Last edited by Polter on July 25th, 2012, 10:00 pm, edited 1 time in total.
 
User avatar
Alan
Posts: 9804
Joined: December 19th, 2001, 4:01 am
Location: California
Contact:

Parallel RNG and distributed MC

July 26th, 2012, 3:01 am

I looked this afternoon at amazon's "Getting Started with EC2 with a Linux/Unix Instance"I can probably get to the point where I have got an instance running and am connected to it remotely.Beyond that, it's been sooo .. long since I used any unix system. Ordered a couple CentOS books,but any recommendations on good "Linux/Unix/CentOS guides"?I can see the rationale for the final compile to be on the amazon machineWill definitely need some detailed help on getting the compiler/linker & boost installed remotely.
Last edited by Alan on July 25th, 2012, 10:00 pm, edited 1 time in total.
 
User avatar
MiloRambaldi
Posts: 273
Joined: July 26th, 2010, 12:27 am

Parallel RNG and distributed MC

July 26th, 2012, 4:52 am

QuoteOriginally posted by: outrunQuoteOriginally posted by: CuchulainnQuoteOriginally posted by: AlanOK, I am heading out now, but will take your advice when I get back, unless there is some update to this thread.The current version Boost is 1.50; boostpro has automagic installer up to 1.47.Yes, 1.47 is fine though, saves the trouble of having to build boost.Building boost has never required more than typing two words on the command line for me Let me know if there is anything you need tested for the Amazon project.
 
User avatar
Alan
Posts: 9804
Joined: December 19th, 2001, 4:01 am
Location: California
Contact:

Parallel RNG and distributed MC

July 26th, 2012, 12:31 pm

QuoteOriginally posted by: outrunThat's good to hear.I can at least explain how to install all the required tools. I've run this list of commands. 1 line at a time, always answer with y<enter> when asked questionssudo yum updatesudo yum groupinstall 'Development Tools'sudo yum install cmakesudo yum install boost sudo yum install boost-devel[The first line updates all currently install software. That took 5 minutes or so.The second line install a predefined bundle of development tools. It includes the gcc 4.4.1 C++ compiler and all related thingsThe 'cmake" program is a tool I use to ease the compilation process.The last two lines are "boost", I'm not sure which is which. Maybe the first is the pre-compiles binaries, and the second the include files?..this is as far as I got.Thank you. In googling these commands to try to understand them, I assume it's "sudo yum install boost-devel"
 
User avatar
Polter
Posts: 2526
Joined: April 29th, 2008, 4:55 pm

Parallel RNG and distributed MC

July 26th, 2012, 2:30 pm

Alan, I can recommend this book for a quick guide: http://shell-simulator.googlecode.com/f ... _SE.pdfand, in general, info gathered here might be useful: http://dontfearthecommandline.blogspot.com/in particular:http://dontfearthecommandline.blogspot. ... s.htmlAlso, this similarly-named (coincidence) manual is pretty decent: http://www.flossmanuals.net/command-line/I can look around for some CentOS-specific books, but the general POSIX stuff + package manager specificity (here: yum) that outrun mentioned should work :-)// BTW, strictly speaking, a Linux (like CentOS) is a Unix-like OS, http://en.wikipedia.org/wiki/Unix-like -- in contrast, BSD or Mac OS X are closer (as fully POSIX-compliant)// A handy general term that covers both Linux and UNIX (and a bit more specific than Unix-like) is POSIX-compliant: http://en.wikipedia.org/wiki/POSIX#POSI ... ng_systems
 
User avatar
Alan
Posts: 9804
Joined: December 19th, 2001, 4:01 am
Location: California
Contact:

Parallel RNG and distributed MC

July 26th, 2012, 3:59 pm

Thanks -- those are very helpful.
 
User avatar
Alan
Posts: 9804
Joined: December 19th, 2001, 4:01 am
Location: California
Contact:

Parallel RNG and distributed MC

July 27th, 2012, 1:40 pm

I am totally confused about the basic arithmetic of the hardware setup at amazon EC2 as it relates to this project.As far as I can tell, the physical machine you can invoke there will have at most 8 cores accessible to multi-threading.So, you can invoke as many threads as you like, but probabably won't get more than a factor of eight speedupfrom one of those. Suppose you invoke the Cluster Compute Eight Extra Large (cc2.8xlarge)According to this link, you have generate an instance with 17024 cores!(Or more likely, somebody invoked O(100) instances of this cluster)But the hardware is a machine with 2 quad core zenon processors.So, how much can multi-threading do vs how much will the results need to be aggregated by other means?
Last edited by Alan on July 26th, 2012, 10:00 pm, edited 1 time in total.
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


Twitter LinkedIn Instagram

JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...


GZIP: On