SERVING THE QUANTITATIVE FINANCE COMMUNITY

 
User avatar
JackBryan
Topic Author
Posts: 147
Joined: August 15th, 2010, 6:15 pm

how to do (non-blocking) communication among boost threads

September 11th, 2011, 9:19 pm

Hi, I am trying to do data-exchange communication among boost::threads. Are there asynchronous (non-blocking) communication APIs in boost ? For example, non_block_send( receiver_address, data_buffer, data_volume, data_tag , send_status ); non_block_recv( sender_address, data_buffer, data_volume, data_tag , recv_status ); if (recv_status == TRUE) getData(data_buffer) ; else wait;Any help is really appreciated. Thanks
 
User avatar
renorm
Posts: 886
Joined: February 11th, 2010, 10:20 pm

how to do (non-blocking) communication among boost threads

September 12th, 2011, 1:04 am

You don't need it, because threads run on shared memory.
 
User avatar
JackBryan
Topic Author
Posts: 147
Joined: August 15th, 2010, 6:15 pm

how to do (non-blocking) communication among boost threads

September 12th, 2011, 1:31 am

Hi, My code is about network flow. Each thread runs a task of a node. At each iteration, each thread needs to send and receive data to and from itsneighboring nodes. Without (non-blocking) communication, how to do that ? They share the same data container ? If yes, when one thread has written data to the container (e.g data), how to notify other threads that the data is ready for read ? If we use a mutex to lock the whole data container (data[]), other threads that do notneed to write the data will also have to wait the mutex. it is not efficient. Any help is really appreciated. Thanks
 
User avatar
renorm
Posts: 886
Joined: February 11th, 2010, 10:20 pm

how to do (non-blocking) communication among boost threads

September 12th, 2011, 2:43 am

Sorry, but I don't get what you are trying to accomplish.In HPC content you usually run 1 process per node and 1 thread per core on every node.Threads (by threads I mean threads such as boost::thread launched by the same process) don't need to send data to each other, because all data is shared. Use mutex or atomic to synchronize threads. It is faster and simpler than interprocess communications.Unlike threads, processes don't share memory explicitly. Use MPI to synchronize processes and for data exchange between processes. Processes belonging to the same node (that is running on the same mainboard) can use shared memory, but in this case threads should be preferred.
Last edited by renorm on September 11th, 2011, 10:00 pm, edited 1 time in total.
 
User avatar
JackBryan
Topic Author
Posts: 147
Joined: August 15th, 2010, 6:15 pm

how to do (non-blocking) communication among boost threads

September 12th, 2011, 2:58 am

I have similar idea as you. I use mutex to lock only data to write and then use condition variable to notify the data receiver. But, how to design the notification signal such that only the receiver node receives it and all otherunrelated nodes (threads) are still running normally ? Is it possible to design a buffer, which has internal mutex to do lock and unlock ? Push the same type buffer into a vector<buffer> such that only vector is locked or unlocked by the mutex. But, how to send notification signal that is only related to the signal receiver ? Any help is really appreciated. Thanks
 
User avatar
Cuchulainn
Posts: 61483
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

how to do (non-blocking) communication among boost threads

September 12th, 2011, 5:16 am

If you are using shared memory (and not something else) it looks as if you need thread notification (wait(), notify()). One application is a producer-consumer pattern.Can you describe the model problem you want to solve? It would make it easier. Describe how data is created, by whom , when, when and by whom accessed. These questions will lead to an answer.Threads are asynchronous/independent units by definition but the term is more usual with non shared memory imo. PC pattern is described herehttp://www.quantnet.com/cplusplus-multithreading-boost/ QuoteMy code is about network flow. Each thread runs a task of a node. OK, I see. What about Boost asio? Basically, you can use connnected and reliable TCP or connectionless unreliable UDP between client and server.
Last edited by Cuchulainn on September 11th, 2011, 10:00 pm, edited 1 time in total.
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
 
User avatar
renorm
Posts: 886
Joined: February 11th, 2010, 10:20 pm

how to do (non-blocking) communication among boost threads

September 12th, 2011, 5:59 am

notify_one will unblock only one of the threads waiting on a conditional variable, but you can't choose which one. Btw, it is still unclear what OP is about. Mutex, threads and conditional variables suggest shared memory, but network flow suggests distributed memory.
Last edited by renorm on September 11th, 2011, 10:00 pm, edited 1 time in total.
 
User avatar
Cuchulainn
Posts: 61483
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

how to do (non-blocking) communication among boost threads

September 12th, 2011, 8:12 am

QuoteOriginally posted by: renormnotify_one will unblock only one of the threads waiting on a conditional variable, but you can't choose which one. Btw, it is still unclear what OP is about. Mutex, threads and conditional variables suggest shared memory, but network flow suggests distributed memory.Maybe OP could us an asio io_service object to which you register client callback functions that then run asychronously.
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
 
User avatar
Cuchulainn
Posts: 61483
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

how to do (non-blocking) communication among boost threads

September 12th, 2011, 8:13 am

QuoteOriginally posted by: renormnotify_one will unblock only one of the threads waiting on a conditional variable, but you can't choose which one. Btw, it is still unclear what OP is about. Mutex, threads and conditional variables suggest shared memory, but network flow suggests distributed memory.Maybe OP could us an asio io_service object to which you register client callback functions that then run asynchronously.
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
 
User avatar
renorm
Posts: 886
Joined: February 11th, 2010, 10:20 pm

how to do (non-blocking) communication among boost threads

September 12th, 2011, 8:39 am

OP probably doesn't need boost asio. It is too low level. OP should use MPI (or Boost.MPI), provided he is dealing with distributed memory.
Last edited by renorm on September 11th, 2011, 10:00 pm, edited 1 time in total.
 
User avatar
zeta
Posts: 1969
Joined: September 27th, 2005, 3:25 pm
Location: Houston, TX
Contact:

how to do (non-blocking) communication among boost threads

September 12th, 2011, 12:46 pm

I've used and would recommend reactor based patterns for this problem, eg., there's some solutions discussed here -> http://www.artima.com/articles/io_design_patterns2.html
 
User avatar
JackBryan
Topic Author
Posts: 147
Joined: August 15th, 2010, 6:15 pm

how to do (non-blocking) communication among boost threads

September 12th, 2011, 2:31 pm

Thanks for your replies.My problem is to solve a network flow problem that has been transformed into a math model. Each thread can run tasks of one or more nodes. At each iteration, a thread is a producer and also a consumer. It produces data that is needed by its neighbour nodes (threads) and also it needs data from its neighbor nodes (theads). The data volume is not large and located in memory not in disk. So, I/O is not helpful here. Currently, the code is multi-threading. When the problem size is very large, I will use multiprocessing and multithreading at thesame time. The problem is partitioned into multiple sub-problems, each of them is run by a distinct process on Linux cluster MPI. On each cluster node (multi-core server), each process is run by multiple threads. I know how to do data exchange by MPI. But, I do not know how to do the similar thingsby multithreading on multicore shared memory. When one node produces data, how to use conditional variable to wake up only a group of threads (not all threads) , which are its neighbor nodes (threads) ? Any helpis really appreciated.
 
User avatar
JackBryan
Topic Author
Posts: 147
Joined: August 15th, 2010, 6:15 pm

how to do (non-blocking) communication among boost threads

September 12th, 2011, 2:34 pm

For multi-threading, what I want is :One thread : non_block_send( receiver_address, data_buffer, data_volume, data_tag , send_status ); Another thread is waken up by the non_block_send() and : non_block_recv( sender_address, data_buffer, data_volume, data_tag , recv_status ); if (recv_status == TRUE) getData(data_buffer) ; else wait;Thanks
 
User avatar
Cuchulainn
Posts: 61483
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

how to do (non-blocking) communication among boost threads

September 12th, 2011, 2:58 pm

QuoteOriginally posted by: JackBryanFor multi-threading, what I want is :One thread : non_block_send( receiver_address, data_buffer, data_volume, data_tag , send_status ); Another thread is waken up by the non_block_send() and : non_block_recv( sender_address, data_buffer, data_volume, data_tag , recv_status ); if (recv_status == TRUE) getData(data_buffer) ; else wait;ThanksHave you tried condition variables in boot for thread notification? That's the way it is done in PC pattern.http://www.cs.mtu.edu/~shene/NSF-3/e-Bo ... OR/CV.html
Last edited by Cuchulainn on September 11th, 2011, 10:00 pm, edited 1 time in total.
http://www.datasimfinancial.com
http://www.datasim.nl

Every Time We Teach a Child Something, We Keep Him from Inventing It Himself
Jean Piaget
 
User avatar
farmer
Posts: 13472
Joined: December 16th, 2002, 7:09 am

how to do (non-blocking) communication among boost threads

September 12th, 2011, 3:18 pm

struct mailbox{char mailwaiting;char mailread;char mail[1000];}mailbox * tohim;mailbox * tome;while (1){ if (tome->mailwaiting > 0) { (tome->mailwaiting = 0; if (tome->mail[0] == 255) { tome->mailread = 1; break; } tome->mailread = 1; } if (tohim->mailread > 0 && tohim->mailwaiting == 0) { tohim->mail[0] = 255; tohim->mailread = 0; tohim->mailwaiting = 1; }}
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


Twitter LinkedIn Instagram

JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...


GZIP: On