Reading The Stdout From Slave Nodes With Ipcluster

I've set up a cluster using ipcluster start --n=8 then accessed it using from IPython.parallel import Client c=Client() dview=c[:] e=[i for i in c] I'm running processes on the s

Solution 1:

stdout is already captured, logged, and tracked, and arrives at Clients as it comes, before the result is complete.

IPython ships with an example script that monitors stdout/err of all engines, which can easily be tweaked to only monitor a subset of this information, etc.

In the Client itself, you can check the metadata dict for stdout/err (Client.metadata[msg_id].stdout) before results are done. Use Client.spin() to flush any incoming messages off of the zeromq sockets, to ensure this data is up-to-date.

If you want stdout to update frequently, make sure you call sys.stdout.flush() to guarantee that the stream is actually published at that point, rather than relying on implicit flushes, which may not happen until the work completes.