Quick MPICH2 tip: passing the “-l” option to MPICH2’s mpiexec (hydra) yields rank-labled output like so:
% mpiexec -l -n 2 ./examples/cpi
[0] Process 0 of 2 is on g2-2.local
[1] Process 1 of 2 is on g2-2.local
[0] pi is approximately 3.1415926544231318, Error is 0.0000000008333387
[0] wall clock time = 0.000210
This is handy for figuring out which process is printing what without reaching in and modifying all of your printf statements to explicitly print the rank as well.
For more complicated output, and especially multiline output that you would get from running valgrind, this interleaved output can be hard to follow. Enter sort_labeled_output:
#!/bin/zsh
# assumes output that matches the following regex: /^\[\d+\] .*$/
exec sort --stable --key=1.2,1.0 -n
The script just stably sorts the output by rank prefix, which makes it much easier to read multiline output.
Given that this is just a thin wrapper around GNU sort, I actually didn’t build this script for a long time. But it turns out that this is a task that I do all the time and reconstructing the sort arguments always distracts me a bit from the debugging task at hand. So the script has been a big productivity win.