9.1. Quick start: Launching applications

Although this section skips many details, it offers examples that will probably work in many environments.

Caution

Note that this section is a “Quick start” — it does not attempt to be comprehensive or describe how to build Open MPI in all supported environments. The examples below may therefore not work exactly as shown in your environment.

Please consult the other sections in this chapter for more details, if necessary.

9.1.1. Using prterun to launch applications

PRRTE supports both prterun and prun to launch applications. For example:

shell$ prterun -n 2 mpi-hello-world
# or
shell$ prte & prun -n 2 mpi-hello-world
# or
shell$ prterun -n 1 mpi-hello-world : -n 1 mpi-hello-world

are all equivalent. For simplicity, the rest of this documentation will simply refer to prterun.

9.1.1.1. Other prterun options

prterun supports the --help option which provides a usage message and a summary of the options that it supports. It should be considered the definitive list of what options are provided.

Several notable options are:

  • --hostfile: Specify a hostfile for launchers (such as the rsh launcher) that need to be told on which hosts to start parallel applications. Note that for compatibility with other launchers, –machinefile is a synonym for --hostfile.

  • --host: Specify a host or list of hosts to run on, including support for relative index syntax.

  • -n: Indicate the number of processes to start.

  • --prtemca or --pmixmca: Set MCA parameters for either PRRTE or the underlying PMIx library.

  • --wdir DIRECTORY: Set the working directory of the started applications. If not supplied, the current working directory is assumed (or $HOME, if the current working directory does not exist on all nodes).

  • -x ENV_VARIABLE_NAME: The name of an environment variable to export to the parallel application. The -x option can be specified multiple times to export multiple environment variables to the parallel application.

Note that the prterun command supports a large number of options. Detailed help on any option can be obtained using the hierarchical help system - e.g., prterun --help map-by.

9.1.2. Launching on a single host

It is common to develop applications on a single laptop or workstation. In such simple “single program, multiple data (SPMD)” cases, use prterun and specify how many processes you want to launch via the -n option:

shell$ prterun -n 6 mpi-hello-world
Hello world, I am 0 of 6 (running on my-laptop))
Hello world, I am 1 of 6 (running on my-laptop)
...
Hello world, I am 5 of 6 (running on my-laptop)

This starts a six-process parallel application, running six copies of the executable named mpi-hello-world.

If you do not specify the -n option, prterun will default to launching as many processes as there are processor cores (not hyperthreads) on the machine.

9.1.3. Launching in a non-scheduled environments (via ssh)

In general, PRRTE requires the following to launch and run applications:

  1. You must be able to login to remote nodes non-interactively (e.g., without entering a password or passphrase).

  2. PRRTE’s daemon executable must be findable (e.g., in your PATH).

  3. PRRTE’s libraries must be findable (e.g., in your LD_LIBRARY_PATH).

prterun accepts a --hostfile option (and its synonym, the --machinefile option) to specify a hostfile containing one hostname per line:

shell$ cat my-hostfile.txt
node1.example.com
node2.example.com
node3.example.com slots=2
node4.example.com slots=10

The optional slots attribute tells PRRTE the maximum number of processes that can be allocated to that node. If slots is not provided, PRRTE — by default — uses the number of processor cores (not hyperthreads) on that node.

Assuming that each of the 4 nodes in my-hostfile.txt have 16 cores:

shell$ prterun --hostfile my-hostfile.txt mpi-hello-world
Hello world, I am 0 of 44 (running on node1.example.com)
Hello world, I am 1 of 44 (running on node1.example.com)
...
Hello world, I am 15 of 44 (running on node1.example.com)
Hello world, I am 16 of 44 (running on node2.example.com)
Hello world, I am 17 of 44 (running on node2.example.com)
...
Hello world, I am 31 of 44 (running on node2.example.com)
Hello world, I am 32 of 44 (running on node3.example.com)
Hello world, I am 33 of 44 (running on node3.example.com)
Hello world, I am 34 of 44 (running on node4.example.com)
...
Hello world, I am 43 of 44 (running on node4.example.com)

You can see the breakdown of how many processes PRRTE launched on each node:

  • node1: 16, because no slots was specified

  • node2: 16, because no slots was specified

  • node3: 2, because slots=2 was specified

  • node2: 10, because slots=10 was specified

Note, however, that not all environments require a hostfile. For example, PRRTE will automatically detect when it is running in batch / scheduled environments (such as Slurm, PBS/Torque, SGE, LoadLeveler), and will use host information provided by those systems.

Also note that if using a launcher that requires a hostfile and no hostfile is specified, all processes are launched on the local host.

9.1.4. Launching in scheduled environments

In scheduled environments (e.g., in a Slurm job, or PBS/Pro, or LSF, or any other schedule), the user tells the scheduler how many MPI processes to launch, and the scheduler decides which hosts to use. The scheduler then passes both pieces of information (the number of processes and the hosts to use) to PRRTE.

There are two ways to launch in a scheduled environment. Nominally, they both achieve the same thing: they launch processes. The main user-observable difference between the two methods is that prterun has many more features than scheduler direct launchers.

9.1.4.1. Using PRRTE’s prterun

When using the full-featured prterun in a scheduled environment, there is no need to specify a hostfile or number of processes to launch. prterun will receive this information directly from the scheduler. Hence, if you want to launch a job that completely “fills” your scheduled allocation (i.e., one process for each slot in the scheduled allocation), you can simply:

# Write a script that runs your application
shell$ cat my-slurm-script.sh
#!/bin/sh
# There is no need to specify -n or --hostfile because that
# information will automatically be provided by Slurm.
prterun mpi-hello-world

You then submit the my-slurm-script.sh script to Slurm for execution:

# Use -n to indicate how many processes you want to run.
# Slurm will pick the specific hosts which will be used.
shell$ sbatch -n 40 my-slurm-script.sh
Submitted batch job 1234
shell$

After Slurm job 1234 completes, you can look at the output file to see what happened:

shell$ cat slurm-1234.out
Hello world, I am 0 of 40 (running on node37.example.com)
Hello world, I am 1 of 40 (running on node37.example.com)
Hello world, I am 2 of 40 (running on node37.example.com)
...
Hello world, I am 39 of 40 (running on node19.example.com)

Note that the Slurm scheduler picked the hosts on which the processes ran.

The above example shows that simply invoking mpirun mpi-hello-world — with no other CLI options — obtains the number of processes to run and hosts to use from the scheduler.

prterun has many more features not described in this Quick Start section. For example, while uncommon in scheduled environments, you can use -n and/or --hostfile to launch in subsets of the overall scheduler allocation. See the prterun help system for more details.