MPI topic: Tools interface

Experimental html version of Parallel Programming in MPI, OpenMP, and PETSc by Victor Eijkhout. download the textbook at https:/
\[ \newcommand\inv{^{-1}}\newcommand\invt{^{-t}} \newcommand\bbP{\mathbb{P}} \newcommand\bbR{\mathbb{R}} \newcommand\defined{ \mathrel{\lower 5pt \hbox{${\equiv\atop\mathrm{\scriptstyle D}}$}}} \] 14.1 : Initializing the tools interface
14.2 : Control variables
14.3 : Performance variables
14.3.1 : Performance experiment sessions
14.4 : Categories of variables
14.5 : Events
Back to Table of Contents

14 MPI topic: Tools interface

Recent versions of MPI have a standardized way of reading out performance variables: the tools interface which improves on the old interface described in section  15.6.2  .

14.1 Initializing the tools interface

crumb trail: > mpi-tools > Initializing the tools interface

The tools interface requires a different initialization routine MPI_T_init_thread

int MPI_T_init_thread( int required,int *provided );
Likewise, there is MPI_T_finalize
int MPI_T_finalize();
These matching calls can be made multiple times, after MPI has already been initialized with MPI_Init or MPI_Init_thread  .

Verbosity level is an integer parameter.


14.2 Control variables

crumb trail: > mpi-tools > Control variables \index[mpi]{control variable|(} \index[mpi]{cvar|see{control variable}}

Control variables that can be used to inspect and/or control the internal workings of MPI. Accessing control variables requires initializing the tools interface; section  14.1  .

We query how many control variables are available with MPI_T_cvar_get_num  .

A description of the control variable can be obtained

// cvar.c
printf("#cvars: %d\n",ncvar);
for (int ivar=0; ivar<ncvar; ivar++) {
  char name[100]; int namelen = 100;
  char desc[256]; int desclen = 256;
  int verbosity,bind,scope;
  MPI_Datatype datatype;
  MPI_T_enum enumtype;
  printf("cvar %3d: %s\n  %s\n",ivar,name,desc);

Remark There is no constant indicating a maximum buffer length for these variables. However, you can do the following:

  1. Call the info routine with NULL values for the buffers, reading out the buffer lengths;
  2. allocate the buffers with sufficient length, that is, including an extra position for the null terminator; and
  3. calling the info routine a second time, filling in the string buffers.

End of remark

Conversely, given a variable name, its index can be retrieved with MPI_T_cvar_get_index :

int MPI_T_cvar_get_index(const char *name, int *cvar_index)
If the name can not be matched, the index is MPI_T_ERR_INVALID_NAME  .

Accessing a control variable is done through a control variable handle  .

int MPI_T_cvar_handle_alloc
   (int cvar_index, void *obj_handle,
    MPI_T_cvar_handle *handle, int *count)
The handle is freed with MPI_T_cvar_handle_free :
int MPI_T_cvar_handle_free(MPI_T_cvar_handle *handle)

Control variable access is done through MPI_T_cvar_read and MPI_T_cvar_write :

int MPI_T_cvar_read(MPI_T_cvar_handle handle, void* buf);
int MPI_T_cvar_write(MPI_T_cvar_handle handle, const void* buf);

\index[mpi]{control variable|)}

14.3 Performance variables

crumb trail: > mpi-tools > Performance variables \index[mpi]{performance variable|(} \index[mpi]{pvar|see{performance variable}}

The realization of the tools interface is installation-dependent, you first need to query how much of the tools interface is provided.

// mpitpvar.c
int npvar;

int name_len=256,desc_len=256,
char var_name[256],description[256]; 
MPI_Datatype datatype; MPI_T_enum enumtype;
for (int pvar=0; pvar<npvar; pvar++) {
  name_len = 256; desc_len=256;
  if (procid==0)
    printf("pvar %d: %d/%s = %s\n",pvar,var_class,var_name,description);


Query the number of performance variables with MPI_T_pvar_get_num :

int MPI_T_pvar_get_num(int *num_pvar);
Get information about each variable, by index, with MPI_T_pvar_get_info :
int MPI_T_pvar_get_info
   (int pvar_index, char *name, int *name_len,
    int *verbosity, int *var_class, MPI_Datatype *datatype,
    MPI_T_enum *enumtype, char *desc, int *desc_len, int *bind,
    int *readonly, int *continuous, int *atomic)
See general remarks about these in section  14.2  .

Given a name, the index can be retried with MPI_T_pvar_get_index :

int MPI_T_pvar_get_index(const char *name, int var_class, int *pvar_index)
Again, see section  14.2  .

14.3.1 Performance experiment sessions

crumb trail: > mpi-tools > Performance variables > Performance experiment sessions

To prevent measurements from getting mixed up, they need to be done in performance experiment session s, to be called `sessions' in this chapter. However see section  8.3  .

Create a session with MPI_T_pvar_session_create

int MPI_T_pvar_session_create(MPI_T_pvar_session *session)
and release it with MPI_T_pvar_session_free :
int MPI_T_pvar_session_free(MPI_T_pvar_session *session)
which sets the session variable to MPI_T_PVAR_SESSION_NULL  .

We access a variable through a handle, associated with a certain session. The handle is created with MPI_T_pvar_handle_alloc :

int MPI_T_pvar_handle_alloc
   (MPI_T_pvar_session session, int pvar_index,
    void *obj_handle, MPI_T_pvar_handle *handle, int *count)
(If a routine takes both a session and handle argument, and the two are not associated, an error of MPI_T_ERR_INVALID_HANDLE is returned.)

Free the handle with MPI_T_pvar_handle_free :

int MPI_T_pvar_handle_free
   (MPI_T_pvar_session session,
    MPI_T_pvar_handle *handle)
which sets the variable to MPI_T_PVAR_HANDLE_NULL  .

Continuous variables (see MPI_T_pvar_get_info above, which outputs this) can be started and stopped with MPI_T_pvar_start and MPI_T_pvar_stop :

int MPI_T_pvar_start(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
int MPI_T_pvar_stop(MPI_T_pvar_session session, MPI_T_pvar_handle handle)
Passing MPI_T_PVAR_ALL_HANDLES to the stop call attempts to stop all variables within the session. Failure to stop a variable returns MPI_T_ERR_PVAR_NO_STARTSTOP  .

Variables can be read and written with MPI_T_pvar_read and MPI_T_pvar_write :

int MPI_T_pvar_read
   (MPI_T_pvar_session session, MPI_T_pvar_handle handle,
    void* buf)
int MPI_T_pvar_write
   (MPI_T_pvar_session session, MPI_T_pvar_handle handle,
    const void* buf)
If the variable can not be written (see the readonly parameter of MPI_T_pvar_get_info  ), MPI_T_ERR_PVAR_NO_WRITE is returned.

A special case of writing the variable is to reset it with

int MPI_T_pvar_reset(MPI_T_pvar_session session, MPI_T_pvar_handle handle)  
The handle value of MPI_T_PVAR_ALL_HANDLES is allowed.

A call to MPI_T_pvar_readreset is an atomic combination of the read and reset calls:

int MPI_T_pvar_readreset
   (MPI_T_pvar_session session,MPI_T_pvar_handle handle, 
    void* buf)

\index[mpi]{performance variable|)}

14.4 Categories of variables

crumb trail: > mpi-tools > Categories of variables

Variables, both the control and performance kind, can be grouped into categories by the MPI implementation.

The number of categories is queried with MPI_T_category_get_num :

int MPI_T_category_get_num(int *num_cat)
and for each category the information is retrieved with MPI_T_category_get_info :
int MPI_T_category_get_info
   (int cat_index,
    char *name, int *name_len, char *desc, int *desc_len, 
    int *num_cvars, int *num_pvars, int *num_categories)
For a given category name the index can be found with MPI_T_category_get_index :
int MPI_T_category_get_index(const char *name, int *cat_index)

The contents of a category are retrieved with MPI_T_category_get_cvars  , MPI_T_category_get_pvars  , MPI_T_category_get_categories :

int MPI_T_category_get_cvars(int cat_index, int len, int indices[])
int MPI_T_category_get_pvars(int cat_index, int len, int indices[])
int MPI_T_category_get_categories(int cat_index, int len, int indices[])

\begin{raggedlist} These indices can subsequently be used in the calls MPI_T_cvar_get_info  , MPI_T_pvar_get_info  , MPI_T_category_get_info  . \end{raggedlist}

If categories change dynamically, this can be detected with MPI_T_category_changed

int MPI_T_category_changed(int *stamp)

14.5 Events

crumb trail: > mpi-tools > Events

// mpitevent.c
int nsource;
int name_len=256,desc_len=256;
char var_name[256],description[256]; 
MPI_T_source_order ordering;
MPI_Count ticks_per_second,max_ticks;
MPI_Info info;
MPI_Datatype datatype; MPI_T_enum enumtype;
for (int source=0; source<nsource; source++) {
  name_len = 256; desc_len=256;

Back to Table of Contents