MPI topic: Tools interface

Experimental html version of Parallel Programming in MPI, OpenMP, and PETSc by Victor Eijkhout. download the textbook at https:/theartofhpc.com/pcse
\[ \newcommand\inv{^{-1}}\newcommand\invt{^{-t}} \newcommand\bbP{\mathbb{P}} \newcommand\bbR{\mathbb{R}} \newcommand\defined{ \mathrel{\lower 5pt \hbox{${\equiv\atop\mathrm{\scriptstyle D}}$}}} \] 14.1 : Initializing the tools interface
14.2 : Control variables
14.3 : Performance variables
14.3.1 : Performance experiment sessions
14.4 : Categories of variables
14.5 : Events
Back to Table of Contents

14 MPI topic: Tools interface

Recent versions of MPI have a standardized way of reading out performance variables: the tools interface

which improves on the old interface described in section  15.6.2  .

14.1 Initializing the tools interface

crumb trail: > mpi-tools > Initializing the tools interface

The tools interface requires a different initialization routine MPI_T_init_thread

int MPI_T_init_thread( int required,int *provided );

Likewise, there is MPI_T_finalize

int MPI_T_finalize();

These matching calls can be made multiple times, after MPI has already been initialized with MPI_Init or MPI_Init_thread  .

Verbosity level is an integer parameter.

MPI_T_VERBOSITY_{USER,TUNER,MPIDEV}_{BASIC,DETAIL,ALL}

14.2 Control variables

crumb trail: > mpi-tools > Control variables

\index[mpi]{control variable|(} \index[mpi]{cvar|see{control variable}}

Control variables that can be used to inspect and/or control the internal workings of MPI. Accessing control variables requires initializing the tools interface; section  14.1  .

We query how many control variables are available with

 .



A description of the control variable can be obtained from
 .



  • An invalid index leads to a function result of MPI_T_ERR_INVALID_INDEX  .
  • Any output parameter can be specified as NULL and MPI will not set this.
  • The bind variable is an object type or MPI_T_BIND_NO_OBJECT  .
  • The enumtype variable is MPI_T_ENUM_NULL if the variable is not an enum type.

// cvar.c
MPI_T_cvar_get_num(&ncvar);
printf("#cvars: %d\n",ncvar);
for (int ivar=0; ivar<ncvar; ivar++) {
  char name[100]; int namelen = 100;
  char desc[256]; int desclen = 256;
  int verbosity,bind,scope;
  MPI_Datatype datatype;
  MPI_T_enum enumtype;
  MPI_T_cvar_get_info
	(ivar,
	 name,&namelen,
	 &verbosity,&datatype,&enumtype,desc,&desclen,&bind,&scope
	 );
  printf("cvar %3d: %s\n  %s\n",ivar,name,desc);

Remark There is no constant indicating a maximum buffer length for these variables. However, you can do the following:

  1. Call the info routine with NULL values for the buffers, reading out the buffer lengths;

  2. allocate the buffers with sufficient length, that is, including an extra position for the null terminator; and

  3. calling the info routine a second time, filling in the string buffers.


End of remark

Conversely, given a variable name, its index can be retrieved with MPI_T_cvar_get_index :

int MPI_T_cvar_get_index(const char *name, int *cvar_index)

If the name can not be matched, the index is MPI_T_ERR_INVALID_NAME  .

Accessing a control variable is done through a control variable handle  .

int MPI_T_cvar_handle_alloc
   (int cvar_index, void *obj_handle,
    MPI_T_cvar_handle *handle, int *count)

The handle is freed with MPI_T_cvar_handle_free :

int MPI_T_cvar_handle_free(MPI_T_cvar_handle *handle)

Control variable access is done through MPI_T_cvar_read and MPI_T_cvar_write :

int MPI_T_cvar_read(MPI_T_cvar_handle handle, void* buf);
int MPI_T_cvar_write(MPI_T_cvar_handle handle, const void* buf);

\index[mpi]{control variable|)}

14.3 Performance variables

crumb trail: > mpi-tools > Performance variables

\index[mpi]{performance variable|(} \index[mpi]{pvar|see{performance variable}}

The realization of the tools interface is installation-dependent, you first need to query how much of the tools interface is provided.

// mpitpvar.c
MPI_Init_thread(&argc,&argv,MPI_THREAD_SINGLE,&tlevel);
MPI_T_init_thread(MPI_THREAD_SINGLE,&tlevel);
int npvar;
MPI_T_pvar_get_num(&npvar);

int name_len=256,desc_len=256,
  verbosity,var_class,binding,isreadonly,iscontiguous,isatomic;
char var_name[256],description[256]; 
MPI_Datatype datatype; MPI_T_enum enumtype;
for (int pvar=0; pvar<npvar; pvar++) {
  name_len = 256; desc_len=256;
  MPI_T_pvar_get_info(pvar,var_name,&name_len,
                      &verbosity,&var_class,
                      &datatype,&enumtype,
                      description,&desc_len,
                      &binding,&isreadonly,&iscontiguous,&isatomic);
  if (procid==0)
    printf("pvar %d: %d/%s = %s\n",pvar,var_class,var_name,description);
}

\begin{raggedlist} Performance variables come in classes:

MPI_T_PVAR_CLASS_STATE

MPI_T_PVAR_CLASS_LEVEL

MPI_T_PVAR_CLASS_SIZE

MPI_T_PVAR_CLASS_PERCENTAGE

MPI_T_PVAR_CLASS_HIGHWATERMARK

MPI_T_PVAR_CLASS_LOWWATERMARK

MPI_T_PVAR_CLASS_COUNTER

MPI_T_PVAR_CLASS_AGGREGATE

MPI_T_PVAR_CLASS_TIMER

MPI_T_PVAR_CLASS_GENERIC \end{raggedlist}

Query the number of performance variables with MPI_T_pvar_get_num :

int MPI_T_pvar_get_num(int *num_pvar);

Get information about each variable, by index, with MPI_T_pvar_get_info :

int MPI_T_pvar_get_info
   (int pvar_index, char *name, int *name_len,
    int *verbosity, int *var_class, MPI_Datatype *datatype,
    MPI_T_enum *enumtype, char *desc, int *desc_len, int *bind,
    int *readonly, int *continuous, int *atomic)

See general remarks about these in section  14.2  .

  • The readonly variable indicates that the variable can not be written.

  • The continuous variable requires use of

    MPI_T_pvar_start and MPI_T_pvar_stop  .

Given a name, the index can be retried with MPI_T_pvar_get_index :

int MPI_T_pvar_get_index(const char *name, int var_class, int *pvar_index)

Again, see section  14.2  .

14.3.1 Performance experiment sessions

crumb trail: > mpi-tools > Performance variables > Performance experiment sessions

To prevent measurements from getting mixed up, they need to be done in performance experiment session s, to be called `sessions' in this chapter. However see section  8.3  .

Create a session with MPI_T_pvar_session_create

int MPI_T_pvar_session_create(MPI_T_pvar_session *session)

and release it with MPI_T_pvar_session_free :

int MPI_T_pvar_session_free(MPI_T_pvar_session *session)

which sets the session variable to MPI_T_PVAR_SESSION_NULL  .

We access a variable through a handle, associated with a certain session. The handle is created with MPI_T_pvar_handle_alloc :

int MPI_T_pvar_handle_alloc
   (MPI_T_pvar_session session, int pvar_index,
    void *obj_handle, MPI_T_pvar_handle *handle, int *count)

(If a routine takes both a session and handle argument, and the two are not associated, an error of MPI_T_ERR_INVALID_HANDLE is returned.)

Free the handle with MPI_T_pvar_handle_free :

int MPI_T_pvar_handle_free
   (MPI_T_pvar_session session,
    MPI_T_pvar_handle *handle)

which sets the variable to MPI_T_PVAR_HANDLE_NULL  .

Continuous variables (see MPI_T_pvar_get_info above, which outputs this) can be started and stopped with MPI_T_pvar_start and MPI_T_pvar_stop :

int MPI_T_pvar_start(MPI_T_pvar_session session, MPI_T_pvar_handle handle);
int MPI_T_pvar_stop(MPI_T_pvar_session session, MPI_T_pvar_handle handle)

Passing MPI_T_PVAR_ALL_HANDLES to the stop call attempts to stop all variables within the session. Failure to stop a variable returns MPI_T_ERR_PVAR_NO_STARTSTOP  .

Variables can be read and written with MPI_T_pvar_read and MPI_T_pvar_write :

int MPI_T_pvar_read
   (MPI_T_pvar_session session, MPI_T_pvar_handle handle,
    void* buf)
int MPI_T_pvar_write
   (MPI_T_pvar_session session, MPI_T_pvar_handle handle,
    const void* buf)

If the variable can not be written (see the readonly parameter of MPI_T_pvar_get_info  ),

MPI_T_ERR_PVAR_NO_WRITE is returned.

A special case of writing the variable is to reset it with

int MPI_T_pvar_reset(MPI_T_pvar_session session, MPI_T_pvar_handle handle)  

The handle value of MPI_T_PVAR_ALL_HANDLES is allowed.

A call to MPI_T_pvar_readreset is an atomic combination of the read and reset calls:

int MPI_T_pvar_readreset
   (MPI_T_pvar_session session,MPI_T_pvar_handle handle, 
    void* buf)

\index[mpi]{performance variable|)}

14.4 Categories of variables

crumb trail: > mpi-tools > Categories of variables

Variables, both the control and performance kind, can be grouped into categories by the MPI implementation.

The number of categories is queried with MPI_T_category_get_num :

int MPI_T_category_get_num(int *num_cat)

and for each category the information is retrieved with MPI_T_category_get_info :

int MPI_T_category_get_info
   (int cat_index,
    char *name, int *name_len, char *desc, int *desc_len, 
    int *num_cvars, int *num_pvars, int *num_categories)

For a given category name the index can be found with MPI_T_category_get_index :

int MPI_T_category_get_index(const char *name, int *cat_index)

The contents of a category are retrieved with MPI_T_category_get_cvars  , MPI_T_category_get_pvars  , MPI_T_category_get_categories :

int MPI_T_category_get_cvars(int cat_index, int len, int indices[])
int MPI_T_category_get_pvars(int cat_index, int len, int indices[])
int MPI_T_category_get_categories(int cat_index, int len, int indices[])

\begin{raggedlist} These indices can subsequently be used in the calls

MPI_T_cvar_get_info  ,

MPI_T_pvar_get_info  ,

MPI_T_category_get_info  . \end{raggedlist}

If categories change dynamically, this can be detected with MPI_T_category_changed

int MPI_T_category_changed(int *stamp)

14.5 Events

crumb trail: > mpi-tools > Events

// mpitevent.c
int nsource;
MPI_T_source_get_num(&nsource);
int name_len=256,desc_len=256;
char var_name[256],description[256]; 
MPI_T_source_order ordering;
MPI_Count ticks_per_second,max_ticks;
MPI_Info info;
MPI_Datatype datatype; MPI_T_enum enumtype;
for (int source=0; source<nsource; source++) {
  name_len = 256; desc_len=256;
  MPI_T_source_get_info(source,var_name,&name_len,
			    description,&desc_len,
			    &ordering,&ticks_per_second,&max_ticks,&info);

Back to Table of Contents