crumb trail: > petsc-tools > Error checking and debugging
crumb trail: > petsc-tools > Error checking and debugging > Debug mode
During installation (see section 31.3 ), there is an option of turning on debug mode. An installation with debug turned on:
crumb trail: > petsc-tools > Error checking and debugging > Error codes
PETSc performs a good amount of runtime error checking. Some of this is for internal consistency, but it can also detect certain mathematical errors. To facilitate error reporting, the following scheme is used.
Every PETSc call returns an error code; typically zero for success, and non-zero for various conditions. You should wrap each such function call in the PetscCall macro:
PetscCall( SomePetscRoutine( arguments ) );(In many codes you may see a macro CHKERRQ ; which was the mechanism pre-\petscstandard{3.18}; see section 38.1.2.2 .) This macro detects any error code, reports it, and exits the current routine.
For a good traceback, surround the executable part of any subprogram with PetscFunctionBeginUser and PetscFunctionReturn , where the latter has the return value as parameter. (The routine PetscFunctionBegin does the same, but should only be used for PETSc library routines.)
crumb trail: > petsc-tools > Error checking and debugging > Error codes > Error throwing
You can effect your own error return by using the variadic function SETERRQ . (Before \petscstandard{3.17} there were separate functions \indexpetsc{SETERRQ1}, \indexpetsc{SETERRQ2}, et cetera.)
Example. We write a routine that sets an error:
// backtrace.c
PetscErrorCode this_function_bombs() {
PetscFunctionBegin;
SETERRQ(PETSC_COMM_SELF,1,"We cannot go on like this");
PetscFunctionReturn(0);
}
Running this gives, in process zero, the output
[0]PETSC ERROR: We cannot go on like this [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.2, Nov, 22, 2019 [0]PETSC ERROR: backtrace on a [computer name] [0]PETSC ERROR: Configure options [all options] [0]PETSC ERROR: #1 this_function_bombs() line 20 in backtrace.c [0]PETSC ERROR: #2 main() line 30 in backtrace.c
Fortran note In Fortran the backtrace is not quite as elegant.
!! backtrace.F90
Subroutine this_function_bombs(ierr)
implicit none
integer,intent(out) :: ierr
SETERRQ(PETSC_COMM_SELF,1,"We cannot go on like this")
ierr = -1
end Subroutine this_function_bombs
[0]PETSC ERROR: ----- Error Message ------------------------------ [0]PETSC ERROR: We cannot go on like this [....] [0]PETSC ERROR: #1 User provided function() line 0 in User fileEnd of Fortran note
Remark
In this example, the use of
PETSC_COMM_SELF
indicates
that this error is individually generated on a process;
use
PETSC_COMM_WORLD
only if the same error would
be detected everywhere.
End of remark
Exercise Look up the definition of SETERRQ1 . Write a routine to compute square roots that is used as follows:
x = 1.5; ierr = square_root(x,&rootx); CHKERRQ(ierr); PetscPrintf(PETSC_COMM_WORLD,"Root of %f is %f\n",x,rootx); x = -2.6; ierr = square_root(x,&rootx); CHKERRQ(ierr); PetscPrintf(PETSC_COMM_WORLD,"Root of %f is %f\n",x,rootx);This should give as output:
Root of 1.500000 is 1.224745 [0]PETSC ERROR: ----- Error Message ---------------------------------------------- [0]PETSC ERROR: Cannot compute the root of -2.600000 [...] [0]PETSC ERROR: #1 square_root() line 23 in root.c [0]PETSC ERROR: #2 main() line 39 in root.c
End of exercise
crumb trail: > petsc-tools > Error checking and debugging > Error codes > Legacy error checking
In PETSc versions pre-\petscstandard{3.18}, errors were handled slightly differently.
PetscErrorCode ierr; ierr = AnyPetscRoutine( arguments ); CHKERRQ(ierr);
Fortran note In the main program, use CHKERRA and SETERRA . Also beware that these error `commands' are macros, and after expansion may interfere with Fortran line length , so they should only be used in .F90 files. End of Fortran note
C++ note The macro CHCKERCXX handles exceptions. End of C++ note
crumb trail: > petsc-tools > Error checking and debugging > Memory corruption
PETSc has its own memory management (section 38.5 ) and this facilitates finding memory corruption errors. The macro CHKMEMQ ( CHKMEMA in void functions) checks all memory that was allocated by PETSc, either internally or throug the allocation routines, for corruption. Sprinkling this macro through your code can detect memory problems before they lead to a segfault .
This testing is only done if the commandline argument -malloc_debug ( -malloc_test in debug mode) is supplied, so it carries no overhead for production runs.
crumb trail: > petsc-tools > Error checking and debugging > Memory corruption > Valgrind
Valgrind is rather verbose in its output. To limit the number of processs that run under valgrind:
mpiexec -n 3 valgrind --track-origins=yes ./app -args : -n 5 ./app -args
crumb trail: > petsc-tools > Program output
PETSc has as variety of mechanisms to export or visualize program data. We will consider a few possibilities here.
crumb trail: > petsc-tools > Program output > Screen I/O
Printing screen output in parallel is tricky. If two processes execute a print statement at more or less the same time there is no guarantee as to in what order they may appear on screen. (Even attempts to have them print one after the other may not result in the right ordering.) Furthermore, lines from multi-line print actions on two processes may wind up on the screen interleaved.
crumb trail: > petsc-tools > Program output > Screen I/O > printf replacements
PETSc has two routines that fix this problem. First of all, often the information printed is the same on all processes, so it is enough if only one process, for instance process 0, prints it. This is done with PetscPrintf .
If all processes need to print, you can use PetscSynchronizedPrintf that forces the output to appear in process order.
To make sure that output is properly flushed from all system buffers use PetscSynchronizedFlush where for ordinary screen output you would use stdout for the file.
Fortran note Fortran does not have the variable-number-of-arguments mechanism from C, so you can only use PetscPrintf on a buffer that you construct with a Write statement:
Character*80 :: message
write(message,10) xnorm,ynorm
10 format("Norm x: ",f6.3,", y: ",f6.3,"\n")
call PetscPrintf(comm,message,ierr)
Fortran note The Fortran calls are only wrappers around C routines, so you can use \n newline characters in the Fortran string argument to PetscPrintf .
The file to flush is typically PETSC_STDOUT . End of Fortran note
Python note Since the print routines use the python print call, they automatically include the trailing newline. You don't have to specify it as in the C calls.
crumb trail: > petsc-tools > Program output > Screen I/O > scanf replacement
Using scanf in Petsc is tricky, since integers and real numbers can be of different sizes, depending on the installation. Instead, use PetscViewerRead , which operates in terms of PetscDataType .
crumb trail: > petsc-tools > Program output > Viewers
In order to export PETSc matrix or vector data structures there is a PetscViewer object type. This is a quite general concept of viewing: it encompasses ascii output to screen, binary dump to file, or communication to a running Matlab process. Calls such as MatView or KSPView accept a PetscViewer argument.
In cases where this makes sense, there is also an inverse `load' operation. See section 32.3.5 for vectors.
Some viewers are predefined, such as PETSC_VIEWER_STDOUT_WORLD for ascii rendering to standard out. (In C, specifying zero or NULL also uses this default viewer; for Fortran use \indexpetsctt{PETSC_NULL_VIEWER}.)
crumb trail: > petsc-tools > Program output > Viewers > Viewer types
For activities such as dumping to file you first need create the viewer with PetscViewerCreate and set its type with PetscViewerSetType .
PetscViewerCreate(comm,&viewer); PetscViewerSetType(viewer,PETSCVIEWERBINARY);Popular types include PETSCVIEWERASCII , PETSCVIEWERBINARY , PETSCVIEWERSTRING , PETSCVIEWERDRAW , PETSCVIEWERSOCKET , PETSCVIEWERHDF5 , PETSCVIEWERVTK ; the full list can be found in include/petscviewer.h .
crumb trail: > petsc-tools > Program output > Viewers > Viewer formats
Viewers can take further format specifications by using PetscViewerPushFormat :
PetscViewerPushFormat
(PETSC_VIEWER_STDOUT_WORLD,
PETSC_VIEWER_ASCII_INFO_DETAIL);
and afterwards a corresponding
PetscViewerPopFormat
Python note
## hdf5.py file_name = "hdf5.dat" viewer = PETSc.Viewer().createHDF5(file_name, 'w', comm) x.view(viewer) viewer = PETSc.Viewer().createHDF5(file_name, 'r', comm) x.load(viewer)
crumb trail: > petsc-tools > Program output > Viewers > Commandline option for viewers
Petsc objects viewers can be activated by calls such as MatView , but often it is more convenient to do this through commandline options, such as mat_view , vec_view , or ksp_view . By default, these output to stdout in ascii form, but this can be controlled by further option values:
program -mat_view binary:matrix.datwhere binary forces a binary dump ( ascii is the default) and a file name is explicitly given.
Binary dump may not be supported for all datatypes, in particular DM . For that case, do
program -dm_view draw \
-draw_pause 20
which pops up an
X11
window,
for the duration of the indicated
pause
.
If a viewer needs to be triggered at a specific location, calls such as VecViewFromOptions can be used. VecSetOptionsPrefix . These routines all have a similar calling sequence:
#include "petscsys.h" PetscErrorCode PetscObjectViewFromOptions(PetscObject obj,PetscObject bobj,const char optionname[]) PetscErrorCode VecViewFromOptions(Vec A,PetscObject obj,const char name[])
\begin{raggedlist} AOViewFromOptions , DMViewFromOptions , ISViewFromOptions , ISLocalToGlobalMappingViewFromOptions , KSPConvergedReasonViewFromOptions , KSPViewFromOptions , MatPartitioningViewFromOptions , MatCoarsenViewFromOptions , MatViewFromOptions , PetscObjectViewFromOptions , PetscPartitionerViewFromOptions , PetscDrawViewFromOptions , PetscRandomViewFromOptions , PetscDualSpaceViewFromOptions , PetscSFViewFromOptions , PetscFEViewFromOptions , PetscFVViewFromOptions , PetscSectionViewFromOptions , PCViewFromOptions , PetscSpaceViewFromOptions , PFViewFromOptions , PetscLimiterViewFromOptions , PetscLogViewFromOptions , PetscDSViewFromOptions , PetscViewerViewFromOptions , SNESConvergedReasonViewFromOptions , SNESViewFromOptions , TSTrajectoryViewFromOptions , TSViewFromOptions , TaoLineSearchViewFromOptions , TaoViewFromOptions , VecViewFromOptions , VecScatterViewFromOptions , \end{raggedlist}
crumb trail: > petsc-tools > Program output > Viewers > Naming objects
A helpful facility for viewing is to name an object: that name will then be displayed when the object is viewed.
Vec i_local; ierr = VecCreate(comm,&i_local); CHKERRQ(ierr); ierr = PetscObjectSetName((PetscObject)i_local,"space local"); CHKERRQ(ierr);giving:
Vec Object: space local 4 MPI processes type: mpi Process [0] [ ... et cetera ... ]
crumb trail: > petsc-tools > Commandline options
PETSc has as large number of commandline options, most of which we will discuss later. For now we only mention -log_summary which will print out profile of the time taken in various routines. For these options to be parsed, it is necessary to pass argc,argv to the PetscInitialize call.
crumb trail: > petsc-tools > Commandline options > Adding your own options
You can add custom commandline options to your program. Various routines such as PetscOptionsGetInt scan the commandline for options and set parameters accordingly. For instance,
// ksp.c PetscBool flag; int domain_size = 100; ierr = PetscOptionsGetInt (NULL,PETSC_NULL,"-n",&domain_size,&flag); PetscPrintf(comm,"Using domain size %d\n",domain_size);
Now executing
mpiexec yourprogram -n 5will
For flags, use PetscOptionsHasName .
Python note In Python, do not specify the initial hyphen of an option name. Also, the functions such as getInt do not return the boolean flag; if you need to test for the existence of the commandline option, use:
hasn = PETSc.Options().hasName("n")
There is a related mechanism using PetscOptionsBegin / PetscOptionsEnd :
// optionsbegin.c
PetscOptionsBegin(comm,NULL,"Parameters",NULL);
PetscCall( PetscOptionsInt("-i","i value",__FILE__,i_value,&i_value,&i_flag) );
PetscCall( PetscOptionsInt("-j","j value",__FILE__,j_value,&j_value,&j_flag) );
PetscOptionsEnd();
if (i_flag)
PetscPrintf(comm,"Option `-i' was used\n");
if (j_flag)
PetscPrintf(comm,"Option `-j' was used\n");
The selling point for this approach is that running your code with
mpiexec yourprogram -helpwill display these options as a block. Together with a ton of other options, unfortunately.
crumb trail: > petsc-tools > Commandline options > Options prefix
In many cases, your code will have only one KSP solver object, so specifying ksp_view or ksp_monitor will display / trace that one. However, you may have multiple solvers, or nested solvers. You may then not want to display all of them.
As an example of the nest solver case, consider the case of a block jacobi preconditioner , where the block is itself solved with an iterative method. You can trace that one with -sub_ksp_monitor .
The sub_ is an option prefix , and you can defined your own with KSPSetOptionsPrefix . (There are similar routines for other PETSc object types.)
Example:
KSPCreate(comm,&time_solver); KSPCreate(comm,&space_solver); KSPSetOptionsPrefix(time_solver,"time_"); KSPSetOptionsPrefix(space_solver,"space_");You can then use options -time_ksp_monitor and such. Note that the prefix does not have a leading dash, but it does have the trailing underscore.
\begin{raggedlist} Similar routines: MatSetOptionsPrefix , PCSetOptionsPrefix , PetscObjectSetOptionsPrefix , PetscViewerSetOptionsPrefix , SNESSetOptionsPrefix , TSSetOptionsPrefix , VecSetOptionsPrefix , and some more obscure ones. \end{raggedlist}
crumb trail: > petsc-tools > Commandline options > Where to specify options
Commandline options can obviously go on the commandline. However, there are more places where they can be specified.
Options can be specified programmatically with PetscOptionsSetValue :
PetscOptionsSetValue( NULL, // for global options "-some_option","value_as_string");
Options can be specified in a file .petscrc in the user's home directory or the current directory.
Finally, an environment variable PETSC_OPTIONS can be set.
The rc file is processed first, then the environment variable, then any commandline arguments. This parsing is done in PetscInitialize , so any values from PetscOptionsSetValue override this.
crumb trail: > petsc-tools > Timing and profiling
PETSc has a number of timing routines that make it unnecessary to use system routines such as getrusage or MPI routines such as MPI_Wtime . The main (wall clock) timer is PetscTime . Note the return type of PetscLogDouble which can have a different precision from PetscReal .
The routine PetscGetCPUTime is less useful, since it measures only time spent in computation, and ignores things such as communication.
crumb trail: > petsc-tools > Timing and profiling > Logging
Petsc does a lot of logging on its own operations. Additionally, you can introduce your own routines into this log.
The simplest way to display statistics is to run with an option log_view . This takes an optional file name argument:
mpiexec -n 10 yourprogram -log_view :statistics.txtThe corresponding routine is PetscLogView .
crumb trail: > petsc-tools > Memory management
Allocate the memory for a given pointer: PetscNew , allocate arbitrary memory with PetscMalloc , allocate a number of objects with PetscMalloc1 (this does not zero the memory allocated, use PetscCalloc1 to obtain memory that has been zeroed); use PetscFree to free.
PetscInt *idxs; PetscMalloc1(10,&idxs); // better than: // PetscMalloc(10*sizeof(PetscInt),&idxs); for (PetscInt i=0; i<10; i++) idxs[i] = f(i); PetscFree(idxs);Allocated memory is aligned to PETSC_MEMALIGN .
The state of memory allocation can be written to file or standard out with PetscMallocDump . The commandline option -malloc_dump outputs all not-freed memory during PetscFinalize .
crumb trail: > petsc-tools > Memory management > GPU allocation
The memories of a CPU and GPU are not coherent. This means that routines such as PetscMalloc1 can not immediately be used for GPU allocation. See section 37.4 for details.