Interfacing deal.II to PETSc and Trilinos

About PETSc, Trilinos, and the deal.II interfaces

deal.II can interface to the PETSc and Trilinos software libraries. Both of these libraries provide lots of functions for linear algebra, among other things, for example implementations of a variety of linear solvers, as well as various different sparse and dense matrix and vector formats. Trilinos also has many subpackages that deal with problems that go far beyond linear algebra, for example nonlinear solvers, automatic differentiation packages, uncertainty propagation engines, etc. Of particular interest to deal.II is their ability to provide this functionality both on sequential and parallel (using MPI) computers. PETSc is written in C; Trilinos is written in C++ and can be considered to be a more modern version of PETSc though both packages are under continuing development at their respective national laboratories.

deal.II has wrapper classes to the linear algebra parts of both packages that provide almost the same interfaces as the built-in deal.II linear algebra classes. We use these interfaces for parallel computations based on MPI since the native deal.II linear algebra classes lack this ability.

Configuring the interfaces to PETSc and Trilinos

The use of PETSc and Trilinos is optional. To use the wrapper classes, you first have to install these packages and point deal.II's ./configure to the installation directories. This happens in similar ways for the two packages:

PETSc

PETSc usually requires you to set the environment variables PETSC_DIR and PETSC_ARCH to a path to PETSc and denoting the architecture for which PETSc is compiled. If these environment variables are set, then deal.II will pick them up during configuration, and store them. It will then also recognize that PETSc shall be used, and enable the wrapper classes.

Alternatively, the --with-petsc=DIR and --with-petsc-arch=ARCH switches to ./configure can be used to override the values of PETSC_DIR and PETSC_ARCH or if these environment variables are not set at all. If you do have a PETSc installation and have set the PETSC_DIR and PETSC_ARCH environment variables but do not wish deal.II to be configured for PETSc use, you should specify --with-petsc=no as a flag during configuration.

Note that in order to use PETSc with deal.II with the current version of deal.II you will need to have at least PETSc version 2.3.0 installed, earlier releases are no longer supported. deal.II version 6.2 has been tested up to PETSc release 3.0.0-p11.

There is an additional caveat: PETSc appears not to co-operate well when using threads and some programs crash when deal.II is compiled in its usual mode supporting multithreading. If you see this sort of behavior, we recommend to try disabling multithreading upon configuration of deal.II using the --disable-threads switch to ./configure.

Installing both PETSc and deal.II together can be a bit of a challenge. A good summary of the relevant steps can be found on the Frequently Asked Questions page.

Trilinos

As above, set the TRILINOS_DIR environment variable to the path to an existing Trilinos installation, or use the --with-trilinos=/path/to/trilinos switch of the deal.II ./configure script. It should point to the path of which the include and lib directories are subdirs. The Trilinos installation needs to provide the same kind of libraries that deal.II is configured for, i.e. if you want deal.II to use shared libraries, then Trilinos needs to use shared libraries as well.

deal.II and its tutorial programs use several of the Trilinos sub-packages.

Trilinos up to and including version 9.0.x

Up to version 9.0, Trilinos was configured and built using autoconf and related tools. Configuring and building Trilinos with a sequence of commands like the following should do the trick:

cd trilinos-9.0.2
mkdir build
cd build
../configure --enable-shared \
      --with-cflags=-fPIC --with-cxxflags=-fPIC --with-fflags=-fPIC \
      --enable-thyra --enable-stratimikos --enable-rtop --enable-teuchos \
      --enable-sacado --enable-ml --enable-ifpack --enable-epetra \
      --enable-belos --enable-aztecoo --enable-amesos \
      --prefix=/home/bangerth/bin/trilinos-9.0.2
make
make install
      
Obviously the path names in the first line and at the end of the configure line need to be adjusted.

Trilinos starting with version 10.0

Note: Trilinos versions 10.6.0 to 10.6.2 are not compatible with deal.II. There are subtle bugs related to parallel matrices and vectors.

Starting with version 10.0, Trilinos uses cmake to configure and build. This is a bit cumbersome because you first have to install a version of cmake that is at least a cmake 2.8 prerelease, and then enter the following slightly longish set of commands:

cd trilinos-10.0.2
mkdir build
cd build

cmake -D Trilinos_ENABLE_OPTIONAL_PACKAGES:BOOL=ON \
      -D Trilinos_ENABLE_Sacado:BOOL=ON \
      -D Trilinos_ENABLE_Stratimikos:BOOL=ON \
      -D CMAKE_BUILD_TYPE:STRING=RELEASE \
      -D CMAKE_CXX_FLAGS:STRING="-g -O3" \
      -D CMAKE_C_FLAGS:STRING="-g -O3" \
      -D CMAKE_FORTRAN_FLAGS:STRING="-g -O5" \
      -D Trilinos_EXTRA_LINK_FLAGS:STRING="-lgfortran" \
      -D CMAKE_VERBOSE_MAKEFILE:BOOL=FALSE \
      -D Trilinos_VERBOSE_CONFIGURE:BOOL=FALSE \
      -D TPL_ENABLE_MPI:BOOL=OFF \
      -D CMAKE_INSTALL_PREFIX:PATH=/w/bangerth/share/x86_64/trilinos-10.4.0 \
      -D BUILD_SHARED_LIBS:BOOL=ON `pwd`/..

make
make install
      
Again, the path into which you want to install Trilinos in the second to last line of the cmake command needs to be adjusted. Obviously, if you want to use Trilinos with MPI on parallel machines, you also need to flip the value of the TPL_ENABLE_MPI flag above. Finally, if your computer has more than one processor core, you may want to add the -jN flag to the calls to make where N is the number of compile jobs you want to run in parallel.

Trilinos sometimes searches for other libraries but can't find them if they are not in the usual directories or have other names. A common example are BLAS or LAPACK. In a case like this, you may have to specifically pass the directories and/or library names under which they can be found to cmake. For example, this may mean to add the following flags to the call above:

      -D BLAS_LIBRARY_NAMES:STRING=goto \
      -D BLAS_LIBRARY_DIRS:STRING=/apps/GotoBLAS/lib64 \
      -D LAPACK_LIBRARY_NAMES:STRING=lapack \
      -D LAPACK_LIBRARY_DIRS:STRING=/apps/lapack-3.2.1/lib64
	  

Configuring for installed Trilinos packages

The above scheme works if Trilinos is installed in its own directory and header and library files are in subdirectories of the directory given to --with-trilinos. This scheme doesn't work, however, if Trilinos has been installed as a regular package on a system, for example into the /usr or /opt directories.

In that case, paths to include and library files may be specified separately using the --with-trilinos-include and --with-trilinos-libs switches to ./configure. Alternatively, this information can also be passed to the configuration script by setting the environment variables TRILINOS_INCDIR, TRILINOS_LIBDIR.


The deal.II mailing list $Date: 2008-10-14 05:36:48 -0500 (Tue, 14 Oct 2008) $