MFEM v4.7.0
Finite element discretization library

MFEM wrapper for Nvidia's multigrid library, AmgX (github.com/NVIDIA/AMGX) More...
#include <amgxsolver.hpp>
Public Types  
enum  AMGX_MODE { SOLVER , PRECONDITIONER } 
Flags to configure AmgXSolver as a solver or preconditioner. More...  
enum  CONFIG_SRC { INTERNAL , EXTERNAL , UNDEFINED } 
Flags to determine whether user solver settings are defined internally in the source code or will be read through an external JSON file. More...  
Public Types inherited from mfem::Operator  
enum  DiagonalPolicy { DIAG_ZERO , DIAG_ONE , DIAG_KEEP } 
Defines operator diagonal policy upon elimination of rows and/or columns. More...  
enum  Type { ANY_TYPE , MFEM_SPARSEMAT , Hypre_ParCSR , PETSC_MATAIJ , PETSC_MATIS , PETSC_MATSHELL , PETSC_MATNEST , PETSC_MATHYPRE , PETSC_MATGENERIC , Complex_Operator , MFEM_ComplexSparseMat , Complex_Hypre_ParCSR , Complex_DenseMat , MFEM_Block_Matrix , MFEM_Block_Operator } 
Enumeration defining IDs for some classes derived from Operator. More...  
Public Member Functions  
AmgXSolver ()  
AmgXSolver (const AMGX_MODE amgxMode_, const bool verbose)  
Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Assumes no MPI parallelism.  
void  InitSerial () 
Initialize the AmgX library for serial execution once the solver configuration has been established through either the AmgXSolver::ReadParameters method or the constructor. The constructor will make this call.  
AmgXSolver (const MPI_Comm &comm, const AMGX_MODE amgxMode_, const bool verbose)  
Configures AmgX with a default configuration based on the AMGX_MODE (AmgXSolver::SOLVER, AmgXSolver::PRECONDITIONER) and verbosity. Pairs each MPI rank with one GPU.  
AmgXSolver (const MPI_Comm &comm, const int nDevs, const AMGX_MODE amgx_Mode_, const bool verbose)  
Configures AmgX with a default configuration based on the AMGX_MODE (AmgXSolver::SOLVER, AmgXSolver::PRECONDITIONER) and verbosity. Creates MPI teams around GPUs to support more ranks than GPUs. Consolidates linear solver data to avoid multiple ranks sharing GPUs. Requires specifying the number of devices in each compute node as nDevs.  
void  InitExclusiveGPU (const MPI_Comm &comm) 
Initialize the AmgX library in parallel mode with exactly one GPU per rank after the solver configuration has been established, either through the constructor or the AmgXSolver::ReadParameters method. If configuring with a constructor, the constructor will make this call.  
void  InitMPITeams (const MPI_Comm &comm, const int nDevs) 
Initialize the AmgX library and create MPI teams based on the number of devices on each node nDevs. If configuring with a constructor, the constructor will make this call, otherwise this will need to be called after the solver configuration has been established through the AmgXSolver::ReadParameters call.  
virtual void  SetOperator (const Operator &op) 
Sets the Operator that is going to be solved via AmgX. Supports operators based on either an MFEM SparseMatrix or HypreParMatrix.  
void  UpdateOperator (const Operator &op) 
Change the input operator that is being solved via AmgX. Supports operators based on either an MFEM SparseMatrix or HypreParMatrix.  
virtual void  Mult (const Vector &b, Vector &x) const 
Utilize the AmgX library to solve the linear system where the "matrix" is the AMG approximation to the operator set by AmgXSolver::SetOperator. If the mode is set to AmgXSolver::PRECONDITIONER the initial guess for the x vector will be set to zero, otherwise the value of x passed in will be used.  
int  GetNumIterations () 
Return the number of iterations that were executed during the last solve phase.  
void  ReadParameters (const std::string config, CONFIG_SRC source) 
Read in the AmgX parameters either through a file or directly through a properly formated string. If source is set to AmgXSolver::EXTERNAL the parameters are loaded from a filename set by config. If source is set to AmgXSolver::INTERNAL the parameters are set directly by the string defined by config.  
void  DefaultParameters (const AMGX_MODE amgxMode_, const bool verbose) 
Set up the AmgX library with the default paramaters.  
void  SetConvergenceCheck (bool setConvergenceCheck_=true) 
Add a check for convergence after applying Mult.  
~AmgXSolver ()  
Close down the AmgX library and free up any MPI Comms set up for it.  
void  Finalize () 
Close down the AmgX library and free up any MPI Comms set up for it.  
Public Member Functions inherited from mfem::Solver  
Solver (int s=0, bool iter_mode=false)  
Initialize a square Solver with size s.  
Solver (int h, int w, bool iter_mode=false)  
Initialize a Solver with height h and width w.  
Public Member Functions inherited from mfem::Operator  
void  InitTVectors (const Operator *Po, const Operator *Ri, const Operator *Pi, Vector &x, Vector &b, Vector &X, Vector &B) const 
Initializes memory for true vectors of linear system.  
Operator (int s=0)  
Construct a square Operator with given size s (default 0).  
Operator (int h, int w)  
Construct an Operator with the given height (output size) and width (input size).  
int  Height () const 
Get the height (size of output) of the Operator. Synonym with NumRows().  
int  NumRows () const 
Get the number of rows (size of output) of the Operator. Synonym with Height().  
int  Width () const 
Get the width (size of input) of the Operator. Synonym with NumCols().  
int  NumCols () const 
Get the number of columns (size of input) of the Operator. Synonym with Width().  
virtual MemoryClass  GetMemoryClass () const 
Return the MemoryClass preferred by the Operator.  
virtual void  MultTranspose (const Vector &x, Vector &y) const 
Action of the transpose operator: y=A^t(x) . The default behavior in class Operator is to generate an error.  
virtual void  AddMult (const Vector &x, Vector &y, const real_t a=1.0) const 
Operator application: y+=A(x) (default) or y+=a*A(x) .  
virtual void  AddMultTranspose (const Vector &x, Vector &y, const real_t a=1.0) const 
Operator transpose application: y+=A^t(x) (default) or y+=a*A^t(x) .  
virtual void  ArrayMult (const Array< const Vector * > &X, Array< Vector * > &Y) const 
Operator application on a matrix: Y=A(X) .  
virtual void  ArrayMultTranspose (const Array< const Vector * > &X, Array< Vector * > &Y) const 
Action of the transpose operator on a matrix: Y=A^t(X) .  
virtual void  ArrayAddMult (const Array< const Vector * > &X, Array< Vector * > &Y, const real_t a=1.0) const 
Operator application on a matrix: Y+=A(X) (default) or Y+=a*A(X) .  
virtual void  ArrayAddMultTranspose (const Array< const Vector * > &X, Array< Vector * > &Y, const real_t a=1.0) const 
Operator transpose application on a matrix: Y+=A^t(X) (default) or Y+=a*A^t(X) .  
virtual Operator &  GetGradient (const Vector &x) const 
Evaluate the gradient operator at the point x. The default behavior in class Operator is to generate an error.  
virtual void  AssembleDiagonal (Vector &diag) const 
Computes the diagonal entries into diag. Typically, this operation only makes sense for linear Operators. In some cases, only an approximation of the diagonal is computed.  
virtual const Operator *  GetProlongation () const 
Prolongation operator from linear algebra (linear system) vectors, to input vectors for the operator. NULL means identity.  
virtual const Operator *  GetRestriction () const 
Restriction operator from input vectors for the operator to linear algebra (linear system) vectors. NULL means identity.  
virtual const Operator *  GetOutputProlongation () const 
Prolongation operator from linear algebra (linear system) vectors, to output vectors for the operator. NULL means identity.  
virtual const Operator *  GetOutputRestrictionTranspose () const 
Transpose of GetOutputRestriction, directly available in this form to facilitate matrixfree RAPtype operators.  
virtual const Operator *  GetOutputRestriction () const 
Restriction operator from output vectors for the operator to linear algebra (linear system) vectors. NULL means identity.  
void  FormLinearSystem (const Array< int > &ess_tdof_list, Vector &x, Vector &b, Operator *&A, Vector &X, Vector &B, int copy_interior=0) 
Form a constrained linear system using a matrixfree approach.  
void  FormRectangularLinearSystem (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, Vector &x, Vector &b, Operator *&A, Vector &X, Vector &B) 
Form a columnconstrained linear system using a matrixfree approach.  
virtual void  RecoverFEMSolution (const Vector &X, const Vector &b, Vector &x) 
Reconstruct a solution vector x (e.g. a GridFunction) from the solution X of a constrained linear system obtained from Operator::FormLinearSystem() or Operator::FormRectangularLinearSystem().  
void  FormSystemOperator (const Array< int > &ess_tdof_list, Operator *&A) 
Return in A a parallel (on truedofs) version of this square operator.  
void  FormRectangularSystemOperator (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, Operator *&A) 
Return in A a parallel (on truedofs) version of this rectangular operator (including constraints).  
void  FormDiscreteOperator (Operator *&A) 
Return in A a parallel (on truedofs) version of this rectangular operator.  
void  PrintMatlab (std::ostream &out, int n, int m=0) const 
Prints operator with input size n and output size m in Matlab format.  
virtual void  PrintMatlab (std::ostream &out) const 
Prints operator in Matlab format.  
virtual  ~Operator () 
Virtual destructor.  
Type  GetType () const 
Return the type ID of the Operator class.  
Public Attributes  
bool  ConvergenceCheck 
Flag to check for convergence.  
Public Attributes inherited from mfem::Solver  
bool  iterative_mode 
If true, use the second argument of Mult() as an initial guess.  
Additional Inherited Members  
Protected Member Functions inherited from mfem::Operator  
void  FormConstrainedSystemOperator (const Array< int > &ess_tdof_list, ConstrainedOperator *&Aout) 
see FormSystemOperator()  
void  FormRectangularConstrainedSystemOperator (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, RectangularConstrainedOperator *&Aout) 
see FormRectangularSystemOperator()  
Operator *  SetupRAP (const Operator *Pi, const Operator *Po) 
Returns RAP Operator of this, using input/output Prolongation matrices Pi corresponds to "P", Po corresponds to "Rt".  
Protected Attributes inherited from mfem::Operator  
int  height 
Dimension of the output / number of rows in the matrix.  
int  width 
Dimension of the input / number of columns in the matrix.  
MFEM wrapper for Nvidia's multigrid library, AmgX (github.com/NVIDIA/AMGX)
AmgX requires building MFEM with CUDA, and AMGX enabled. For distributed memory parallism, MPI and Hypre (version 16.0+) are also required. Although CUDA is required for building, the AmgX solver is compatible with a MFEM CPU device configuration.
The AmgXSolver class is designed to work as a solver or preconditioner for existing MFEM solvers. The AmgX solver class may be configured in one of three ways:
Serial  Takes a SparseMatrix solves on a single GPU and assumes no MPI communication.
Exclusive GPU  Takes a HypreParMatrix and assumes each MPI rank is paired with an Nvidia GPU.
MPI Teams  Takes a HypreParMatrix and enables flexibility between number of MPI ranks, and GPUs. Specifically, MPI ranks are grouped with GPUs and a matrix consolidation step is taken so the MPI root of each team performs the necessary AmgX library calls. The solution is then broadcasted to appropriate ranks. This is particularly useful when configuring MFEM's device as CPU. This work is based on the AmgXWrapper of Chuang and Barba. Routines were adopted and modified for setting up MPI communicators.
Examples 1/1p in the examples/amgx directory demonstrate configuring the wrapper as a solver and preconditioner, as well as configuring and running with exclusive GPU or MPI teams modes.
This work is partially based on:
PiYueh Chuang and Lorena A. Barba (2017). AmgXWrapper: An interface between PETSc and the NVIDIA AmgX library. J. Open Source Software, 2(16):280, doi:10.21105/joss.00280
See https://github.com/barbagroup/AmgXWrapper.
Definition at line 69 of file amgxsolver.hpp.
Flags to configure AmgXSolver as a solver or preconditioner.
Definition at line 74 of file amgxsolver.hpp.
Flags to determine whether user solver settings are defined internally in the source code or will be read through an external JSON file.
Enumerator  

INTERNAL  Configuration will be read directly from a string. 
EXTERNAL  Configure will be read from a specified file. 
UNDEFINED 
Definition at line 93 of file amgxsolver.hpp.
mfem::AmgXSolver::AmgXSolver  (  ) 
Definition at line 37 of file amgxsolver.cpp.
mfem::AmgXSolver::AmgXSolver  (  const AMGX_MODE  amgxMode_, 
const bool  verbose ) 
Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Assumes no MPI parallelism.
Definition at line 40 of file amgxsolver.cpp.
mfem::AmgXSolver::AmgXSolver  (  const MPI_Comm &  comm, 
const AMGX_MODE  amgxMode_,  
const bool  verbose ) 
Configures AmgX with a default configuration based on the AMGX_MODE (AmgXSolver::SOLVER, AmgXSolver::PRECONDITIONER) and verbosity. Pairs each MPI rank with one GPU.
Definition at line 54 of file amgxsolver.cpp.
mfem::AmgXSolver::AmgXSolver  (  const MPI_Comm &  comm, 
const int  nDevs,  
const AMGX_MODE  amgx_Mode_,  
const bool  verbose ) 
Configures AmgX with a default configuration based on the AMGX_MODE (AmgXSolver::SOLVER, AmgXSolver::PRECONDITIONER) and verbosity. Creates MPI teams around GPUs to support more ranks than GPUs. Consolidates linear solver data to avoid multiple ranks sharing GPUs. Requires specifying the number of devices in each compute node as nDevs.
Definition at line 68 of file amgxsolver.cpp.
mfem::AmgXSolver::~AmgXSolver  (  ) 
Close down the AmgX library and free up any MPI Comms set up for it.
Definition at line 84 of file amgxsolver.cpp.
void mfem::AmgXSolver::DefaultParameters  (  const AMGX_MODE  amgxMode_, 
const bool  verbose ) 
Set up the AmgX library with the default paramaters.
[in]  amgxMode_  AmgXSolver::PRECONDITIONER, AmgXSolver::SOLVER. 
[in]  verbose  true, false. Specifies the level of verbosity. 
When configured as a preconditioner, the default configuration is to apply two iterations of an AMG V cycle with AmgX's default smoother (block Jacobi).
When configured as a solver the preconditioned conjugate gradient method is used with the AMG Vcycle and a block Jacobi smoother is used as a preconditioner.
Definition at line 202 of file amgxsolver.cpp.
void mfem::AmgXSolver::Finalize  (  ) 
Close down the AmgX library and free up any MPI Comms set up for it.
Definition at line 994 of file amgxsolver.cpp.
int mfem::AmgXSolver::GetNumIterations  (  ) 
Return the number of iterations that were executed during the last solve phase.
Definition at line 987 of file amgxsolver.cpp.
void mfem::AmgXSolver::InitExclusiveGPU  (  const MPI_Comm &  comm  ) 
Initialize the AmgX library in parallel mode with exactly one GPU per rank after the solver configuration has been established, either through the constructor or the AmgXSolver::ReadParameters method. If configuring with a constructor, the constructor will make this call.
Definition at line 124 of file amgxsolver.cpp.
void mfem::AmgXSolver::InitMPITeams  (  const MPI_Comm &  comm, 
const int  nDevs ) 
Initialize the AmgX library and create MPI teams based on the number of devices on each node nDevs. If configuring with a constructor, the constructor will make this call, otherwise this will need to be called after the solver configuration has been established through the AmgXSolver::ReadParameters call.
Definition at line 153 of file amgxsolver.cpp.
void mfem::AmgXSolver::InitSerial  (  ) 
Initialize the AmgX library for serial execution once the solver configuration has been established through either the AmgXSolver::ReadParameters method or the constructor. The constructor will make this call.
Definition at line 89 of file amgxsolver.cpp.
Utilize the AmgX library to solve the linear system where the "matrix" is the AMG approximation to the operator set by AmgXSolver::SetOperator. If the mode is set to AmgXSolver::PRECONDITIONER the initial guess for the x vector will be set to zero, otherwise the value of x passed in will be used.
Implements mfem::Operator.
Definition at line 906 of file amgxsolver.cpp.
void mfem::AmgXSolver::ReadParameters  (  const std::string  config, 
CONFIG_SRC  source ) 
Read in the AmgX parameters either through a file or directly through a properly formated string. If source is set to AmgXSolver::EXTERNAL the parameters are loaded from a filename set by config. If source is set to AmgXSolver::INTERNAL the parameters are set directly by the string defined by config.
Definition at line 190 of file amgxsolver.cpp.
void mfem::AmgXSolver::SetConvergenceCheck  (  bool  setConvergenceCheck_ = true  ) 
Add a check for convergence after applying Mult.
Definition at line 197 of file amgxsolver.cpp.

virtual 
Sets the Operator that is going to be solved via AmgX. Supports operators based on either an MFEM SparseMatrix or HypreParMatrix.
Implements mfem::Solver.
Definition at line 863 of file amgxsolver.cpp.
void mfem::AmgXSolver::UpdateOperator  (  const Operator &  op  ) 
Change the input operator that is being solved via AmgX. Supports operators based on either an MFEM SparseMatrix or HypreParMatrix.
Definition at line 886 of file amgxsolver.cpp.
bool mfem::AmgXSolver::ConvergenceCheck 
Flag to check for convergence.
Definition at line 87 of file amgxsolver.hpp.