MFEM wrapper for Nvidia's multigrid library, AmgX (github.com/NVIDIA/AMGX) More...

#include <amgxsolver.hpp>

Inheritance diagram for mfem::AmgXSolver:

Collaboration diagram for mfem::AmgXSolver:

Public Types
enum	AMGX_MODE { SOLVER , PRECONDITIONER }
	Flags to configure AmgXSolver as a solver or preconditioner. More...

enum	CONFIG_SRC { INTERNAL , EXTERNAL , UNDEFINED }
	Flags to determine whether user solver settings are defined internally in the source code or will be read through an external JSON file. More...

Public Types inherited from mfem::Operator
enum	DiagonalPolicy { DIAG_ZERO , DIAG_ONE , DIAG_KEEP }
	Defines operator diagonal policy upon elimination of rows and/or columns. More...

enum	Type { ANY_TYPE , MFEM_SPARSEMAT , Hypre_ParCSR , PETSC_MATAIJ , PETSC_MATIS , PETSC_MATSHELL , PETSC_MATNEST , PETSC_MATHYPRE , PETSC_MATGENERIC , Complex_Operator , MFEM_ComplexSparseMat , Complex_Hypre_ParCSR , Complex_DenseMat , MFEM_Block_Matrix , MFEM_Block_Operator }
	Enumeration defining IDs for some classes derived from Operator. More...

Public Member Functions
	AmgXSolver ()

	AmgXSolver (const AMGX_MODE amgxMode_, const bool verbose)
	Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Assumes no MPI parallelism.

void	InitSerial ()
	Initialize the AmgX library for serial execution once the solver configuration has been established through either the AmgXSolver::ReadParameters method or the constructor. The constructor will make this call.

	AmgXSolver (const MPI_Comm &comm, const AMGX_MODE amgxMode_, const bool verbose)
	Configures AmgX with a default configuration based on the AMGX_MODE (AmgXSolver::SOLVER, AmgXSolver::PRECONDITIONER) and verbosity. Pairs each MPI rank with one GPU.

	AmgXSolver (const MPI_Comm &comm, const int nDevs, const AMGX_MODE amgx_Mode_, const bool verbose)
	Configures AmgX with a default configuration based on the AMGX_MODE (AmgXSolver::SOLVER, AmgXSolver::PRECONDITIONER) and verbosity. Creates MPI teams around GPUs to support more ranks than GPUs. Consolidates linear solver data to avoid multiple ranks sharing GPUs. Requires specifying the number of devices in each compute node as nDevs.

void	InitExclusiveGPU (const MPI_Comm &comm)
	Initialize the AmgX library in parallel mode with exactly one GPU per rank after the solver configuration has been established, either through the constructor or the AmgXSolver::ReadParameters method. If configuring with a constructor, the constructor will make this call.

void	InitMPITeams (const MPI_Comm &comm, const int nDevs)
	Initialize the AmgX library and create MPI teams based on the number of devices on each node nDevs. If configuring with a constructor, the constructor will make this call, otherwise this will need to be called after the solver configuration has been established through the AmgXSolver::ReadParameters call.

virtual void	SetOperator (const Operator &op)
	Sets the Operator that is going to be solved via AmgX. Supports operators based on either an MFEM SparseMatrix or HypreParMatrix.

void	UpdateOperator (const Operator &op)
	Change the input operator that is being solved via AmgX. Supports operators based on either an MFEM SparseMatrix or HypreParMatrix.

virtual void	Mult (const Vector &b, Vector &x) const
	Utilize the AmgX library to solve the linear system where the "matrix" is the AMG approximation to the operator set by AmgXSolver::SetOperator. If the mode is set to AmgXSolver::PRECONDITIONER the initial guess for the x vector will be set to zero, otherwise the value of x passed in will be used.

int	GetNumIterations ()
	Return the number of iterations that were executed during the last solve phase.

void	ReadParameters (const std::string config, CONFIG_SRC source)
	Read in the AmgX parameters either through a file or directly through a properly formated string. If source is set to AmgXSolver::EXTERNAL the parameters are loaded from a filename set by config. If source is set to AmgXSolver::INTERNAL the parameters are set directly by the string defined by config.

void	DefaultParameters (const AMGX_MODE amgxMode_, const bool verbose)
	Set up the AmgX library with the default paramaters.

void	SetConvergenceCheck (bool setConvergenceCheck_=true)
	Add a check for convergence after applying Mult.

	~AmgXSolver ()
	Close down the AmgX library and free up any MPI Comms set up for it.

void	Finalize ()
	Close down the AmgX library and free up any MPI Comms set up for it.

Public Member Functions inherited from mfem::Solver
	Solver (int s=0, bool iter_mode=false)
	Initialize a square Solver with size s.

	Solver (int h, int w, bool iter_mode=false)
	Initialize a Solver with height h and width w.

Public Member Functions inherited from mfem::Operator
void	InitTVectors (const Operator Po, const Operator Ri, const Operator *Pi, Vector &x, Vector &b, Vector &X, Vector &B) const
	Initializes memory for true vectors of linear system.

	Operator (int s=0)
	Construct a square Operator with given size s (default 0).

	Operator (int h, int w)
	Construct an Operator with the given height (output size) and width (input size).

int	Height () const
	Get the height (size of output) of the Operator. Synonym with NumRows().

int	NumRows () const
	Get the number of rows (size of output) of the Operator. Synonym with Height().

int	Width () const
	Get the width (size of input) of the Operator. Synonym with NumCols().

int	NumCols () const
	Get the number of columns (size of input) of the Operator. Synonym with Width().

virtual MemoryClass	GetMemoryClass () const
	Return the MemoryClass preferred by the Operator.

virtual void	MultTranspose (const Vector &x, Vector &y) const
	Action of the transpose operator: `y=A^t(x)`. The default behavior in class Operator is to generate an error.

virtual void	AddMult (const Vector &x, Vector &y, const real_t a=1.0) const
	Operator application: `y+=A(x)` (default) or `y+=a*A(x)`.

virtual void	AddMultTranspose (const Vector &x, Vector &y, const real_t a=1.0) const
	Operator transpose application: `y+=A^t(x)` (default) or `y+=a*A^t(x)`.

virtual void	ArrayMult (const Array< const Vector * > &X, Array< Vector * > &Y) const
	Operator application on a matrix: `Y=A(X)`.

virtual void	ArrayMultTranspose (const Array< const Vector * > &X, Array< Vector * > &Y) const
	Action of the transpose operator on a matrix: `Y=A^t(X)`.

virtual void	ArrayAddMult (const Array< const Vector * > &X, Array< Vector * > &Y, const real_t a=1.0) const
	Operator application on a matrix: `Y+=A(X)` (default) or `Y+=a*A(X)`.

virtual void	ArrayAddMultTranspose (const Array< const Vector * > &X, Array< Vector * > &Y, const real_t a=1.0) const
	Operator transpose application on a matrix: `Y+=A^t(X)` (default) or `Y+=a*A^t(X)`.

virtual Operator &	GetGradient (const Vector &x) const
	Evaluate the gradient operator at the point x. The default behavior in class Operator is to generate an error.

virtual void	AssembleDiagonal (Vector &diag) const
	Computes the diagonal entries into diag. Typically, this operation only makes sense for linear Operators. In some cases, only an approximation of the diagonal is computed.

virtual const Operator *	GetProlongation () const
	Prolongation operator from linear algebra (linear system) vectors, to input vectors for the operator. `NULL` means identity.

virtual const Operator *	GetRestriction () const
	Restriction operator from input vectors for the operator to linear algebra (linear system) vectors. `NULL` means identity.

virtual const Operator *	GetOutputProlongation () const
	Prolongation operator from linear algebra (linear system) vectors, to output vectors for the operator. `NULL` means identity.

virtual const Operator *	GetOutputRestrictionTranspose () const
	Transpose of GetOutputRestriction, directly available in this form to facilitate matrix-free RAP-type operators.

virtual const Operator *	GetOutputRestriction () const
	Restriction operator from output vectors for the operator to linear algebra (linear system) vectors. `NULL` means identity.

void	FormLinearSystem (const Array< int > &ess_tdof_list, Vector &x, Vector &b, Operator *&A, Vector &X, Vector &B, int copy_interior=0)
	Form a constrained linear system using a matrix-free approach.

void	FormRectangularLinearSystem (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, Vector &x, Vector &b, Operator *&A, Vector &X, Vector &B)
	Form a column-constrained linear system using a matrix-free approach.

virtual void	RecoverFEMSolution (const Vector &X, const Vector &b, Vector &x)
	Reconstruct a solution vector x (e.g. a GridFunction) from the solution X of a constrained linear system obtained from Operator::FormLinearSystem() or Operator::FormRectangularLinearSystem().

void	FormSystemOperator (const Array< int > &ess_tdof_list, Operator *&A)
	Return in A a parallel (on truedofs) version of this square operator.

void	FormRectangularSystemOperator (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, Operator *&A)
	Return in A a parallel (on truedofs) version of this rectangular operator (including constraints).

void	FormDiscreteOperator (Operator *&A)
	Return in A a parallel (on truedofs) version of this rectangular operator.

void	PrintMatlab (std::ostream &out, int n, int m=0) const
	Prints operator with input size n and output size m in Matlab format.

virtual void	PrintMatlab (std::ostream &out) const
	Prints operator in Matlab format.

virtual	~Operator ()
	Virtual destructor.

Type	GetType () const
	Return the type ID of the Operator class.

Public Attributes
bool	ConvergenceCheck
	Flag to check for convergence.

Public Attributes inherited from mfem::Solver
bool	iterative_mode
	If true, use the second argument of Mult() as an initial guess.

Additional Inherited Members
Protected Member Functions inherited from mfem::Operator
void	FormConstrainedSystemOperator (const Array< int > &ess_tdof_list, ConstrainedOperator *&Aout)
	see FormSystemOperator()

void	FormRectangularConstrainedSystemOperator (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, RectangularConstrainedOperator *&Aout)
	see FormRectangularSystemOperator()

Operator *	SetupRAP (const Operator Pi, const Operator Po)
	Returns RAP Operator of this, using input/output Prolongation matrices Pi corresponds to "P", Po corresponds to "Rt".

Protected Attributes inherited from mfem::Operator
int	height
	Dimension of the output / number of rows in the matrix.

int	width
	Dimension of the input / number of columns in the matrix.

Detailed Description

MFEM wrapper for Nvidia's multigrid library, AmgX (github.com/NVIDIA/AMGX)

AmgX requires building MFEM with CUDA, and AMGX enabled. For distributed memory parallism, MPI and Hypre (version 16.0+) are also required. Although CUDA is required for building, the AmgX solver is compatible with a MFEM CPU device configuration.

The AmgXSolver class is designed to work as a solver or preconditioner for existing MFEM solvers. The AmgX solver class may be configured in one of three ways:

Serial - Takes a SparseMatrix solves on a single GPU and assumes no MPI communication.

Exclusive GPU - Takes a HypreParMatrix and assumes each MPI rank is paired with an Nvidia GPU.

MPI Teams - Takes a HypreParMatrix and enables flexibility between number of MPI ranks, and GPUs. Specifically, MPI ranks are grouped with GPUs and a matrix consolidation step is taken so the MPI root of each team performs the necessary AmgX library calls. The solution is then broadcasted to appropriate ranks. This is particularly useful when configuring MFEM's device as CPU. This work is based on the AmgXWrapper of Chuang and Barba. Routines were adopted and modified for setting up MPI communicators.

Examples 1/1p in the examples/amgx directory demonstrate configuring the wrapper as a solver and preconditioner, as well as configuring and running with exclusive GPU or MPI teams modes.

This work is partially based on:

Pi-Yueh Chuang and Lorena A. Barba (2017). AmgXWrapper: An interface between PETSc and the NVIDIA AmgX library. J. Open Source Software, 2(16):280, doi:10.21105/joss.00280

See https://github.com/barbagroup/AmgXWrapper.

Definition at line 69 of file amgxsolver.hpp.

Member Enumeration Documentation

◆ AMGX_MODE

enum mfem::AmgXSolver::AMGX_MODE

Flags to configure AmgXSolver as a solver or preconditioner.

Enumerator
SOLVER	Use the preconditioned conjugate gradient method with the AMG V-cycle used as a proconditioner. With the default configuration a block Jacobi smoother is used.
PRECONDITIONER	Directly apply iterations of the AMG V cycle to the matrix With the default configuration this will be 2 iterations with block Jacobi smoother.

Definition at line 74 of file amgxsolver.hpp.

◆ CONFIG_SRC

enum mfem::AmgXSolver::CONFIG_SRC

Flags to determine whether user solver settings are defined internally in the source code or will be read through an external JSON file.

Enumerator
INTERNAL	Configuration will be read directly from a string.
EXTERNAL	Configure will be read from a specified file.
UNDEFINED

Definition at line 93 of file amgxsolver.hpp.

Constructor & Destructor Documentation

◆ AmgXSolver() [1/4]

mfem::AmgXSolver::AmgXSolver ( )

Definition at line 37 of file amgxsolver.cpp.

◆ AmgXSolver() [2/4]

mfem::AmgXSolver::AmgXSolver	(	const AMGX_MODE	amgxMode_,
		const bool	verbose )

Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Assumes no MPI parallelism.

Definition at line 40 of file amgxsolver.cpp.

◆ AmgXSolver() [3/4]

mfem::AmgXSolver::AmgXSolver	(	const MPI_Comm &	comm,
		const AMGX_MODE	amgxMode_,
		const bool	verbose )

Configures AmgX with a default configuration based on the AMGX_MODE (AmgXSolver::SOLVER, AmgXSolver::PRECONDITIONER) and verbosity. Pairs each MPI rank with one GPU.

Definition at line 54 of file amgxsolver.cpp.

◆ AmgXSolver() [4/4]

mfem::AmgXSolver::AmgXSolver	(	const MPI_Comm &	comm,
		const int	nDevs,
		const AMGX_MODE	amgx_Mode_,
		const bool	verbose )

Configures AmgX with a default configuration based on the AMGX_MODE (AmgXSolver::SOLVER, AmgXSolver::PRECONDITIONER) and verbosity. Creates MPI teams around GPUs to support more ranks than GPUs. Consolidates linear solver data to avoid multiple ranks sharing GPUs. Requires specifying the number of devices in each compute node as nDevs.

Definition at line 68 of file amgxsolver.cpp.

◆ ~AmgXSolver()

mfem::AmgXSolver::~AmgXSolver ( )

Close down the AmgX library and free up any MPI Comms set up for it.

Definition at line 84 of file amgxsolver.cpp.

Member Function Documentation

◆ DefaultParameters()

void mfem::AmgXSolver::DefaultParameters	(	const AMGX_MODE	amgxMode_,
		const bool	verbose )

Set up the AmgX library with the default paramaters.

Parameters

[in]	amgxMode_	AmgXSolver::PRECONDITIONER, AmgXSolver::SOLVER.
[in]	verbose	true, false. Specifies the level of verbosity.

When configured as a preconditioner, the default configuration is to apply two iterations of an AMG V cycle with AmgX's default smoother (block Jacobi).

When configured as a solver the preconditioned conjugate gradient method is used with the AMG V-cycle and a block Jacobi smoother is used as a preconditioner.

Definition at line 202 of file amgxsolver.cpp.

◆ Finalize()

void mfem::AmgXSolver::Finalize ( )

Close down the AmgX library and free up any MPI Comms set up for it.

Definition at line 994 of file amgxsolver.cpp.

◆ GetNumIterations()

int mfem::AmgXSolver::GetNumIterations ( )

Return the number of iterations that were executed during the last solve phase.

Definition at line 987 of file amgxsolver.cpp.

◆ InitExclusiveGPU()

void mfem::AmgXSolver::InitExclusiveGPU ( const MPI_Comm & comm )

Initialize the AmgX library in parallel mode with exactly one GPU per rank after the solver configuration has been established, either through the constructor or the AmgXSolver::ReadParameters method. If configuring with a constructor, the constructor will make this call.

Definition at line 124 of file amgxsolver.cpp.

◆ InitMPITeams()

void mfem::AmgXSolver::InitMPITeams	(	const MPI_Comm &	comm,
		const int	nDevs )

Initialize the AmgX library and create MPI teams based on the number of devices on each node nDevs. If configuring with a constructor, the constructor will make this call, otherwise this will need to be called after the solver configuration has been established through the AmgXSolver::ReadParameters call.

Definition at line 153 of file amgxsolver.cpp.

◆ InitSerial()

void mfem::AmgXSolver::InitSerial ( )

Initialize the AmgX library for serial execution once the solver configuration has been established through either the AmgXSolver::ReadParameters method or the constructor. The constructor will make this call.

Definition at line 89 of file amgxsolver.cpp.

◆ Mult()

void mfem::AmgXSolver::Mult	(	const Vector &	b,
		Vector &	x ) const

virtual

Utilize the AmgX library to solve the linear system where the "matrix" is the AMG approximation to the operator set by AmgXSolver::SetOperator. If the mode is set to AmgXSolver::PRECONDITIONER the initial guess for the x vector will be set to zero, otherwise the value of x passed in will be used.

Implements mfem::Operator.

Definition at line 906 of file amgxsolver.cpp.

◆ ReadParameters()

void mfem::AmgXSolver::ReadParameters	(	const std::string	config,
		CONFIG_SRC	source )

Read in the AmgX parameters either through a file or directly through a properly formated string. If source is set to AmgXSolver::EXTERNAL the parameters are loaded from a filename set by config. If source is set to AmgXSolver::INTERNAL the parameters are set directly by the string defined by config.

Definition at line 190 of file amgxsolver.cpp.

◆ SetConvergenceCheck()

void mfem::AmgXSolver::SetConvergenceCheck ( bool setConvergenceCheck_ = true )

Add a check for convergence after applying Mult.

Definition at line 197 of file amgxsolver.cpp.

◆ SetOperator()

void mfem::AmgXSolver::SetOperator ( const Operator & op )

virtual

Sets the Operator that is going to be solved via AmgX. Supports operators based on either an MFEM SparseMatrix or HypreParMatrix.

Implements mfem::Solver.

Definition at line 863 of file amgxsolver.cpp.

◆ UpdateOperator()

void mfem::AmgXSolver::UpdateOperator ( const Operator & op )

Change the input operator that is being solved via AmgX. Supports operators based on either an MFEM SparseMatrix or HypreParMatrix.

Definition at line 886 of file amgxsolver.cpp.

Member Data Documentation

◆ ConvergenceCheck

bool mfem::AmgXSolver::ConvergenceCheck

Flag to check for convergence.

Definition at line 87 of file amgxsolver.hpp.

The documentation for this class was generated from the following files:

linalg/amgxsolver.hpp
linalg/amgxsolver.cpp

Public Types

Public Member Functions

Public Attributes

Additional Inherited Members