#include <amgxsolver.hpp>

Inheritance diagram for mfem::AmgXSolver:

Collaboration diagram for mfem::AmgXSolver:

Public Types
enum	AMGX_MODE { SOLVER, PRECONDITIONER }
	Flags to configure AmgXSolver as a solver or preconditioner. More...

enum	CONFIG_SRC { INTERNAL, EXTERNAL, UNDEFINED }

Public Types inherited from mfem::Operator
enum	DiagonalPolicy { DIAG_ZERO, DIAG_ONE, DIAG_KEEP }
	Defines operator diagonal policy upon elimination of rows and/or columns. More...

enum	Type { ANY_TYPE, MFEM_SPARSEMAT, Hypre_ParCSR, PETSC_MATAIJ, PETSC_MATIS, PETSC_MATSHELL, PETSC_MATNEST, PETSC_MATHYPRE, PETSC_MATGENERIC, Complex_Operator, MFEM_ComplexSparseMat, Complex_Hypre_ParCSR, Complex_DenseMat, MFEM_Block_Matrix, MFEM_Block_Operator }
	Enumeration defining IDs for some classes derived from Operator. More...

Public Member Functions
	AmgXSolver ()

	AmgXSolver (const AMGX_MODE amgxMode_, const bool verbose)

void	InitSerial ()

	AmgXSolver (const MPI_Comm &comm, const AMGX_MODE amgxMode_, const bool verbose)

	AmgXSolver (const MPI_Comm &comm, const int nDevs, const AMGX_MODE amgx_Mode_, const bool verbose)

void	InitExclusiveGPU (const MPI_Comm &comm)

void	InitMPITeams (const MPI_Comm &comm, const int nDevs)

virtual void	SetOperator (const Operator &op)

void	UpdateOperator (const Operator &op)

virtual void	Mult (const Vector &b, Vector &x) const
	Operator application: `y=A(x)`. More...

int	GetNumIterations ()

void	ReadParameters (const std::string config, CONFIG_SRC source)

void	DefaultParameters (const AMGX_MODE amgxMode_, const bool verbose)

void	SetConvergenceCheck (bool setConvergenceCheck_=true)
	Add a check for convergence after applying Mult. More...

	~AmgXSolver ()

void	Finalize ()

Public Member Functions inherited from mfem::Solver
	Solver (int s=0, bool iter_mode=false)
	Initialize a square Solver with size s. More...

	Solver (int h, int w, bool iter_mode=false)
	Initialize a Solver with height h and width w. More...

Public Member Functions inherited from mfem::Operator
void	InitTVectors (const Operator Po, const Operator Ri, const Operator *Pi, Vector &x, Vector &b, Vector &X, Vector &B) const
	Initializes memory for true vectors of linear system. More...

	Operator (int s=0)
	Construct a square Operator with given size s (default 0). More...

	Operator (int h, int w)
	Construct an Operator with the given height (output size) and width (input size). More...

int	Height () const
	Get the height (size of output) of the Operator. Synonym with NumRows(). More...

int	NumRows () const
	Get the number of rows (size of output) of the Operator. Synonym with Height(). More...

int	Width () const
	Get the width (size of input) of the Operator. Synonym with NumCols(). More...

int	NumCols () const
	Get the number of columns (size of input) of the Operator. Synonym with Width(). More...

virtual MemoryClass	GetMemoryClass () const
	Return the MemoryClass preferred by the Operator. More...

virtual void	MultTranspose (const Vector &x, Vector &y) const
	Action of the transpose operator: `y=A^t(x)`. The default behavior in class Operator is to generate an error. More...

virtual void	AddMult (const Vector &x, Vector &y, const double a=1.0) const
	Operator application: `y+=A(x)` (default) or `y+=a*A(x)`. More...

virtual void	AddMultTranspose (const Vector &x, Vector &y, const double a=1.0) const
	Operator transpose application: `y+=A^t(x)` (default) or `y+=a*A^t(x)`. More...

virtual void	ArrayMult (const Array< const Vector > &X, Array< Vector > &Y) const
	Operator application on a matrix: `Y=A(X)`. More...

virtual void	ArrayMultTranspose (const Array< const Vector > &X, Array< Vector > &Y) const
	Action of the transpose operator on a matrix: `Y=A^t(X)`. More...

virtual void	ArrayAddMult (const Array< const Vector > &X, Array< Vector > &Y, const double a=1.0) const
	Operator application on a matrix: `Y+=A(X)` (default) or `Y+=a*A(X)`. More...

virtual void	ArrayAddMultTranspose (const Array< const Vector > &X, Array< Vector > &Y, const double a=1.0) const
	Operator transpose application on a matrix: `Y+=A^t(X)` (default) or `Y+=a*A^t(X)`. More...

virtual Operator &	GetGradient (const Vector &x) const
	Evaluate the gradient operator at the point x. The default behavior in class Operator is to generate an error. More...

virtual void	AssembleDiagonal (Vector &diag) const
	Computes the diagonal entries into diag. Typically, this operation only makes sense for linear Operators. In some cases, only an approximation of the diagonal is computed. More...

virtual const Operator *	GetProlongation () const
	Prolongation operator from linear algebra (linear system) vectors, to input vectors for the operator. `NULL` means identity. More...

virtual const Operator *	GetRestriction () const
	Restriction operator from input vectors for the operator to linear algebra (linear system) vectors. `NULL` means identity. More...

virtual const Operator *	GetOutputProlongation () const
	Prolongation operator from linear algebra (linear system) vectors, to output vectors for the operator. `NULL` means identity. More...

virtual const Operator *	GetOutputRestrictionTranspose () const
	Transpose of GetOutputRestriction, directly available in this form to facilitate matrix-free RAP-type operators. More...

virtual const Operator *	GetOutputRestriction () const
	Restriction operator from output vectors for the operator to linear algebra (linear system) vectors. `NULL` means identity. More...

void	FormLinearSystem (const Array< int > &ess_tdof_list, Vector &x, Vector &b, Operator *&A, Vector &X, Vector &B, int copy_interior=0)
	Form a constrained linear system using a matrix-free approach. More...

void	FormRectangularLinearSystem (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, Vector &x, Vector &b, Operator *&A, Vector &X, Vector &B)
	Form a column-constrained linear system using a matrix-free approach. More...

virtual void	RecoverFEMSolution (const Vector &X, const Vector &b, Vector &x)
	Reconstruct a solution vector x (e.g. a GridFunction) from the solution X of a constrained linear system obtained from Operator::FormLinearSystem() or Operator::FormRectangularLinearSystem(). More...

void	FormSystemOperator (const Array< int > &ess_tdof_list, Operator *&A)
	Return in A a parallel (on truedofs) version of this square operator. More...

void	FormRectangularSystemOperator (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, Operator *&A)
	Return in A a parallel (on truedofs) version of this rectangular operator (including constraints). More...

void	FormDiscreteOperator (Operator *&A)
	Return in A a parallel (on truedofs) version of this rectangular operator. More...

void	PrintMatlab (std::ostream &out, int n, int m=0) const
	Prints operator with input size n and output size m in Matlab format. More...

virtual void	PrintMatlab (std::ostream &out) const
	Prints operator in Matlab format. More...

virtual	~Operator ()
	Virtual destructor. More...

Type	GetType () const
	Return the type ID of the Operator class. More...

Public Attributes
bool	ConvergenceCheck
	Flag to check for convergence. More...

Public Attributes inherited from mfem::Solver
bool	iterative_mode
	If true, use the second argument of Mult() as an initial guess. More...

Additional Inherited Members
Protected Member Functions inherited from mfem::Operator
void	FormConstrainedSystemOperator (const Array< int > &ess_tdof_list, ConstrainedOperator *&Aout)
	see FormSystemOperator() More...

void	FormRectangularConstrainedSystemOperator (const Array< int > &trial_tdof_list, const Array< int > &test_tdof_list, RectangularConstrainedOperator *&Aout)
	see FormRectangularSystemOperator() More...

Operator *	SetupRAP (const Operator Pi, const Operator Po)
	Returns RAP Operator of this, using input/output Prolongation matrices Pi corresponds to "P", Po corresponds to "Rt". More...

Protected Attributes inherited from mfem::Operator
int	height
	Dimension of the output / number of rows in the matrix. More...

int	width
	Dimension of the input / number of columns in the matrix. More...

Detailed Description

MFEM wrapper for Nvidia's multigrid library, AmgX (github.com/NVIDIA/AMGX)

AmgX requires building MFEM with CUDA, and AMGX enabled. For distributed memory parallism, MPI and Hypre (version 16.0+) are also required. Although CUDA is required for building, the AmgX solver is compatible with a MFEM CPU device configuration.

The AmgXSolver class is designed to work as a solver or preconditioner for existing MFEM solvers. The AmgX solver class may be configured in one of three ways:

Serial - Takes a SparseMatrix solves on a single GPU and assumes no MPI communication.

Exclusive GPU - Takes a HypreParMatrix and assumes each MPI rank is paired with an Nvidia GPU.

MPI Teams - Takes a HypreParMatrix and enables flexibility between number of MPI ranks, and GPUs. Specifically, MPI ranks are grouped with GPUs and a matrix consolidation step is taken so the MPI root of each team performs the necessary AmgX library calls. The solution is then broadcasted to appropriate ranks. This is particularly useful when configuring MFEM's device as CPU. This work is based on the AmgXWrapper of Chuang and Barba. Routines were adopted and modified for setting up MPI communicators.

Examples 1/1p in the examples/amgx directory demonstrate configuring the wrapper as a solver and preconditioner, as well as configuring and running with exclusive GPU or MPI teams modes.

This work is partially based on:

Pi-Yueh Chuang and Lorena A. Barba (2017). AmgXWrapper: An interface between PETSc and the NVIDIA AmgX library. J. Open Source Software, 2(16):280, doi:10.21105/joss.00280

See https://github.com/barbagroup/AmgXWrapper.

Definition at line 69 of file amgxsolver.hpp.

Member Enumeration Documentation

◆ AMGX_MODE

enum mfem::AmgXSolver::AMGX_MODE

Flags to configure AmgXSolver as a solver or preconditioner.

Enumerator
SOLVER
PRECONDITIONER

Definition at line 74 of file amgxsolver.hpp.

◆ CONFIG_SRC

enum mfem::AmgXSolver::CONFIG_SRC

Flags to determine whether user solver settings are defined internally in the source code or will be read through an external JSON file.

Enumerator
INTERNAL
EXTERNAL
UNDEFINED

Definition at line 83 of file amgxsolver.hpp.

Constructor & Destructor Documentation

◆ AmgXSolver() [1/4]

mfem::AmgXSolver::AmgXSolver ( )

Definition at line 33 of file amgxsolver.cpp.

◆ AmgXSolver() [2/4]

mfem::AmgXSolver::AmgXSolver	(	const AMGX_MODE	amgxMode_,
		const bool	verbose
	)

Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Assumes no MPI parallism.

Definition at line 36 of file amgxsolver.cpp.

◆ AmgXSolver() [3/4]

mfem::AmgXSolver::AmgXSolver	(	const MPI_Comm &	comm,
		const AMGX_MODE	amgxMode_,
		const bool	verbose
	)

Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Pairs each MPI rank with one GPU.

Definition at line 50 of file amgxsolver.cpp.

◆ AmgXSolver() [4/4]

mfem::AmgXSolver::AmgXSolver	(	const MPI_Comm &	comm,
		const int	nDevs,
		const AMGX_MODE	amgx_Mode_,
		const bool	verbose
	)

Configures AmgX with a default configuration based on the AmgX mode, and verbosity. Creates MPI teams around GPUs to support MPI ranks > GPUs. Consolidates linear solver data to avoid multiple ranks sharing GPUs. Requires specifying number of devices in each compute node.

Definition at line 64 of file amgxsolver.cpp.

◆ ~AmgXSolver()

mfem::AmgXSolver::~AmgXSolver ( )

Definition at line 80 of file amgxsolver.cpp.

Member Function Documentation

◆ DefaultParameters()

void mfem::AmgXSolver::DefaultParameters	(	const AMGX_MODE	amgxMode_,
		const bool	verbose
	)

Parameters

[in]	amgxMode_	AmgXSolver::PRECONDITIONER, AmgXSolver::SOLVER.
[in]	verbose	true, false. Specifies the level of verbosity.

When configured as a preconditioner, the default configuration is to apply two iterations of an AMG V cycle with AmgX's default smoother (block Jacobi).

As a solver the preconditioned conjugate gradient method is used. The AMG V-cycle with a block Jacobi smoother is used as a preconditioner.

Definition at line 198 of file amgxsolver.cpp.

◆ Finalize()

void mfem::AmgXSolver::Finalize ( )

Definition at line 990 of file amgxsolver.cpp.

◆ GetNumIterations()

int mfem::AmgXSolver::GetNumIterations ( )

Definition at line 983 of file amgxsolver.cpp.

◆ InitExclusiveGPU()

void mfem::AmgXSolver::InitExclusiveGPU ( const MPI_Comm & comm )

Once the solver configuration has been established, either through the constructor or the ReadParameters method, InitSerial will initialize the library. If configuring with constructor, the constructor will make this call.

Definition at line 120 of file amgxsolver.cpp.

◆ InitMPITeams()

void mfem::AmgXSolver::InitMPITeams	(	const MPI_Comm &	comm,
		const int	nDevs
	)

Once the solver configuration has been established, either through the ReadParameters method, InitMPITeams will initialize the library and create MPI teams based on the number of devices on each node (nDevs). If configuring with constructor, the constructor will make this call.

Definition at line 149 of file amgxsolver.cpp.

◆ InitSerial()

void mfem::AmgXSolver::InitSerial ( )

Once the solver configuration has been established through either the ReadParameters method or the constructor, InitSerial will initialize the library. If configuring with constructor, the constructor will make this call.

Definition at line 85 of file amgxsolver.cpp.

◆ Mult()

void mfem::AmgXSolver::Mult	(	const Vector &	x,
		Vector &	y
	)		const

virtual

Operator application: y=A(x).

Implements mfem::Operator.

Definition at line 902 of file amgxsolver.cpp.

◆ ReadParameters()

void mfem::AmgXSolver::ReadParameters	(	const std::string	config,
		CONFIG_SRC	source
	)

Definition at line 186 of file amgxsolver.cpp.

◆ SetConvergenceCheck()

void mfem::AmgXSolver::SetConvergenceCheck ( bool setConvergenceCheck_ = true )

Add a check for convergence after applying Mult.

Definition at line 193 of file amgxsolver.cpp.

◆ SetOperator()

void mfem::AmgXSolver::SetOperator ( const Operator & op )

virtual

Sets Operator for AmgX library, either MFEM SparseMatrix or HypreParMatrix

Implements mfem::Solver.

Definition at line 859 of file amgxsolver.cpp.

◆ UpdateOperator()

void mfem::AmgXSolver::UpdateOperator ( const Operator & op )

Replaces the matrix coefficients in the AmgX solver.

Definition at line 882 of file amgxsolver.cpp.

Member Data Documentation

◆ ConvergenceCheck

bool mfem::AmgXSolver::ConvergenceCheck

Flag to check for convergence.

Definition at line 77 of file amgxsolver.hpp.

The documentation for this class was generated from the following files:

linalg/amgxsolver.hpp
linalg/amgxsolver.cpp

Public Types

Public Member Functions

Public Attributes

Additional Inherited Members