The MFEM Device class abstracts hardware devices such as GPUs, as well as programming models such as CUDA, OCCA, RAJA and OpenMP. More...

#include <device.hpp>

Public Member Functions
	Device ()
	Default constructor. Unless Configure() is called later, the default Backend::CPU will be used.

	Device (const std::string &device, const int device_id=0)
	Construct a Device and configure it based on the device string. See Configure() for more details.

	~Device ()
	Destructor.

void	Configure (const std::string &device, const int device_id=0)
	Configure the Device backends.

void	Print (std::ostream &os=mfem::out)
	Print the configuration of the MFEM virtual device object.

Static Public Member Functions
static void	SetMemoryTypes (MemoryType h_mt, MemoryType d_mt)
	Set the default host and device MemoryTypes, h_mt and d_mt.

static bool	IsConfigured ()
	Return true if Configure() has been called previously.

static bool	IsAvailable ()
	Return true if an actual device (e.g. GPU) has been configured.

static bool	IsEnabled ()
	Return true if any backend other than Backend::CPU is enabled.

static bool	IsDisabled ()
	The opposite of IsEnabled().

static int	GetId ()
	Get the device ID of the configured device.

static int	GetDeviceCount ()
	Get the number of available devices (may be called before configuration).

static std::string	GetUUID (const int device_id=0)

static bool	Allows (unsigned long b_mask)
	Return true if any of the backends in the backend mask, b_mask, are allowed.

static auto	GetRajaResource ()

static MemoryType	GetHostMemoryType ()
	Get the current Host MemoryType. This is the MemoryType used by most MFEM classes when allocating memory used on the host.

static MemoryClass	GetHostMemoryClass ()
	Get the current Host MemoryClass. This is the MemoryClass used by most MFEM host Memory objects.

static MemoryType	GetDeviceMemoryType ()
	Get the current Device MemoryType. This is the MemoryType used by most MFEM classes when allocating memory to be used with device kernels.

static MemoryType	GetMemoryType ()
	(DEPRECATED) Equivalent to GetDeviceMemoryType().

static MemoryClass	GetDeviceMemoryClass ()
	Get the current Device MemoryClass. This is the MemoryClass used by most MFEM device kernels to access Memory objects.

static MemoryClass	GetMemoryClass ()
	(DEPRECATED) Equivalent to GetDeviceMemoryClass().

static void	SetGPUAwareMPI (const bool force=true)
	Manually set the status of GPU-aware MPI flag for use in MPI communication routines which have optimized implementations for device buffers.

static bool	GetGPUAwareMPI ()
	Get the status of GPU-aware MPI flag.

static MemoryType	QueryMemoryType (const void *ptr)

static int	NumMultiprocessors (int device_id)
	The number of hardware compute units/streaming multiprocessors available on a given compute device device_id.

static int	NumMultiprocessors ()
	Same as NumMultiprocessors(int), for the currently active device.

static int	WarpSize (int device_id)
	The number of threads in a warp on a given compute device device_id.

static int	WarpSize ()
	Same as WarpSize(int), for the currently active device.

static void	DeviceMem (size_t free, size_t total)
	Gets the free and total memory on the device.

Friends
class	MemoryManager

Detailed Description

The MFEM Device class abstracts hardware devices such as GPUs, as well as programming models such as CUDA, OCCA, RAJA and OpenMP.

This class represents a "virtual device" with the following properties:

At most one object of this class can be constructed and that object is controlled by its static methods.
If no Device object is constructed, the static methods will use a default global object which is never configured and always uses Backend::CPU.
Once configured, the object cannot be re-configured during the program lifetime.
MFEM classes use this object to determine where (host or device) to perform an operation and which backend implementation to use.
Multiple backends can be configured at the same time; currently, a fixed priority order is used to select a specific backend from the list of configured backends. See the Backend class and the Configure() method in this class for details.

Definition at line 128 of file device.hpp.

Constructor & Destructor Documentation

◆ Device() [1/2]

mfem::Device::Device ( )

Default constructor. Unless Configure() is called later, the default Backend::CPU will be used.

Note: At most one Device object can be constructed during the lifetime of a program.; This object should be destroyed after all other MFEM objects that use the Device are destroyed.

Definition at line 76 of file device.cpp.

◆ Device() [2/2]

mfem::Device::Device	(	const std::string &	device,
		const int	device_id = 0 )

inline

Construct a Device and configure it based on the device string. See Configure() for more details.

Note: At most one Device object can be constructed during the lifetime of a program.; This object should be destroyed after all other MFEM objects that use the Device are destroyed.

Definition at line 185 of file device.hpp.

◆ ~Device()

mfem::Device::~Device ( )

Destructor.

Definition at line 158 of file device.cpp.

Member Function Documentation

◆ Allows()

static bool mfem::Device::Allows ( unsigned long b_mask )

inlinestatic

Return true if any of the backends in the backend mask, b_mask, are allowed.

This method can be used with any of the Backend::Id constants, the Backend::*_MASK, or combinations of those.

Definition at line 271 of file device.hpp.

◆ Configure()

void mfem::Device::Configure	(	const std::string &	device,
		const int	device_id = 0 )

Configure the Device backends.

The string parameter device must be a comma-separated list of backend string names (see below). The device_id argument specifies the ID of the actual devices (e.g. GPU) to use.

The available backends are described by the Backend class.
The string name of a backend is the lowercase version of the Backend::Id enumeration constant with '_' replaced by '-', e.g. the string name of 'RAJA_CPU' is 'raja-cpu'. The string name of the debug backend (Backend::Id 'DEBUG_DEVICE') is exceptionally set to 'debug'.
The 'cpu' backend is always enabled with lowest priority.
The current backend priority from highest to lowest is: 'ceed-cuda', 'occa-cuda', 'raja-cuda', 'cuda', 'ceed-hip', 'hip', 'debug', 'occa-omp', 'raja-omp', 'omp', 'ceed-cpu', 'occa-cpu', 'raja-cpu', 'cpu'.
The following backend aliases are also available: 'ceed-gpu', 'occa-gpu', 'raja-gpu', and 'gpu' where they alias their respective '-cuda' or '-hip' backends depending on the MFEM build-time configuration.
Multiple backends can be configured at the same time.
Only one 'occa-*' backend can be configured at a time.
The backend 'occa-cuda' enables the 'cuda' backend unless 'raja-cuda' is already enabled.
The backend 'occa-omp' enables the 'omp' backend (if MFEM was built with MFEM_USE_OPENMP=YES) unless 'raja-omp' is already enabled.
Only one 'ceed-*' backend can be configured at a time.
The backend 'ceed-cpu' delegates to a libCEED CPU backend the setup and evaluation of the operator.
The backend 'ceed-cuda' delegates to a libCEED CUDA backend the setup and evaluation of operators and enables the 'cuda' backend to avoid transfers between host and device.
The backend 'ceed-hip' delegates to a libCEED HIP backend the setup and evaluation of operators and enables the 'hip' backend to avoid transfers between host and device.
The 'debug' backend should not be combined with other device backends.

Note: If the device is actually enabled, this method will also update the current host/device MemoryType and MemoryClass.

Definition at line 191 of file device.cpp.

◆ DeviceMem()

void mfem::Device::DeviceMem	(	size_t *	free,
		size_t *	total )

static

Gets the free and total memory on the device.

Definition at line 704 of file device.cpp.

◆ GetDeviceCount()

int mfem::Device::GetDeviceCount ( )

static

Get the number of available devices (may be called before configuration).

Definition at line 427 of file device.cpp.

◆ GetDeviceMemoryClass()

static MemoryClass mfem::Device::GetDeviceMemoryClass ( )

inlinestatic

Get the current Device MemoryClass. This is the MemoryClass used by most MFEM device kernels to access Memory objects.

Definition at line 306 of file device.hpp.

◆ GetDeviceMemoryType()

static MemoryType mfem::Device::GetDeviceMemoryType ( )

inlinestatic

Get the current Device MemoryType. This is the MemoryType used by most MFEM classes when allocating memory to be used with device kernels.

Definition at line 298 of file device.hpp.

◆ GetGPUAwareMPI()

static bool mfem::Device::GetGPUAwareMPI ( )

inlinestatic

Get the status of GPU-aware MPI flag.

Definition at line 319 of file device.hpp.

◆ GetHostMemoryClass()

static MemoryClass mfem::Device::GetHostMemoryClass ( )

inlinestatic

Get the current Host MemoryClass. This is the MemoryClass used by most MFEM host Memory objects.

Definition at line 293 of file device.hpp.

◆ GetHostMemoryType()

static MemoryType mfem::Device::GetHostMemoryType ( )

inlinestatic

Get the current Host MemoryType. This is the MemoryType used by most MFEM classes when allocating memory used on the host.

Definition at line 289 of file device.hpp.

◆ GetId()

static int mfem::Device::GetId ( )

inlinestatic

Get the device ID of the configured device.

Definition at line 258 of file device.hpp.

◆ GetMemoryClass()

static MemoryClass mfem::Device::GetMemoryClass ( )

inlinestatic

(DEPRECATED) Equivalent to GetDeviceMemoryClass().

Deprecated: Use GetDeviceMemoryClass() instead.

Definition at line 310 of file device.hpp.

◆ GetMemoryType()

static MemoryType mfem::Device::GetMemoryType ( )

inlinestatic

(DEPRECATED) Equivalent to GetDeviceMemoryType().

Deprecated: Use GetDeviceMemoryType() instead.

Definition at line 302 of file device.hpp.

◆ GetRajaResource()

static auto mfem::Device::GetRajaResource ( )

inlinestatic

Definition at line 276 of file device.hpp.

◆ GetUUID()

std::string mfem::Device::GetUUID ( const int device_id = 0 )

static

Gets a string representation of the GPU UUID. 0 <= device_id < GetDeviceCount()

Definition at line 723 of file device.cpp.

◆ IsAvailable()

static bool mfem::Device::IsAvailable ( )

inlinestatic

Return true if an actual device (e.g. GPU) has been configured.

Definition at line 249 of file device.hpp.

◆ IsConfigured()

static bool mfem::Device::IsConfigured ( )

inlinestatic

Return true if Configure() has been called previously.

Definition at line 246 of file device.hpp.

◆ IsDisabled()

static bool mfem::Device::IsDisabled ( )

inlinestatic

The opposite of IsEnabled().

Definition at line 255 of file device.hpp.

◆ IsEnabled()

static bool mfem::Device::IsEnabled ( )

inlinestatic

Return true if any backend other than Backend::CPU is enabled.

Definition at line 252 of file device.hpp.

◆ NumMultiprocessors() [1/2]

int mfem::Device::NumMultiprocessors ( )

static

Same as NumMultiprocessors(int), for the currently active device.

Definition at line 764 of file device.cpp.

◆ NumMultiprocessors() [2/2]

int mfem::Device::NumMultiprocessors ( int device_id )

static

The number of hardware compute units/streaming multiprocessors available on a given compute device device_id.

Definition at line 746 of file device.cpp.

◆ Print()

void mfem::Device::Print ( std::ostream & os = mfem::out )

Print the configuration of the MFEM virtual device object.

Definition at line 319 of file device.cpp.

◆ QueryMemoryType()

MemoryType mfem::Device::QueryMemoryType ( const void * ptr )

static

Query the device driver for what memory type a given ptr is allocated with.

Definition at line 622 of file device.cpp.

◆ SetGPUAwareMPI()

static void mfem::Device::SetGPUAwareMPI ( const bool force = true )

inlinestatic

Manually set the status of GPU-aware MPI flag for use in MPI communication routines which have optimized implementations for device buffers.

Definition at line 315 of file device.hpp.

◆ SetMemoryTypes()

void mfem::Device::SetMemoryTypes	(	MemoryType	h_mt,
		MemoryType	d_mt )

static

Set the default host and device MemoryTypes, h_mt and d_mt.

The host and device MemoryTypes are also set to be dual to each other.

These two MemoryTypes are used by most MFEM classes when allocating memory used on host and device, respectively.

This method can only be called before Device construction and configuration, and the specified memory types must be compatible with the subsequent Device configuration.

Definition at line 296 of file device.cpp.

◆ WarpSize() [1/2]

int mfem::Device::WarpSize ( )

static

Same as WarpSize(int), for the currently active device.

Definition at line 792 of file device.cpp.

◆ WarpSize() [2/2]

int mfem::Device::WarpSize ( int device_id )

static

The number of threads in a warp on a given compute device device_id.

Definition at line 775 of file device.cpp.

Friends And Related Symbol Documentation

◆ MemoryManager

friend class MemoryManager

friend

Definition at line 131 of file device.hpp.

The documentation for this class was generated from the following files:

general/device.hpp
general/device.cpp

Public Member Functions

Static Public Member Functions

Friends

Detailed Description

Constructor & Destructor Documentation

◆ Device() [1/2]

◆ Device() [2/2]

◆ ~Device()

Member Function Documentation

◆ Allows()

◆ Configure()

◆ DeviceMem()

◆ GetDeviceCount()

◆ GetDeviceMemoryClass()

◆ GetDeviceMemoryType()

◆ GetGPUAwareMPI()

◆ GetHostMemoryClass()

◆ GetHostMemoryType()

◆ GetId()

◆ GetMemoryClass()

◆ GetMemoryType()

◆ GetRajaResource()

◆ GetUUID()

◆ IsAvailable()

◆ IsConfigured()

◆ IsDisabled()

◆ IsEnabled()

◆ NumMultiprocessors() [1/2]

◆ NumMultiprocessors() [2/2]

◆ Print()

◆ QueryMemoryType()

◆ SetGPUAwareMPI()

◆ SetMemoryTypes()

◆ WarpSize() [1/2]

◆ WarpSize() [2/2]

Friends And Related Symbol Documentation

◆ MemoryManager