Version 1.8
Released November 2nd, 2006
$Date: 2008/10/11 04:34:38 $
$Revision: 1.44 $
Editor: Dan Bonachea
http://www.cs.berkeley.edu/~bonachea/gasnet
http://upc.lbl.gov/
Copyright © 2002-2004, Dan Bonachea.
Selected portions adapted from:
Permission is granted to freely distribute this specification and use it in creating GASNet clients or implementations. The authoritative version of the GASNet specification is maintained by Dan Bonachea and any proposed changes should be submitted for review.
Published by LBNL FTG and U.C. Berkeley
This GASNet specification describes a network-independent and language-independent high-performance communication interface intended for use in implementing the runtime system for global address space languages (such as UPC or Titanium). GASNet stands for "Global-Address Space Networking".
The interface is divided into 2 layers - the GASNet core API and the GASNet extended API:
The core API is the minimum interface that must be implemented on each network when porting to a new system, and we provide a network-independent reference implementation of the extended API which is written purely in terms of the core API to ease porting and quick prototyping. Implementors for NIC's that provide some hardware support for higher-level messaging operations (e.g. support for servicing remote reads/writes on the NIC without involving the main CPU) are encouraged to also implement an appropriate subset of the extended API directly on the network of interest (bypassing the core API) to achieve maximal performance for those operations (but this is an optimization and is not required to have a working system). Most clients will use calls to the extended API functions to implement the bulk of their communication work (thereby ensuring optimal performance across platforms). However the client is also permitted to use the core active message interface to implement non-trivial language-specific or compiler-specific communication operations which would not be appropriate in a language-independent API (e.g. implementing distributed language-level locks, distributed garbage collection, collective memory allocation, etc.).
Note the extended API interface is meant primarily as a low-level compilation target, not a library for hand-written code - as such, the goals of expressiveness and performance generally take precedence over readability and minimality.
gasnet_
GASNET_
gasnet.h and
linking the appropriate library
gasnet_init(), and its associated
local memory space and system resources. The basic unit of control when
interfacing with GASNet.
Client code must #define exactly one of GASNET_PAR, GASNET_PARSYNC or
GASNET_SEQ when compiling the GASNet library and the client code (before
including gasnet.h) to indicate the threading environment.
GASNET_PAR
GASNET_PARSYNC
GASNET_SEQ
Implementor's Note:
|
Many GASNet core functions return 0 on success (GASNET_OK), or else they
return errors from the following list, as specified by each function:
GASNET_OK = 0 (no error)
GASNET_ERR_RESOURCE
GASNET_ERR_BAD_ARG
GASNET_ERR_NOT_INIT
GASNET_ERR_BARRIER_MISMATCH
GASNET_ERR_NOT_READY
Except where otherwise noted, errors that occur during a call to the extended API are fatal.
Many of the core API functions will return GASNET_ERR_RESOURCE to
indicate a generic failure in the hardware or communications system,
GASNET_ERR_BAD_ARG to indicate an illegal client argument, or
GASNET_ERR_NOT_INIT to indicate that gasnet_attach() has not been called.
If any node of a GASNet job crashes, aborts, or suffers a fatal hardware error, GASNet should make every attempt to ensure that the remaining nodes of the job are terminated in a timely manner to prevent creation of orphaned processes.
| char * gasnet_ErrorName (int errval) | |
| char * gasnet_ErrorDesc (int errval) |
|
|
gasnet_node_t
gasnet_handle_t
gasnet_handler_t
gasnet_handlerarg_t
gasnet_token_t
gasnet_register_value_t
SIZEOF_GASNET_REGISTER_VALUE_T is a preprocess-time literal integer constant
(i.e. not sizeof())indicating the size of this type in bytes
gasnet_handlerentry_t
gasnet_attach()
GASNET_SPEC_VERSION_MAJOR
GASNET_SPEC_VERSION_MINOR
GASNET_RELEASE_VERSION_MAJOR
GASNET_RELEASE_VERSION_MINOR
GASNET_RELEASE_VERSION_PATCH
GASNET_VERSION (deprecated)
GASNET_CONFIG_STRING
GASNET_MAXNODES
gasnet_node_t.
GASNET_ALIGNED_SEGMENTS
gasnet_attach()
guarantees that the remote-access memory segment will be aligned at the same virtual
address on all nodes. Defined to 0 otherwise.
GASNET_PAGESIZE
gasnet_exit
The core API consists of:
Job startup in GASNet is a two-step process. GASNet programs should start by calling gasnet_init() as the first statement in their main() function, which bootstraps the nodes and establishes command-line arguments and the job environment. All nodes then call the gasnet_attach() function to initialize the network and register shared memory segments.
GASNet initialization may register some UNIX signal handlers (e.g. to support
interrupt-based implementations or aggressive segment registration policies). Client
code which registers signal handlers must be careful not to preempt any
GASNet-registered signal handlers (even for seemingly fatal signals such as
SIGABRT) - the only signal which the client may always safely catch is
SIGQUIT.
Any GASNet library implementation can be built in one of the following three
configurations, which affects the behavior of remote-access memory segment registration
during gasnet_attach().
The gasnet.h header file will define the appropriate preprocessor symbol
to indicate which configuration is active.
GASNET_SEGMENT_FAST
gasnet_getMaxLocalSegmentSize().
GASNET_SEGMENT_LARGE
gasnet_getMaxLocalSegmentSize(), and should be comparable to the maximum
total data size allowed for processes on the given system.
GASNET_SEGMENT_EVERYTHING
Implementor's Note:
|
| int gasnet_init (int *argc, char ***argv) |
|
Bootstraps a GASNet job and performs any system-specific setup required. Called by all GASNet-based applications upon startup to bootstrap the nodes,
before any other processing takes place. Must be called before
any calls to any other functions in this specification, and before any
investigation of the command-line parameters passed to the program in
argc/argv, which may be modified or augmented by this call.
The semantics of any code executing before the call to Upon return from All other GASNet calls are prohibited until after a successful gasnet_attach().
|
typedef struct {
gasnet_handler_t index; // == 0 for don't care
void (*fnptr)();
} gasnet_handlerentry_t;
| int gasnet_attach (gasnet_handlerentry_t *table, int numentries, uintptr_t segsize, uintptr_t minheapoffset) |
|
Initializes the GASNet network system and performs any system-specific setup required. table is an array of numentries gasnet_handlerentry_t elements used for registering
active-message handlers provided by the client code. Clients that never explicitly
call the active-message request functions in the core API need not register
any handlers, and may pass a segsize and minheapoffset are used to communicate the desired size and
location of the remote-access memory data segment for the local node that
will be used for all remote accesses (i.e. using the data transfer functions
of the extended API) or as the target of any Long active-messages in
the core API. The client passes the desired size of this area in bytes as
segsize, which must be a multiple of GASNet will attempt to place the data segment in an area of the virtual memory space
whose pages are currently unused (e.g. by calling If the GASNet implementation defines the macro In the segsize and minheapoffset are ignored in the
A successful call acts as a global
barrier and blocks until all other nodes which are part of this parallel job
have successfully called |
Implementor's Note:
|
| uintptr_t gasnet_getMaxLocalSegmentSize () |
|
Retrieve an approximate, optimistic maximum size in bytes for the remote-access memory segment
that may be provided to The return value of this function may depend on
current system resource usage, and may return different values on different nodes
of a job, according to current system utilization. The value returned will
always be a multiple of The value returned is an optimistic approximation of the segment size
which can be acquired by On many implementations, this function will return different values in the
This function has undefined behavior after |
| uintptr_t gasnet_getMaxGlobalSegmentSize () |
|
Returns a global minimum value
that would be returned by a call to This function has undefined behavior after |
| void gasnet_exit (int exitcode) |
|
Terminate the current GASNet job and return the given exitcode to the
console which invoked the job (in a system-specific way). This call is not
a collective operation, meaning any node may call it at any time after
initialization. It causes the system to flush all I/O, release all resources
and terminate the job for all active nodes. If several nodes and/or threads
call it simultaneously with different exit codes within a given
synchronization phase, the result provided to the console will be one of the
provided exit codes (chosen arbitrarily). This function should be called at
the end of GASNet will send a |
| gasnet_node_t gasnet_mynode () |
|
returns the unique, 0-based node index representing this node in the current GASNet job |
| gasnet_node_t gasnet_nodes () |
|
returns the number of nodes in the current GASNet job |
typedef struct {
void *addr;
uintptr_t size;
} gasnet_seginfo_t;
| int gasnet_getSegmentInfo (gasnet_seginfo_t *seginfo_table, int numentries) |
|
Query the segment base addresses and sizes for all the nodes in the job.
seginfo_table is an array of Note that when GASNET_ALIGNED_SEGMENTS=1, the base addresses are guaranteed to be
equal (i.e. all remote-access segments start at the same virtual addresses).
However, in any case
the segment sizes may differ across nodes, and specifically they may differ from the
size requested by the client in the |
| char * gasnet_getenv (const char *name) |
|
Has the same semantics as the POSIX |
Active message communication is formulated as logically matching request and reply operations. Upon receipt of a request message, a request handler is invoked; likewise, when a reply message is received, the reply handler is invoked. Request handlers can reply at most once to the requesting node. If no explicit reply is made, the layer may generate one (to an implicit do-nothing reply handler). Thus a request handler can call reply at most once, and may only reply to the requesting node. Reply handlers cannot request or reply.
Here is a high-level description of a typical active message exchange between two nodes, A and B:
gasnet_AMRequest*() to send a request to B.
The call includes arguments, data payload, the node index of B and the index of
the request handler to run on B when the request arrives
gasnet_AMRequest*() call.
The request handler does some work on the arguments, and usually finishes
by calling gasnet_AMReply*() to issue a reply message before it exits (replying is
optional in GASNet, but required in AM2 - if the request handler does not
reply then no further actions are taken).
gasnet_AMReply*() takes the token passed to the request handler, arguments and data
payload, and the index of the reply handler to run when the reply message
arrives. It does not take a node index because a request handler is only
permitted to send a reply to the requesting node
gasnet_AMReply*() call.
The reply handler does some work on the arguments and then exits. It is
not permitted to send further messages.
The message layer will deliver requests and replies to destination nodes
barring any catastrophic errors (e.g. node crashes). From a sender's point
of view, the request and reply functions block until the message is sent. A
message is defined to be sent once it is safe for the caller to reuse the storage
(registers or memory) containing the message (one notable exception to this
policy is gasnet_RequestLongAsyncM()). In implementations which copy or
buffer messages for transmission, the definition still holds: message sent
means the layer has copied the message and promises to deliver the copy with
its "best effort", and the original message storage may be reused. By best
effort, the message layer promises it will take care of all the details
necessary to transmit the message. These details include any retransmission
attempts and buffering issues on unreliable networks.
However, in either case, sent does not imply received. Once control returns from a request or reply function, clients cannot assume that the message has been received and handled at the destination. The message layer only guarantees that if a request or reply is sent, and, if the receiver occasionally polls for arriving messages, then the message will eventually be received and handled. From a receiver's point of view, a message is defined to be received only once its handler function is invoked. The contents of partially received messages and messages whose handlers have not executed are undefined.
If the client sends an AM request or AM reply to a handler index which has not been registered on the destination node, GASNet will print an implementation-defined error message and terminate the job. It is implementation-defined whether this checking happens on the sending or receiving node.
There are three categories of active messages:
Short Active Message
gasnet_AMMaxArgs())
void handler(gasnet_token_t token,
gasnet_handlerarg_t arg0, ... gasnet_handlerarg_t argM-1);
Medium Active Message
gasnet_AMMaxMedium() bytes in length), that will
be made available to the handler when it is run on the remote node.
void handler(gasnet_token_t token,
void *buf, size_t nbytes,
gasnet_handlerarg_t arg0, ... gasnet_handlerarg_t argM-1);
Long Active Message
gasnet_AMMaxLong() bytes in length) which is destined
for a particular predetermined address in the segment of the remote node
(often implemented using RDMA hardware assistance)
void handler(gasnet_token_t token,
void *buf, size_t nbytes,
gasnet_handlerarg_t arg0, ... gasnet_handlerarg_t argM-1);
For more discussion on these three categories, see the Appendix.
The number of handler arguments (M) is specified upon issuing a request or reply by choosing the request/reply function of the appropriate name. The category of message and value of M used in the request/reply message sends determines the appropriate handler prototype, as detailed above. If a request or reply is sent to a handler whose prototype does not match the requirements as detailed above, the result is undefined.
Implementor's Note:
|
These functions are used to query the maximum size messages of each category supported by a given implementation. These are likely to be implemented as macros for efficiency of client code which uses them (within packing loops, etc.)
| size_t gasnet_AMMaxArgs () |
|
Returns the maximum number of handler arguments (i.e. M) that may be passed
with any AM request or reply function. This value is guaranteed to be at
least |
| size_t gasnet_AMMaxMedium () |
|
Returns the maximum number of bytes that can be sent in the payload of a single medium AM request or reply. This value is guaranteed to be at least 512 bytes on any implementation. |
| size_t gasnet_AMMaxLongRequest () |
|
Returns the maximum number of bytes that can be sent in the payload of a single long AM request. This value is guaranteed to be at least 512 bytes on any implementation. Implementations which use RDMA to implement long messages are likely to support a much larger value. |
| size_t gasnet_AMMaxLongReply () |
|
Returns the maximum number of bytes that can be sent in the payload of a single long AM reply. This value is guaranteed to be at least 512 bytes on any implementation. Implementations which use RDMA to implement long messages are likely to support a much larger value. |
In the function descriptions below, M is to be replaced with a number in [0 ... gasnet_AMMaxArgs()]
| int gasnet_AMRequestShortM ( gasnet_node_t dest, gasnet_handler_t handler, gasnet_handlerarg_t arg0, ..., gasnet_handlerarg_t argM-1 ); |
|
Send a short AM request to node dest,
to run the handler registered on the destination node at handler table index handler,
with the given M arguments.
|
| int gasnet_AMRequestMediumM ( gasnet_node_t dest, gasnet_handler_t handler, void *source_addr, size_t nbytes, gasnet_handlerarg_t arg0, ..., gasnet_handlerarg_t argM-1 ) |
|
Send a medium AM request to node dest, to run the handler registered on the destination node at handler table index handler, with the given M arguments. The message also carries a data payload copied from the local node's memory
space as indicated by source_addr and nbytes
(which need not fall within the registered data segment on the local node).
The value of nbytes must be no larger than the value returned by
Upon receipt, the receiver invokes the appropriate request handler function
with a pointer to temporary storage containing the data payload
(in a buffer which is suitably aligned to hold any datatype), the number
of data bytes transferred, and the M integer arguments. The dynamic scope of
the storage is the same as the dynamic scope of the handler. The data should
be copied if it is needed beyond this scope.
Returns |
| int gasnet_AMRequestLongM ( gasnet_node_t dest, gasnet_handler_t handler, void *source_addr, size_t nbytes, void *dest_addr, gasnet_handlerarg_t arg0, ..., gasnet_handlerarg_t argM-1 ); |
|
Send a long AM request to node dest, to run the handler registered on the destination node at handler table index handler, with the given M arguments. The message also carries a data payload copied from the local node's memory
space as indicated by source_addr and nbytes
(which need not fall within the registered data segment on the local node).
The value of nbytes must be no larger than the value returned by
If dest is the current node (i.e. loopback) and the source and destination
memory overlap, the result is undefined.
|
| int gasnet_AMRequestLongAsyncM ( gasnet_node_t dest, gasnet_handler_t handler, void *source_addr, size_t nbytes, void *dest_addr, gasnet_handlerarg_t arg0, ..., gasnet_handlerarg_t argM-1 ); |
|
|
Implementor's Note:
|
The following active message reply functions may only be called from the context of a running active message request handler, and a reply function may be called at most once from any given request handler (it is an error to do otherwise).
| int gasnet_AMReplyShortM ( gasnet_token_t token, gasnet_handler_t handler, gasnet_handlerarg_t arg0, ..., gasnet_handlerarg_t argM-1 ); |
|
Send a short AM reply to the indicated handler on the requesting node (i.e. the
node responsible for this particular invocation of the request handler), and
include the given M arguments.
Upon receipt, the receiver invokes the appropriate active message reply
handler function with the M integer arguments.
Returns |
| int gasnet_AMReplyMediumM ( gasnet_token_t token, gasnet_handler_t handler, void *source_addr, size_t nbytes, gasnet_handlerarg_t arg0, ..., gasnet_handlerarg_t argM-1 ); |
|
Send a medium AM reply to the indicated handler on the requesting node (i.e. the
node responsible for this particular invocation of the request handler), with
the given M arguments and given data payload copied from the local node's
memory space (source_addr need not fall within the registered data segment
on the local node).
The value of nbytes must be no larger than the value returned by
Upon receipt, the receiver invokes the appropriate reply handler function
with a pointer to temporary storage containing the data payload, the number
of data bytes transferred, and the M integer arguments. The dynamic scope of
the storage is the same as the dynamic scope of the handler. The data should
be copied if it is needed beyond this scope.
Returns |
| int gasnet_AMReplyLongM ( gasnet_token_t token, gasnet_handler_t handler, void *source_addr, size_t nbytes, void *dest_addr, gasnet_handlerarg_t arg0, ..., gasnet_handlerarg_t argM-1 ); |
|
Send a long AM reply to the indicated handler on the requesting node (i.e. the
node responsible for this particular invocation of the request handler), with
the given M arguments and given data payload copied from the local node's
memory space (source_addr need not fall within the registered data segment
on the local node).
The value of nbytes must be no larger than the value returned by
Upon receipt, the receiver
invokes the appropriate reply handler function with a pointer into the
memory segment where the data was placed, the number of data bytes
transferred, and the M integer arguments.
Returns |
| int gasnet_AMPoll () |
|
An explicit call to service the network, process pending messages and run
handlers as appropriate.
Most of the message-sending primitives in GASNet poll the network
implicitly.
Purely polling-based implementations of GASNet may require occasional calls
to this function to ensure progress of remote nodes during compute-only
loops. Any client code which spin-waits for the arrival of a message should
call this function within the spin loop to optimize response time.
This call may be a no-op on some implementations (e.g. purely interrupt-based
implementations).
Returns |
#define GASNET_BLOCKUNTIL(cond) ???
This is a macro which implements a busy-wait/blocking polling loop in the way most efficient for the current GASNet core implementation. The macro blocks execution of the current thread and services the network until the provided condition becomes true. cond is an arbitrary C expression which will be evaluated by the macro one or more times as active messages arrive until the condition evaluates to a non-zero value. cond is an expression whose value is altered by the execution of an AM handler which the client thread is waiting for - GASNet may safely assume that the value of cond will only change while an AM handler is executing.
Example usage:
int doneflag = 0;
gasnet_AMRequestShort1(..., &doneflag); // reply handler sets doneflag to 1
GASNET_BLOCKUNTIL(doneflag == 1);
Note that code like this would be illegal and could cause node 0 to sleep forever:
static int doneflag = 0;
node 0: node 1:
GASNET_BLOCKUNTIL(doneflag == 1); gasnet_put_val(0, &doneflag, 1, sizeof(int));
because gasnet_put_val (and other extended API functions) might not be
implemented using AM handlers. Also note that cond may be evaluated
concurrently with handler execution, so the client is responsible for negotiating
any atomicity concerns between the cond expression and handlers (for example, protecting
both with a handler-safe lock if the cond expression reads two or more values which are all
updated by handlers). Finally, note that unsynchronized handler code which modifies
one or more locations and then performs a flag write to signal a different thread
may need to execute a local memory barrier before the flag write to ensure correct
ordering on non-sequentially-consistent SMP hardware.
Implementor's Note:
|
| int gasnet_AMGetMsgSource (gasnet_token_t token, gasnet_node_t *srcindex) |
|
Can be called by handlers to query the source of the message being handled.
The token argument must be the token passed into the handler on entry.
Returns |
Handlers may run asynchronously with respect to the main computation (in an
implementation which uses interrupts to run some or all handlers), and they
may run concurrently with each other on separate threads (e.g. in a CLUMP
implementation where several threads may be polling the network at once). An
implementation using interrupts may result in handler code running within a
signal handler context. Some implementations may even choose to run handlers
on a separate private thread created by GASNet (making handlers asynchronous
with respect to all client threads). Note that polling-based GASNet
implementations are likely to poll (and possibly run handlers) from within
any GASNet call (i.e. not just gasnet_AMPoll()). Because of all this,
handler code should run quickly and to completion without making blocking
calls, and should not make assumptions about the context in which it is
being run (special care must be taken to ensure safety in a signal handler
context, see below).
Regardless, handlers themselves are not interruptible - any given thread
will only be running a single AM handler at a time and will never be
interrupted to run another AM handler (there is one exception to this rule -
the gasnet_AMReply*() call in a request handler may cause reply handlers to
run synchronously, which may be necessary to avoid deadlock in some
implementations. This should not be a problem since gasnet_AMReply*() is
often the last action taken by a request handler). Handlers are
specifically prohibited from initiating random network communication to
prevent deadlock - request handlers must generate at most one reply (to the
requestor) and make no other communication calls (including polling), and
reply handlers may not communicate or poll at all.
The asynchronous nature of handlers requires two mechanisms to make them safe: a mechanism to ensure signal safety for GASNet implementations using interrupt-based mechanisms, and a locking mechanism to allow atomic updates from handlers to data structures shared with the client threads and other handlers.
(see http://www.cs.berkeley.edu/~bonachea/upc/ for a more detailed discussion on handler atomicity)
Traditionally, code running in signal handler context is extremely
circumscribed in what it can do: e.g. none of the standard pthreads/System V
synchronization calls are on the list of signal-safe functions (for such a
list see POSIX System Interfaces 2.4, IEEE Std 1003.1-2001).
Note that even most "thread-safe" libraries will break or deadlock if
called from a signal handler by the same thread currently executing a
different call to that library in an earlier stack frame. One specific case
where this is likely to arise in practice is calls to malloc()/free(). To
overcome these limitations, and allow our handlers to be more useful, the
normal limitations on signal handlers will be avoided by allowing the client
thread to temporarily disable the network interrupts that run handlers. All
function calls that are not signal-safe and could possibly access state
shared by functions also called from handlers MUST be called within a GASNet
"No-Interrupt Section":
| void gasnet_hold_interrupts () | |
| void gasnet_resume_interrupts () |
|
|
There is a strict set of conventions governing the use of No-Interrupt Sections which must be followed in order to ensure correct operation on all GASNet implementations. Clients which violate any of these rules may be subject to intermittent crashes, fatal errors or network deadlocks.
gasnet_hold_interrupts() and gasnet_resume_interrupts() need not be
called from within a handler context - handlers are run within an implicit
No-Interrupt Section, and gasnet_hold_interrupts() and gasnet_resume_interrupts()
calls are ignored within a handler context.
Note that due to the previous rule, these are also the only GASNet functions that may legally be called within a handler context (andgasnet_mynode(),gasnet_nodes(),gasnet_hsl_*(),gasnet_exit(),gasnet_AMReply*()
gasnet_AMReply*()
is only legal in a request handler).
gasnet_hsl_lock within a No-Interrupt Section (subject to the rules
in section see Restrictions on Handler-Safe Locks).
Implementor's Note:
|
In order to support handlers atomically updating data structures accessed by the main-line client code and other handlers, GASNet provides the Handler-Safe Lock (HSL) mechanism. As the name implies, these are a special kind of lock which are distinguished as being the only type of lock which may be safely acquired from a handler context. There is also a set of restrictions on their usage which allows this to be safe (see below). All lock-protected data structures in the client that need to be accessed by handlers should be protected using a Handler-Safe Lock (i.e. instead of a standard POSIX mutex).
gasnet_hsl_t is an opaque type representing a Handler-Safe Lock.
HSL's operate analogously to POSIX mutexes, in that they are always
manipulated using a pointer.
gasnet_hsl_t hsl = GASNET_HSL_INITIALIZER;
| void gasnet_hsl_init (gasnet_hsl_t *hsl) | |
| void gasnet_hsl_destroy (gasnet_hsl_t *hsl) |
|
Similarly to POSIX mutexes, HSL's can be created in two ways. They can be
statically declared and initialized using the |
| void gasnet_hsl_lock (gasnet_hsl_t *hsl) | |
| int gasnet_hsl_trylock (gasnet_hsl_t *hsl) | |
| void gasnet_hsl_unlock (gasnet_hsl_t *hsl) |
|
Lock and unlock HSL's.
Note that under the All HSL locking/unlocking calls must follow the usage rules documented in the next section. |
There is a strict set of conventions governing the use of HSL's which must be followed in order to ensure correct operation on all GASNet implementations. Amongst other things, the restrictions are designed to ensure that HSL's are always held for a strictly bounded amount of time, to ensure that acquiring them from within a handler can't lead to deadlock. Clients which violate any of these rules may be subject to intermittent crashes, fatal errors or network deadlocks.
gasnet_hold_interrupts() and gasnet_resume_interrupts()
are ignored while holding an HSL.
gasnet_AMReply*()
gasnet_hsl_lock()
or gasnet_hsl_trylock(hsl) on a
lock already held by the current thread) and attempting to do so will lead
to undefined behavior. It is permitted for a thread to acquire more than
one HSL, although the traditional cautions about the possibility of deadlock
in the presence of multiple locks apply (e.g. the common solution is to
define a total order on locks and always acquire them in a monotonically
ascending sequence).
Implementor's Note:
|
Errors in calls to the extended API are considered fatal and abort the job
(by sending a SIGABORT signal) after printing an appropriate error message.
These comments apply to all put/get functions:
gasnet_attach()), or the results are undefined
gasnet_node_t) and a void * virtual memory address,
which logically represent a global pointer to the given address on the given
node. These global pointers need not be remote - the node rank passed to
these functions may in fact be the rank of the current node -
implementations must support this form of loopback, and should probably
attempt to optimize it by avoiding network traffic for such purely local
operations.
| void gasnet_get (void *dest, gasnet_node_t node, void *src, size_t nbytes) | |
| void gasnet_put (gasnet_node_t node, void *dest, void *src, size_t nbytes) |
|
Blocking get/put operations for aligned data. The get operation fetches nbytes bytes from the address src on node node and places them at dest in the local memory space. The put operation sends nbytes bytes from the address src in the local address space, and places them at the address dest in the memory space of node node. A call to these functions blocks until the transfer is complete, and the contents of the destination memory are undefined until it completes. If the contents of the source memory change while the operation is in progress the result will be implementation-specific. The src and dest addresses (whether local or remote) must be properly aligned for accessing objects of size nbytes. nbytes must be >= 0 and has no maximum size, but implementations will likely optimize for small powers of 2. |
| void gasnet_get_bulk (void *dest, gasnet_node_t node, void *src, size_t nbytes) | |
| void gasnet_put_bulk (gasnet_node_t node, void *dest, void *src, size_t nbytes) |
|
Blocking get/put operations for bulk (unaligned) data. These function similarly to the aligned get/put operations above, except the data is permitted to be unaligned, and implementations are likely to optimize for larger sizes of nbytes. |
| void gasnet_memset (gasnet_node_t node, void *dest, int val, size_t nbytes) |
|
Blocking operation that has the same effect as if the dest node had executed the POSIX call
|
The following functions provide non-blocking, split-phase memory access to shared data.
All such non-blocking operations require an initiation (generally a put or get) and a subsequent synchronization on the completion of that operation before the result is guaranteed.
There are two basic categories of non-blocking operations, defined by the synchronization mechanism used:
Successful synchronization of a non-blocking get operation means the local result is ready to be examined, and will contain a value held by the source location at some time in the interval between the call to the initiation function and the successful completion of the synchronization (note this specifically allows implementations to delay the underlying read until the synchronization operation is called, provided they preserve the blocking semantics of the synchronization function).
Successful synchronization of a put operation means the source data has been written to the destination location and get operations issued subsequently by any thread (or load instructions issued by the destination node) will receive the new value or a subsequently written value (assuming no other threads are writing the location)
Note that the order in which non-blocking operations complete is intentionally unspecified - the system is free to coalesce and/or reorder non-blocking operations with respect to other blocking or non-blocking operations, or operations initiated from a separate thread - the only ordering constraints that must be satisfied are those explicitly enforced using the synchronization functions (i.e. the non-blocking operation is only guaranteed to occur somewhere in the interval between initiation and successful synchronization on that operation).
Implementors should attempt to make the non-blocking initiation operations return as quickly as possible - however in some cases (e.g. when a large number of non-blocking operations have been issued or the network is otherwise busy) it may be necessary to block temporarily while waiting for the network to become available. In any case, all implementations must support at least 2^16-1 non-blocking operations in-progress - that is, the client is free to issue up to 2^16-1 non-blocking operations before issuing a sync operation, and the implementation must handle this correctly without deadlock or livelock.
The explicit-handle non-blocking data transfer functions return a
gasnet_handle_t value to represent the non-blocking operation in flight.
gasnet_handle_t is an opaque scalar type whose contents are implementation-defined,
with one exception - every implementation must provide a scalar value corresponding
to an "invalid" handle (GASNET_INVALID_HANDLE) and furthermore this value
must be the result of setting all the bytes in the gasnet_handle_t datatype
to zero. Implementators are free to define the gasnet_handle_t type to be
any reasonable and appropriate size, although they are recommended to use a
type which fits within a single standard register on the target
architecture. In any case, the datatype should be wide enough to express at least
2^16-1
different handle values, to prevent limiting the number of
non-blocking operations in progress due to the number of handles available. It
is legal for clients to pass gasnet_handle_t values into function callees
or back to function callers.
In the case of multithreaded clients (GASNET_PAR or GASNET_PARSYNC),
gasnet_handle_t values are thread-specific. In other words, it is an error
to obtain a handle value by initiating a non-blocking operation on one
thread, and later pass that handle into a synchronization function from a
different thread.
Any explicit-handle, non-blocking operation may return GASNET_INVALID_HANDLE
to indicate it was possible to complete the operation immediately without
blocking (e.g. operations where the "remote" node is actually the local
node)
It is always an error to discard the gasnet_handle_t value for an
explicit-handle operation in-flight - i.e. to initiate an operation and never
synchronize on its completion.
| gasnet_handle_t gasnet_get_nb (void *dest, gasnet_node_t node, void *src, size_t nbytes) | |
| gasnet_handle_t gasnet_put_nb (gasnet_node_t node, void *dest, void *src, size_t nbytes) |
|
Non-blocking get/put functions for aligned data. These functions operate
similarly to their blocking counterparts, except they initiate a
non-blocking operation and return immediately with a handle ( |
| gasnet_handle_t gasnet_get_nb_bulk (void *dest, gasnet_node_t node, void *src, size_t nbytes) | |
| gasnet_handle_t gasnet_put_nb_bulk (gasnet_node_t node, void *dest, void *src, size_t nbytes) |
|
Non-blocking get/put functions for bulk (unaligned) data. For the put version, the source memory may not be safely overwritten until a successful synchronization for the operation. If the contents of the source memory change while the operation is in progress the result will be implementation-specific. These otherwise behave identically to the non-bulk variants (but are likely to be optimized for large transfers). |
| gasnet_handle_t gasnet_memset_nb (gasnet_node_t node, void *dest, int val, size_t nbytes) |
|
Non-blocking operation that has the same effect as if the dest node had executed the POSIX call
The synchronization behavior is identical to a non-blocking, explicit-handle put operation (the
|
GASNet supports two basic types of synchronization for non-blocking
operations - trying (polling) and waiting (blocking). All explicit-handle
synchronization functions take one or more gasnet_handle_t values as input
and either return an indication of whether the operation has completed or
block until it completes.
| void gasnet_wait_syncnb (gasnet_handle_t handle) | |
| int gasnet_try_syncnb (gasnet_handle_t handle) |
|
Synchronize on the completion of a single specified explicit-handle
non-blocking operation that was initiated by the calling thread.
It is legal to pass It is an error to pass a |
| void gasnet_wait_syncnb_all (gasnet_handle_t *handles, size_t numhandles) | |
| int gasnet_try_syncnb_all (gasnet_handle_t *handles, size_t numhandles) |
|
Synchronize on the completion of an array of non-blocking explicit-handle
operations (all of which were initiated by this thread). numhandles
specifies the number of handles in the provided array of handles.
Both functions will modify the provided array to reflect completions -
handles whose operations have completed are overwritten with the value
It is legal to pass the value |
| void gasnet_wait_syncnb_some (gasnet_handle_t *handles, size_t numhandles) | |
| int gasnet_try_syncnb_some (gasnet_handle_t *handles, size_t numhandles) |
|
These operate analogously to the Both functions ignore |
gasnet_memset_nbi
| void gasnet_get_nbi (void *dest, gasnet_node_t node, void *src, size_t nbytes) | |
| void gasnet_put_nbi (gasnet_node_t node, void *dest, void *src, size_t nbytes) | |
| void gasnet_get_nbi_bulk (void *dest, gasnet_node_t node, void *src, size_t nbytes) | |
| void gasnet_put_nbi_bulk (gasnet_node_t node, void *dest, void *src, size_t nbytes) | |
| void gasnet_memset_nbi (gasnet_node_t node, void *dest, int val, size_t nbytes) |
|
Non-blocking get/put functions for aligned and unaligned (bulk) data. These functions operate similarly to their explicit-handle counterparts, except they do not return a handle and must be synchronized using the implicit-handle synchronization operations. The contents of the destination memory address are undefined until a synchronization completes successfully for the non-blocking operation. As with the explicit-handle variants, the source memory for the non-bulk put operation may be safely overwritten once the initiation function returns, but the bulk put version requires the source memory to remain unchanged until the operation has been successfully completed using a synchronization.
|
The following functions are used to synchronize implicit-handle non-blocking operations.
In the case of multithreaded clients, implicit-handle synchronization functions only synchronize the implicit-handle non-blocking operations initiated from the calling thread. Operations initiated by other threads sharing the GASNet interface proceed independently and are not synchronized. Implicit-handle synchronization functions will synchronize operations initiated within other function frames by the calling thread (but this cannot affect the correctness of correctly synchronized code).
gasnet_try_syncnbi_puts, gasnet_try_syncnbi_all
| void gasnet_wait_syncnbi_gets () | |
| void gasnet_wait_syncnbi_puts () | |
| void gasnet_wait_syncnbi_all () | |
| int gasnet_try_syncnbi_gets () | |
| int gasnet_try_syncnbi_puts () | |
| int gasnet_try_syncnbi_all () |
|
These functions implicitly specify a set of non-blocking operations on which
to synchronize. They synchronize on a set of outstanding non-blocking
implicit-handle operations initiated by this thread - either all such gets,
all such puts, or all such puts and gets (where outstanding is defined as
all those implicit-handle operations which have been initiated (outside an
access region) but not yet completed through a successful implicit
synchronization). The wait variants block until all operations in this
implicit set have completed (indicating these operations have been
successfully synchronized). The try variants test whether all operations in
the implicit set have completed, and return If there are no outstanding implicit-handle operations, these
synchronization functions all return immediately (with |
Implementor's Note:
|
In some cases, it may be useful or desirable to initiate a number of
non-blocking shared-memory operations (possibly without knowing how many at
compile-time) and synchronize them at a later time using a single, fast
synchronization.
Simple implicit handle synchronization may not be appropriate for this
situation if there are intervening implicit accesses which are not to be
synchronized.
This situation could be handled using explicit-handle non-blocking
operations and a list synchronization (e.g. gasnet_wait_syncnb_all()), but
this may not be desirable because it requires managing an array of handles
(which could have negative cache effects on performance, or could be
expensive to allocate when the size is not known until runtime).
To handle these cases, we provide "implicit access region" synchronization,
described below.
| void gasnet_begin_nbi_accessregion (); | |
| gasnet_handle_t gasnet_end_nbi_accessregion (); |
|
|
Sample code:
gasnet_begin_nbi_accessregion(); // begin the access region
gasnet_put_nbi(...); // becomes assoc. with access region
while (...) {
gasnet_put_nbi(...); // becomes assoc. with access region
}
// unrelated explicit-handle operation not assoc. with access region
h2 = gasnet_get_nb(...);
gasnet_wait_syncnb(h2);
// end the access region and get the handle
handle = gasnet_end_nbi_accessregion();
.... // other code, which may include unrelated implicit-handle
// operations+syncs, or other regions, etc
// wait for all the operations assoc. with access region to complete
gasnet_wait_syncnb(handle);
Register-memory operations allow client code to avoid forcing communicated data to pass through the local memory system. Some interconnects may be able to take advantage of this capability and launch remote puts directly from registers or recieve remote gets directly into registers.
| void gasnet_put_val (gasnet_node_t node, void *dest, gasnet_register_value_t value, size_t nbytes); | |
| gasnet_handle_t gasnet_put_nb_val (gasnet_node_t node, void *dest, gasnet_register_value_t value, size_t nbytes); | |
| void gasnet_put_nbi_val (gasnet_node_t node, void *dest, gasnet_register_value_t value, size_t nbytes); |
|
Register-to-remote-memory put - these functions take the value to be put as
input parameter to avoid forcing outgoing values to local memory in client
code.
Otherwise, the behavior is identical to the memory-to-memory versions of put above.
Requires: |
| gasnet_register_value_t gasnet_get_val (gasnet_node_t node, void *src, size_t nbytes); |
|
This function returns the fetched value to avoid
forcing incoming values through local memory (on architectures
which pass the return value in a register).
Otherwise, the behavior is identical to the memory-to-memory blocking get.
Requires: |
This operates similarly to the blocking form of value get, but is split-phase. Non-blocking value gets are synchronized independently of all other operations in GASNet.
typedef ??? gasnet_valget_handle_t;
| gasnet_valget_handle_t gasnet_get_nb_val (gasnet_node_t node, void *src, size_t nbytes); | |
| gasnet_register_value_t gasnet_wait_syncnb_valget (gasnet_valget_handle_t handle); |
|
|
The following functions can be used to execute a parallel split-phase
barrier with the given barrier identifier across all nodes in the job.
Note that the barrier wait/notify functions should only be called once (i.e.
by one representative thread) on each node per barrier phase.
The client must synchronize its own accesses to the barrier functions and
ensure that only one thread is ever inside a GASNet barrier function at a
time (esp. gasnet_barrier_try()).
#define GASNET_BARRIERFLAG_ANONYMOUS ???
#define GASNET_BARRIERFLAG_MISMATCH ???
| void gasnet_barrier_notify (int id, int flags) |
|
Execute the notification for a split-phase barrier, with a barrier value id. This is a non-blocking operation that completes immediately after noting the barrier value. No synchronization is performed on outstanding non-blocking memory operations. Generates a fatal error if this is the second call to
If flags == 0 then this is a "named" barrier notify that carries the given
id value.
If flags == |
| int gasnet_barrier_wait (int id, int flags) |
|
Execute the wait for a split-phase barrier, with a barrier value.
This is a blocking operation that returns only after all remote nodes have
called Generates a fatal error if there were no preceding calls to
Returns |
| int gasnet_barrier_try (int id, int flags) |
|
Generates a fatal error if there were no preceding calls to
|
When compiled in the GASNET_PAR or GASNET_PARSYNC configurations, GASNet is
capable of handling multiple client threads. It is likely that GASNet
implementations will need to distinguish these threads, specifically they
may need to store some metadata associated with each client thread.
Unfortunately, the overhead of discovering the identity of a particular
client thread making a GASNet call (hereafter termed "thread discovery") can
have a non-trivial overhead on some threading systems (e.g. the cost of
calling pthread_self() or pthread_getspecific()). Many of the simpler GASNet
functions could have their performance dominated by this cost if they need
to perform thread discovery on every call.
The following macros provide a way for the client to amortize the cost of
thread discovery over many GASNet calls made by the same thread. This is an
optimization which is totally optional - clients need not make any of the
calls below to have a working system, although GASNet performance is likely
to suffer without it in a GASNET_PAR or GASNET_PARSYNC configuration.
typedef void *gasnet_threadinfo_t;
gasnet_threadinfo_t is an opaque pointer representing the internal GASNet
metadata associated with a particular client thread.
#define GASNET_GET_THREADINFO() ???
Returns a value of type gasnet_threadinfo_t which represents the GASNet
internal metadata associated with the current client thread. This
gasnet_threadinfo_t value can be passed into or out of functions and may be
posted for GASNet's use with GASNET_POST_THREADINFO(). May be called from
anywhere in the client program, at any time after GASNet initialization. It
is erroneous to hand-off this gasnet_threadinfo_t value to a different
client thread.
#define GASNET_POST_THREADINFO(info) ???
This macro may optionally be placed at the top of functions which make
calls to GASNet. It has no runtime semantics, but it may provide a
performance boost on some implementations (especially in functions which
make multiple calls to the extended API - e.g. it provides the
implementation with a place for minimal per-function initialization or
temporary storage that may be helpful in amortizing implementation-specific
overheads).
When used, it must appear only at the very beginning of a function or block
(before any declarations or calls to the API in that function). It may not
appear as a global declaration. The info argument must be a
gasnet_threadinfo_t value acquired from a previous call to
GASNET_GET_THREADINFO() on this thread.
#define GASNET_BEGIN_FUNCTION() ???
A convenience macro that may optionally be placed at the top of functions which repeatedly make GASNet calls, to amortize the overhead of thread discovery on some implementations.
It has behavior equivalent to GASNET_POST_THREADINFO(GASNET_GET_THREADINFO()),
however some implementations may choose to lazily postpone performing thread discovery
until the first place where it is actually needed.
| int gasnet_set_waitmode (int wait_mode) |
|
Optional call which gives the GASNet implementation a hint about how aggressively threads within blocking GASNet calls should contend for CPU resources. wait_mode must be one of the following recognized values:
Wait mode is a per-node hint which is permitted to differ across GASNet nodes. Returns |
The GASNet core API was originally based on Active Messages 2.0 (as described in A. Mainwaring and D. Culler in "Active Message Applications Programming Interface and Communication Subsystem Organization"), however we've removed some of the generality which is not required (and can lead to performance degradation and more implementation effort), and stripped it down to the bare essentials required for active messages in a purely SPMD environment. The final spec more closely resembles the "Generic Active Message Interface Specification v.1.1", by D.Culler et al., however we describe the differences from AM2.0 for readers familiar with that specification (and because we envision a number of the GASNet core implementations being simply a thin wrapper over the existing AM2.0 implementations on a number of platforms).
Here are a summary of the changes (informal style.. this is not really part of the spec):
AM_Init, AM_Terminate,
AM_AllocateBundle, AM_AllocateEndpoint, AM_FreeEndpoint, AM_FreeBundle,
AM_MoveEndpoint, AM_GetXferM, AM_GetDestEndpoint
gasnet_attach(), and the
maximum number of handlers is fixed at 256 (including handler 0, the error
handler)
AM_SetHandler and
AM_SetHandlerAny, AM_GetNumHandlers, AM_SetNumberHandlers, AM_MaxNumHandlers
gasnet_attach() (using a uintptr_t to
allow entire VA space)
AM_SetSeg and AM_MaxSegLength
(still have AM_GetSeg)
gasnet_attach requests a size larger than what underlying
AM_SetSeg can provide, then we turn off large AM Xfers and emulate
gasnet_Xfer using medium messages)
dest_offset argument to the Xfer functions is changed to a void *
address
AM_Map, AM_MapAny,
AM_Unmap, AM_SetTag, AM_GetTag, AM_GetTranslationName, AM_GetTranslationTag,
AM_GetTranslationInUse, AM_MaxNumTranslations, AM_GetNumTranslations,
AM_SetNumTranslations, AM_GetMsgTag
en_t * argument to AM_GetSourceEndpoint is now an gasnet_node_t *
and returns the node rank of the sender (the now-opaque token could be
implemented as the integer node index itself, although we allow
implementations to still use it as a ptr to metadata if required)
AM_RequestXferAsyncM has more useful semantics (may block)
AM_SetExpectedResources no longer exists
AM_PAR (multi-threaded) access mode
(GASNET_PAR configuration)
(void*)'s can be sent)
cons: handler code needs to be rewritten for 64-bit platforms to perform packing/unpacking
AM_GetEventMask and AM_SetEventMask no longer exist
AM_WaitSema is replaced with GASNET_BLOCKUNTIL()
ReplyXfer in favor of GetXfer
ReplyXfer's (with software
flow control & reliability)
AM_MaxLong into AM_MaxLongRequest, and
AM_MaxLongReply
GetXfer doesn't add any expressiveness - really want a way to get
from remote segment into arbitrary local memory address
Newcomers to Active Messages and GASNet occasionally express confusion over the concepts of Short, Medium and Long AM's. Despite the somewhat misleading naming convention, the three categories of messages may actually bear only a loose correllation to the actual message/data sizes. The important distinctions are semantic, and sufficiently minor that one might imagine replacing the three categories with a single, more general type of AM that provides the functionality of each GASNet AM category as a special case.
Specifically, a GASNet Short AM can be seen as a special case of a Medium or Long AM where the payload has length zero. Furthermore, the only important semantic distinction between Medium and Long AM's are that Medium AM's provide the payload to the handler in a temporary network buffer, whereas Long AM's write the payload (often using RDMA) to a sender-specified location in the user memory segment of the target node before running the handler (each semantic is useful for different usage scenarios).
Hence, some users may find it helpful to consider building "unified" AM request/reply functions such as suggested below:
/* unified request function */
int unified_AMRequestM(
gasnet_node_t dest, gasnet_handler_t handler,
void *buf, size_t buf_len,
void *dest_addr,
int32 arg0, int32 arg1, ...) {
if (buf == NULL)
return gasnet_AMRequestShortM(dest,handler,arg0,arg1,...);
else if (dest_addr == NULL)
return gasnet_AMRequestMediumM(dest,handler,buf,buf_len,arg0,arg1,...);
else
return gasnet_AMRequestLongM(dest,handler,buf,buf_len,dest_addr,arg0,arg1,...);
}
gasnet_AMMaxArgs()
dest_addr == NULL requires buf_len <= gasnet_AMMaxMedium(),
and the payload is delivered in a temporary buffer
dest_addr != NULL requires buf_len <= gasnet_AMMaxLongRequest()
(or gasnet_AMMaxLongReply() for replies),
and the payload is written into the target node segment at dest_addr
buf == NULL:
void handler_nopayload(gasnet_token_t token,
gasnet_handlerarg_t arg0, gasnet_handlerarg_t arg1...);
buf != NULL:
void handler_withpayload(gasnet_token_t token,
void *buf, size_t buf_len,
gasnet_handlerarg_t arg0, gasnet_handlerarg_t arg1...);
GASNET_ALIGNED_SEGMENTS: gasnet_ErrorName
gasnet_AMGetMsgSource: gasnet_AMGetMsgSource
gasnet_AMMaxArgs: gasnet_AMMaxArgs
gasnet_AMMaxLongReply: gasnet_AMMaxLongReply
gasnet_AMMaxLongRequest: gasnet_AMMaxLongRequest
gasnet_AMMaxMedium: gasnet_AMMaxMedium
gasnet_AMPoll: gasnet_AMPoll
gasnet_AMReplyLongM: gasnet_AMReplyLongM
gasnet_AMReplyMediumM: gasnet_AMReplyMediumM
gasnet_AMReplyShortM: gasnet_AMReplyShortM
gasnet_AMRequestLongAsyncM: gasnet_AMRequestLongAsyncM
gasnet_AMRequestLongM: gasnet_AMRequestLongM
gasnet_AMRequestMediumM: gasnet_AMRequestMediumM
gasnet_AMRequestShortM: gasnet_AMRequestShortM
gasnet_attach: gasnet_attach
gasnet_barrier_notify: gasnet_barrier_notify
gasnet_barrier_try: gasnet_barrier_try
gasnet_barrier_wait: gasnet_barrier_wait
GASNET_BARRIERFLAG_ANONYMOUS: Barriers
GASNET_BARRIERFLAG_MISMATCH: Barriers
GASNET_BEGIN_FUNCTION: GASNET_BEGIN_FUNCTION
gasnet_begin_nbi_accessregion: gasnet_begin_nbi_accessregion
GASNET_BLOCKUNTIL: GASNET_BLOCKUNTIL
GASNET_CONFIG_STRING: gasnet_ErrorName
gasnet_end_nbi_accessregion: gasnet_begin_nbi_accessregion
GASNET_ERR_BAD_ARG: Introduction
GASNET_ERR_BARRIER_MISMATCH: Introduction
GASNET_ERR_NOT_INIT: Introduction
GASNET_ERR_NOT_READY: Introduction
GASNET_ERR_RESOURCE: Introduction
gasnet_ErrorDesc: gasnet_ErrorName
gasnet_ErrorName: gasnet_ErrorName
gasnet_exit: gasnet_exit
gasnet_get: gasnet_get
gasnet_get_bulk: gasnet_get_bulk
gasnet_get_nb: gasnet_get_nb
gasnet_get_nb_bulk: gasnet_get_nb_bulk
gasnet_get_nb_val: gasnet_get_nb_val
gasnet_get_nbi: gasnet_get_nbi
gasnet_get_nbi_bulk: gasnet_get_nbi
GASNET_GET_THREADINFO: GASNET_GET_THREADINFO
gasnet_get_val: gasnet_get_val
gasnet_getenv: gasnet_getenv
gasnet_getMaxGlobalSegmentSize: gasnet_getMaxGlobalSegmentSize
gasnet_getMaxLocalSegmentSize: gasnet_getMaxLocalSegmentSize
gasnet_getSegmentInfo: gasnet_getSegmentInfo
gasnet_hold_interrupts: gasnet_hold_interrupts
gasnet_hsl_destroy: gasnet_hsl_init
gasnet_hsl_init: gasnet_hsl_init
GASNET_HSL_INITIALIZER: gasnet_hsl_init
gasnet_hsl_lock: gasnet_hsl_lock
gasnet_hsl_t: gasnet_hsl_t
gasnet_hsl_trylock: gasnet_hsl_lock
gasnet_hsl_unlock: gasnet_hsl_lock
gasnet_init: gasnet_init
GASNET_MAXNODES: gasnet_ErrorName
gasnet_memset: gasnet_memset
gasnet_memset_nb: gasnet_memset_nb
gasnet_memset_nbi: gasnet_get_nbi
gasnet_mynode: gasnet_mynode
gasnet_nodes: gasnet_nodes
GASNET_OK = 0 (no error): Introduction
GASNET_PAGESIZE: gasnet_ErrorName
GASNET_PAR: Introduction
GASNET_PARSYNC: Introduction
GASNET_POST_THREADINFO: GASNET_POST_THREADINFO
gasnet_put: gasnet_get
gasnet_put_bulk: gasnet_get_bulk
gasnet_put_nb: gasnet_get_nb
gasnet_put_nb_bulk: gasnet_get_nb_bulk
gasnet_put_nb_val: gasnet_put_val
gasnet_put_nbi: gasnet_get_nbi
gasnet_put_nbi_bulk: gasnet_get_nbi
gasnet_put_nbi_val: gasnet_put_val
gasnet_put_val: gasnet_put_val
GASNET_RELEASE_VERSION_MAJOR: gasnet_ErrorName
GASNET_RELEASE_VERSION_MINOR: gasnet_ErrorName
GASNET_RELEASE_VERSION_PATCH: gasnet_ErrorName
gasnet_resume_interrupts: gasnet_hold_interrupts
GASNET_SEGMENT_EVERYTHING: Job Control Interface
GASNET_SEGMENT_FAST: Job Control Interface
GASNET_SEGMENT_LARGE: Job Control Interface
GASNET_SEQ: Introduction
gasnet_set_waitmode: gasnet_set_waitmode
GASNET_SPEC_VERSION_MAJOR: gasnet_ErrorName
GASNET_SPEC_VERSION_MINOR: gasnet_ErrorName
gasnet_try_syncnb: gasnet_wait_syncnb
gasnet_try_syncnb_all: gasnet_wait_syncnb_all
gasnet_try_syncnb_some: gasnet_wait_syncnb_some
gasnet_try_syncnbi_all: gasnet_wait_syncnbi_gets
gasnet_try_syncnbi_gets: gasnet_wait_syncnbi_gets
gasnet_try_syncnbi_puts: gasnet_wait_syncnbi_gets
gasnet_valget_handle_t: Non-Blocking Value Get (explicit-handle)
GASNET_VERSION (deprecated): gasnet_ErrorName
GASNET_WAIT_BLOCK: gasnet_set_waitmode
GASNET_WAIT_SPIN: gasnet_set_waitmode
GASNET_WAIT_SPINBLOCK: gasnet_set_waitmode
gasnet_wait_syncnb: gasnet_wait_syncnb
gasnet_wait_syncnb_all: gasnet_wait_syncnb_all
gasnet_wait_syncnb_some: gasnet_wait_syncnb_some
gasnet_wait_syncnb_valget: gasnet_get_nb_val
gasnet_wait_syncnbi_all: gasnet_wait_syncnbi_gets
gasnet_wait_syncnbi_gets: gasnet_wait_syncnbi_gets
gasnet_wait_syncnbi_puts: gasnet_wait_syncnbi_gets