Copyright © 2001 Southern Storm Software, Pty Ltd.
Permission to distribute unmodified copies of this work is hereby granted.
Implementations of the Common Language Runtime (CLR) on Windows provide a rich set of tools to converting between the well-defined world of CLR conventions, and the ill-defined world of Windows API conventions. Attributes are added to method declarations that identify the DLL that contains the function, and the mechanism to use to convert CLR types into parameter types for the function. These attributes are converted into flags and token structures within the metadata of the program.
Implementations of the CLR on Unix will need a similar set of tools. However,
the problem is exacerbated by the variations in CPU types that are common
in the Unix community. Windows API's typically use types such as
WORD and DWORD, which are fixed in size for
all platforms. Unix API's typically use types such as int
and long which are variable in size.
This document proposes a standard mechanism for "PInvoke" under Unix. The goal is to create a system that "just works", even where variations in type sizes occur.
| 32/32 system | int and long
are both 32 bits in size. (Sometimes
referred to as "ILP32") |
| 32/64 system | int is 32 bits in size and
long is 64 bits in size.
(Sometimes referred to as "LP64") |
| 64/64 system | int and long
are both 64 bits in size. |
We use the convention "native X" to refer to the native
C type "X". e.g. "native unsigned int",
"native char", etc.
lseek system call. It is typically declared in C as some
variant of the following:
extern long lseek(int fd, long offset, int whence);
On recent systems, the off_t typedef is used in place
of long, but the underlying definition is the same.If we used the obvious approach, we might be tempted to declare this in C# as:
[DllImport("libc")]
extern long lseek(int fd, long offset, int whence);
Here, we have reused the DllImport attribute to indicate
the shared object from which to import the function. This will ensure
that a PInvoke record is created in the metadata with the module
reference set appropriately.
However, the above definition will not work on 32/32 systems (the most
common at present). This is because the C# long type is
64 bits in length, not the 32 that the 32/32 C API expects. It also will not
work on 64/64 systems because the C# int type is 32 bits
in length, not the 64 that the 64/64 C API expects.
There are actually three "correct" definitions for lseek:
32/32 systems:Which definition we use is dependent upon the underlying operating system and CPU type. Any application that uses this declaration will need to be compiled in three different variants. This situation is untenable, especially if we intend to wrap large API's such as GTK+.32/64 systems:[DllImport("libc")]
extern int lseek(int fd, int offset, int whence);64/64 systems:[DllImport("libc")]
extern long lseek(int fd, long offset, int whence);[DllImport("libc")]
extern long lseek(long fd, long offset, long whence);
CLI allows attributes to be added to parameters to modify their
marshaling characteristics. This is typically used to convert
C# types into the foreign type systems used by the Win32 SDK and COM.
One of the "unmanaged types" that can be specified is SysInt,
which indicates an integer value of the correct native size for the
underlying platform. We can thus modify the definition
of lseek as follows:
[DllImport("libc")]
extern long lseek([MarshalAs(UnmanagedType.SysInt)] int fd,
long offset, [MarshalAs(UnmanagedType.SysInt)] int whence);
Alternatively, the built-in value type IntPtr can
be used (suggested by Jay Freeman):
[DllImport("libc")]
extern long lseek(IntPtr fd, long offset, IntPtr whence);
This takes care of the int parameters for the method.
ECMA specifies UnmanagedType.SysInt and
UnmanagedType.SysUInt, but there are no equivalents for
the native long type. Also, it may not always work.
The native integer types used by the CLR are specified to be
the most efficient integer sizes for the CPU in question. On
a 64-bit CPU, this will normally be 64-bits, even though
most 64-bit C compilers for Unix will define int
to be 32-bit. Hence, it may not be possible to rely upon
UnmanagedType.SysInt or IntPtr to
provide the correct behaviour on all platforms.
Three different proposals seem to offer a solution to this problem. They are described in the following subsections.
NativeTypeSizes is introduced
which can be associated with a method (the namespace for this type
is yet to be determined). We would modify our lseek
example as follows:
[DllImport("libc"), NativeTypeSizes(true)]
extern long lseek(int fd, long offset, int whence);
The presence of this attribute with a value of true indicates
to a compliant CLR that types should be marshaled to their obvious
native counterparts. We now have one definition that works for all
Unix platforms without requiring multiple library and application builds.
If a method contains a mixture of fixed-size and variable-size parameters,
then explicit marshaling can be applied to the fixed-size parameters.
Consider the following C declaration (assuming an appropriate typedef
for int32):
extern int foo(long x, int32 y);
In this case, we would declare the C# counterpart as follows:
[DllImport("libfoo"), NativeTypeSizes(true)]
extern int foo(long x, [MarshalAs(UnmangedType.I4)] int y);
The explicit marshaling attribute on y will override
the default behaviour of marshaling to SysInt.Bob Salita suggested hijacking the existing calling convention specifications. Windows allows methods to be labelled as "cdecl", "stdcall", or "winapi". We could hijack one of these values to indicate native type size handling. However, this could be confusing to programmers and may cause problems should we ever discover a legitimate need for the original meanings.
PosixProto attribute that specifies the full alternative
prototype for the method, using Posix types. Miguel's version of
the example was:
[DllImport("libc"), PosixProto(int, off_t, int)]
extern long lseek(int fd, long offset, int whence);
Here, PosixProto is an attribute with a variable number of
parameters that takes type names as arguments. If we add the type for
the return value (missing from Miguel's example), and make it strictly
conforming to C# syntax, we would get something like the following:
[DllImport("libc"), PosixProto(typeof(off_t), typeof(int), typeof(off_t), typeof(int))]
extern long lseek(int fd, long offset, int whence);
where PosixProto and off_t are defined as follows:
public class PosixProto : Attribute
{
public PosixProto(Type returnType, params Type[] paramTypes);
};
public struct off_t {}
Using this information, the runtime engine can convert between the
engine type (e.g. long) and the underlying Posix type
(e.g. off_t). The runtime engine is configured at
compile time to know how to marshal between engine types and
the standard Posix types.
The beauty of this system is that it can be extended to other ANSI C and
Posix types, including the "nasty" ones such as struct stat.
A drawback of this system is that the runtime engine must be pre-configured
with all types of interest.
[PosixType]
public struct off_t
{
private long value;
public off_t(long x) { value = x; }
public static operator implicit off_t(long x)
{ return new off_t(x); }
public static operator implicit long(off_t x)
{ return x.value; }
};
[PosixType]
public struct int_t
{
private int value;
public int_t(int x) { value = x; }
public static operator implicit int_t(int x)
{ return new int_t(x); }
public static operator implicit int(int_t x)
{ return x.value; }
};
[DllImport("libc")]
extern off_t lseek(int_t fd, off_t offset, int_t whence);
The presense of the PosixType attribute on the value type
definition indicates that special marshaling behaviour is required for
PInvoke methods. The implicit conversions ensure that the resulting
types act like the regular numeric types in ordinary usage.
This scheme can also be used to modify how larger structures are laid out.
Consider the struct timeval type:
struct timeval
{
long tv_sec;
long tv_usec;
};
Because the size of long varies between systems, we
will have problems with Posix functions such as gettimeofday.
The CLR layout of the structure will not match the underlying C library.
Using an appropriate definition of long_t, we can write
this in C# as follows:
public struct timeval
{
public long_t tv_sec;
public long_t tv_usec;
};
Because long_t has the PosixType attribute,
compliant CLR's know to lay out the field as a native long,
rather than a full CLR long. The gettimeofday
function can then be declared as follows:
[DllImport("libc")]
extern unsafe int_t gettimeofday(timeval *tv, timezone *tz);
There is one drawback of using types such as int_t and
off_t. Values must be converted to and from normal CLR types
using the implicit conversion operators. This involves some additional
overhead in the bytecode that the interpreter must handle. However,
JIT's should be able to optimize this overhead away by looking for
references to Posix-enabled conversion operators.
struct stat
type. It isn't possible to define a C# struct using types such as
int_t and long_t because some systems have
entire fields missing, or different orders for the fields.
[PosixStructType]
public struct stat
{
public dev_t st_dev;
public ino_t st_ino;
public mode_t st_mode;
public nlink_t st_nlink;
public uid_t st_uid;
public gid_t st_gid;
public dev_t st_rdev;
public dev_t st_rdev;
public off_t st_size;
public blksize_t st_blksize;
public time_t st_atime;
public time_t st_ctime;
public time_t st_mtime;
};
This definition does not match any known Unix flavour, including
Linux: there are normally extra padding and unused fields. We've
omitted these extra fields on purpose to help demonstrate how they
are handled.
The presence of the PosixStructType attribute indicates that
the runtime engine is responsible for laying out this structure.
It adjusts the offsets and types of each field so that they correctly
match those of the underlying struct stat type.
Any extra fields are not accessible by the C# application.
The layout algorithm is based on field name. We have purposely reversed
the usual order of the st_mtime and st_ctime
fields above. The runtime engine will remap the field offsets so that
they match the underlying type correctly.
Note: we did not place the PosixStructType attribute on
the timeval definition in the previous section.
If the default layout algorithm will give the correct
result on all systems, there is no need to label the type.
NativeTypeSizes(true)" is specified.
| C# Type | Default | Native |
bool |
int8 |
native int |
sbyte |
int8 |
int8 |
byte |
uint8 |
uint8 |
short |
int16 |
native short |
ushort |
uint16 |
native unsigned short |
char |
uint16 |
native wchar_t |
int |
int32 |
native int |
uint |
uint32 |
native unsigned int |
long |
int64 |
native long |
ulong |
uint64 |
native unsigned long |
float |
float32 |
native float |
double |
float64 |
native double |
System.Object |
Object handle | Object handle |
System.String |
See below | See below |
System.Enum |
Underlying type | Underlying type |
System.ValueType |
Struct passed by value | Struct passed by value |
Objects of type System.String are handled differently depending
upon the kind of parameter or return value:
native char array
and pass a pointer to it as const native char *.
If the string value is null, then the native value
NULL will be passed to the function.native char array and
pass a pointer to it as a native char *. Upon return,
create a new string with the contents of the array.native char
array and pass a pointer to it as native char *.
If the string value is null, then an empty string
is passed in. Upon return, create a new string with the final
contents of the array.const native char *, and create a new string with
the contents of the return value. If the return value was
the native value NULL, then it is converted into
the string value null.MallocStringReturn attribute if the
string was malloc'ed by the function and the value should
be free'd automatically once it has been converted into
System.String. For example:
[DllImport("libc"), return:MallocStringReturn(true)]
extern string get_current_dir_name();
When an explicit MarshalAs attribute is supplied, the values are
marshaled as follows:
| UnmanagedType | NativeType |
Bool |
native int |
VariantBool |
native char |
I1 |
int8 |
U1 |
uint8 |
I2 |
int16 |
U2 |
uint16 |
I4 |
int32 |
U4 |
uint32 |
I8 |
int64 |
U8 |
uint64 |
SysInt |
native int |
SysUInt |
native unsigned int |
R4 |
float32 |
R8 |
float64 |
RPrecise |
native long double |
LPStr |
const native char * |
LPWStr |
const uint16 * |
LPTStr |
const native wchar_t * |
Interface |
Runtime Object Handle |
Struct |
Structure ValueType passed by value |
LPStruct |
Structure ValueType passed by reference |
LPVoid |
native void * |
Note: we are using LPTStr slightly differently than
in Windows. Windows uses it to select between either Ansi or Unicode
variants of a function. We are using it to convert strings into
arrays of native wchar_t elements, where the size of
native wchar_t is system-specific.
String conversions use the currently prevailing locale to convert
the Unicode System.String values into native char
arrays.
Because most Unix API's take native char * strings as parameters,
we always marshal strings in that manner unless explicitly overridden
using LPWStr or LPTStr. For example:
[DllImport("libc"), NativeTypeSizes(true)]
extern uint wcslen([MarshalAs(LPTStr)] string str);
This converts into a call to the following C function:
extern unsigned int wcslen(const wchar_t *str);
If a System.String object is marshaled as Interface,
then the object handle is passed directly. This should only be used with
functions that are aware of the runtime engine's mechanism for representing
strings.
We also handle Bool and VariantBool slightly
differently than in Windows. The most common boolean types in Unix
API's are native int and native char, with
the former more common. Windows use 0 and -1 as boolean values with
VariantBool. This usage is extremely rare in Unix, and can
be handled using the integer types if necessary.
size_t,
and time_t. While there are common definitions for these
types (e.g. unsigned int and long), it is
possible that they may have unusual sizes on some systems.There really isn't any good way to handle unusual type sizes cleanly. Two possibilities come to mind: one using custom marshalers and the other using attributes. For example:
The second is probably a little better: if the runtime engine does not understand the attribute, it will fall back to the default handling. In the example above, the default will "just work" on the majority of platforms.[DllImport("libc"), NativeTypeSizes(true),
return:MarshalAs(UnmanagedType.CustomMarshaler, MarshalType="size_t")]
extern uint getpagesize();
[DllImport("libc"), NativeTypeSizes(true), return:AnsiCType("size_t")]
extern uint getpagesize();
Both methods require the runtime engine to have a list of standard ANSI C types. If this is extended to include other standard Unix types, the list could become quite large. A large list increases the chance that different engines will have different lists, and hence will not interoperate successfully.
More investigation is required.
mscorlib) contain a
large number of InternalCall methods. These are similar to PInvoke
methods, except that the destination of the call is implicitly supplied
by the runtime engine.InternalCall methods typically operate closer to the runtime engine than PInvoke methods, and so their marshaling characteristics are a little different.
Objects of type System.String are passed as object handles to
InternalCall methods unless explicitly overridden by a MarshalAs
attribute. i.e. they are not automatically converted into
native char arrays.
All InternalCall methods have an extra parameter inserted at the front
of their parameter list to hold a handle to the currently executing
thread. For example, consider the following method within
System.Environment:
[MethodImpl(MethodImplOptions.InternalCall)]
extern private Object nativeCaptureStackTrace(int skip);
This obtains an object that represents a stack trace within the currently
executing thread. The C counterpart to this would be declared as follows:
ObjHandle nativeCaptureStackTrace(ThreadHandle thread, int32 skip);
where ObjHandle represents the type used by the runtime engine
to represent an object handle (probably a pointer of some kind), and
ThreadHandle represents the type used by the runtime engine
to represent a thread handle (probably also a pointer).
The underlying implementation can use thread to look up the
internal runtime data structures to obtain the requested information.
The approach of adding an extra parameter to InternalCall methods avoids the use of global variables to obtain a link back to the runtime engine. PInvoke methods do not have this extra parameter because they typically refer to functions outside of the runtime engine itself.
We do not define the specifics of object handles and thread handles further in this document. Some future document may define a standard "Dotnet Native Interface" for use by InternalCall methods to interact with the runtime engine.
IntPtr may not always work.