                   Netmessage - Network Object Serialization

1. Foreword

   This document is a work in progress.  Some of the documentation found
   herein may be out of date, and several of the netmessages have not
   been documented as of yet.

   Plugin developers SHOULD consult class definitions to ensure that the
   documentation is infact correct.  Questions regarding netmessages may
   be sent via e-mail to the Olympus development list,

      olympus-dev@mountlinux.com

   Documentation on the APIs of individual client-side netmessages and
   the protocol details will be split into seperate files in a future
   release, once the netmessage protocol is complete.

1.1. Introduction

   The transport of data in Olympus is accomplished using a class called
   netmessage.

   Classes derived from the base netmessage class are referred to as
   netmessages.

   Each individual netmessage is purpose specific and has at least one
   unique identification number as defined in section 2.1.

2. Object Identification Numbers

   Olympus uses a simple ordering scehme for defining netmessage object
   identification numbers.

   Object identification numbers are treated as 32-bit unsigned integers
   by Olympus, as a result of which, they have a large possible range.
   
   These numbers have been allocated as follows:

      1-20     Network connection management
      21-40    Olympus user management
      41-60    Olympus database management
      61-100   Olympus general utility messages
      101-120  File management
      121-140  Package management
      141-160  Process management
      161-200  System monitoring
      201-225  Reserved for future use
      226-255  General status identifiers
      256+     Reserved for future use

2.1. Object Identification Number Definitions

   The following object identification numbers have been defined for
   Olympus.

    1-20: Network Connection Negotiation

       1    COMMAND_VERSION
       2    COMMAND_ALGINIT
       3    COMMAND_KEXINIT
       4    COMMAND_KEXREPLY
       5    COMMAND_USERAUTH

   21-40: Olympus user management

      21    COMMAND_USERS

   41-60: Olympus database management

      41    COMMAND_DBSELECT
      42    COMMAND_DBINSERT
      43    COMMAND_DBDELETE
      44    COMMAND_DBUPDATE

   61-100: Olympus general utility messages

      61    COMMAND_STATUS
      62    COMMAND_PING

   101-120: File management

      101   COMMAND_FETCHFILE
      102   COMMAND_REPLACEFILE
      103   COMMAND_FILEINFO
      104   COMMAND_REMOVEFILE
      105   COMMAND_LINKFILE
      106   COMMAND_KEYFILEINFO

   121-140: Package management

      The package management netmessages are still being developed, no
      numbers have been defined in this range.

   141-160: Process management

      141   COMMAND_FETCHXML
      142   COMMAND_EXEC
      143   COMMAND_EXECDATA

      The next major revision of the process management netmessages will
      make them considerably more robust, amongst the many improvements,
      there will be new messages for signal dispatching, for indiciating
      process termination, and the seperation of output streams.

   161-200: System monitoring

      161   COMMAND_SYSINFO

   Object identification number definitions may also be found in a file
   in the Olympus source tree:

      olympus/src/common/include/commid.h

   For compatibility with other versions of Olympus, these numbers MUST
   remain constant once defined.

2.2. Message Operation Codes

   These numbers have been allocated as follows:

      1-20     Olympus protocol negotiation
      21-40    Algorithm negotiation
      41-60    Key exchange
      61-80    User authentication
      81-100   Reserved for future use
      101-125  Data message
      126-150  Error message
      150-200  Reserved for future use
      201-225  Olympus protocol generic
      226-255  Olympus protocol error
      256+     Reserved for future use

   These are used in the transport and connection protocols, they do not
   belong here and will be moved to a different document after the new
   transport layer is complete.

3. Netmessage Status Codes

   These numbers have been allocated as follows:

      1-25     Success
      26-50    Request error
      51-75    Olympus user error
      76-100   Olympus database error
      101-150  Olympus server communication/link error (or succes?)
      151-255  System and general errors
      256+     Reserved for future use

3.1. Status Code Definitions

4. Netmessages
4.1. Data Type Definitions

   Modifications are currently underway to make the data type scheme
   more robust.

5. Netmessage Definitions
5.1. nmExec

   The nmExec class is used to open and manipulate processes on a target
   host.

   The COMMAND_EXEC and COMMAND_EXECDATA object identifiers are used for
   the nmExec class.

   The following constructor is used when opening a new process:

    nmExec(transport *socket, uint32 sessionID, const char *_key1,
           const char *_key2, const char *_args = "");

   When a process creation request is made, the key1 and key2 parameters
   are referenced against the database on a target host to find the path
   of a corresponding program. The optional parameter, args, is used to
   specify any arguments that should be given to the program when the
   process is opened. For example, to start an Apache webserver:

       key1           = "apache"
       key2           = "start"
       args           = ""

   A target host will always respond to a process creation command with
   an object of type COMMAND_EXEC. This response includes a status code
   and the process ID of the new process. The process ID is used as the
   reference to the process in any further communications.

   The following message status codes may be returned in a response to
   a process creation request:

   NM_SUCCESS
      A new process was successfully created.

   NM_EDBNOENT
      No database entry was found corresponding with the specified key
      pair.

   NM_EINVAL
      An invalid argument was specified or unmatched '"'.

   NM_EUNKNOWN
      Internal server error; an unexpected error occurred.

   NM_EAGAIN
      Internal server error; there was insufficient memory available for
      the new process or too many processes are open.

   NM_ENOMEM
      Internal server error; there was insufficient memory available for
      the new process.

   NM_EMFILE
      Internal server error; failed to allocate file descriptors for the
      pipes because too many descriptors are in use by Olympus.

   NM_ENFILE
      Internal server error; failed to allocate file descriptors for the
      pipes because the system file table is full.

   Management of the new child process is performed by the host it was
   created on, although some process manipulation functionality exists
   in the nmExec class.

   When data is available for reading from a child process, it is read
   and returned in an object of type COMMAND_EXECDATA. The same object
   is used for notification of process termination and any errors that
   may occur after the process has been created.

   When an object of type COMMAND_EXECDATA is sent, the following status
   codes may be given:

   NM_DATA
      Data was read from the process and has been returned with this
      object.

   NM_PROCFIN
      Process closed.

   NM_EBADRQC
      Invalid request code specified.

   NM_ESRCH
      Failed write or close because the specified process ID does not
      exist or is not owned by the current user.

   NM_EUNKNOWN
      Internal server error; an unexpected error occurred. The process
      has been closed.

   NM_EBADF
      Internal server error; failed read or write because a file
      descriptor used was not open for reading or writing. The process
      has been closed.

   NM_EINVAL
      Internal server error; failed read or write because and invalid
      file descriptor was used. The process has been closed.

   NM_EPIPE
      Internal server error; failed write because the process is not
      reading from the pipe. The process has been closed.

   NM_ECHILD
      Internal server error; attempted to close a process that does not
      exist or is not a child of the server.

   There are several ways to send process manipulation requests to a
   parent host, all of which are accomplished through the use of the
   COMMAND_EXECDATA object type.

   The following constructor is used to write data to a process:

    nmExec(transport *socket, uint32 sessionID, uint32 _pid,
           byte *_data, int len);

   It is also possible to send a simple process manipulation request to
   a target host. This may be done using the following constructor:

    nmExec(transport *socket, uint32 sessionID, uint32 _pid,
           byte _request);

   The following process manipulation request codes and their actions
   are currently defined for use in nmExec:

       1    NMEXEC_CLOSE      = close process
       2    NMEXEC_WRITE      = write to process

   Several methods of retrieving member data exist for when an object is
   received:

   uint16 nmExec::nmstatus()
      Returns the message status code.

   uint32 nmExec::processid()
      Returns the process ID.

   const char *nmExec::rkey1()
      Returns the key1 which was used in the request.

   const char *nmExec::rkey2()
      Returns the key2 which was used in the request.

   int32 nmExec::procstatus()
      Returns the process termination status.

   uint32 nmExec::datalen()
      Returns the length in bytes of the data read from the process.

   const byte *nmExec::data()
      Returns a pointer to a byte array containing data read from the
      process or NULL if no data was received.

   const char *nmExec::datastr()
      Returns a pointer to a null-terminated string containing data read
      from the process or NULL if no data was received.

   A nmExec process creation request packet is transported as follows:

      byte[32]  netmessage packet header
      string    key1
      string    key2
      string    args

   A nmExec process creation response packet is transported as follows:

      byte[32]  netmessage packet header
      string    key1
      string    key2
      uint16    status
      uint32    process id

   A nmExec process manipulation request packet is transported as
   follows:

      byte[32]  netmessage packet header
      uint32    process id
      byte      request
      uint32    data length
      byte[dl]  data; dl = data length

   A nmExec process information or data packet is transported as
   follows:

      byte[32]  netmessage packet header
      uint16    status
      uint32    process id
      int32     process termination status
      uint32    data length
      byte[dl]  data; dl = data length

5.2. nmFetchFile

   The nmFetchFile class is used to retrieve one or more regular files
   from a peer.

   The COMMAND_FETCHFILE object identifier is used for the nmFetchFile
   class.

   The following constructor is used when making a file request:

    nmFetchFile(transport *socket, uint32 sessionID, const char *_key1,
                const char *_key2);

   When a file retrieval request is made, the key1 and key2 parameters
   are referenced against the database on the receiving host to find a
   corresponding entry containing a file path. For example, to retrieve
   the inetd configuration file from a host:

       key1           = "inetd"
       key2           = "config"

   When a file in a database entry exists, at a minimum a record of the
   existence of the file will be included in the retrieval response.
   Each record includes the name of the file, a status code and the file
   itself if it is a regular file and no errors occur when handling it.

   If the filename component of the database entry contains any wildcard
   characters ('*' or '?'), it is treated as a wildcard expression and
   all of the files in the directory are matched against it. A record of
   each matching file will be included in the retrieval response.

   The following message status codes may be returned in a response to
   a file retrieval request:

   NM_SUCCESS
      Request successful.

   NM_EDBNOENT
      No database entry was found corresponding with the specified key
      pair.

   NM_EDBBADENT
      A database entry was found but it did not contain a file path.

   NM_ENOENT
      A component of the path in the database entry does not exist, no
      such file or directory.

   NM_ENOTDIR
      A component of the path prefix in the database entry is not a
      directory.

   NM_ELOOP
      Too many symbolic links encountered while traversing the path in
      the database entry.

   NM_ENAMETOOLONG
      The path in the database entry was too long.

   NM_EUNKNOWN
      Internal server error; an unexpected error occurred.

   NM_EACCESS
      Internal server error; permission denied.

   NM_EMFILE
      Internal server error; failed to allocate file descriptor to open
      directory because too many descriptors are in use by Olympus.

   NM_ENFILE
      Internal server error; failed to allocate file descriptor to open
      directory because the system file table is full.

   NM_ENOMEM
      Internal server error; out of memory.

   As mentioned above, a status code is given with each individual file
   record. The following status codes may be given:

   NM_DATA
      A file has been returned with this record.

   NM_ENOTFILE
      The file exists but is not a regular file.

   NM_ELOOP
      Too many symbolic links were encountered while traversing path.

   NM_EUNKNOWN
      Internal server error; an unexpected error occurred.

   NM_EACCESS
      Internal server error; permission denied.

   NM_ENAMETOOLONG
      Internal server error; file name too long.

   NM_ENOMEM
      Internal server error; out of memory.

   Individual file records are stored in a structure named fileent. The
   nmFetchFile::fileent structure is defined as follows:

      struct nmFetchFile::fileent
      {
         char *name;     // name of the file
         uint16 status;  // file retrieval status code
         char *file;     // local path of received file
      };

   If no file was received, file will be set to NULL.

   Several methods of retrieving member data exist for when an object is
   received:

   uint16 nmFetchFile::nmstatus()
      Returns the message status code.

   const char *nmFetchFile::rkey1()
      Returns the key1 which was used in the request.

   const char *nmFetchFile::rkey2()
      Returns the key2 which was used in the request.

   uint32 nmFetchFile::filecount()
      Returns the number of file records received.

   struct nmFetchFile::fileent *nmFetchFile::operator[](int pos)
      Returns a pointer to a structure representing a file record at the
      specified position from the list of records received. Returns NULL
      if a record does not exist at the specified position.

   struct nmFetchFile::fileent *nmFetchFile::getent()
      Returns a pointer to a structure representing the file record at
      the current position in the file list and then advances the list
      to the next record. Returns NULL if no entries remain in the list.

   void nmFetchFile::rewind()
      Repositions the file record list to the beginning of the list.

   A nmFetchFile request packet is transported as follows:

      byte[32]  netmessage packet header
      string    key1
      string    key2

   A nmFetchFile response packet is transported as follows:

      byte[32]  netmessage packet header
      string    key1
      string    key2
      uint16    status
      uint32    count

   If the message status code indicates success and the count is greater
   than 0, the following is included in the response count times:

      string    filename
      uint16    status
      file      file

5.3. nmFiles

   The nmFiles class is used to retrieve status and other information
   for one or more files from a peer.

   The COMMAND_FILEINFO object identifier is used to identify the
   nmFiles class.

   The following constructor is used when making a request:

    nmFiles(transport *socket, uint32 sessionID, const char *_path,
            uint16 _request = 0);

   The path parameter is the path to a file on the receiving host. It is
   preferred that the path not have a trailing '/' character, as it will
   result in additional processing time on the receiving host.

   The request parameter is an optional set of flags ORed together that
   determine additional information to be included in the response.

   The following base request flags are currently defined for use with
   nmFiles:

      0x0001   NMFRQ_RLNK     = expand symbolic links
      0x0002   NMFRQ_NAME     = include user and group names
      0x0040   NMFRQ_DLST     = list directory contents without wildcard
                                pattern matching
      0x0080   NMFRQ_DLSTGLOB = list directory contents with wildcard
                                pattern matching

   When a directory listing is requested by setting the NMFRQ_DLSTGLOB
   or NMFRQ_DLST flag, additional flags are required specifying which
   file types to include in the file list:

      0x0100   NMFRQ_REG      = include regular files
      0x0200   NMFRQ_DIR      = include directories
      0x0400   NMFRQ_LNK      = include symbolic links
      0x0800   NMFRQ_CHR      = include character devices
      0x1000   NMFRQ_BLK      = include block devices
      0x2000   NMFRQ_FIFO     = include named pipes
      0x4000   NMFRQ_SOCK     = include sockets

   Another macro is provided for requesting all file types be included
   in the file list:

      0x7f00   NMFRQ_ALL      = include all files

   The file type inclusion flags described above have no effect when the
   NMFRQ_DLSTGLOB and NMFRQ_DLST flags are not set.

   The following message status codes may be returned in a response to
   a file status request:

   NM_SUCCESS
      Request successful.

   NM_ENOTDIR
      Invalid path specified; a component of the path prefix is not a
      directory. This will also occur if the NMFRQ_DLSTGLOB or
      NMFRQ_DLST is set and the path is not a directory.

   NM_ENOENT
      Invalid path specified; no such file or directory or path is an
      empty string.

   NM_ELOOP
      Invalid path specified; too many symbolic links encountered while
      traversing the path.

   NM_ENAMETOOLONG
      Invalid path specified; file name too long.

   NM_EUNKNOWN
      Internal server error; an unexpected error occured.

   NM_EACCESS
      Internal server error; permission denied.

   A record for each file within the request parameters is included with
   the response to a status request. Each record includes a file name,
   a status code representing the results of a call to lstat(), and the
   file status information if the operation is successful.

   When the NM_NAME flag is set a user name and group name corresponding
   to the user ID and group ID obtained by lstat() is also included with
   each record.

   When the NM_RLNK flag is set and a file is a symbolic link, the value
   of the link along with a status code representing the underlying file
   operation is included with the record.

   The following status codes may be returned with either file operation
   status or link resolution status for each record:

   NM_SUCCESS
      File operation successful; results included with record.

   NM_EUNKNOWN
      Internal server error; an unexpected error occured.

   NM_EIO
      Internal server error; an I/O error occurred while reading from
      the file system.

   NM_ENOENT
      Internal server error; no such file or directory.

   NM_ENOTDIR
      Internal server error; a component of the path is not a directory.

   NM_ELOOP
      Internal server error; too many symbolic links were encountered in
      translating the path.

   NM_EACCESS
      Internal server error; permission denied.

   NM_ENOMEM
      Internal server error; out of memory.

   NM_ENAMETOOLONG
      Internal server error; file name too long.

   Individual file records are stored in a structure named fileent. The
   nmFiles::fileent structure is defined as follows:

      struct nmFiles::fileent
      {
         char *name;          // name of the file
         uint16 status;       // file operation status
         char *linkname;      // symbolic link value
         uint16 linkstatus;   // symbolic link operation status
         uint32 st_mode;      // protection
         uint64 st_size;      // total size, in bytes
         uint32 st_uid;       // user ID of owner
         char *uname;         // corresponding user name
         uint32 st_gid;       // group ID of owner
         char *gname;         // corresponding group name
         uint32 st_atime;     // time of last access
         uint32 st_mtime;     // time of last modification
         uint32 st_ctime;     // time of last change
      };

   The linkname and linkstatus members are given values only if the file
   is a symbolic link and the NMFRQ_RLNK flag was specified.

   Several methods of retrieving member data exist for when an object is
   received:

   uint16 nmFiles::nmstatus()
      Returns the message status code.

   const char *nmFiles::rpath()
      Returns the path which was used in the request.

   uint16 nmFiles::rcode()
      Returns the request code which was used in the request.

   const char *nmFiles::rdir()
      Returns the canonicalized absolute directory name for the results,
      or NULL if none was received.

      All symbolic links in the directory name have been expanded and
      references to '/./' and '/../' segments and extra '/' characters
      are resolved. A trailing '/' character is appended onto the end.

   uint32 nmFiles::filecount()
      Returns the number of file records received.

   struct nmFiles::fileent *nmFiles::operator[](int pos)
      Returns a pointer to a structure representing a file record at the
      specified position from the list of records received. Returns NULL
      if a record does not exist at the specified position.

   struct nmFiles::fileent *nmFiles::getent()
      Returns a pointer to a structure representing the file record at
      the current position in the file list and then advances the list
      to the next record. Returns NULL if no entries remain in the list.

   void nmFiles::rewind()
      Repositions the file record list to the beginning of the list.

   A nmFiles request packet is transported as follows:

      byte[32]  netmessage packet header
      uint16    request flags
      string    path

   A nmFiles response packet is transported is follows:

      byte[32]  netmessage packet header
      uint16    request flags
      string    path
      uint16    status
      uint32    count

   If the message status code indicates success and the count is greater
   than 0, additional data is included in the response:

      string    directory

   The following is included in the response count times:

      string    filename
      uint16    file operation status
      string    link value; included only if NMFRQ_RLNK is set
      uint16    link read status; included only if NMFRQ_RLNK is set
      uint32    st_mode
      uint64    st_size
      uint32    st_uid
      string    username; included only if NMFRQ_NAME is set
      uint32    st_gid
      string    groupname; included only if NMFRQ_NAME is set
      uint32    st_atime
      uint32    st_mtime
      uint32    st_ctime

5.4. nmPing

   The nmPing class is used to determine the length of time a response
   takes to return from a target host.

   The COMMAND_PING object identifier is used to identify the nmPing
   class.

   The following constructor is used when making a request:

    nmPing(transport *socket, uint32 sessionID, uint32 objectID);

   nmPing provides a public data member which is used to calculate the
   difference between ping times in milliseconds when a response is
   received:

      uint32 elapsedTime = elapsed time in millseconds

   A nmPing request packet is sent as follows:

      byte[32]  netmessage packet header
      uint32    seconds
      uint32    microseconds

   A nmPing response packet is sent as follows:

      byte[32]  netmessage packet header
      uint32    seconds
      uint32    microseconds

5.5. nmSysInfo

   The nmSysInfo class is used to retrieve system information from a
   target host.

   The COMMAND_SYSINFO object identifier is used for the nmSysInfo
   class.

   The following constructor is used when making a request:

    nmSysInfo(transport *socket, uint32 sessionID, uint32 objectID);

   nmSysInfo provides all of the data members that are received in a
   response as part of its public interface. These members can be broken
   down into three seperate categories:

   CPU Information

      uint32 cpu_user    = ticks in user mode
      uint32 cpu_system  = ticks in system mode
      uint32 cpu_nice    = ticks in niced tasks
      uint32 cpu_idle    = ticks in idle tasks

   Physical Memory Information

      uint32 mem_total   = total physical memory available, in kb
      uint32 mem_free    = free physical memory, in kb
      uint32 mem_used    = physical memory in use, in kb
      uint32 mem_buffers = buffered physical memory, in kb
      uint32 mem_cached  = cached physical memory, in kb

   Swap Memory Information

      uint32 swap_total  = total swap space available, in kb
      uint32 swap_free   = free swap memory, in kb

   A nmSysInfo request packet is sent as follows:

      byte[32]  netmessage packet header

   A nmSysInfo response packet is sent as follows:

      byte[32]  netmessage packet header
      uint32    cpu_user
      uint32    cpu_system
      uint32    cpu_nice
      uint32    cpu_idle
      uint32    mem_total
      uint32    mem_free
      uint32    mem_used
      uint32    mem_buffers
      uint32    mem_cached
      uint32    swap_total
      uint32    swap_free
