Central to the design of the Package Forge system is the concept of a build job which is what the user submits. In essence a build job has a unique identifier and consists of a set of, one or more, source packages along with some instructions which tell the target build systems how and where the packages should be built and submitted.
A build job is represented by the PkgForge::Job
class. This has the following important attributes:
This can be any string the user chooses which has not been previously
seen by the system. The only limitation on the contents is that placed
by the column definition in the job table of the registry database,
this requires it to be a maximum of 50 characters long. If the user
does not select their own tracking identifier then a Universally
Unique Identifiers (UUID) is generated using
the Data::UUID
module.
This controls the set of platforms for which the build job should be submitted. By default, if the user does not express an opinion, all supported, active platforms will be selected. As well as the straightforward, positive inclusive approach of just listing the suitable target platforms (e.g. "sl5, f13") it is possible to exclude particular unsuitable platforms from the whole set (e.g. "!f13").
In PkgForge terminology a platform name is based only on the distribution name and version (e.g. "f13" or "sl5"). The platform name does not contain any reference to the architecture. Note that this is different from LCFG which appends an "_64" for the x86_64 architecture. The target architectures are handled separately, see below for details.
It should be noted that there are no limits on the strings which can be placed in the list of platforms. Anything which refers to a platform which does not exist will be ignored, no errors are reported. However, a helpful warning is given on the web interface when it spots that a submitted job refers to unsupported or inactive platforms.
The architecture list is used in a very similar way to the platform list. This controls the set of architectures for which the build job should be submitted. By default, if the user does not express an opinion, all architectures for the selected platforms will be selected. Again, as well as the straightforward, positive inclusive approach of just listing the suitable target architectures (e.g. "i386, x86_64") it is possible to exclude particular unsuitable architectures from the whole set (e.g. "!x86_64").
It should be noted that there are no limits on the strings which can be placed in the list of architectures. Anything which refers to a platform which does not exist will be ignored, no errors are reported. However, a helpful warning is given on the web interface when it spots that a submitted job refers to unsupported or inactive architectures.
Further to this, it is worth noting that there are some limits on what can be achieved with this method of specifying the platform and architecture lists. If a job should be built for different architectures on different platforms (e.g. i386-only on sl5 but all on f13) then it will need to be submitted as multiple separate jobs. It is not expected that this will cause much difficulty.
Clearly the most important part of a build job is the set of source
packages which should be built. A job may contain one or more source
packages of multiple file types. Any file type which has an associated
Perl class which implements the PkgForge::Source
Moose
role can be added to a job. Individual build daemons will filter the
set of source packages and only attempt to build from those which are
appropriate (e.g. an RPM builder might only choose SRPMs). This
feature makes it very easy for a user to submit a job for all
platforms without needing prior knowledge as to which will support a
particular file type.
When a job is submitted the source packages are validated and the SHA1 sums are calculated. These are then used throughout the system to ensure that the source package files remain consistent. The validation varies from type to type but includes basics such as checks that the file exists and is of a non-zero size. For example, the SRPM source package type also checks that the package is a valid source RPM.
The builds of the source packages will be attempted in the order that they are specified by the user. When building a set of packages some platforms will stop immediately when an error is encountered, on others the build will move onto the next package and the failed package will be retried later. See the documentation for the relevant build daemon for full details.
The package bucket is LCFG/DICE terminology for a directory in which built packages are stored. In the Package Forge build system the bucket is typically used for two related issues. The first is the selection of the build system configuration to be used and the second is the selection of the location into which built packages will be stored.
There are no restrictions on what the user can set as the bucket for a submitted job. The knowledge of which buckets are supported is only available to the build daemons (and varies on each platform) so the incoming queue processor cannot check for validity. Depending on the build system being used an incorrect bucket name could trigger an error and cause a complete failure to build the job.
The bucket name must be specified by the user. Where it is not applicable a build daemon may choose to completely ignore the bucket though, in which case it can be set to anything the user chooses. For example, an external site using their own pkgforge system may choose to put all packages into a single location or may not use the pkgsubmit tool.
The RPM builder uses mock to make packages. On these platforms the
mock configuration is closely related to the directory to be used for
storing built packages. Packages in certain directories should only
have build-dependencies on packages in other particular
directories. For example, on F13 the LCFG packages in
the lcfg
bucket should only depend on software which is
in F13 or already in the bucket. There should not be any dependencies
on packages in buckets with license restrictions
(e.g. inf
or uoe
). The list of buckets is
site-dependent, see local documentation for details.
Currently Package Forge only supports sending reports via email. The report attribute is used to specify the (comma-separated) list of email addresses to which a report should be sent after each build job is completed. If nothing is specified then no reports will be sent.
Due to the way in which a job is split into multiple tasks, one for
each target platform, a separate report is sent for each task. So, for
example, one submitted job may generate four reports
(sl5/i386
,
sl5/x86_64
, f13/i386
, f13/x86_64
). There
is no way to easily aggregate the messages as the data is stored
separately for each task.
A build job is submitted as a directory which contains all the source packages and a meta-data manifest file. This directory is placed into a standard location in the file-system where the user has write access. In the School of Informatics the file-system used is AFS so that a build job may be submitted from anywhere.
Build jobs are submitted using the submit
command of
the pkgforge
command-line tool. Here is the current short
help-text for the command:
pkgforge submit [-aBcpr] [long options...] --verbose Verbose output -c --configfile The configuration file for this application -a --archs The architectures for which packages should be built --target Location into which jobs should be submitted --id The unique identifier for this job -p --platforms The platforms on which packages should be built -B --bucket The bucket to which the packages will be submitted. -r --report Email addresses to which build reports will be sent
Most of these options (--archs
, --platforms
,
--id
, --bucket
and --report
)
have already been discussed above. It is unlikely that it will ever be
necessary to set the target option, this controls where the new build
job should be stored and the default should be correct. It is entirely
possible to use the submission tool with multiple separate pkgforge
systems though in which the target option is essential. The only other
option which might come in handy is that for changing the
configfile. Again, if multiple pkgforge systems are being used it
might be necessary to have different configurations for each. The
standard configuration file handling is discussed below.
Submitting a job for only F13 (both i386 and x86_64 architectures):
$ pkgforge submit --bucket lcfg --platform f13 foobar-1-2.src.rpm
Submitting a job for all platforms except SL5 (again, for all architectures):
$ pkgforge submit --bucket world --platform '!sl5' foobar-1-2.src.rpm
Email reports are needed:
$ pkgforge submit --bucket devel --report somebody@example.org foobar-1-2.src.rpm
For the job submission tool the following configuration files are examined and, if they exist, the contents are used to set the default values for the submission client options. Values for options set in files which are later in the list override any values from files which are earlier in the list.
/etc/pkgforge/pkgforge.yml
/etc/pkgforge/submit.yml
~/.pkgforge/pkgforge.yml
~/.pkgforge/submit.yml
Default values for any submit option can be set, the configuration file format is YAML. The easiest approach is to look at the system-wide configuration files and copy the applicable settings into the local directory. For example, this makes it possible to avoid having to specify the email addresses for reports every time a job is submitted.