OpenPKG Project
OpenPKG ProjectDocumentationArticle

Package Once, Use Everywhere

Cross-platform Unix software packaging with OpenPKG

You might prefer Open Source software for its well-known advantages, but sometimes regret the downside of its associated open and distributed development when trying to apply it manually to your work environment.

To keep a work environment stable and secure, it's often necessary to locate the latest version of an application on the Internet and collect its most recent patches containing security and bug fixes. However, there is often no single location for these files. After the search finishes, a system administrator must build and install the new binaries on every Unix box in the network, and might find that the application needs slight tweaking on each of them.

Then after a round of laborious build manipulation, it might not be clear that the application will run as intended on each of the different platforms. If the application is a daemon even more work awaits, because most Unix flavors have their own method of starting and stopping daemons.

If the previous operation succeeds after all, the system administrator might be left to wonder why it is necessary to clutter up the system with residual installation files and how do the vendors think the residue should be removed? Also, if the /var file system later runs out of disk space due to an overgrown application log, do the vendors think that the responsibility lies with the system administrator to implement log file rotation himself? Such redundant basic tasks can and should be avoided.

What can a system administrator do when facing obstacles like these? He might start by sticking with vendor packages only. Unfortunately, there may be several different Unix variants to administer. Worse yet, no existing vendor packaging facility supports multiple installation of an application, a feature useful for multiple project coexistence on a single machine and independent package testing. Another missing feature regards the management of coexisting packaged and unpackaged software.

Finally, if the system administrator wants a software installation and configuration guide for his vacation replacement, he might as well give up. This is usually impossible, because every vendor packaging approach is different and cannot be uniformly applied across all major Unix platforms.

OpenPKG to the rescue

OpenPKG is a large Open Source software packaging project. The project began in November 2000, and has grown to be a collaborative software development effort managed and maintained by many. The project aims to create a modular and flexible Unix subsystem for cross-platform software packaging and installation.

More specifically, the goals of OpenPKG stem from the historical problem often faced in the daily operation of a datacenter environment. The major Unix platforms in operation in datacenters include FreeBSD, Linux and Solaris. To satisfy the demands of such a diverse Unix computing environment, the installation and maintenance of software packages across these platforms greatly benefits from a unified and consistent approach.

OpenPKG is not limited to the three major platforms just mentioned, however. OpenPKG has more ambitious goals and runs on most major Unix platforms. To achieve such cross-platform portability, OpenPKG provides a subsystem on top of the underlying Unix system as shown in Figure 1. It covers every essential server software component from shells, editors, compilers, and up to network daemons and add-on applications. Hence, the intended target community consists of system administrators faced with a large and diverse set of Unix servers.

Internally, OpenPKG leverages the existing packaging technology of the popular RedHat Package Manager (RPM). However, the RPM software included with OpenPKG is specially extended to become more unique and self-contained. The nearly 1000 available OpenPKG packages are really just RPM packages under the hood, but were developed from scratch in an OpenPKG standard approach. The packages are clean and robust, because they follow extremely strict style guidelines and environment requirements.

To meet these OpenPKG guidelines and standards, a package must be built from pristine vendor sources in a non-privileged temporary environment. It has to work under an arbitrary file system location, has to follow a strict file system layout, and must be fully self-contained within its OpenPKG instance. Furthermore, the package must be maximum independent from external Unix facilities, has to install with a reasonable and ready to go configuration, and has to use log file rotations and other such administrative wonders.

These well specified package-building guidelines yield several benefits to OpenPKG users. OpenPKG users can install an instance (the OpenPKG subsystem and user-chosen packages) under any file system location, and even install multiple such instances on a single Unix system.

Incidentally, the main OpenPKG project environment is hosted on a machine with fourteen other ongoing software projects, each with their own dedicated OpenPKG instance. To separately satisfy each project's needs, the associated OpenPKG instance serves every required software component from Postfix to BIND, and INN to Apache. Each project can therefore run in its own isolated environment, much like on a virtual machine — but without any virtualization overhead.

Covering the whole OpenPKG package lifecycle

OpenPKG follows an approach of minimum OS intrusion and maximum standalone presence. It tries hard to smooth out the differences between the underlying vendor solutions. Sometimes this means that it has to break with well known standards in order to provide a cleaner solution. This is the most reasonable approach, and leads to a consistent cross-platform solution.

A question often asked is, "why does OpenPKG use RPM as the underlying packaging technology when other alternatives exist?" There are indeed other similar packaging technologies available to projects like OpenPKG, with the most prominent alternatives to RPM being the Debian dpkg/apt combination, FreeBSD ports, and System V pkgadd. According to some, RPM may not be the greatest packaging system since sliced bread. However, RPM along with its OpenPKG extensions is the only solution covering the whole package lifecycle in a fully consistent way.

To elaborate, the OpenPKG package lifecycle starts with automated tracking, fetching, unpacking, patching, and building the source package from pristine vendor sources. It goes on to build the binary package in an unprivileged environment, and finishes its life term with the installation, upgrade, and uninstallation of the binary package on the target OpenPKG instance. This all works in a self-contained environment, and is driven by concise yet complete package specifications (RPM .spec files). So, to finally answer the question, OpenPKG adopts RPM as its underlying packaging technology because no other can fulfill these steep requirements.

It is important to note that OpenPKG is primarily about packaging, not porting. A requirement of the OpenPKG packaging philosophy is that the vendor software has to be inherently portable to begin with. Minor platform porting issues are fixed by the OpenPKG packagers, but fundamental changes are not considered. In fact, the main reason for some platforms lacking full OpenPKG support is that the amount of overhead in building software on them is not within reason.

It may surprise some that OpenPKG officially discourages the use of binary packages, and only provides them for bootstrapping (development tools not available) and emergency (tight time constraints) purposes. Experience has shown that installing binary packages built from source packages on the target machine outperforms other binary methods in respect to security and robustness.

There are simply too many subtle differences between most build and install systems that can influence the binary at run time and cause trouble. Some important run time parameters, such as the maximum size of shared memory segments, are compiled into the binary on the build machine. Among many examples of such a run time build dependency is a situation in which an Apache package is built with mod_ssl and OSSP mm. The dependency details of such a combination are overwhelming, when sorting out the run time parameters. To avoid such trouble, OpenPKG believes that the only reasonable solution is to always start with source packages.

Bootstrapping OpenPKG for the first time

The OpenPKG bootstrapping process is wrapped into a tricky shell script that when run, will create a new instance of OpenPKG. This process is as self-contained as possible, and requires a minimum amount of operating system support and tools to unpack and compile itself. In the best case, the script will search the $PATH for the development tools tar, make and cc and use them in its processing. If any of these tools are missing, an alternative approach exists in which a shell script containing binaries provides the missing tools. FIXME?

The first step in bootstrapping involves dedicating a unique file system prefix to the instance along with user and group ids. The generic bootstrap building script called openpkg-version-release.src.sh requires these arguments and creates a platform specific bootstrap installation script named openpkg-version-release.arch-os-tag.sh. When run, this script installs the OpenPKG instance under the specified prefix with all files owned by the user and group (Figure 2).

This bootstrapping process links the OpenPKG instance with the underlying Unix system in a very mild way with only a few anchor points. Subsequent package installations do not touch the system at all, and if OpenPKG itself is uninstalled, the anchor points vanish.

After creating a self-contained hierarchy the bootstrap process registers itself as the openpkg package, and can thus be upgraded or treated just like any other package. To make upgrade an already bootstrapped OpenPKG instance easier, .rpm versions of the bootstrap package are available as well.

A step by step example of a complete installation and uninstallation of an OpenPKG instance with a RSYNC server package is given in Listing 4. To understand the OpenPKG RPM commands used, see the quick reference in Table 2.

OpenPKG file system layout

Every file system standard sucks. OpenPKG's file system aims to suck less (Figure 2). Basically, its package area resembles the traditional layout found under /usr on most popular Unix systems. Additionally, it contains its own OpenPKG RPM package management information in a sub-area for purposes of self-containment plus a local area for adding unpackaged components.

OpenPKG does break with tradition in one aspect of its file system layout, however. It unconventionally uses a separate subdirectory of prefix/etc/, prefix/share/ and prefix/var/ for each installed package. These subdirectories are easy to manage, because each is named after its associated package. This provides for a better structure than the usual mess of files, and every OpenPKG package adheres to this layout scheme (even when requiring a lot of effort to override the different vendor package intentions).

Looking again at the RSYNC example in Listing 4, you can conclude right away that the RSYNC configuration is in prefix/etc/rsync/, and it logs to somewhere in prefix/var/rsync/. Such ease of maintenance makes backups easier, moving whole instances without hassle, and more.

Managing OpenPKG packages

When building packages, the temporary files are placed into subdirectories of prefix/RPM/ by default. A package builder can obtain the necessary subdirectory access by either being a member of the associated OpenPKG group, logging in under the management user id of the OpenPKG instance, or logging in as root. A carefully written ~/.rpmmacros file can alternatively redirect the paths to a specified location (see the default macros %_sourcedir, %_specdir, %_builddir, %_tmppath, %_rpmdir, %_srcrpmdir in prefix/etc/openpkg/rpmmacros) and this way allow even an arbitrary user to build packages.

To build a binary package pkg-bin from a source package pkg-src, use openpkg rpm --rebuild pkg-src. OpenPKG RPM will read the .spec information of the pkg-src, build the package based on the information, and place the resulting binary package in prefix/RPM/PKG/pkg-bin.

To finally install the binary package so that it becomes part of the OpenPKG instance, just use openpkg rpm -Uvh pkg-bin. Strictly speaking this upgrades the package. To OpenPKG RPM, installation is nothing more than the special case of upgrading from nothing.

As a side note, some packages provide alternative build variants through boolean variables named with_name. To determine which variables are available (if any at all), run "openpkg rpm -qpi pkg-src | grep with_". To build a binary package using such variables, add --with name or --withoutname to the openpkg rpm --rebuild command to override the default value.

OpenPKG RPM is very clever when it comes to keeping configuration files during an upgrade, as shown in Table 1. An old configuration file is kept if the system administrator stuck to default configuration which remained the same, or if the configuration was changed but coincidentally matches the default configuration of the new package. In practice, a administrator-changed configuration must be reapplied in few cases of package upgrade.

In any case, should a configuration file not be kept, OpenPKG RPM saves the old configuration file with the extension .rpmsave before saving a new default in its place. This ensures that changes to a default configuration can be recovered and reapplied so that an upgraded package runs correctly. Should a new default configuration file replace an old one that retains its original (but old) default, OpenPKG RPM will rename it with the extension .rpmorig.

To make this delightful mechanism work properly, the configuration files of each package have to be explicitly tagged. OpenPKG packages all follow this principle, which further contributes to OpenPKG's robust nature. OpenPKG RPM does the intuitive right thing by making sure that a changed configuration file is kept in place if possible and if not, then preserves it for manual consideration and application.

Finally, after the installation of a package, you can query a lot of its information. The command openpkg rpm -qi pkg-name summarizes a single installed package, while openpkg rpm -qa lists the names of all installed packages. openpkg rpm -qlv pkg-name lists all the files associated with a package and openpkg rpm -qf prefix/path/to/file reveals to which package the given file belongs. You can even check a package's integrity using openpkg rpm -V pkg-name to verify which files have been tampered with or somehow munged. For more details on this refer to Table 2.

The OpenPKG run-command facility

You might have noticed that in the previous example installation of RSYNC, the server was started using the command openpkg rc rsync start. The workhorse behind this simple statement is the powerful OpenPKG run-command facility. Run-commands for every package are conveniently named prefix/etc/rc.d/rc.pkg-name. What each of them offers is the functionality of several shell script segments encapsulated in a single file. The sections of a run-command file are identified by left-aligned labels prefixed with '%'. Listing 2 shows rc.rsync as an example.

The openpkg rc command takes pkg-name as the first argument and one or more section labels as additional arguments. The run command segments corresponding with the desired section labels are then extracted from the rc.pkg-name file and executed in the order given on the command line. The reserved package name all serves as a wildcard and refers to all installed OpenPKG packages, causing the processing of all run-command files in a specified order. In this case, the run-command facility will order the run-command processing according to the priority field (-p number) of the given section label in each run-command file. Another popular field in a section label is -u user, which directs the script code to execute with the privileges of user.

Most sections in a run-command file have arbitrary labels intended to be used as command line arguments to the run-command facility. However, some sections have special meaning. The section labels of these are reserved names used internally by the run-command facility. For example, the %common section functions as a library and contains script code useful to some or all of the other sections. Its script code is run before any other script code.

Just like its cousin, the %common section, the %config section can appear only one time in each run-command file. It contains variables used to configure the behavior of the other sections residing in the same run-command file. This means that the logging and enabling variables in a %config section will only affect the associated package, for example. Such variables can be overridden in prefix/etc/rc.conf in a per-hierarchy scope, however. Technically, the run-command facility assembles a large script file from the %config section, the prefix/etc/rc.conf file, the %common section, and finally the user-defined section given as an argument (in that order). The fat script it then executed.

The sections %monthly, %weekly, %daily, %hourly and %quarterly also have special meaning, as the OpenPKG bootstrap process sets up cron jobs to execute them accordingly. Another label often seen is %env, which is intended to be used with the --eval option explained below.

   Regarding configuration through variables, it should be mentioned that the rc.pkg-name file is intentionally not tagged as a configuration file and will be overwritten on updates with no questions asked. The prefix/etc/rc.conf file is tagged as a configuration file and is intended for overriding variables.

With OpenPKG, all daemon packages are released with scripts that recognize the value of a variable pkg-name_enable (default value "yes"). Setting this variable to "no" disables all run-commands of the daemon in question. As seen with the RSYNC server example, this can be quite useful when installing a package just to get a client piece. Should the server piece not be of interest, then a simple variable shuts it off completely. Similarly, to disable the automatic startup of all daemons in a hierarchy, just add a openpkg_rc_all="no" to prefix/etc/rc.conf. In this case, daemons can still be started manually. This feature may be of interest to system administrators wanting control over daemons with finer granularity.

The OpenPKG run-command facility has many other interesting features. Use openpkg rc --query variable to see the effective value of any configured variable, or use openpkg rc --config to see a complete list of all available variables with their default and effective values. Last but not least the run-command facility offers a very handy feature to allow packages to extend the user shell environment. For instance the bootstrap package openpkg uses this to add the OpenPKG instance into your PATH, MANPATH, INFOPATH, etc. Just execute eval `prefix/bin/openpkg rc --eval openpkg env` to perform this environment extension for your current shell session.

OpenPKG RPM vs. RedHat RPM

As mentioned, OpenPKG is based on a uniquely adjusted and extended RPM-based packaging facility which allows for very concise and clean package specifications and building of every package in an unprivileged environment. Is this any different than what the RedHat, SuSE, or Mandrake implementations offer? To understand the added value of the OpenPKG implementation, let's take as example the OpenPKG packaging of the RSYNC program. The OpenPKG packaging consists of three files: the OpenPKG RPM specification (rsync.spec, Listing 1), the run-commands (rc.rsync, Listing 2), and the default daemon configuration (rsync.conf, Listing 3). In comparison with the RPM-based RSYNC package of other vendors, the OpenPKG RPM-based package is full featured yet very concise and clean. This is due to the use of OpenPKG RPM extensions and strict style guidelines.

To offer more portable and concise shell scripting, OpenPKG RPM uses GNU shtool. All manual installation and patching is done with the shtool command. A companion tool, rpmtool, complements shtool with OpenPKG RPM and OS-specific features. The rpmtool allows all OpenPKG packages to generate their file list (%files) on the fly and makes the packaging information smaller. It reduces the required maintenance when vendor version updates occur as well.

OpenPKG RPM additionally provides a set of local macros (%{l_xxx}) to abstract system specifics and also help in removing redundancy from packaging specifications. For example, the %{l_prefix} is the file system prefix of the associated OpenPKG instance. Using OpenPKG's local macros offers a clear advantage, because packages no longer need hard-coded path prefixes and can therefore be built for arbitrary OpenPKG instances.

Macros exist for the most often used build variables. The %{l_cc} macro expands to either prefix/bin/cc (in case the OpenPKG gcc package is installed) or defaults to just cc. The same goes for "%{l_cflags -O}": it expands to the optimized C compiler flags. In case gcc is installed, it expands to "-O2 -pipe". Otherwise, it expands to just "-O" by default. The variables %{l_make} and %{l_mflags} work together in a similar way. If %{l_make} points to a known make which supports parallel building and the underlying system has more than one CPU, then "%{l_mflags -O}" expands to the necessary flags to leverage the system's multiple processing power. For example, on a 2 CPU FreeBSD machine with BSD make, "%{l_mflags -O}" expands to "-j4 -B" while on a 4 CPU Linux machine with GNU make, "%{l_mflags -O}" expands to "--no-print-directory -j8".

Many packages also deploy a tricky OpenPKG-specific package build option feature based on OpenPKG RPM's %define macro and the Provides header: A package specification foo.spec can contain zero or more "%option opt-name opt-default-value" lines which expand to both a "%ifndef opt-name %define opt-name opt-default-value %endif" construct and a "Provides: foo::opt-name = opt-value" header definition. First, this allows the package itself to conditionally build with variations through the use of following '%if "%{opt-name}" == "opt-value"%endif' constructs inside the foo.spec file.

The compared effective option value is either opt-default-value or the opt-override-value from a "--define 'opt-name opt-override-value'" or "--with opt-name" or or "--without opt-name" command line options. Second, the resulting source RPM can be queried with "openpkg rpm -qp --provides" or "rpm -qpi" in order to lookup the provided options and their default values. The same can be done for the binary RPM to lookup the options and their effective values which were used to built the package. Third, another package depending on foo usually just uses "[Build]PreReq:foo". In OpenPKG it also can depend on a particular build variation of foo by using "[Build]PreReq:foo, foo::opt-name = opt-value". The consequent of this functionality provides both a clean, precise and flexible packaging.

Additionally, all OpenPKG packages follow exactly the same style as the RSYNC example (see Listing 1, Listing 2, and Listing 3). The header order, indentation, etc. is fully standardized, and allows developers to easily query and even semi-automatically edit package information directly from the source. Incidentally, the indices on the OpenPKG FTP server and the OpenPKG release engineering procedures are auto generated by exploiting this standard scheme.

Every OpenPKG package is able to build in an unprivileged (non-root user) environment and with read-only access to an OpenPKG instance. This allows safe (no development system intrusion) and precise (no trashed or missing files) packaging. Such security and precision is achieved by consistently using the BuildRoot feature of RPM for all packages. In short, this means that when rolling a binary package the software is redirected to install into a shadow area (prefix/RPM/TMP/pkg-name-root/prefix). The package is then made from the shadow area just as if it were located in the real file system location (prefix). This important improvement to the standard RPM behavior may sound trivial and easy to achieve, but is actually one of the trickiest steps in packaging software for OpenPKG.

Sometimes (as with RSYNC), it is just a matter of overriding variables (prefix in the example) on the "make install" step. Other times the solution is more involved. For some OpenPKG packages it takes a lot of effort to find a reasonable way to redirect the vendor installation to the BuildRoot location, but the extra effort is always worthwhile and results in safer and more precise packaging.

Finally, OpenPKG's RPM implementation provides proxy packages, a tricky and appealing mechanism for reusing the packages of a master OpenPKG instance. Proxy packages can reside in multiple slave OpenPKG instances, and allow the system administrator to avoid redundant building and maintaining of the same software package in multiple OpenPKG instances. For example, gcc is typically required by many packages at build time. A gcc OpenPKG package is usually needed in every OpenPKG instance.

A savvy system administrator will install a single gcc package in a master OpenPKG instance and then only proxy packages (pointing to the real gcc) in the other OpenPKG instances by running slave-prefix/bin/openpkg makeproxy on the gcc binary RPM of the master instance. OpenPKG's RPM will then produce a binary RPM package for the slace instance, containing a shadow tree resembling the contents of the master instance. The shadow tree is technically nothing more than symbolic links to the master (non-proxy) package's files and directories. This mechanism can save a lot of time and storage, however it should only be applied to packages with global configuration dependencies only or with no configuration dependencies at all.

Integrating unpackaged software

No matter how many packages OpenPKG provides, the world will always have appealing yet unpackaged software. Ambitious system administrators can package the software themselves for local purposes and even contribute such new packages to the OpenPKG community. Alternatively, the local subdirectory of an OpenPKG instance exists for the purpose of containing unpackaged software, and can be instrumental in integrating a base of OpenPKG packages with other unpackaged software in an easy to maintain way. OpenPKG also provides a corresponding lsync tool to aid such integration.

To integrate unpackaged software into an OpenPKG instance, each unpackaged software component may be installed into the bin, sbin, man, info, include and lib subdirectories of prefix/local/PKG/pkg-name/ and then virtually linked into the corresponding top-level directories under prefix/local/ by running prefix/bin/openpkg lsync. This easy to administer strategy leads to a very clean and maintainable OpenPKG instance, even with its new coexisting unpackaged software in prefix/local/. This especially makes it easy to uninstall a package. Just remove prefix/local/PKG/pkg-name/ with all its contents and run openpkg lsync again.

This strategy even allows for installation of different versions of the same software. Just install into prefix/local/PKG/pkg-name-version/ and add a symbolic link pointing from prefix/local/PKG/pkg-name/ to this directory. This works, because openpkg lsync skips subdirectories of prefix/local/PKG/ with version numbers attached. To upgrade an older foo-0.7.41 to foo-0.7.42, just repeat the installation in the same way, altering the symlink prefix/local/PKG/foo to point to foo-0.7.42 instead and running openpkg lsync again. openpkg lsync will automatically update symlinks, creating new links if required and removing outdated dangling ones. As might be guessed, it is just as easy to go back to the old version if the new one keeps dumping core or something like that. For an example of such multiple unpackaged software installation see Figure 3.

OpenPKG release engineering

A carefully crafted release process is part of the OpenPKG project, and the fruits of the whole project are available to the public according to Open Source standards. All sources (package specifications, source patches, website sources, the handbook, this article, etc) are located in a publicly readable central CVS repository, which can be browsed anonymously by conventional cvs commands or through the website for added convenience. Additionally, all developer commits to this repository are tracked and summarized with postings to public mailing lists and public newsgroups. People can easily follow all developments by subscribing to the list or reading the newsgroup.

For stability and to reduce conflicts between development milestones, OpenPKG has three release branches (which technically directly map to CVS branches.) These are OpenPKG-SOLID, OpenPKG-STABLE and OpenPKG-CURRENT. OpenPKG-SOLID is the security update branch of the last public OpenPKG release. OpenPKG-STABLE is the stable branch from whose contents the next public release is made. OpenPKG-CURRENT is the current state of the development branch, and contains packages of beta-grade stability.

In any case, the branch from which a package was built can easily be determined by the OpenPKG RPM file name, because they follow a consistent naming scheme: pkg-name-version-YYYYMMDD (for CURRENT), pkg-name-version-N.YYYYMMDD (for N-STABLE), pkg-name-version-N.M.X (for N.M.X-SOLID). Once such a source OpenPKG RPM package is built, the new binary OpenPKG RPM package filename contains even more information, such as operating system, hardware, and the OpenPKG instance tag.

The OpenPKG developer team is extremely very fast in keeping OpenPKG-CURRENT packages up to date and in sync with the latest upstream vendor versions. This is due to the fact that the versions of all externally available vendor sources are fully automatically tracked on a bi-daily basis. An OpenPKG package for a new vendor software version is often available before the software is even announced on Freshmeat.net.

Finally, OpenPKG takes security very seriously. Experience has shown that "security through obscurity" does not work, and that public disclosure leads to quicker and better solutions to security problems. In that vein, OpenPKG tries to release fixed packages as fast as possible after a vulnerability was discovered. The OpenPKG security release and advisory process notifies the community by publishing official security advisories in the security section of the website and on the mailing lists.

Conclusion

OpenPKG is an Open Source software project existing since November 2000. The implementation relies on RPM 4 for its basic packaging mechanism, but offers more than RPM alone. To meet its goal of becoming a modular and flexible Unix subsystem for cross-platform software packaging and installation, OpenPKG includes tricky bootstrapping logic that installs a customized implementation of RPM 4 on any of the supported target platforms.

OpenPKG is in production use since April 2001 in datacenter environments of large ISPs. Since its public release in January 2002, OpenPKG users have profited from an increase from 220 to nearly 1000 software packages. The project is continuously improved by a diverse team of developers who also daily update and add packages.

The base of OpenPKG software packages is expected to increase even more, partly due to the ease of writing specifications and building packages. Most OpenPKG users find it deceivingly simple to build a basic package. New users interested in such packaging can use the RSYNC example in this article as a blueprint. Accordingly, package contributions are always appreciated by the members of the OpenPKG project and the members of the OpenPKG Foundation.

References

OpenPKG: http://www.openpkg.org/ ftp://ftp.openpkg.org/

OpenPKG Community Forums: mailto:openpkg-users@openpkg.org mailto:openpkg-dev@openpkg.org nntp://news.openpkg.org/openpkg.users nntp://news.openpkg.org/openpkg.dev

About the authors

Ralf S. Engelschall is a computer scientist and Open Source software hacker. He is the author of well-known software like Apache mod_ssl, GNU Pth, and GNU Shtool and the founder of Open Source software projects like OpenSSL, OpenPKG, and OSSP. He can be contacted at: rse@engelschall.com

Thomas Lotterer is a network professional and consultant working as a Unix software developer. He gained experience in cross-platform system integration and software distribution by working previously as a system administrator and technical trainer. Today, Thomas works actively on the OpenPKG and OSSP projects. He can be contacted at: thomas@lotterer.net

Christoph Schug is a senior Unix system administrator. His revolutionary ideas and visions often result in additional lines in Ralf's TODO list. When not in the office, Christoph might be found in the Alps steering the screaming and smoking tires of his Miata MX-5 roadster. He can be contacted at: chris@schug.net

Michael Schloh von Bennewitz is a software engineer. He was an active contributor to both the OpenPKG and OSSP projects. He can be contacted at: michael@schloh.com

(Last Modified: 2006-09-26)

Figure 1: The OpenPKG subsystem principle

Figure 2: The OpenPKG file system layout

Figure 3: File system layout of non-OpenPKG software installed

Listing 1: OpenPKG packaging for rsync (rsync.spec: OpenPKG RPM specification)
##
##  rsync.spec — OpenPKG RPM Package Specification
##  Copyright © 2000-2006 OpenPKG Foundation e.V. <http://openpkg.net/>
##  Copyright © 2000-2006 Ralf S. Engelschall <http://engelschall.com/>
##
##  Permission to use, copy, modify, and distribute this software for
##  any purpose with or without fee is hereby granted, provided that
##  the above copyright notice and this permission notice appear in all
##  copies.
##
##  THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED
##  WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
##  MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
##  IN NO EVENT SHALL THE AUTHORS AND COPYRIGHT HOLDERS AND THEIR
##  CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
##  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
##  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
##  USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
##  ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
##  OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
##  OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
##  SUCH DAMAGE.
##

#   package information
Name:         rsync
Summary:      Remote Filesystem Synchronization
URL:          http://rsync.samba.org/
Vendor:       Andrew Tridgell & Paul Mackerras
Packager:     OpenPKG
Distribution: OpenPKG
Class:        CORE
Group:        Filesystem
License:      GPL
Version:      2.6.8
Release:      20060904

#   package options
%option       with_lastmatch  no
%option       with_timelimit  no

#   list of sources
Source0:      http://rsync.samba.org/ftp/rsync/rsync-%{version}.tar.gz
Source1:      rsync.conf
Source2:      rc.rsync
Patch0:       rsync.patch

#   build information
Prefix:       %{l_prefix}
BuildRoot:    %{l_buildroot}
BuildPreReq:  OpenPKG, openpkg >= 20060823
PreReq:       OpenPKG, openpkg >= 20060823
AutoReq:      no
AutoReqProv:  no

%description
    Rsync is a replacement for rcp(1) that has many more features.
    Rsync uses the "rsync algorithm" which provides a very fast method
    for bringing remote files into sync. It does this by sending just
    the differences in the files across the link, without requiring
    that both sets of files are present at one of the ends of the link
    beforehand. At first glance this may seem impossible because the
    calculation of differences between two files normally requires local
    access to both files.

%track
    prog rsync = {
        version   = %{version}
        url       = http://rsync.samba.org/ftp/rsync/
        regex     = rsync-(\d+\.\d+\.\d+)\.tar\.gz
    }

%prep
    #   unpack vendor sources
    %setup -q
    %patch -p0
%if "%{with_lastmatch}" == "yes"
    %{l_patch} -p1 <patches/last-match.diff
%endif
%if "%{with_timelimit}" == "yes"
    %{l_patch} -p1 <patches/time-limit.diff
%endif

%build
    #   configure vendor sources
    CC="%{l_cc}" \
    CFLAGS="%{l_cflags -O}" \
    ./configure \
        --prefix=%{l_prefix} \
        --with-rsyncd-conf=%{l_prefix}/etc/rsync/rsync.conf \
        --disable-debug \
        --disable-locale \
        --with-included-popt \
        --with-rsh=ssh

    #   build vendor sources
    %{l_make} %{l_mflags -O}

%install
    #   perform vendor installation
    rm -rf $RPM_BUILD_ROOT
    %{l_make} %{l_mflags} install \
        prefix=$RPM_BUILD_ROOT%{l_prefix} \
        exec_prefix=$RPM_BUILD_ROOT%{l_prefix}

    #   post-adjust vendor installation
    strip $RPM_BUILD_ROOT%{l_prefix}/bin/* >/dev/null 2>&1 || true
    mv  $RPM_BUILD_ROOT%{l_prefix}/man/man5/rsyncd.conf.5 \
        $RPM_BUILD_ROOT%{l_prefix}/man/man5/rsync.conf.5

    #   provide default (deamon) configuration
    %{l_shtool} mkdir -f -p -m 755 \
        $RPM_BUILD_ROOT%{l_prefix}/etc/rsync
    %{l_shtool} install -c -m 644 %{l_value -s -a} \
        -e 's;@version@;%{version};g' \
        %{SOURCE rsync.conf} $RPM_BUILD_ROOT%{l_prefix}/etc/rsync/
    %{l_shtool} install -c -m 644 /dev/null \
        $RPM_BUILD_ROOT%{l_prefix}/etc/rsync/rsync.passwd
    %{l_shtool} mkdir -f -p -m 755 \
        $RPM_BUILD_ROOT%{l_prefix}/var/rsync

    #   provide run-command script
    %{l_shtool} mkdir -f -p -m 755 \
        $RPM_BUILD_ROOT%{l_prefix}/etc/rc.d
    %{l_shtool} install -c -m 755 %{l_value -s -a} \
        %{SOURCE rc.rsync} $RPM_BUILD_ROOT%{l_prefix}/etc/rc.d/

    #   determine binary package files
    %{l_rpmtool} files -v -ofiles -r$RPM_BUILD_ROOT \
        %{l_files_std} '%config %{l_prefix}/etc/rsync/rsync.*'

%files -f files

%clean
    rm -rf $RPM_BUILD_ROOT

%post
    #   after upgrade, restart service
    [  -eq 2 ] || exit 0
    eval `%{l_rc} rsync status 2>/dev/null`
    [ ".$rsync_active" = .yes ] && %{l_rc} rsync restart
    exit 0

%preun
    #   before erase, stop service and remove log files
    [  -eq 0 ] || exit 0
    %{l_rc} rsync stop 2>/dev/null
    rm -f $RPM_INSTALL_PREFIX/var/rsync/rsync.log* >/dev/null 2>&1 || true
    exit 0

Listing 2: OpenPKG packaging for rsync (rc.rsync: run-commands)
#!@l_prefix@/bin/openpkg rc
##
##  rc.rsync — Run-Commands
##

%config
    rsync_enable="$openpkg_rc_def"
    rsync_daemon="no"
    rsync_flags=""
    rsync_nicelevel="10"
    rsync_log_prolog="true"
    rsync_log_epilog="true"
    rsync_log_numfiles="10"
    rsync_log_minsize="1M"
    rsync_log_complevel="9"

%common
    rsync_cfgfile="@l_prefix@/etc/rsync/rsync.conf"
    rsync_logfile="@l_prefix@/var/rsync/rsync.log"
    rsync_pidfile="@l_prefix@/var/rsync/rsync.pid"
    rsync_signal () {
        [ -f $rsync_pidfile ] && kill - `cat $rsync_pidfile`
    }

%status -u @l_susr@ -o
    rsync_usable="unknown"
    rsync_active="no"
    rcService rsync enable yes && \
        rsync_signal 0 && rsync_active="yes"
    echo "rsync_enable=\"$rsync_enable\""
    echo "rsync_usable=\"$rsync_usable\""
    echo "rsync_active=\"$rsync_active\""

%start -u @l_susr@
    rcService rsync enable yes || exit 0
    rcService rsync active yes && exit 0
    rcVarIsYes rsync_daemon || exit 0
    nice -n $rsync_nicelevel @l_prefix@/bin/rsync \
         $rsync_flags --daemon --config=$rsync_cfgfile
    sleep 2

%stop -u @l_susr@
    rcService rsync enable yes || exit 0
    rcService rsync active no  && exit 0
    rcVarIsYes rsync_daemon || exit 0
    rsync_signal TERM
    sleep 2

%restart -u @l_susr@
    rcService rsync enable yes || exit 0
    rcService rsync active no  && exit 0
    rcVarIsYes rsync_daemon || exit 0
    rc rsync stop start

%daily -u @l_susr@
    rcService rsync enable yes || exit 0
    rcVarIsYes rsync_daemon || exit 0
    shtool rotate -f \
        -n ${rsync_log_numfiles} -s ${rsync_log_minsize} -d \
        -z ${rsync_log_complevel} -m 644 -o @l_susr@ -g @l_mgrp@ \
        -P "${rsync_log_prolog}" \
        -E "${rsync_log_epilog}; rc rsync restart" \
        $rsync_logfile

Listing 3: OpenPKG packaging for rsync (rsync.conf: default configuration)
##
##  rsync.conf — OpenPKG rsync configuration (daemon mode only)
##

secrets file       = @l_prefix@/etc/rsync/rsync.passwd
lock file          = @l_prefix@/var/rsync/rsync.lck
pid file           = @l_prefix@/var/rsync/rsync.pid
log file           = @l_prefix@/var/rsync/rsync.log
    
address            = 127.0.0.1
port               = 873

max connections    = 20
timeout            = 60
strict modes       = yes
use chroot         = yes
read only          = yes
ignore nonreadable = yes
transfer logging   = yes
log format         = "%o %h [%a] %m (%u) %f %l"
dont compress      = *.bz2 *.gz *.zip *.z *.rpm *.deb *.tgz *.iso
list               = no
uid                = @l_nusr@
gid                = @l_ngrp@

[pub]
comment            = Public area of @l_prefix@ OpenPKG instance
path               = @l_prefix@/pub/
exclude            = .*
read only          = yes
list               = yes

Listing 4: Bootstrapping OpenPKG instance under /openpkg
1. manually download the "openpkg" package

user$ cd /tmp
user$ ftp ftp.openpkg.org
Connected to ftp.openpkg.org.
220 ftp.openpkg.org OpenPKG Anonymous FTP Server ready.
Name (ftp.openpkg.org): anonymous
331 Anonymous login ok, send your email address as your password.
Password: you@example.com
230- […] Welcome to OpenPKG.org! […]
230 Anonymous access granted, restrictions apply.
ftp> bin
200 Type set to I.
ftp> cd stable/2.20060622/UPD
ftp> get openpkg-2.20060815-2.20060815.src.sh
ftp> bye
221 Goodbye.

2. build and install "openpkg" package to bootstrap instance

user$ sh openpkg-2.20060615-2.20060615.src.sh \
      --prefix=/openpkg --user=openpkg --group=openpkg
user$ su –
root# sh openpkg-2.20060615-2.20060615.*-*-*.sh
root$ rm -f openpkg-2.20060615-2.20060615.*

3. build, install and start "rsync" package

root# /openpkg/bin/openpkg rpm --rebuild \
      ftp://ftp.openpkg.org/stable/2.20060622/SRC/rsync-2.6.8-2.20060622.src.rpm
root# /openpkg/bin/openpkg rpm -Uvh /openpkg/RPM/PKG/rsync-2.6.8-2.20060622.*-*-*.rpm
root# /openpkg/bin/openpkg rc rsync start
root# exit

4. check that RSYNC service is working

user$ /openpkg/bin/rsync rsync://localhost/
pub             Public area of /openpkg OpenPKG instance

5. cleanup, shutdown and deinstall everything again

user$ su –
root$ rm -f /openpkg/RPM/PKG/*
root# /openpkg/bin/openpkg rc rsync stop
root# /openpkg/bin/rpm -e rsync openpkg
root# exit

Listing 5: Installation of non-OpenPKG software
user$ su –
root# mkdir -p /openpkg/local/PKG/foo-0.7.42/SRC
root# cd /openpkg/local/PKG/foo-0.7.42/SRC
root# wget ftp://ftp.example.com/foo-0.7.42.tar.gz
root# gunzip <foo-0.7.42.tar.gz | tar xf –
root# cd foo-0.7.42
root# ./configure --prefix=/openpkg/local/PKG/foo-0.7.42
root# make
root# make install
root# cd /openpkg/local/PKG
root# ln -s foo-0.7.42 foo
root# /openpkg/bin/openpkg lsync

Table 1: RPM handling of configuration files
old in   old on     new in   new on 
package  filesystem package  filesystem approach
======== ========== ======== ========== ===========================================
X        X          X        X          keep old config
X        X'         X'       X'         keep old config
X        X'         X        X'         keep old config
X        X          Y        Y          install new config
X        X'         Y        Y          install new config, preserve X' as .rpmsave
–        X          Y        Y          install new config, preserve X  as .rpmorig

Table 2: OpenPKG Administration Commands (Quick Reference)
$ <prefix>/bin/openpkg rpm -Uvh <pkg-src>                unpack source package
$ <prefix>/bin/openpkg rpm -bb <pkg-spec>                build binary package from unpacked sources
$ <prefix>/bin/openpkg rpm --clean --rmsource <pkg-spec> cleanup after building binary package
$ <prefix>/bin/openpkg rpm --rebuild <pkg-src>           build binary package from source package

# <prefix>/bin/openpkg rpm -Uvh <pkg-bin>                install/upgrade package
# <prefix>/bin/openpkg rpm -Uvh --nodeps <pkg-bin>       install/upgrade package (ignore dependencies)
# <prefix>/bin/openpkg rpm -Uvh --force <pkg-bin>        install/upgrade package (no checks at all)
# <prefix>/bin/openpkg rpm -Uvh --oldpackage <pkg-bin>   install/downgrade package
# <prefix>/bin/openpkg rpm -Fvh <pkg-bin>                upgrade package (if already installed only)

# <prefix>/bin/openpkg rpm -e <pkg-name>                 uninstall/erase package
# <prefix>/bin/openpkg rpm -e --nodeps <pkg-name>        uninstall/erase package (ignore dependencies)

$ <prefix>/bin/openpkg rpm -qpi <pkg-bin>                list information about binary package
$ <prefix>/bin/openpkg rpm -qplv <pkg-bin>               list all files a binary package will install
$ <prefix>/bin/openpkg rpm -qa                           list all installed packages and their versions
$ <prefix>/bin/openpkg rpm -qi <name>                    list information about an installed package
$ <prefix>/bin/openpkg rpm -qlv <name>                   list all files a package has installed
$ <prefix>/bin/openpkg rpm -qf <prefix>/…              list package owning a particular file
$ <prefix>/bin/openpkg rpm -qc <name>                    list all config files of an installed package

$ <prefix>/bin/openpkg rpm --checksig <pkg-rpm>          verify integrity and origin of a package
$ <prefix>/bin/openpkg rpm -Va                           check the integrity of all packages
$ <prefix>/bin/openpkg rpm -V <name>                     check the integrity of a particular package
$ <prefix>/bin/openpkg rpm -Va --nofiles                 check all package dependencies only

$ <prefix>/bin/openpkg rc --config                       show all %config variables and values
$ <prefix>/bin/openpkg rc --query <variable>             query a particular %config variable
# <prefix>/bin/openpkg rc <pkg_name> start               start a particular package
# <prefix>/bin/openpkg rc <pkg_name> stop                stop  a particular package
# <prefix>/bin/openpkg rc <pkg_name> stop start          restart a particular package
# <prefix>/bin/openpkg rc all stop start                 restart all packages
$ eval `<prefix>/bin/openpkg rc --eval all env`          setup shell environment of all packages 

Legend: #          command has to be executed with     root privileges
        $          command should be executed withough root privileges
        <prefix>   e.g. /openpkg
        <pkg-name> e.g. rsync
        <pkg-spec> e.g. rsync.spec
        <pkg-rpm>  e.g. rsync-2.6.8-2.20060622.*.rpm
        <pkg-src>  e.g. rsync-2.6.8-2.20060622.src.rpm
        <pkg-bin>  e.g. rsync-2.6.8-2.20060622.ix86-freebsd6.1-openpkg.rpm

Table 4: OpenPKG summary
- Open source software
– Cross-platform solution for FreeBSD, Linux, Solaris and others
– Subsystem on top of underlying Unix system
– Minimal OS intrusion
– Fully self-contained with minimum OS dependencies
– Primarily targeted at server environments
– Based on a customized RPM 4
– Support for entire package lifecycle
– Build and run-time dependency control
– Builds software from pristine vendor sources
– All OpenPKG RPM package specifications written from scratch
– Clear and concise specifications follow a strict style
– Supports integration of unpackaged software also
– Tricky bootstrapping with no requirement for a preinstalled RPM
– Strict and consistent file system layout
– All packages installed with reasonable preconfiguration
– Users benefits from packager's knowledge
– Cryptographically signed packages
– Package integrity verification
– Support for building packages as unprivileged user
– Installation under arbitrary file system prefix
– Allows multiple instances on the same host
– Fully uninstalls packages
– Powerful package queries
– Flexible run-command facility
– Mature technology in datacenter production use since April 2001
– Variety of nearly 1000 packages available