rPath Technical Guide:Packaging with Conary
From rPath Wiki
Introductory Concepts
This document provides a current conceptual explanation of Conary adapted from earlier writings and updated to fit current usage. If you are new to Conary, use the links in each section to learn more about each concept.
The Conary software provisioning system is a means to package, deliver, and maintain software. Conary packaging technology is the open-source package management system at the heart of rPath's products and services.
Conary provides new features to package management, addressing some of the common issues of Linux system administration. Employing repositories, a branch structure with shadowing, and changesets, Conary offers the advantages of a revision control system to improve system maintenance tasks such as updating current software, installing new software, maintaining configurations during software updates, and resolving dependency issues. These features extend beyond previous software packaging technologies, creating a system in which many traditional Linux software maintenance tasks are automated and efficient, significantly reducing the time spent on system management.
Conary also offers a finer breakdown of system components and a structure designed to be easily streamlined. This streamlining can be used to build a dedicated software appliance in which the only installed pieces of the operating system are those necessary to support a particular application. rPath Linux is built using Conary packaging technology, and rBuilder is designed for developing and distributing Conary-based appliances.
Repository
A Conary repository is a database that contains source and binary files organized in packages which consist of any number of components. This database is accessible over a network (including over the Internet if needed) by Conary-based systems, and it is the means by which these systems can install and maintain their software. The repository contains multiple packages in multiple versions along multiple development branches, and Conary ensures that system maintenance uses the appropriate version on the appropriate branch when updating a system. Access to a package in the database requires enough information to distinguish the exact version required, but much of this information is maintained in configuration files and installed package information to further automate the update process.
Repository Contents
A repository provides the packages that can be installed on a Conary-based system. Other types of content in the repository include components, groups, and filesets, all of which represent a type of trove, or versioned item in the repository.
Packages
A package represents a piece of software that can be installed on a Conary-based system. When software is packaged for Conary, building the package creates components, and the package contains those components. The Conary QuickReference provides instructions for installing, updating, and removing packages on a Conary-based system.
When a user installs a package, Conary installs all the components of that package. Conary will also indicate any dependencies the package requires, and the user can add an option to let Conary automatically resolve those dependencies. When Conary installs a package with this option, it additionally installs only the components necessary to resolve the dependency.
When a user updates installed packages, Conary only retrieves files that have changed since the last update on the package. For example, if the component foo:runtime has five files, and only one of those files has changed between two versions, then only that one file will be updated when updating between the two versions.
From the respository's perspective, Conary aims for similar efficiency. The new version of a package does not contain duplicates of files in previous versions. Only if a file has changed will its update information be saved to the repository.
Not all Conary users will need to package software, but those that do may find it helpful to understand Conary packaging concepts prior to packaging. Packaging software for Conary includes:
- Writing a recipe to tell Conary how to build the package
- Cooking the recipe and testing the package repeatedly in an effort to develop the recipe
- Committing the package to a repository and cooking the package in the repository
The Conary New Package Tutorial introduces this packaging process to both new packagers and experienced Linux packagers who are new to Conary.
Components
A component is a grouping of files that are needed to install some specific part of software, such as the application, libraries, or documentation. When a package is built, Conary automatically assigns files to components (though this assignment can be overridden if desired). Components are identified using the package name to which it applies, a colon (:), and a suffix describing the component (such as package_name:lib and package_name:docs).
A source component (:source) contains files and is named like other components, but Conary treats source components differently during development and maintenance of packages. Though it is named using the original package for which it was built, a source component is versioned differently, and it is not considered part of any package. A source component may be used to build multiple packages, and it not installable like other components.
Troves
A trove is a collection of files or a collection other troves. The term trove is used often in Conary discussions. A trove could refer to a component, a package, a group, or a fileset.
A trove is identified by name, version, and flavor; these three values make the trove unique in the repository. When a file in a trove is first saved to a repository, it is assigned a unique identifier so that it can be tracked by that identifier instead of by name. The file identifier allows Conary to handle continued updates to the file, including filename changes, while preserving its unique existence in the trove.
Groups
A group contains a grouping of other kinds of troves (components, packages, filesets, or groups) that are found in the same repository. Groups are mainly used to pull together packages and components that should be installed together to create a complete infrastructure for software. A group name has the prefix group-. The Conary QuickReference provides instructions for installing and updating groups on a Conary-based system.
Groups are versioned like other troves and can be updated in the same way. The advantage of creating a group of packages and components is that these troves can be updated and maintained together. The group identifies specific versions of each of its contents. Installing a group means installing those specific versions of packages and components. Because of this, installing or updating a group may not result in receiving the latest available version of its contents because a package or component in the group may have a later version available outside of the group. Instead, installing and updating by group ensures that the group maintains the versions that were intended. Groups allow a developer to distribute software that is assured to work together. If packages or components in a group are updated, the group update will only affect those updated troves.
For example, suppose you have installed group-foobar on your Conary-based system which consists of packages foo-1.2.1 and bar-2.1.1. Suppose you are told that newer version foo-2.0.1 is now available, and you perform a system-wide update on your system to ensure you have the latest version. Because foo was installed as part of the group and not on its own, your version of foo remains 1.2.1. Then, suppose group-foobar is updated to use bar-2.2.1. Another system update results in a group update; while foo remains in version 1.2.1, bar is updated to version 2.2.1.
Filesets
A fileset contains only files from components in the repository. Filesets can be created to designate a custom group of files in the repository outside of the component, package, and group purposes. A fileset name has the prefix fileset-. This type of trove is rarely used outside of low-level Conary development.
Repository Access
Conary uses client-side repository access features that enhance Conary's functionality.
Repository Map
A repository map setting on a Conary-based system maps a hostname from a label to a full URL for a networked repository. When software is installed using Conary, the label is part of the information that identifies the software and its source. If the repository changes locations, the repository map can be updated to ensure that Conary finds the relocated repository when a user runs software updates. Repository map settings are maintained in a Conary configuration file on the Conary-based system.
Install Label Path
Though it is an option, Conary does not require a user to specify a label when installing a trove from a repository. When a user names the trove (with or without specifying a revision), the repositories listed in the system's install label path are searched in order to locate that trove and fulfill the request. As needed, repositories can be appended to or removed from this path. Install label path settings are maintained in the installLabelPath line of a Conary configuration file on the Conary-based system.
Mirrors
Conary provides the option to mirror a single repository onto another Conary system. This can be useful when backing up a repository or when making privately-developed software packages available in a publically accessible repository. Conary users can create a standalone repository and mirror to that repository. Additionally, project owners on an rBuilder Online (rBO) project can create a mirror of the rBO repository identified with that project. On the software appliance scale, appliance developers and maintainers can use the rPath Appliance Platform Update Service appliance to manage mirrors provided to multiple consumers (such as software vendors or customers).
Label Structure
Conary uses labels to simplify update paths for software. This structure is aimed at eliminating the guesswork when identifying:
- Whether the package is "newer" or "older"
- The appropriate upstream development path the current package has followed to its current version
Conary identifies each component with a version string that identifies the originating respository and revision number. When a user performs system update activities, Conary uses this string as the update path to follow for each trove.
Releated page: Conary:Version String
Label Affinity
Conary supports divergent development by allowing that development to occur on different branches. Such development in the Conary repository creates a branched structure and a need to identify which label to use when updating packages installed from the repository. When a trove is installed on a Conary-based system, the updates for that trove will come from the same label in the respository from which it was installed. This concept, called label affinity, ensures the update path is consistent, even when the development is not linear. The identifiers that Conary creates for ensuring label affinity are hidden from users, and it is not necessary to know them when updating software on a Conary-based system.
Conary recognizes an appended version string with a slash and labels as a unique identifier for a development branch. When the branch is created, the branch name becomes the label for that branch. This label inherits the namespace, so it can be referred to not only with the longer slash-separated strings, but also in place of the original tag. In the example, 1 is the branch name, and the full label for the branch is conary.rpath.com@rpl:1.
/conary.rpath.com@rpl:devel//1/1.5.0-2-3 or conary.rpath.com@rpl:1
If a package is installed from this label, the label name ensures that Conary uses the correct update path in the repository.
Historically, Conary had separate branch and shadow creations. Explicit branch creation was deprecated so that shadows alone create a repository's branch structure.
Versions
As previously described, Conary structures packaged software in components, and each component contains one or more files. The version string provides a globally unique identifier for the component. See Conary Version String which describes the version string and its parts, including the commonly used branch, label, and revision values.
Flavors
Along with the name and version, the flavor of a trove makes it unique among troves that can be installed on a Conary-based system. In the same way that the version identifies where and when in a sequence the trove was built, the flavor identifies how it was built. This information is part of the build instructions and includes selecting options in the packaged software, such as designating an architecture or ensuring support for a particular database. Flavors are typically invisible to users installing software because Conary automatically selects the most appropriate flavor for a package for the target system. Conary provides an option to override this automatic choice to users wishing to select a particular flavor of the package.
Related page: Conary:Flavors
Shadows
Conary provides the shadow concept as a means of creating parallel development branches. Shadows can keep track of upstream changes from a parent branch, and developers can choose to merge changes from any new revision on the parent at any time. In this way, developers can be responsible for their own changes without maintaining the entire development stream.
Related pages: Conary:Shadows | Conary:Merge
Changesets
A Conary changeset is a file that describes modifications to be made to a system in order to apply a Conary update. A changeset can include how particular files have changed as well as how their associated troves have changed. Conary uses changesets to make the update process more efficient and less intrusive to a Conary-based system. By applying only the changes that are needed, files that have not changed are not needlessly updated, and locally-changed files are merged with updated files instead of overwritten.
A changeset represents the difference between two versions. An absolute changeset is the difference between having no version installed on a system and installing the desired version. A relative changeset is the difference between a currently installed version and the desired version. A relative changeset could imply either an upgrade to a newer version or a downgrade to an older version.
Changesets are typically invisible to the user making updates to a Conary-based system. During an update operation, Conary creates the changeset dynamically before downloading and applying that changeset. The changeset may be cached by the repository server, but it is not saved as a separate file on the target system. In some instances, a system administrator or software developer may generate a static changeset file to distribute a desired set of changes.
One use of a static changeset is a local changeset, which is a relative changeset containing the differences between the trove as installed and configured on the local system and the trove as it exists in the repository. System administrators can distribute certain configuration and permission changes on a trove by creating and distributing a local changeset. The local changeset can be distributed to target machines with the same trove version, and it can be committed to a branch of the repository from which target machines can install the necessary update.
Rollbacks
As a result of the use of changesets in Conary, a Conary-based system provides the user an opportunity to roll back changes that have been made to the system as the result of a Conary update. Conary maintains a list of rollbacks, and the system administrator can roll back a trove or the entire system when necessary to bring the system back to a particular state.
Understanding Flavors
A name, version string, and flavor together identify a unique trove (such as a package or group) in a Conary repository. A single revision of a trove can be built with multiple flavors, each representing a particular scenario, such as usage on a 64-bit architecture or deployment as a VMware virtual machine.
A flavor is made up of flavor specifications, and each specification indicates some special condition for which the trove was built.
For example, one flavor specification is "vmware" which indicates that it only applies for VMware images. For appliance groups, this specification determines whether VMware tools are part of the appliance. If this specification is not explicitly included when a group is built, Conary assumes a default value of "preferred not vmware" (written "~!vmware"). If the specification is provided when building the group, Conary will built the group as "vmware" or "preferred vmware" (written "~vmware"). This specification will become part of the string that makes up the group's flavor.
Query and Read Flavors
When querying information about a trove, its associated flavor specification is a string of values separated by commas in square brackets. Use the --flavors option to reveal the flavors for a trove:
$> conary q group-core --flavors group-core=1.1-0.14-2[~Mesa.dri,~MySQL-python.threadsafe,X,~!alternatives,~!ati, ~!bootstrap,~buildtests,desktop,~!dom0,~!domU,emacs,gcj,gnome,~!grub.static,gtk, ipv6,~!kernel.debug,~!kernel.debugdata,~!kernel.numa,~!kernel.pae,~kernel.smp, krb,ldap,nptl,~!nvidia,~!openssh.smartcard,~!openssh.static_libcrypto,pam,pcre, perl,~!pie,~!postfix.mysql,python,qt,readline,sasl,~!selinux,~sqlite.threadsafe, ssl,tcl,tcpwrappers,tk,~!xen,~xorg-server.dmx,~xorg-server.dri, ~xorg-server.xnest is: x86(i486,i586,i686,~!sse2)]
(Normally flavor specifications are presented as a single line; it has been split into multiple lines here for better readability.)
When reading a flavor, note the following mnemonics for each flavor specification:
- A flavor with no preceding mark means requires
- A flavor preceded by ~ means prefers
- A flavor preceded by ! means not required
- A flavor preceded by ~! means not preferred
More recent versions of Conary shorten these flavors to show only the specifications that are relevant to the unique trove.
Use Flavors Specifications when Building
rPath Linux (rPL) and the rPath Appliance Platform Linux Service (rLS) have components throughout with various flavor specifications. This provides packagers and appliance builders with the option to indicate a flavor specification as a way to have Conary automatically include the appropriate parts of rPL or rLS. Two common uses of this is to build for 64-bit architecture and to build for virtual environments.
For example, Example Appliance uses the appliance group "group-example." Example Appliance vendors need to provide the appliance for both 32-bit and 64-bit architectures, and for both hardware-based installs and VMware virtual machine deployments. To ensure that a the group is built for all combinations of conditions, it is built with four flavors. This results in four builds of the same version, each with a different flavor. The build command looks something like this:
$> rmake build 'group-example[is: x86]' 'group-example[vmware is: x86]' 'group-example[is: x86_64]' 'group-example[vmware is:x86_64]'
 | If a flavor is not specified when the package or group is build, Conary cooks the flavor that is most appropriate for the current buildFlavor (in the Conary configuration for current context). |
Create Flavor Specifications when Packaging
Packagers can use conditional statements in package or group recipe to create additional flavor specifications besides those already included in Conary. These specifications are used when building the package or group to indicate which parts of the conditional should apply to the build.
An example of using this would be to build a single package with multiple configurations. Suppose the package example-package needs a configuration for marketing users and a different configuration for IT users. The recipe for example-package can use a conditional that identifies adding two specific configurations as well as adding a default configuration. The following is an example of such a recipe:
class ExamplePackage(PackageRecipe): name = 'example-package' version = '1.0' Flags.marketing = False #default flavor is not marketing Flags.it = False #default flavor is not it def setup(): # Assert that marketing and it configurations cannot both be included assert not (Flags.marketing and Flags.it) # Set a macro identifying where to unpack the archive for the application r.macros.exampledir = '%(servicedir)s/%(name)s' r.addArchive('http://www.example.com/download/%(name)s_%(upver)s_install.tar.gz', dir='%(exampledir)s/') # In this example, no additional commands are required to set up the application # Include the configuration appropriate to the build flavor if Flags.marketing: r.addSource('marketing-config', dest'%(exampledir)s/config.file') elif Flags.it: r.addSource('it-config', dest'%(exampledir)s/config.file') else: r.addSource('default-config', dest'%(exampledir)s/config.file')
The following build command includes the arguments necessary to ensure example-package is built for three flavors at once, using the "prefer" and "prefer not" options with each flavor specification. Note that because these specifications are part of the example-package recipe class, they require "example-package" as part of each flavor specification:
$> rmake build 'example-package[~example-package.marketing,~!example-package.it]' 'example-package[~!example-package.marketing,~example-package.it]' 'example-package[~!example-package.marketing,~!example-package.it]'
Because the default flavor is both not marketing and not it, the third argument represents the default flavor of the group, installed when the package is installed without a flavor stated:
#> conary update example-package
When installing to use the Marketing configuration, install example-package as follows:
#> conary update example-package[example-package.marketing]
When installing to use the IT configuration, install example-package as follows:
#> conary update example-package[example-package.it]
Using Conary
This page provides a quick reference for Conary commands used by system administrators for day-to-day maintenance for Conary-based systems. rPath recommends using rPath Appliance Platform Agent for appliance administration when it is available, and the commands shown here can be used on desktop distributions or appliances when command line access is necessary.
NOTE: You can print this page through your browsers' file menu which should contain a "print" menu entry.
| Get information about installed and available software | |
| conary query | |
|
$> conary query example | Query installed software for information about its components and update path Read more |
| conary repquery | |
|
$> conary repquery | more | Query software available for installation from Conary repositories Read more |
| conary verify | |
|
$> conary verify --all | Verify whether changes have been made to installed files Read more |
| conary config | |
|
$> conary config | Display Conary configuration details in the current scope Read more |
| Update and roll back software and the entire system | |
| conary update | |
|
#> conary update example=conary.rpath.com@rpl:1 | Install and update software from a Conary repository, including updating to an earlier version Read more |
| conary updateall | |
|
#> conary updateall | Update the top-level groups and any additional troves not under those groups. --apply-critical installs only critical updates needed by Conary itself. Read more |
| conary remove | |
|
#> conary remove /usr/share/example/example.doc | Remove a file from the system and from Conary management, ensuring it is not reinstalled during Conary updates of the package or group containing it Read more |
| conary erase | |
|
#> conary erase example | Uninstall software (package or group) Read more |
| conary rollback | |
|
#> conary rollback 1 | Roll back a system to a prior state, before one or more conary update commands Read more |
| conary rblist | |
|
#> conary rblist | more | List the available rollback operations and the update operations that would be reversed during each rollback Read more |
| Update the kernel | |
| conary pin | |
|
#> conary pin example | Pin an installed trove (typically a component, package, or group) to prevent it from being modified or removed during a conary udpate or conary erase; kernels are pinned by default Read more |
| conary unpin | |
|
#> conary unpin example | Unpin an installed trove (typically a component, package, or group) to allow it to be modified by conary udpate or conary erase Read more |
| Work with Changeset Files | |
| conary showcs | |
|
$> conary showcs example-1.0.ccs | Show the contents of a changeset created by Conary and used to install, modify, or remove software on a Conary-based system; contents include what files will be added, changed, or removed when the changeset is applied with conary update Read more |
Pinning Troves with Conary
Use conary pin to pin a trove so that it cannot be erased. This means that a conary erase command on the pinned trove will fail and the trove will remain on the system.
Use conary unpin to unpin the trove and allow for the erase.
A trove's pinning status is displayed as part of the output produced by adding the --info option to the conary query command. In the following example, the zile package is not pinned as indicated by the False value beside Pinned:
# conary q zile --info Name : zile Build time: Fri Jun 30 20:20:24 2006 Version : 2.2.14-1-1 Label : contrib.rpath.org@rpl:devel Size : 234766 Pinned : False Flavor : is: x86
The user can pin the zile trove using conary pin:
# conary pin zile # conary q zile --info Name : zile Build time: Fri Jun 30 20:20:24 2006 Version : 2.2.14-1-1 Label : contrib.rpath.org@rpl:devel Size : 234766 Pinned : True Flavor : is: x86
Verify the trove is pinned by querying for the trove info again to confirm the True value beside Pinned.
In the example, the user cannot remove the zile trove because it is pinned:
# conary erase zile Not erasing zile - it is pinned. To erase this zile, run: conary unpin 'zile=/contrib.rpath.org@rpl:devel/2.2.14-1-1[is: x86]' conary erase 'zile=/contrib.rpath.org@rpl:devel/2.2.14-1-1[is: x86]'
Follow the instructions displayed by Conary to erase the trove:
# conary unpin 'zile=/contrib.rpath.org@rpl:devel/2.2.14-1-1[is: x86]'
conary erase 'zile=/contrib.rpath.org@rpl:devel/2.2.14-1-1[is: x86]'
Applying update job:
Erase zile(:data :doc :runtime)=2.2.14-1-1
Although the conary unpin and conary erase commands displayed by Conary include the trove's full version string, you only need to include as much of the version necessary to uniquely identify the trove. In the example above, if there was only one version of zile installed on the system, you would only need conary unpin zile and conary erase zile to unpin and erase the trove. However, if there are two versions of a trove installed at the same time, you will require more than just the ability to pin a trove:
# conary update zile Not removing old zile as part of update - it is pinned. Therefore, the new version cannot be installed. To upgrade zile, run: conary unpin 'zile=/contrib.rpath.org@rpl:devel/2.2.8-1-1[is: x86]' and then repeat your update command
The update failed even though there is a newer version available. The reason for this failure is a result of the nature of the packaged software. The software in our example (zile) is not packaged to support "side-by-side" installation, meaning that any two versions of the package cannot have any files with exactly the same path name. Some ways to make sure there are no path conflicts are:
- Include a version-specific string as part of each filename
- Install files into directories whose filenames include a version-specific string
For the example, the files packaged as part of zile have not used either of these methods to avoid path conflicts. Therefore, it is not possible to have more than one version installed at a time. However, all the files packaged as part of the kernel use these methods, making it possible to have more than one version of the kernel installed simultaneously. Because of this aspect of kernel packages, pinning prevents older kernel troves from being updated while providing the opportunity to remove those older troves after a new kernel package is verified to be running correctly.
See the following resources for more information about updating and erasing kernel troves:
Starting to Package with Conary
Before stepping through this tutorial, you may want to see the following resources:
Also, prior to stepping through this tutorial, you will need a configured development environment. Use the following tutorials to establish such a development environment:
Establish a Project Repository
To package software for Conary package management, you need a Conary repository to which you have write permissions. Each rBuilder project includes a repository, and users can set up stand-alone repositories. This tutorial steps through using rBuilder Online, which sets up a free repository for each project.
If you have not already done so, do the following:
Set Up a Context
Set up a context for packaging as described in the Conary Contexts Tutorial. Use the information from your rBuilder Online project as follows:
- Repository hostname: The repository hostname is an abbreviated URL for your project that uses the project name and the domain name of the rBuilder installation. For rBuilder Online, this should be projectname.rpath.org where projectname is the name of your project in all lower-case letters. Navigate to http://projectname.rpath.org in a web browser to verify that you are directed to your new project's main page.
- Project Repository URL: The URL to your rBuilder Online project repository should be https://www.rpath.org/rbuilder/repos/projectname/ where projectname is the name of your project in all lower-case letters.
- Project Branch Name: Click edit project beside your project's name on its main project page in rBO, and record the value in the Default Branch Name text box. Though this can be any two colon-separated values, use corp:example-1-devel for this tutorial.
| | The user root cannot perform all of the package-building activities described in this tutorial. The cvc cook commands which have a potential to cause inadvertent changes to users' systems are disallowed from being run as root for security reasons. rPath thus recommends that you perform regular package-building activities as a user other than root on your local Conary-based system. If you have a sudo package installed on your system, you can execute software installation commands as any user by preceding the command with sudo on the command line. Add your user to the sudoers list (/etc/sudoers) to take advantage of this option. |
Create a New Package
Create a new package by using the cvc newpkg command in your context to create the package filespace and writing the recipe used to cook the package.
The package example for this tutorial will package a small application which requires a very basic recipe: Goom. In the directory of your newly created context, create the new package filespace.
[devuser@mymachine projectname]$ cvc newpkg goom
Change directories to the new goom subdirectory and create a new recipe file:
[devuser@mymachine projectname]$ cd goom
Create a text file called goom.recipe in the goom subdirectory using a text editor of your choice. Use the instructions in the package recipe step to edit this new file.
Package Recipe
The Conary package recipe provides all the instructions Conary needs to go from downloading the source to creating a package that will install as desired on a Conary-based system. Recipes range in complexity depending on the variety of options and conditions that must be considered for configuring and installing the software.
The Goom recipe is very basic because it can be built with very few instructions. Most of its instructions call established tools that automatically unpack and install typical source code. Edit your new goom.recipe file to match the following example:
class Goom(AutoPackageRecipe): name = 'goom' version = '2k4_0' buildRequires = [] def unpack(r): r.macros.upstreamVersion = r.macros.version.replace('_','-') r.addArchive('http://superb-east.dl.sourceforge.net/sourceforge/' '%(name)s/%(name)s-%(upstreamVersion)s-src.tar.gz')
In this recipe, AutoPackageRecipe takes the place of what would typically be just PackageRecipe (a super class). This tells Conary where to get the archive using a method called unpack.
One problem that Goom presents is using the hyphen (-) character in the version string. Because Conary does not allow hyphens, you are creating a macro called upstreamVersion that allows you to pull the correctly named upstream source.
The r.addArchive line provides the full URL to the upstream source using the macro values.
| | Note that the URL was broken into two lines for easy reading. On a single line, the URL would be in a single pair of quotes between the parentheses. This and other syntax is part of the Python programming language in which your recipe is written. For future recipe writing, you may wish to familiarize yourself with Python using the tutorial at Python.org. |
For future packaging, see the Develop a Package Recipe HOWTO which provides stepwise instructions for developing recipes using the information provided in Conary:Recipe and Conary:Package Recipe Classes. You may also wish to consult Conary API Documentation for lower-level recipe development.
Cooking for Packages
There are two types of cooking to be aware of when working with packages: cooking the recipe and cooking the package.
Cook the recipe to check whether the package will build, to examine what additional instructions you may need in your recipe, and to install and test the changeset prior to committing the new package to the repository. When you cook the recipe, a changeset is created that includes the packaged software. Use this changeset to test the installation of the software before committing the changeset to the repository.
Commit and cook the package to finally add the package to the repository from which it can be installed. The tutorial sections which provide instructions for this include more information about what happens when you perform these actions.
| | Another use of the term cook is in regards to groups. This is a separate type of cook used in appliance building. Cooking a group is not covered in this tutorial. |
Cook the Recipe and Test
Use the following command to cook the receipe.
[devuser@mymachine goom] cvc cook goom.recipe
The messages displayed during the cook include information about downloading, making, component creation, and writing a changeset. For this tutorial, warning messages in the output indicate modifications that can be made to the buildRequires portion of the recipe. For the tutorial, Goom displays some messages suggesting modifications to the recipe, but you will not need to change your recipe to continue.
In typical package building, you will modify the recipe and repeat cooking the recipe as many times as necessary to handle these warnings and other issues indicated by the cook messages. This is the most important and involving step in the package building process, and mastering recipe development is the key to mastering package building in Conary. As previously mentioned, see the Develop a Package Recipe HOWTO for further information about developing recipes.
Ensure things built correctly by installing the changeset file packagename-packageversion.ccs where packagename and packageversion match the package you have built. For the Goom example, it should be goom-2k4_0.ccs, and you may have seen a line in the output after the build that indicated this file was written.
Install the changeset using the following command.
[devuser@mymachine goom]$ sudo conary update goom-2k4_0.ccs
| | This command assumes the current user has rights to install software on the local system. As mentioned at the start of the tutorial, you may use sudo to do this if you are not the root user. |
If the installation worked correctly, you should have libgoom2.so in /usr/lib (or in /usr/lib64 for 64-bit architectures), and you should should have a directory full of useful information at /usr/share/doc/goom-2k4_0/.
For future package building, repeat the steps of updating the recipe, cooking the recipe, and applying the changeset until the installation works in the way you want your package to work.
Commit and Cook the Package
After the package has built and installed correctly (or as you wish for it to work), commit the recipe to the repository, and cook the package in the repository.
Commit
Change to the directory containing the recipe and use cvc add and cvc ci to commit the recipe. For the Goom recipe in this tutorial, using the following commands.
[devuser@mymachine goom-2k4_0]$ cd ~/conary/projectname/goom/ [devuser@mymachine goom]$ cvc add goom.recipe [devuser@mymachine goom]$ cvc ci
You will only need the cvc add command one time after each cvc newpkg. After the package is added and committed, any changes can be committed with cvc ci without a preceding add command.
Conary starts a session of the default text editor for the system, prompting for a commit message. If this is the vi text editor, press i to start inserting text, type New package (or something similar), and then save and quit vi. (Commands for saving and quitting in vi: Press Esc, type :wq, and press Enter.)
Verify the commit progress is displayed briefly and replaced with the next command prompt.
| | rmake commit may be used in lieu of cvc commit or cvc ci for checking in newly packages. |
| | You may occassionally see the cvc commit command in place of the cvc ci in some cases, and both commands will perform the same task. Add a --message option with an argument in single quotes to provide the commit message at the commandline instead of in the subsequent vi session. For the Goom steps, you could replace cvc ci with this command: cvc ci --message 'New package' |
| | If you are using local source files instead of a source URL, cvc ci will prompt you to use cvc add on those files in addition to the recipe file. |
Cook
Cook the package in the repository so all the troves will be created and the package will be both installable from the repository and available to add to software appliances in rBuilder projects. For the the Goom package in this tutorial, use the following commands.
[devuser@mymachine goom]$ cvc cook goom
Like the recipe cook, the package cook also displays messages during the process. Verify the last line of the output reads Changeset committed to the repository.
You can also verify the package is now available in your rBO project by navigating to the project's main page, clicking Browse Repository, and verifying the goom package is listed.
Install the Package from the Repository
After you have created the package, including committing it to the repository and cooking the package in the repository, any user who has access to install from that respository can use conary update to install the package.
Complete this tutorial by installing your new Goom package from your rBO repository. Use the following command.
[devuser@mymachine goom]$ conary update goom
Using Conary Contexts
In Conary, a configuration context, or just context, is an association between the build environment and a Conary repository. A single context consists of:
- A context entry in a Conary configuration file
- A corresponding project directory within which cvc context has been executed for the specific context entry
Create Context Entries
Developers can create the text file .conaryrc in the user home directory to establish configuration contexts. The following directives can be added to apply globally to anything in the user's home directory. These are useful when working on several projects with the same repository (rBuilder) user, and would otherwise be specified separately in each context entry:
contact <email_address_or_url> name <user_preferred_name> user *.<rBuilder_domain> <rBuilder_username> <rBuilder_password>
- Use an email address or URL which can be used to contact you in place of <email_address_or_url>.
- Use a preferred name, such as your first and last name, in place of <user_preferred name>.
- Use one user line for each rBuilder domain:
- Replace <rBuilder_domain> with the domain associated with your rBuilder system or group of systems (such as rpath.com for rBuilder Online).
- Replace <rBuilder_username> with your rBuilder username.
- Replace <rBuilder_password> with your rBA password.
A context entry in a configuration file comes after these global directives and starts with a context name in brackets: [example]. The lines following that line and prior to the next context entry are part of that context entry. At a minimum, a context entry in a configuration file can direct Conary to a default build label to which to commit packages and groups (buildLabel entries). Additional directives can be added as needed, such as repositoryMap and installLabelPath entries particular to the context.
The following is an example context entry for the project example:
[example] buildLabel example.rpath.org@corp:devel installLabelPath example.rpath.org@corp:devel
If projects are further organized by branches, each branch should use its own context entry (such as example-1-devel and example-1-test). Conary will use the argument provided to the context command to associate a project directory with a context entry.
| | Any directives in the context entry that duplicate directives at the top of the configuration file will override those global directives for commands executed within the context (directory). |
Set Up Each Context
With context entries in place, change to each corresponding project directory and use cvc context to associate the directory with its context entry. For the example context, the commands would be as follows:
$> cd ~/conary/example $> cvc context example
Use Context Entries Beyond the Context
The following are the ways you might refer to the configuration context entries:
- Use cvc context <context_name> in a directory so that context is used for all Conary operations in the directory.
- Use the --context option with a Conary command to name the context in which to run the command, such as in:
$> conary rq sample --context example
This option overrides the current context, no matter what the current working directory. - Set an environment variable, CONARY_CONTEXT, naming the context you wish to use.
- Use the context directive at the top of the Conary configuration file to specify a default context to be used.
| | If the environment variable is specified, and the --context option used, the option takes precedence. |
Identify the Current Context
View the context currently available in the cumulative Conary configuration using conary config with the --show-contexts option:
$> conary config --show-contexts
Use the cvc context without an argument to determine information about the current context:
$> cvc context [example] buildLabel example.rpath.org@corp:devel installLabelPath example.rpath.org@corp:devel
Reproducible Builds using rMake
| rMake facilitates building packages consistently across computers with dissimilar environments. rPath Linux and Foresight Linux development teams use rMake extensively for better control over their builds. To use rMake, first install and configure it for Conary-based systems
Some reasons to use rMake include the following:
| rMake Reference | |
Getting Started with rMakeUse the following links as a reference for getting started with a rMake-based system:
| Package SoftwareUsers can package software to take advantage of Conary and rMake features. Use the following links to find instructions and reference for Conary packaging:
| |
rMake Workflow
The following sequence shows a typical workflow using rMake during package or group development. The commands and other actions used at each step vary for each rMake build, typically depending on the packages and groups included in the build.
| 1 - Check out for updates | |
 | The developer checks out a package or group from an upstream repository into an established local build environment.
In this example, the developer has checked out version 1.1 of package example. |
| 2 - Update in the local build environment | |
| The developer modifies checked-out package or group in the local build environment. The developer may choose to include test builds to ensure the modifications build successfully prior to using rMake. For Conary-based systems, this includes modifying recipes, using the cvc commands to cook locally, and applying changesets to a test environment. | |
| 3 - Run rmake build | |
| The developer runs rmake build with appropriate arguments and other options to build with rMake. The developer may use the options to take advantage of various rMake features. The actions in 4 through 7 below occur as part of the rMake build. | |
| 4 - rMake shadows from the original (upstream) repository | |
 | rMake starts the build by shadowing the source component from the original (upstream) repository into the local rMake repository.
In the example, example version 1.1 of the source is shadowed from the same upstream repository from which the developer originally checked it out. |
| 5 - rMake creates and commits a changeset | |
 | rMake creates a changeset from modifications in the local build environment, and it commits that changeset to the local rMake repository. In Conary, this is similar to committing a changeset to a repository with cvc prior to cooking the package or group in that repository.
In the example, rMake generates a 1.2 version changeset of example from the local build environment and applies it to the local rMake repository. |
| 6 - rMake uses build requirements to make a chroot | |
 | rMake uses the build requirements in the package and group recipes to establish a temporary chroot on the local system. rMake will build in the chroot rather than on the local filesystem. This feature ensures independence between the local build environment and the environment desired for building the packages and groups.
In the example, rMake creates a chroot with the build requirements for the updated example. The example shows that this includes obtaining resources from various repositories. In Conary, these build requirements are listed in the file example.recipe. |
| 7 - rMake builds into the rMake repository | |
 | rMake uses the contents of the chroot and builds into the rMake repository. When the build completes successfully, the chroot is deleted and the updated package or group resides in the local rMake repository.
In the example, the rMake build results in example version 1.2. |
| 8 - Run rmake commit | |
 | After the rMake build is completed successfully, the developer runs rmake commit with the appropriate arguments to commit the rMake build to the original (upstream) repository. Unlike cvc commit in Conary, the rMake commit makes a clone while incrementing the version of the package or group in the upstream repository. Because rMake commits may include the building of multiple items, the commit is directed toward the job number generated by the rMake build.
In the example, the developer runs rmake commit to commit the newly built example to the upstream repository from which the developer originally shadowed. |
See rMake:Workflow to view how these pieces are used in typical development operations.
rMake tools are available from the command line after rMake is installed on a designated build system. Additionally, rPath's rBuilder tool incorporates rMake functionality for building appliances and derivative distributions (see rMake with rBuilder).
rMake chroot and Users
Each rMake chroot is created during an rMake build using the build requirements specified by the package or group to be built. rMake caches the troves downloaded for the chroot in /var/rmake/cscache. The chroot is deleted by default when the rMake build completes successfully, and the cached files can be deleted without adverse affects.
rMake runs as the user rmake. rMake creates the chroot with the help of a setuid program chroothelper, and it then runs inside of the chroot as user rmake-chroot. All the files in the chroot are owned by the user rmake, but the user in the chroot is rmake-chroot. With this structure, a developer can avoid inadvertently destroying the root while building, but it still does not have to create the chroot as the root user.
rMake with rBuilder
rBuilder incorporates rMake functions in its interface and makes those available from a link on the project panel. rBuilder displays an rMake panel on the right side of each rBuilder page while the rMake build is constructed.
To use rMake in rBuilder, rMake must be installed and started on the developer's local machine. See rBuilder:Appliance Development for reference and instruction on using rMake together with rBuilder.
rMake Installation and Configuration
Developers can install and configure rMake for Conary-based systems using the instructions provided on this page.
Install rMake
Install the rmake package on a Conary-based system which will be used for development:
#> conary update rmake=conary.rpath.com@rpl:1
Installation requires that the target system is using Conary version 1.1.19 or greater. If necessary, add the --resolve option to pull in dependencies (info packages).
Configure rMake
If the Conary-based system is rPath Linux, Foresight Linux, or a derivative distribution that uses group-dist as the group that defines its release, the system should need no configuration prior to starting and using the rmake server.
If the distribution does not use group-dist, you may have to edit /etc/rmake/rmakerc and set the value for resolveTroves. For example, if group-appliance is the group that defines the distribution's release, the following line should be added to /etc/rmake/rmakerc:
resolveTroves group-appliance
Start the rMake Server
After successful installation of the rMake packages, start the rMake server so that packages may be built with the rMake. Use the following command to start the rMake server:
#> /sbin/service rmake start
You should receive a notification similar to the following upon successful start of the rmake service:
Starting rMake Server and Repository:13:56:59 - [rmake-server] - using Conary in /usr/lib/python2.4/site-packages/conary
[ OK ]
If the rmake service still has not started, you can find detailed log data in /var/log/rmake/repos.log and /var/log/rmake/rmake-server.log.
rMake for RPM-Based Distributions
Developers can use the Yum package management tool to install rMake RPM packages for various RPM-based distributions. rPath's Yum repository is located at http://yum.rpath.com. rPath recommends configuring the Yum installation to pull from the rPath Yum repository appropriate for the RPM-based distribution.
 | RPMS for rMake are out of date. Therefore, please use them at your won risk as they may be unusable or have out dated dependencies. |
Obtain and Configure rMake for a Particular Distribution
Use the following sections for reference.
Fedora Core Releases
Put the following content in a file named rpath.repo inside your system's /etc/yum.repos.d/ directory:
[rpath] name=rPath Repository for Fedora $releasever - $basearch baseurl=http://yum.rpath.com/fedora-$releasever/$basearch/ enabled=1
| | Note: Copy the contents into the file verbatim; do not substitute $releasever or $basearch - yum will do the right thing
|
You will also need to make sure that you have the Fedora Extras repository enabled. Please check your /etc/yum.repos.d/fedora-extras.repo and make sure that the [extras] repository is set as enabled=1. For example:
[extras] name=Fedora Extras $releasever - $basearch #baseurl=http://download.fedora.redhat.com/pub/fedora/linux/extras/$releasever/$basearch/ mirrorlist=http://fedora.redhat.com/download/mirrors/fedora-extras-$releasever enabled=1 ...
CentOS 4
Put the following content in a file named rpath.repo inside your system's /etc/yum.repos.d/ directory:
[rpath] name=rPath Repository for CentOS $releasever - $basearch baseurl=http://yum.rpath.com/centos-$releasever/$basearch/ enabled=1 gpgCheck=0
| | Note: Copy the contents into the file verbatim; do not substitute $releasever or $basearch - yum will do the right thing
|
Red Hat Enterprise Linux 4
TBD
Install rMake
After obtaining and configuring rMake, install rMake with the following command:
$> yum install rmake
The Yum installation resolves dependencies and pulls in additional required packages for rMake.
Building with rMake
rMake can be used to build four types of things: packages, recipes, groups, and redirects. The following sections provide some of the build considerations for each of these.
Packages
A developer can use package names with rmake build to build them in a single rMake build job. The developer can include a version specifier and flavor for each package if desired.
rMake handles the ordering when two or more packages are in the same build. rMake determines this order by loading the recipes, reading what packages they create, and comparing this to the build requirements for each package. rMake identifies which recipes do not require any of the other packages in their build requirements, and it builds those packages first. After these required packages are built, the other packages are built using the newly built versions of the required packages (which exist in the chroot during the rMake build).
For example, if a developer is building a new version of glib and gtk together, rMake will build glib first as a build requirement for gtk, and then rMake will build gtk using the new glib in the rMake chroot. If glib fails to build, gtk subsequently fails because rMake assumes it should use the new glib.
Recipes
A developer can specify a recipe with rmake build. rMake shadows the package for the recipe to the local rMake repository and builds the recipe there. When it is finished, rMake clones both the binaries and the source to the upstream repository from which it shadowed. This allows the developer to cook a recipe with rMake, and then commit the recipe and binaries to the upstream repository at the same time.
A developer can also build recipes that are not source checkouts, and those recipes will be committed to the project that is the current buildLabel.
Groups
A developer can build a group with rmake build --recurse --match=-group-foo. Unlike with packages and recipes, rMake builds groups individually and not as part of a larger job. This is because the groups search the respository for things they include or to which they point, and it could be potentially confusing to rMake to have another item cooked alongside a group.
For example, suppose a developer cooks group-foo, which contains a reference to package foo, together with the foo package itself. The package foo gets cooked before group-foo, and it is stored in the rMake internal repository after it is cooked. However, rMake does not know to search its internal repository for foo; it searches the same repository as if it were cooked from the command line without rMake. Therefore, the developer's efforts would result in a group-foo that contained a reference to an old foo, not the one that was just cooked.
A developer chooses rmake build --recurse --binary-search to instruct Conary to find the matching binary version of the group and rebuild the packages within that group with the flavors with which they were previously built.
Redirects
A developer can build a redirect with rmake build, and rMake imposes the same restriction of building individually instead of part of a larger job.
This page provides a quick reference for rMake commands used by package builders. rPath recommends using rPath Appliance Platform Agent for appliance administration when it is available, and the commands shown here can be used on desktop distributions or appliances when command line access is necessary.
NOTE: Use the Print as PDF link in the TOOLBOX to generate a printable form of this and any other page.
| Informational Display | ||
|
$> rmake config | Common information options for rMake. | |
| Job Manipulation | ||
|
#> rmake build | Common job manipulation commands for rMake. | |
| Miscellaneous Commands | ||
|
#> rmake archive | Miscellaneous commands for rMake. | |
| Common Commands | ||
|
$> rmake build group-name --recurse | Common commands used with rMake | |
Recipe Conventions, Examples, and Templates
Recipe Conventions
Recipes are written on Conary's code base, which is in the Python programming language. Python knowledge is encouraged, but not required. rPath recommends that packagers use sample recipes and the recipe templates as a resource to start a new recipe, and to follow the conventions described here.
Be sure to use the Recipe Structure page to supplement this reference.
Basic Recipe Syntax
Use the following syntax guidelines to keep Conary recipes consistent in form. Many of these overlap with typically recommended Python syntax:
- Use 4 space indentation, not tabs
NOTE: In the vi text editor, packages can set the tab key to enter four spaces with each use: set expandtab softabstop=4 sw=4 - Keep line lengths to 78 characters, exceeding this for strings without natural breaks
- When breaking up a function, indent the next line of the function to the open parenthesis ( ( ) opening the function, or (if more lines are needed) increase indentation one level
- Avoid excessive ( \ ) in strings when possible
- Use grouped single quotes ( '' ) instead of multiple-line triple-double quotes ( """ ... """ ) for function arguments, such as in the following example:
r.Configure('options here' ' --more-option=here' ' --still-more-options')
If you are using \n at the e