From rPath Wiki

Migrating to rPath's Best Practices

Introduction

The purpose of this document is to outline how users of rBuilder versions prior to version 5.2 can migrate their appliance development efforts to align with rPath's best practices, and thereby take full advantage of rBuilder's latest features.

This document assumes the reader is familiar with basic Conary concepts and rBuilder usage.

NOTE: This document cannot describe the migration process for all possible appliance configurations. Therefore, if at any time you are not sure how to proceed, please contact rPath for technical assistance.

Current Situation

Long-time users of rPath's products and technologies have tended to develop methodologies that vary according to the specific circumstances each user faced.

However, as rPath has continued to improve rBuilder, it became clear that, for the majority of situations, it was possible to develop a single best practice that would work for the majority of users, and to gear new rBuilder development to support this methodology.

rBuilder's support for this best practice began for new users with rBuilder 5; with rBuilder 5.2, existing users can begin the process of migrating their appliances to align with rPath's best practice.

Advantages to Migration

By migrating to rPath's best practices, appliance developers gain a number of benefits:

• A well-defined "platform" on which to base the appliance
• Support for staged development
• One-time definition of all desired image formats and their options
• A consistent, easy-to-use command-line interface for appliance development
• A rich, highly-visual web interface for appliance development

Let's review each benefit in more detail.

Well-Defined Platform

Because each appliance created with rBuilder includes "just enough operating system" (JeOS), it is important to strictly define exactly where those operating system components come from, and to control how changes to those components are to be integrated into the appliance being developed.

In rBuilder, a collection of operating system components is referred to as a "platform". rPath provides a number of different platforms, but once a platform is identified for a given appliance, all operating system components come from that platform (unless you explicitly specify other places from which to obtain them).

As rPath makes updates available to each platform, appliances based on that platform can consume those updates when it is convenient to do so. In this way, an appliance in active development can consume platform updates immediately, while an appliance undergoing final pre-release testing can defer the consumption of platform updates that could potentially derail the release process.

Support For Staged Development

Software development usually takes place under a variety of scenarios. For any one major revision of an application's codebase (usually represented by the application's major version number), the following activities are likely to take place simultaneously:

• Active development of new features and/or fixing customer-reported bugs
• Bug fixing of issues uncovered during pre-release testing

rBuilder addresses this reality with "stages" -- a method of unambiguously associating the development status of any package in the appliance's Conary repository. By default, rBuilder creates the following stages:

• Development -- For developing new features and/or fixing customer-reported bugs
• QA -- For pre-release testing, and -- optionally -- bug-fixing
• Release -- For groups/packages that have been (or will be) released to customers

Because platform selection, stages, and image definitions are all associated with a specific major version, simultaneous new development, bugfixing, and pre-release testing can be undertaken for each of the appliance's existing major versions.

Up-Front Definition of All Desired Image Formats/Options

The end-goal of software development is to make the final product available to the people that will use it. In today's IT environment, this can mean producing multiple image formats for users that might want to deploy the appliance on any one of several different virtualization environments, or direct hardware installation -- and in either 32- or 64-bit variants.

rBuilder, in addition to supporting a wide variety of image formats for a wide variety of deployment scenarios, allows the user to define a set of images and architectures that should be built, and to undertake this definition once (though changes and additions can easily be made at any time, if necessary). This also ensures that the appliances packages will be made available in every flavor (configuration) necessary to support the desired images/architectures.

Once the set of image formats has been defined, the creation of all the necessary images can be undertaken with a single command or mouse-click.

The Appliance Creation Process

When an appliance has been configured (with major version number, platform, and desired image formats all specified) it's then necessary to do the actual work of building the appliance. rBuilder provides two ways of doing this:

• The rBuild command-line utility
• The rBuilder web UI

Since each method differs in terms of ease-of-use and capabilities, let's review each one in a bit more detail.

The rBuild Command-Line Utility

The rBuild command-line utility is a tool that makes various rPath tools and technology available via a single command -- "rbuild". Through this command, the following major operations can take place:

• Building packages
• Building groups
• Building images
• Moving (also known as promoting) packages/groups between stages
• Consuming platform updates (also known as rebasing)

Because rBuild is stage-aware, it can handle both building on a particular stage, as well as promoting groups and packages from one stage to another (for example, when a Development-stage build is deemed ready for pre-release testing on the QA stage). Additionally, the rbuild command can easily be integrated into any build system capable of executing shell commands.

A Rich Web Interface

rBuilder's web interface provides a highly-visual, point-and-click means of creating and building an appliance. The following operations can be performed using rBuilder's web UI:

• Building packages (*)
• Building groups
• Building images
• Consuming platform updates

(* Note that, at present, package building using rBuilder's web UI is restricted to the packaging of ready-to-run binaries in .rpm, .deb, .tar, or .zip formats.)

The Structure of a New Appliance

When an entirely new appliance is created on rBuilder 5.2, a number of artifacts related to the appliance are configured in such a way that the appliance can be built and maintained, if desired, from rBuilder's web UI, or from the command-line (via rBuild).

The artifacts include:

• A product definition
• An appliance group recipe
• An appliance manifest

Let's look at each one in more detail.

Product Definition

The product definition is stored in the appliance's Conary repository in the product-definition:source trove. The product definition appears on the appliance's "Release" label (more about this below), and contains the following files:

• product-definition.recipe
• product-definition.xml

The product-definition.recipe file is, as noted in its comments, a simple stub that makes it possible for users to commit hand-made changes to the product-definition.xml file.

The product-definition.xml file defines all aspects of the appliance, including:

• Major version
• The name of the appliance's group recipe
• The development stages, with Conary labels for each
• Desired image definitions
• The platform type and version currently being consumed

Let's explore each in more detail.

Major Version

The major version appears within the <productVersion> element.

Appliance Group Recipe

The name of the appliance group recipe appears within the <imageGroup> element, and takes the following format:

                       group-<uniqueName>-appliance

(Where <uniqueName> represents the contents of the "Unique Name" field specified when the appliance was originally created. This is also known as the "Product Short Name" in rBuilder's older user interface.)

Development Stages

The development stages are grouped within the <stages> element, each of which appear as attributes of the <stage> element. By default, the following stages are defined, each with its own suffix to be appended to the appliance's repository name and namespace:

• Development ("-devel")
• QA ("-qa")
• Release ("")

Therefore, for an appliance whose major version is "1", repository name is "foo.example.com" and namespace is "bar", the Development, QA, and Release stage label would be:

foo.example.com@bar:1-devel
foo.example.com@bar:1-qa
foo.example.com@bar:1

Image Definitions

There is no default value for image definitions in product-definition.xml -- the value depends on the image(s) selected when the appliance was created time. However, the image definition(s) that have been selected appear grouped under the <buildDefinition> element, each within its own <build> element.

By default, each <build> element also includes one <stage> element for each of the three default stages (Development, QA, and Release). This means that, although this capability is not yet exposed by either the rBuilder web UI or the rBuild command-line utility, it is possible to have different sets of image definitions for each stage (for example, to support the creation of an image strictly for developmental or testing purposes).

Platform

All platform-related information is grouped under the <platform> element. The default values here depend on the platform selected by the user, as well as by the current version of that platform at the time the product definition was created.

The <searchPaths> element contains one or more <searchPath> elements, each of which represents part of the platform the appliance will consume. Note that each <searchPath> element includes troveName, version, and label attributes to unambiguously define the trove to be consumed by the platform.

The <searchPath> elements also include two other attributes:

• isResolveTroves
• isGroupSearchPathTroves

The value of the isResolveTroves attribute controls whether the trove referenced by the <searchPath> element should be used for providing build-time packages, while the value of the isGroupSearchPathTroves attribute controls whether the trove referenced by the <searchPath> element should be used to provide packages to be added to the appliance's group and, therefore, to the appliance itself.

Appliance Group Recipe

The default appliance group recipe (which, as stated earlier is named "group-<uniqueName>-appliance") is of class OverrideRecipe, which is subclassed from FactoryRecipeClass, and defines the addRecipePackages() method. Here is the entire recipe (minus comments and docstrings):

class OverrideRecipe(FactoryRecipeClass):
    def addRecipePackages(r):

As the name (and associated docstring) implies, addRecipePackages() is where packages can be added to the appliance. However, it is also where other group-related operations can be undertaken, such as using r.removeComponents() to trim unwanted components from the appliance. Comments present in the default appliance group recipe outline a number of the more common customizations that are possible.

It is also possible to completely rewrite this recipe; however, this could have implications related to the ability to fully take advantage of the rBuilder web UI; this will be discussed in more depth below.

Appliance Manifest

The appliance manifest file is named "appliance-manifest.xml" and is used by recipes that are subclassed from FactoryRecipeClass (such as the default appliance group recipe) to declare the following settings:

• Appliance group recipe name
• Major version
• Search path
• Troves

Let's look at each of these settings in more detail.

Appliance Group Recipe Name

The name of the appliance group recipe appears within the <troveName> element.

Major Version

The major version appears within the <troveVersion> element.

Search Path

The search path is used by rBuilder's web UI to control the troves that are to be displayed as "available troves" during the appliance content editing process. It is also used when the group is being built to find packages not explicitly identified by the user, but required to produce a complete appliance.

The search path is spread between two different elements:

• <searchPaths> -- Platform-provided search path entries appear here
• <extraSearchPaths> -- Optional user-defined search path entries appear here

In both cases, each entry is contained within a <searchPath> element.

The contents of the search path vary according to the specific platform selected. For example, when rPath Linux 2 is selected, the following search path entries appear (contained within <searchPath> elements):

group-os=conary.rpath.com@rpl:2/2.0.1-0.1-4
group-rapa=raa.rpath.org@rpath:rapa-3/3.1.3-0.1-1

(Note that the actual version numbers will differ, due to updates to the platform.)

In this example, the entry for group-os corresponds to rPath Linux 2 itself, with the second entry (group-rapa) corresponding to the inclusion of the rPath Platform Agent.

As these two search path entries indicate, the search path can include a full version string. In this way, it is possible to guarantee the state of the platform being consumed. However, a search path entry does not necessarily need to contain a full version string; for example, rBuilder Online's "contrib" repository contains many useful, though unsupported packages. When used to select packages compatible with rPath Linux 2, its search path entry would look like this:

contrib.rpath.org@rpl:2

Troves

The packages that have been explicitly selected by the user for inclusion in the appliance appear within the <explicitTroves> element, which wraps a set of zero of more <trove> elements, each of which specifies a trove (package) to be added to the appliance.

By default, the <explicitTroves> element is empty; it is normally populated via the rBuilder web UI (specifically, the Appliance Content Editor dialog).

Planning Your Migration

The steps to be taken to migrate an existing appliance to rPath's best practices depend on the following variables:

• How the appliance group is currently defined
• How the appliance is currently being developed and built
• Which features of rBuild and/or rBuilder's web UI are to be used

By taking these variables into account, the migration process that makes the most sense for a particular appliance can be determined.

Additional details appear below.

Current Group Structure

The way the appliance's top-level group is constructed has an impact on the migration process, as it affects which rBuild and rBuilder web UI features can be used.

In general, the group must be a subclass of ApplianceGroupRecipe to take advantage of rBuild's capabilities, while using both rBuild and the rBuilder web UI requires that the group be a subclass of FactoryRecipeClass.

Appliance Development Workflow

The way the appliance has been developed and built must also be taken into consideration; primarily from the standpoint of how it integrates with the development organization's existing workflow.

If the organization's current workflow uses automation, then it is best to take advantage of rBuild's ability to be scripted. In this situation, rBuild replaces the older "rbuilder" command-line utility.

On the other hand, while rBuilder's web UI is not suited for automation, it can be a good match for organizations where appliance development happens occasionally, or may be undertaken on a more ad-hoc basis by differing groups of developers.

Desired rBuild/Web UI Features

The following sections explore the migration steps from the perspective of enabling use of the rBuild command-line utility and the rBuilder web UI features.

Migration to Support the rBuild Command-Line Utility

The migration steps necessary to take an existing appliance, and make the maintenance of it possible via rBuild revolves around the creation of a product-definition:source trove, and associating the appliance's group with it.

Migration includes the following steps:

• Creating a product definition
• Modifying the appliance's group recipe

Let's look at these steps in more detail.

Creating a Product Definition

The rBuilder web UI is used to create the product-definition:source trove. In the web UI's terminology, this is referred to as adding or creating a "major version".

How this is done depends on whether you're going to migrate by creating an entirely new appliance, or by converting the repository containing an existing appliance.

If an entirely new appliance is being created, a product-definition:source trove will automatically be created, with "1" as the default major version (although this can be changed under the Advanced options section of the appliance creation dialog).

If the repository containing the existing appliance does not include a product-definition:source trove, rBuilder 5.2 will refer to the repository containing the existing appliance as a "Repository". This repository can be converted to an appliance and, as part of the conversion process, provided with a product-definition:source trove. The conversion is done by following these steps:

• Using the new rBuilder web UI, navigate to the desired repository by clicking on the "Repositories" tab in the Resources sidebar.
• Click on the "Images" tab on the far right. In the upper left-hand corner, there will be a "Convert to appliance" button.
• Click on the "Convert to appliance" button.
• Confirm that you wish to perform the conversion.
• Enter the desired data in the following fields:
  • Appliance name -- A human-readable short description of your appliance
  • Image types -- The list of image types that rBuilder can generate. Select one or more as desired.
  • Platform -- The operating environment from which packages will be selected to provide your appliance with just enough operating system.

NOTE: Select the platform previously used by your platform. Do not change platforms during the migration process; this can be done after you've completed your migration.

• • Major version -- The most-significant portion of your appliance's version number.
• Click on the "Convert" button

Once the conversion process completes, the repository now contains a product-definition:source trove, and is considered by the rBuilder web UI as an "Appliance" with the major version you specified. As an Appliance, it will now appear when clicking on the "Appliances" tab in the Resources sidebar.

Modifying Group Recipe

Next, the appliance's original group must be modified to be compatible with the product definition. There are two aspects to compatibility:

• Group recipe name
• Group recipe type

Group Recipe Name

The name of the group recipe must be in the form:

group-<uniqueName>-appliance

(Where <uniqueName> represents the contents of the "Unique Name" field specified when the appliance was originally created. This is also known as the "Product Short Name" in rBuilder's older user interface.)

To ensure that the group recipe is appropriately named, use the Appliance Content Editor feature of the rBuilder web UI to create a group. When the Appliance Content Editor is displayed, click the "Done" button, and then create a new build.

NOTE: If your appliance's original group recipe was already named "group-<uniqueName>-appliance, the above steps will cause it to be superseded with an empty template recipe. This will be corrected below.

Additionally, after the above steps complete, a file named appliance-manifest.xml will also be present in the group's source component; do not remove it.

At this point, you have created an appliance containing only the rPath Platform Agent management interface; in the next section, we will integrate the recipe from your existing appliance.

Group Recipe Type

To take full advantage of the rBuild command-line interface and the capabilities of the product definition, the appliance's group must be a subclass of either ApplianceGroupRecipe or FactoryRecipeClass. If your appliance's group is already a subclass of ApplianceGroupRecipe, and you do not want to use the rBuilder web UI, completely replace the contents of the newly-created group recipe with the contents of your appliance's original group recipe.

Once this has been done, the rBuilder command-line utility can be used for appliance development.

If, on the other hand, you also want to take advantage of the rBuilder web UI, your appliance's group recipe must be a subclass of FactoryRecipeClass. The process of converting your appliance's group recipe to be a subclass of FactoryRecipeClass is discussed in more detail below.

Migration to Support the rBuild Command-Line Utility and rBuilder Web UI

If, in addition to using the rBuild command-line utility, you also want to use the rBuilder web UI to edit the appliance contents, it is necessary to restructure the appliance's current group recipe. The restructuring must include the following steps:

• Modify group recipe to be "class OverrideRecipe(FactoryRecipeClass)"
• Review/Modify <searchPath> elements in appliance-manifest.xml
• Populate <explicitTroves> element in appliance-manifest.xml
• Populate addRecipePackages() method

Let's look at these steps in more detail.

Modify Group Recipe Type

The appliance's group recipe must be rewritten. Start by using the following recipe as a template:

class OverrideRecipe(FactoryRecipeClass):
    def addRecipePackages(r):

Initially, the addRecipePackages() method should be empty.

Review/Modify <searchPaths> Element

The appliance-manifest.xml file contains one or more <searchPath> elements. The <searchPath> elements within the <searchPaths> section represent the location of the packages coming from the selected platform.

If there are other troves required by the appliance, and these troves do not come from either the platform, or the appliance's Conary repository, their location can be added to the <extraSearchPaths> element, each entry within its own <searchPath> element. This can be done by using the Appliance Content Editor feature of the rBuilder web UI.

When the search path is properly set, the desired packages will appear in the "Available packages" section of the Appliance Content Editor.

Populate <explicitTroves> Element

Next, the packages that comprise the appliance must be added to the appliance-manifest.xml file. This can be done by using the Appliance Content Editor feature of the rBuilder web UI. By moving the desired packages from the "Available packages" to the "Selected packages" section, the <explicitTroves> element will be populated.

Populate addRecipePackages() Method

If the appliance's original recipe only added packages via r.add(), and all these packages have been listed in the <explicitTroves> element, then addRecipePackages() can remain empty. However, if the original recipe contained additional operations, they should be added to addRecipePackages(). This can be done via the Recipe tab in the Appliance Content Editor feature of the rBuilder web UI.

At this point, the appliance can be maintained either via the rBuild command-line utility or the rBuilder web UI.