rBuilder Online/Group Builder Reinvented Products

From rPath Wiki

Group Builder Reinvented: Products

rPath has fixed a lot of bugs by replacing the old "Group Builder" web interface in rBuilder Online with a new feature: Products. The new Product feature solves similar problem in a different way. It is a better way, but since it is different, anyone who has used Group Builder and become accustomed to it will have a few changes to make. This document describes the reasons for the change and documents how to adjust.

The first obvious difference between building an appliance as a "Product" in rBuilder and using the old "Group Builder" tool is that with Group Builder, you can choose any package on rBuilder Online with a "shopping cart" model, but with a Product, rBuilder presents fewer choices by default; you are generally presented with only one of each package that you might want to add, instead of starting by searching all the packages anywhere in rBuilder Online by default. As of this writing, an rBuilder package search for "mysql" returns almost a thousand items due to many customized versions of MySQL on rBuilder Online, but when you are building your Product you will be presented with only one version instead of a bewildering variety.

The second obvious difference is that with a Product, you have the opportunity to test your development work before making it available to users; with Group Builder, every time you made a change, it instantly showed up for all users without the opportunity for testing first.

The third obvious difference is that this process now has integrated command-line support for some things that previously were supported only by a web interface. Over time, expect more command-line support.

Choosing Packages

Why limit the number of packages that show up by default?

Besides the sometimes bewildering variety of choices that have become available as rBuilder Online has been used to build many products, another reason for limiting choice is that there are multiple platforms available on rBuilder Online and they are not compatible with each other. If you try to add a package that was built for a different platform, you end up getting complicated and involved error messages because Conary tries its best to resolve dependencies between packages that it doesn't know are incompatible and gives error messages that do not give you the information you really need: the different packages you have selected mix about as well as oil and water. Packages built for rPath Linux 2 won't work on rPath Linux 1; Packages built for Ubuntu Hardy are unlikely to work on either release of rPath Linux, etc.

In order to show you choices that are actually built to work with the base platform you choose, we have implemented a new feature. You define a "Product" in your repository, and you explicitly choose the "Platform" your product is based on. Then the extra packages you can select in the web interface are extra packages from your platform that are not included by default, as well as packages that are directly part of your product.

Adding Packages

That begs the question: How do you get additional packages into your product?

Web Interface

From the Web interface, there are two options.

First, you can upload archives by going to your Product's home page and clicking on "Maintain packages". These are archives of files to be installed on the system, not source code to build. They can be compressed or uncompressed tar archives, zip files, RPM packages, or dpkg packages.

Second, you can choose packages that have already been built and exist somewhere in rBuilder Online by adding new items to the search path for your Product. To do this:

1. Follow the link to the new UI (later, this will become the defalt or only UI).
2. Click the browse appliance button at the upper right hand corner of the screen.
3. Click on the tile for the appliance you want to work with.
4. Click on the appliance contents icon to add/remove packages and search paths.
5. Click the expander icon next to "Search Paths" to add/remove search paths.
6. You can now add labels (e.g. contrib.rpath.org@rpl:2), groups (including at least the label and optionally the version – see the examples in the lines above the entry box that come from your platform), or even individual packages (just like groups, at least the label is required, and the version may also be provided).

Command Line

Other options do require using the command line. However, if you have previously used command-line tools such as Conary and rMake and found them complex, you'll be glad to know that the process has been simplified. With the new Product functionality in rBuilder comes a new tool called rBuild which uses the Product definition in rBuilder Online to do configuration appropriately.

The key requirement for the command line is a Conary-based development system. This can be a virtual image such as the one published on devimage.rpath.org running under VMware, Xen, or some other virtualization technology; or can be an operating system such as Foresight Linux or devimage running directly on hardware. The key requirement is that it contain the rBuild tool. The other major requirement is that if you are building a product for which you will build any 64-bit images, your development image or system must be 64-bit. This tutorial will assume that you are using one of the devimage.rpath.org images, and that it is up to date.

The two main ways to add a package with rBuild are to create an entirely new package that you build from source, or create a "shadow" of some package on rBuilder Online -- but rebuild it to make sure that it works with the platform on which your product is based. These two things require mostly the same steps. Since this tutorial is about replacing Group Builder, it will concentrate on the "shadow", which is most like the Group Builder option, but most of the work is the same for creating your own packages that you build from scratch.

Using rBuild is kind of like using a source control tool. You check out a subdirectory tree that represents the product you are building, and you run commands in that subdirectory tree to do things with your product, including adding, modifying, and building packages.

First, follow the launch and setup instructions at http://docs.rpath.com/quickstart-devimage.html including setting up your "devuser" user. (You may choose a different name for the account; adjust these instruction accordingly.)

Second, create a "Product version" on rBuilder Online. You do this with the "Create a new product version" link on the home page for your product. You will have a name and a version for your product. Remember both of them. When editing your product version, it is very important to define some images. rBuild decides what to build based on the image types that you want to create for your product. You can always add more images later, but you will need at least one to start using rBuild. If any of the images you define are 64-bit images, you will need to use a 64-bit development environment.

Log in to your development image, and initialize a directory for your new product version. Run the command: rbuild init myProductName myProductVersion "myProductName" and "myProductVersion" are from the product version you just defined. This document will use the example product:

• rbuild init example 1

rBuild will tell you the name of the directory it created, such as "example-1"

The first time you do this, rBuild will interview you for the information it needs to work for you, including your user name and password on rBuilder Online. For rBuilder Online, when it asks for the URL for your rBuilder, use https://www.rpath.org After it collects the information it needs, it will then finish the job.

Look in the directory you checked out:

• cd example-1
• ls -l

You will see Development, QA, and Release subdirectories. The Development subdirectory is where you build software, the QA subdirectory is where you build images that are intended for testing, and the Release subdirectory is where you build images for final release. There's far more to it than that, but this is enough to start with.

Because this tutorial is about building packages, change to the Development directory:

• cd Development

Now you are finally set up to use a package from some other product in rBuilder Online -- and then use the tools included in your development image to rebuild the package to work with the platform you are using. The way you use a package from elsewhere is called "creating a shadow":

• rbuild checkout --shadow rapa-custom=devimage.rpath.org@rpath:devimage-1-devel

Now you tell rbuild to build all the versions of rapa-custom that you will need for all of the images you have created:

• rbuild build packages

Assuming that the package builds successfully, you can now add it to your product using the Web user interface -- it will now show up just like packages that are included in the platform your product is based on, when you select "Manage the ... appliance" link under "Manage This Product" on your product home page.

From the "Manage the ... appliance" link, rBuilder Online will rebuild the groups that define the contents of your appliance. You can also do this from the command line with the command

• rbuild build groups

When you are ready to update to the latest version of the platform, including new bug fixes and especially security updates, you can do this in one of two ways: You can "Edit product version ..." from your product home page and click the "UPDATE" button to the left of "Appliance Platform", or you can run the command

• rbuild rebase

from anywhere in your example-1 product subdirectory tree.

In either case, after updating to the latest version of the platform, you will need to rebuild groups in the same way as before.

If you add more image types (also available from "Edit product version ..." in the Web user interface) you will need to rebuild your packages and groups, because the new image may have added configuration or instruction set variations (we call them "flavors" ) that need to be represented in your packages and groups.

Controlling Deployment

The option to test changes before deploying them to users currently requires using the rBuild command line, though it is relatively simple to do. The main thing to understand is that we use the word "stage" for the steps your Product goes through and you "promote" your work from one stage to the next: You build your product in the "Development" stage, you promote to the "QA" stage to do testing, and then you promote from the "QA" stage to the "Release" stage for normal users.

Note that if you do not want to make use of this capability, you do not have to; your users will just use your "Development" stage and the name won't matter very much. If you later decide to add this stage management, your users will be able to switch to using your "QA" or "Release" stage whenever they want to.

Promoting from one stage to the next is always done based on groups. When you promote from Development to QA for testing, you will promote the latest versions of the groups that you have built. If you have built newer versions of packages that haven't been included in any groups, those newer packages will not be promoted. That is, if you have run the "rbuild build groups" command, followed by editing a package and running "rbuild build packages" to build a new version of your package, the new package you built will not be promoted.

To promote from Development to QA, you need to be in your Development directory, and run the "rbuild promote" command:

• cd example-1/Development
• rbuild promote

To build QA images for testing, you need to be in the QA directory and run the "rbuild build images" command:

• cd ../QA
• rbuild build images

Note that it will build all the images that you have defined in the Web user interface, one at a time. When it has finished, it will make a release that includes all of those images, and you will be able to click on "Publish" in the Web user interface to make those images available, or run the "rbuild publish" command to publish the images you just built. (You may choose not to publish testing images; you may choose to make them available only to testers.)

When your testing is successful, you will be able to build images for general release. From the QA directory:

• rbuild promote
• cd ../Release
• rbuild build images
• rbuild publish