rMake:Build Software

From rPath Wiki

In place of cvc cook on a package in the repository, use rMake to build packages in a consistent build environment.

Advantages of rMake:

• rMake creates a chroot for every build and builds with components from the repository
  • This removes the reliance on local development tools that cvc introduces
  • Builds are exactly repeatable on any system
  • Common development tools are defined for each project
• rMake allows you to build and test binaries before committing source to the repository. This includes building sets of binaries to ensure compatability between them.
• rMake will build multiple packages in the appropriate order according to their dependencies upon each other.

For more information on rMake, please see rMake Architecture

rMake Build Environment

To setup a rMake build environment, the following is needed:

• A Conary-based system (such as rPath Linux)
• The conary-build package installed on that system; use conary q to determine if it is installed, and run conary update with root privileges to install it if necessary:
  
• The conary-build package is needed for rPath Linux only. Foresight Linux users have this trove included in group-devel and can skip to the next bullet point.
  #> conary q conary-build
  #> sudo conary update conary-build
• The group-devel package to be installed on that system; use conary q to determine if it is installed, and run conary update with root priveleges to install it if necessary:
  #> conary q group-devel
  #> sudo conary update group-devel
  #> sudo conary update rmake
  #> sudo service rmake start
  

The user root cannot perform all of the package-building activities described in this tutorial. The rmake commands which have a potential to cause inadvertent changes to users' systems are disallowed from being run as root for security reasons. rPath thus recommends that you perform regular package-building activities as a user other than root on your local Conary-based system. If you have a sudo package installed on your system, you can execute software installation commands as any user by preceding the command with sudo on the command line. Add your user to the sudoers list (/etc/sudoers) to take advantage of this option.

• A repository to which you have write permissions

After your new project is created, record the following information to be used later in this document.

• Project URL: This should be projectname.rpath.org where projectname is the name of your project in all lower-case letters. Navigate to http://projectname.rpath.org to verify that you are directed to your new project's information page.
• Project Repository URL: The URL to your project's repository should be https://www.rpath.org/rbuilder/repos/projectname/ where projectname is the name of your project in all lower-case letters.
• Project Branch Name: Click edit project beside your project's name on its main project page in rBO, and record the value in the Default Branch Name text box. For this tutorial, you will use the default rpl:devel branch name.

rMake Context

• rMake reads Conary configuration files and contexts
• Use the Conary Package Tutorial to setup a Conary build environment and context.
• Values set in an rMake context (inside rmakerc) will override those set in a Conary context (inside conaryrc).
• The context in rmakerc is applied after those in conaryrc; thus, a buildLabel set in a rMake config file will override one set in an Conary context when using rMake.
• Using the --context option with the rmake command will override those set in both conaryrc and rmakerc. See man rmake for more information on the --context flag.

rMake specifically reads Conary configuration files /etc/conaryrc and ~/.conaryrc. rMake also supports configuration of contexts in its own configuration file, rmakerc. Note that for simplicity, rMake reads conary configuration values therefore, all currently documented Conary configuration values affect rMake operation.

Configuring rMake Server

The rMake server is the back-end for rMake, and controls such elements as rMake's database, repository, and changeset cache. It can be configured with individual options inside /etc/rmake/serverrc. The options are contained in the man page for the rmake service and can be viewed with the command man rmake-server.

No changes to default settings are required.

Configuring rMake

rMake uses information set in /etc/rmake/rmakerc and allows you to customize options for building in ~/.rmakerc which resides in the home directory of a system user.

When building software with rMake it relies on the installLabelPath set by the current Conary context (set by cvc context command). As an example, if a context was set to build software for conary.rpath.org@rpl:1, rMake would utilize this context and build software for that specific context.

Like Conary, rMake has three scopes for configuration:

1. System-wide: /etc/rmake/rmakerc
2. User-specific: $HOME/.rmakerc
3. Project-specific: rmakerc in the project directory

Use the command rmake config to examine all the current settings for rMake. More information is also available via the rMake man page viewable with the command man rmake.

Building with rMake

There are two different approaches when building software with rMake. The first is an appliance approach to building with rMake. With this approach, rMake becomes a tool with which any appliance is built and maintained. Appliance building with rMake is covered in the Appliance Guide (link forthcoming).

The second approach when building software with rMake is package development and maintenance. By using rMake, one can guarantee that local build requirements are always when contributing software which guarantees software being downloaded by others is truly usable on any system.

In this section, the second approach will be covered. It is assumed that a Conary and rMake build environment has already been setup for development on your system. If a build environment has not been setup, please read how to setup a build environement.

Building Recipes

Determining what type of recipe you will need (CPackage, Python, etc.) is covered in the Conary Package Recipe How-to. This document will use a simple C application called alltray. Alltray is an application that allows programs to minimize to the tray area of Linux even if minimization isn't supported. Alltray is also a simple C application in that it needs a standard linux configure, make, and make install to run on a system. Using the Package Recipe How-to above, select the simple C recipe template. In a terminal, execute the following commands:

#> cvc newpkg alltray
#> cd alltray

Once inside the alltray directory, create a file named alltray.recipe and copy/paste the following:

 
#
# Copyright (c) 2007 rPath, Inc. 
# This file is distributed under the terms of the MIT License.
# A copy is available at http://www.rpath.com/permanent/mit-license.html
#
class AllTray(AutoPackageRecipe):
    name = 'alltray'
    version = '0.69'
 
    buildRequires = []
 
    def unpack(r):
        r.addArchive('mirror://sourceforge/%(name)s/'
                     '%(name)s-%(version)s.tar.gz')

Save and close the alltray.recipe file. Now that the recipe has been created and is ready to be built.

cvc cook for Build Requirements

A handy step when using rMake to build recipes is to first use cvc cook alltray.recipe to identify the build requirements needed. Original Conary commands such as cvc cook are still useful for testing and troubleshooting some packages, but rmake build should be used in most build activities.

cvc commands should only be used on a system using or based on rPath Linux. This ensures that standard build requirements are met easily. Using a cvc cook command will build a package or recipe locally on your system; if the system lacks the build requirements for the recipe to cook the build requirements will need to be installed before completing the package build. A rPath Linux based system contains many common build requirements which eliminates the need to install them locally in order to utilize the cook command.

Issue the command cvc cook alltray.recipe and take note of the warnings that display at the end of the build:

warning: reportMissingBuildRequires: Suggested buildRequires additions: ['GConf:devel', 'ORBit2:devel', 'atk:devel', 'cairo:devel', 'desktop-file-utils:runtime', 'expat:devel', 'fontconfig:devel', 'freetype:devel', 'glib:devel', 'gtk:devel', 'libX11:devel', 'libXrender:devel', 'libpng:devel', 'pango:devel', 'pkgconfig:devel', 'zlib:devel']
+ looking up pathids from repository history
+ pathId lookup complete
Created component: alltray:data /project.rpath.org@namespace:tag//local@local:COOK/0.69-1-0.1 is: x86
Created component: alltray:debuginfo /project.rpath.org@namespace:tag//local@local:COOK/0.69-1-0.1 is: x86
Created component: alltray:devellib /project.rpath.org@namespace:tag//local@local:COOK/0.69-1-0.1 is: x86
Created component: alltray:doc /project.rpath.org@namespace:tag//local@local:COOK/0.69-1-0.1 is: x86
Created component: alltray:lib /project.rpath.org@namespace:tag//local@local:COOK/0.69-1-0.1 is: x86
Created component: alltray:runtime /project.rpath.org@namespace:tag//local@local:COOK/0.69-1-0.1 is: x86
Changeset written to: alltray-0.69.ccs
#> 

Copy and paste the build requirements into the alltray.recipe file inside the buildRequires = []. Save the recipe and issue the cvc cook alltray.recipe command again. Look back through the output on the screen for any errors or warnings. Resolve any errors or warnings (missing build requirements) using the method show above. After resolving, cvc cook. When the cook has completed successfully, rMake may be used to build and commit the package. Upon completion, your recipe should look as follows:

 
#
# Copyright (c) 2007 rPath, Inc. 
# This file is distributed under the terms of the MIT License.
# A copy is available at http://www.rpath.com/permanent/mit-license.html
#
class AllTray(AutoPackageRecipe):
    name = 'alltray'
    version = '0.69'
 
    buildRequires = ['GConf:devel', 'ORBit2:devel', 'atk:devel', 'cairo:devel', 'desktop-file-utils:runtime', 'expat:devel', 'fontconfig:devel', 'freetype:devel', 'glib:devel', 'gtk:devel', 'libX11:devel', 'libXrender:devel', 'libpng:devel', 'pango:devel', 'pkgconfig:devel', 'zlib:devel']
 
    def unpack(r):
        r.addArchive('mirror://sourceforge/%(name)s/'
                     '%(name)s-%(version)s.tar.gz')

Troubleshooting rMake Build Errors

Steps for troubleshooting rMake build errors are as follows:

1. Query the rMake build Logs
2. Search for keyword failures
3. Add buildRequires to the recipe
4. Repeat steps 1-3 until you have a successful build of the recipe

Query the rMake Build Logs

When rMake fails to build a recipe or package, it is often the result of missing build requirements. Look at the example errors below:

error: Error building recipe alltray:source=/project.rpath.org@namespace:tag//hostname.yourdomain.com@local:tag/0.69-0.1[is: x86(~cmov,~i486,~i586,~i686,~mmx,~nx,~sse,~sse2)]: Shell command "cd '/tmp/rmake/builds/alltray/alltray-0.69/';  CLASSPATH="" CFLAGS="-O2 -march=i686 -g" CXXFLAGS="-O2 -march=i686 -g " CPPFLAGS="" LDFLAGS="-g" CC=gcc CXX=g++  ./configure --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info   " exited with exit code 1
15:35:22 checkResults    - SocketAuth(pid=6444, uid=201, gid=201)
[15:35:23] [19] - State: Failed
[15:35:23] [19] - Failed while building: Build job had failures:
   * alltray:source: Error building recipe alltray:source=/project.rpath.org@namespace:tag//your.domain.com@local:tag/0.69-0.1[is: x86(~cmov,~i486,~i586,~i686,~mmx,~nx,~sse,~sse2)]: Shell command "cd '/tmp/rmake/builds/alltray/alltray-0.69/';  CLASSPATH="" CFLAGS="-O2 -march=i686 -g" CXXFLAGS="-O2 -march=i686 -g " CPPFLAGS="" LDFLAGS="-g" CC=gcc CXX=g++  ./configure --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info   " exited with exit code 1

This output is helpful to tell us that the build failed, but not necessarily what failed in the build. Make note of the number inside the '[]. This number is the rMake build number for this iteration of building. Use this build number to find what failed:

#> rmake query --logs 19 | less

The build number in the example above is 19. This build number may be different on your local system. Make note of the build number when using the rmake query command to ensure that the logs you display are from the correct rmake build.

Search For Keyword Failures

When the output from the log is displayed on the screen, press the '/' key. This key allows you to search in the logs. After pressing '/' input the search phrase; for example, look for the word error. Use the 'n' key to move to the next search result. The first true error (not the liberror package) appears as follows:

When looking for problems during building, search for the following keywords: error, missing, not found.

Add buildRequires To The Recipe

Open alltray.recipe and look for the buildRequires = [] entry. Add in any build requirements not met that are called out by the rMake build process by inputting the build requirement name buildRequires = ['buildreq1', 'buildreq2'] (replace buildreq1 and buildreq2 with your build requirements called out by rMake). Note the use of quotes inside the brackets. Conary and rMake are very specific on syntax inside recipes and will require each build requirement to be surrounded by single quotes and separated by commas. After inserting the text, save and close the recipe.

Continue and Repeat

Begin building the recipe again:

#> rmake build alltray.recipe --reuse

The --reuse flag allows rmake to reuse the same chroot that was created previously allowing for a much faster build and better utilization of space in your build environment.

While the recipe is building, note the incremental build number in brackets. When the recipe fails to build again, look through the logs by repeating the command issued above:

#> rmake query --logs 20 | less

Make sure to increase the build number accordingly so that you are looking at the correct build log. Repeat the troubleshooting steps above until rMake reports a successful build of the recipe. Remember to use the keyword searching methods described above.

Commit the rMake Build

After successful build of the alltray.recipe, you are now ready to build and commit your package. With rMake, the --message flag can be used with commit for a commit message:

#> rmake commit XX --message="First commit package of alltray"

replace XX above with the build number of the successful alltray.recipe build previously done. As an example:

#> rmake commit 25 --message="First commit package of alltray"
                                                                           
Committed job 25:
    alltray:source=/project.rpath.org@namespace:tag//your.host.com@local:tag/0.69-0.6[is: x86(~cmov,~i486,~i586,~i686,~mmx,~nx,~sse,~sse2)] ->
       alltray=/project.rpath.org@namespace:tag/0.69-1-1[is: x86]
       alltray:source=/project.rpath.org@namespace:tag/0.69-1[]

+ Replacing CONARY file /home/$USER/conary/project/alltray after initial commit
#> 

The alltray package is now built and committed to the project repository. Install the package you just built and test the functionality.

#> sudo conary update alltray=project.rpath.org@namespace:tag
Password:
Resolving dependencies...The following updates will be performed:
    Install alltray(:data :devellib :doc :lib :runtime)=0.69-1-1
continue with update? [Y/n] 
Applying update job:                       
    Install alltray(:data :devellib :doc :lib :runtime)=0.69-1-1