rPath Appliance Platform Agent:Flip-flop Update Plugin

From rPath Wiki

The rPath Appliance Platform Agent (rAPA) default update mechanism can be overridden by the Flip-flop Update and Factory Reset plugins. This change is at the discretion of the appliance developers. The Flip-flop Update plugin allows an administrator to download an update to the appliance and test it prior to committing to replace the system's current state.

Using the Flip-flop Update Plugin Interface

See the Upload Update Images page in the Appliance Administration pages for information on how to use the Flip-flop Update plugin in the web interface.

Appliance Developer Notes

The Flip-flop Update plugin uses two or more system partitions and a special boot loader configuration to provide a failsafe in case a new system update causes issues on an appliance. rPath recommends using the UpdateTroves plugin to provide the benefits of Conary during updates. The Flip-flop Update plugin provides an alternative to this recommendation for those that place a high priority on the time it takes to revert to a last known good state.

If the device that is to be used for the target partition is in use at the beginning of a flipflop operation, the flipflop will fail and return an error. One way to check if a device is in use is to use the system command fuser with -m option.

Appliance developers may also include the Factory Reset plugin and a factory reset partition alongside the flip-flop update. The factory reset requires the flip-flop update, but developers can hide the Flip-flop Update plugin ("System Update" menu item) in the user interface to make use of the factory reset without exposing the flip-flop update. See the Factory Reset section for plugin-specific development requirements.

Developers should add a trove to the appliance group that specifies all the mount points to be used by the plugin.

Updates generated for consumption by the Flip-flop Update plugin may be digitally signed. The Flip-flop plugin verifies the digital signature before applying the update. Setting the Flip-flop Update plugin to use signed update images requires further configuration and specific steps as covered in the documentation below. By default, the Flip-flop Update plugin does not accept signed images.

Developers should add a trove to the appliance group that specifies all the mount points to be used by the plugin. Developers should add the gpg:runtime trove into the appliance if signed images will be used.

Partitions

The following partition structure supports the flip-flop boot feature:

• Boot partition
• Operating system partition 1
• Operating system partition 2
• Application data partition
• Swap partition
• Factory reset partition (if provided by the appliance developers, see the Factory Reset plugin)

Process

During a flip-flop update, the appliance steps through the following operations:

1. Upload the update image
2. Unmount the alternate system partition
3. Format the alternate system partition
4. Unpack the update image to the alternate system partition
5. Copy designated files from the original system partition to the alternate system partition
6. Configure GRUB to boot once to the alternate system partition
7. Boot to the alternate partition

After the appliance boots to the alternate partition, system administrators can test the update, and then choose to finalize the update or revert to the previous state.

Configuration

Appliances designed to use the flip-flop update have some configuration that supports the feature. This configuration includes configuration for:

• Appliance installation, including system and data partition setup and GRUB configuration
• Plugin-specific configuration files

System and Data Partitions

The Flip-flop Update plugin designates two system partitions by default: Partition A and Partition B. The appliance installs on one of these two partitions and makes it the default partition to which to boot the appliance.

The data partition includes application data that should remain consistent during flip-flop updates. Developers should ensure all application data is permanently stored on the data partition. Otherwise, developers must designate additional files to be copied between partitions during the update (applianceconfig script called in the plugin configuration).

Appliance developers can use a kickstart file during installation to partition the system and install the appliance in the appropriate partitions. See Kickstart for Flip-flop Update.

GRUB

On the appliance, the master boot record (/dev/sda or /dev/hda) contains a grub configured to read from the /mainboot configuration (mount point default, configured by the flipflop.mounts.maingrub directive). GRUB on the appliance chain loads the two system partitions. The following is a sample for /mainboot/grub/grub.conf used on such an appliance:

#boot=/dev/sda
default=0
timeout=9
title Partition A
        rootnoverify (hd0,0)
        chainloader +1
title Partition B
        rootnoverify (hd0,1)
        chainloader +1

Appliance developers should install GRUB to the first system partition instead of the master boot record when installing through Anaconda. This and other GRUB configuration can be handled with a kickstart file at installation. See Kickstart for Flip-flop Update.

Kickstart for Flip-flop Update

The following is sample data for a kickstart file for automating the appliance installation (Anaconda installer) for appliances using the Flip-flop Update plugin. Modify the media-template package for your appliance to include the kickstart file:

bootloader --location=partition
clearpart --all
part /mainboot --fstype ext3 --size=20 --asprimary
part / --fstype ext3 --size=2048 --asprimary
part /alt --fstype ext3 --size=2048 --asprimary
part /factory --fstype ext3 --size=2048
part swap --size=1000 --grow --maxsize=2000
part /srv --fstype ext3 --size=100 --grow

...
%post --nochroot
disk=hda #Or sda depending on hardware
#echo "Remounting /dev/${disk}1" |tee /tmp/post.log
mount /tmp/${disk}1 /mnt/sysimage -o remount,ro

#echo "Unmounting /dev/${disk}2" |tee -a /tmp/post.log
umount /tmp/${disk}2

echo "Copying /tmp/${disk}1 to /tmp/${disk}2" |tee -a /tmp/post.log
dd if=/tmp/${disk}1 of=/tmp/${disk}2

echo "Remounting /dev/${disk}1" |tee -a /tmp/post.log
mount /tmp/${disk}1 /mnt/sysimage -o remount,rw
mount -t ext3 /tmp/${disk}2 /mnt/sysimage/alt

echo "Modifying fstab" |tee -a /tmp/post.log
cp /mnt/sysimage/etc/fstab /mnt/sysimage/etc/fstab.orig
sed -e 's/LABEL=\/[0-9]*\([ \t]*\/ \)/\/dev\/'$disk'1\1/' -e 's/LABEL=\/alt[0-9]*\([ \t]*\/alt\)/\/dev\/'$disk'2\1/' -e 's/LABEL=\/factory[0-9]*\([ \t]*\/factory\)/\/dev\/'$disk'5\1/' -e 's/LABEL=\/srv[0-9]*\([ \t]*\/srv\)/\/dev\/'$disk'7\1/' -e 's/LABEL=\/mainboot[0-9]*\([ \t]*\/mainboot\)/\/dev\/'$disk'3\1/' </mnt/sysimage/etc/fstab.orig >/mnt/sysimage/etc/fstab

cp /mnt/sysimage/alt/etc/fstab /mnt/sysimage/alt/etc/fstab.orig
sed -e 's/LABEL=\/[0-9]*\([ \t]*\/ \)/\/dev\/'$disk'2\1/' -e 's/LABEL=\/alt[0-9]*\([ \t]*\/alt\)/\/dev\/'$disk'1\1/' -e 's/LABEL=\/factory[0-9]*\([ \t]*\/factory\)/\/dev\/'$disk'5\1/' -e 's/LABEL=\/srv[0-9]*\([ \t]*\/srv\)/\/dev\/'$disk'7\1/' -e 's/LABEL=\/mainboot[0-9]*\([ \t]*\/mainboot\)/\/dev\/'$disk'3\1/' </mnt/sysimage/alt/etc/fstab.orig >/mnt/sysimage/alt/etc/fstab

echo "Modifying grub configuration" |tee -a /tmp/post.log
mkdir /mnt/sysimage/mainboot/grub
cp /mnt/sysimage/boot/grub/* /mnt/sysimage/mainboot/grub/
cat <<EOT >/mnt/sysimage/mainboot/grub/grub.conf
#boot=/dev/sda
default=0
timeout=10
title System Image 1
    rootnoverify (hd0,0)
    chainloader +1
title System Image 2
    rootnoverify (hd0,1)
    chainloader +1
EOT

cp /mnt/sysimage/boot/grub/grub.conf /mnt/sysimage/boot/grub/grub.conf.orig
sed -e 's/root=LABEL=\/[0-9]*/root=\/dev\/'$disk'1/' </mnt/sysimage/boot/grub/grub.conf.orig >/mnt/sysimage/boot/grub/grub.conf
sed -e 's/root=LABEL=\/[0-9]*/root=\/dev\/'$disk'2/' </mnt/sysimage/alt/boot/grub/grub.conf.orig >/mnt/sysimage/alt/boot/grub/grub.conf

echo "Installing grub" |tee -a /tmp/post.log

echo "install (hd0,2)/grub/stage1 (hd0) (hd0,2)/grub/stage2 (hd0,2)/grub/grub.conf" | grub --batch
echo "install (hd0,0)/boot/grub/stage1 (hd0,0) (hd0,0)/boot/grub/stage2 (hd0,0)/boot/grub/grub.conf" | grub --batch
echo "install (hd0,1)/boot/grub/stage1 (hd0,1) (hd0,1)/boot/grub/stage2 (hd0,1)/boot/grub/grub.conf" | grub --batch

#Optionally set up the factory images
echo "Creating Factory Reset Datasets"
/mnt/sysimage/bin/tar -czvplf /mnt/sysimage/factory/systemimage.tgz -C /mnt/sysimage . >> /tmp/post.log
/mnt/sysimage/bin/tar -czvplf /mnt/sysimage/factory/dataimage.tgz -C /mnt/sysimage/srv . >> /tmp/post.log

#copy the post log into the system image.
mkdir -p /mnt/sysimage/root/postdebug
cp /tmp/post.log /mnt/sysimage/root/postdebug

Plugin Configuration

The directives below from flipflop.cfg may require customizations in custom.cfg to configure the flip-flop update plugin. Some of these selections are based on changes to the kickstart file used during installation:

If an exception is raised while running the user copy script as defined by flipflop.commands.applianceconfig, the stderr (if applicable) will be in the status message. The status message is accessible via the flipflop.Update.getProcessingMessage() XMLRPC interface and the rAPA backend service logs

flipflop.update_check_url

If this URL is set, the plugin displays a Check for Updates link that directs the user to this URL.

flipflop.autofinalize

This settings provides that when the system boots and Appliance starts, the alternate partition is marked as the last known good without requiring a manual Finalize step by the user.
NOTE: rPath recommends developers do not use auto-finalize.

flipflop.partitions.boot

The flipflop process cycles between two system partitions specified by this setting, the devices here must be listed in the same order in which they are mentioned in the main grub.conf file, such as ['/dev/sda1', '/dev/sda2'].

flipflop.mounts.boot

This is the mount points of the partitions defined in flipflop.partitions.boot. The first item [0] is the running system image, [-1] is the previous system image, and [1] is the next mount point to use.

flipflop.mounts.maingrub

This is the location where the main GRUB parition is mounted, such as /mainboot. rPath recommends that this partition is mounted read-only.

flipflop.commands.applianceconfig

This is the script that should be run to transition all important appliance-specific configuration from original partition to alternate partition, relative to the mount point of the alternate partition. For example, if B is mounted at /alt, and the file to run is /alt/sbin/save_appliance_config, use /sbin/save_appliance_config. The script cannot contain a space in the filename, and all options passed to the script should be added with space-separated arguments following.

flipflop.files.system

These are files that should be copied from original partition to alternate partition. The default setting includes the files listed in the Copied Settings section. rPath recommends developers add to the list if needed and not to remove from the list.

Copied Settings

The following files are managed by the Appliance Platform during the flip-flop update:

/etc/passwd
/etc/shadow
/etc/group
/etc/gshadow
/etc/modprobe.conf
/etc/hosts
/etc/resolv.conf
/etc/sysconfig/network-scripts/ifcfg-*
/etc/localtime
/etc/sysconfig/network
/etc/sysconfig/clock
/etc/sysconfig/i18n
/etc/sysconfig/iptables
/etc/sysconfig/hwconf
/etc/sysconfig/firstboot
/etc/sysconfig/grub
/etc/sysconfig/keyboard
/etc/sysconfig/mouse

Also included is Appliance data in /var/lib/raa/raadb.

The Flip-flop Update plugin will call the script configured in the config file designated by failsafe.commands.applianceconfig to save any data that the appliance developer needs to be copied from A to B.

Signed Image Settings

As noted previously, the gpg:runtime trove should be included with the appliance.

The directives below are default settings from the [/flipflop] section of the custom.cfg configuration file. Appliance developers must change this value to True to enable signed update images in their appliances.

flipflop.validate_signature = False

This is the setting that tells the Flip-flop Plugin to validate updates for signatures

flipflop.validate_signature.key_ring = "/var/lib/raa/keyring"

Path to the location that contains the keys to validate the signed image against

If flipflop.validate_signature = True, a signed image is required, and that image must be signed directly by a key contained in "/var/lib/raa/keyring/pubring.gpg".

• Further Information: Creating/Signing Flip-flop images; Check, test, and commit signed updates; Signed Image Workflow

Update Images

To generate an update image to work with the Flip-flop Update plugin, create a compressed tar file from the same appliance group as the appliance image. Make the compressed tar file available in accordance the release management structure for the appliance.

Factory Reset Notes

The Factory Reset plugin requires the Flip-flop Update plugin and some additional configuration requirements:

• A factory reset image
• Adjustments to flipflop.cfg
• Plugin-specific configuration

Though the alternate partition is available in grub as with the flip-flop update, the factory reset also changes the data partition, eliminating the ability to revert to the previous system state.

All plugins that are included in an appliance are enabled by default. Use the Plugins page for information about disabling or hiding included plugins.

Factory Reset Image

The factory reset image includes a system image and a data image, each in the form of a compressed tar file:

System image

The factory reset reformats the system partition and unpacks this file into that partition. The default name expected by the plugin is systemimage.tgz and can be configured with the factoryreset.image.system directive.

Data image

The factory reset reformats the data partition, mounts it, and unpacks this file into that partition. The default name expected by the plugin is dataimage.tgz and can be configured with the factoryreset.image.data directive.

Generate a these files from an installed version of the appliance by running the following commands, such as in the kickstart file, to install the appliance:

$> tar czplf /factory/systemimage.tgz /
$> tar czplf /factory/dataimage.tgz /srv

In the tar command, the p option preserves permissions and the l option instructs tar to use one file system.

Factory Reset Configuration

When creating custom configuration for Factory Reset, package customizations in custom.cfg as instructed on the plugins page. This includes any changes to flipflop.cfg as well as to factoryreset.cfg. The following is a copy of factoryreset.cfg:

[global]

[/factoryreset]
# Partition
#Devices for the factory and data partitions respectively
factoryreset.partition.factory = '/dev/sda5'
factoryreset.partition.data = '/dev/sda7'

#mount points for the factory and data partitions respectively
factoryreset.mount.factory = '/factory'
factoryreset.mount.data = '/srv'

#images for the factory and data filesystems respectively, used as
#<factoryreset.mount.system> + '/' + <factoryreset.image.system>
factoryreset.image.system = 'systemimage.tgz'
factoryreset.image.data = 'dataimage.tgz'

#This script will be run before the system is reset, for example to stop any services that may blow up with the data directory dissappearing.
#The script may not contain whitespace in the filename.  All options passed to the script should be added below.  i.e. "/sbin/pre_appliance_reset Arg1 Arg2 'argument with spaces'"
factoryreset.commands.prereset = '/sbin/pre_appliance_reset'

The following directives in flipflop.cfg will also affect the Factory Reset plugin:

• flipflop.partitions.boot
• flipflop.mounts.boot
• flipflop.mounts.maingrub