qlnx-psets Home Page

The qlnx-psets tool born to help dealing with the nice data generated by the combination of bkcvs repository (kudos to bitmover.com to supply such data and to kernel.org to host the rsync server) and cvsps (kudos to David Mansfield). The data that is dumped on the CVS repository, while being a complete mirror of the real bk repository, it is quite difficult to work out due to the limitations of CVS in the operations that I am usually doing on such data. Those typically leads to cheacking out e given version (on a patch set base) and adding/removing patch sets. The qlnx-psets tool uses cvsps to collect a log and a series of patch sets, and it uses such data to generate user configurations. It might be worth to be noticed, that the qlnx-psets tool can be used on any CVS repository, not only Linux kernels (with some limitations regarding kernel-version -> patchset mapping).


Setup and Use

To start using qlnx-psets, you need to start by installing cvsps, that you can find at:

http://www.cobite.com/cvsps

Download the package and install it by following the instructions given by the authors (and say thanks to them since cvsps really brings back order to the CVS caos). The whole process starts by rsync'ing the bkcvs CVS repository from kernel.org, that in my machines usually goes like:

$ cat bksync.sh
#!/bin/sh

rsync -vzrlt --delete rsync.kernel.org::pub/scm/linux/kernel/bkcvs /usr/src/


The shell script above fetches the bkcvs CVS repository into the /usr/src/bkcvs directory. Since rsync is based upon a smart protocol, updates of the local repository are typically fast and not bandwidth hungry. The qlnx-psets tool supports a few command line options, that can also be fed using a configuration file (to avoid retyping):

qlnx-psets.pl [--qroot DIR] [--root DIR] [--repo NAME] [--outdir DIR]
              [--cfgfile FILE] [--no-cvs-direct] [--no-bkcvs] [--hard-links] [--help]
              cmd [arg [, arg]]

--qroot DIR       = Sets the tool root directory to DIR (mandatory)
--root DIR        = Sets the CVS root to DIR (mandatory)
--repo NAME       = Selects the repository inside the CVS root to NAME (mandatory)
--outdir DIR      = Sets the output directory to DIR (default .)
--cfgfile FILE    = Use the config file FILE to load options
--no-cvs-direct   = Unselect the CVS-direct mode of cvsps
--no-bkcvs        = Unselect the special --bkcvs mode of cvsps
--hard-links      = Use hard links to replicate caches
--help            = Prints this help

Mapping between command line options and configurations file options are:

--qroot           => $qlnx_qroot (QLNX_QROOT)
--root            => $qlnx_root (QLNX_ROOT)
--repo            => $qlnx_repo (QLNX_REPO)
--outdir          => $qlnx_outdir (QLNX_OUTDIR)
--no-cvs-direct   => $qlnx_cvsdirect (either empty or "--cvs-direct") (QLNX_CVSDIRECT)
--no-bkcvs        => $qlnx_bkcvs (either empty or "--bkcvs") (QLNX_BKCVS)
--hard-links      => $qlnx_usehl (QLNX_USEHL)

My example configuration file looks like:

# Maps to --qroot (defaults to environment QLNX_QROOT)
$qlnx_qroot = "/home/nobkup/qlnx-psets";

# Maps to --root (defaults to environment QLNX_ROOT)
$qlnx_root = "/usr/src/bkcvs";

# Maps to --repo (defaults to environment QLNX_REPO)
$qlnx_repo = "linux-2.5";

# Maps to --outdir (defaults to environment QLNX_OUTDIR)
$qlnx_outdir = ".";

# Maps to --no-cvs-direct (defaults to environment QLNX_CVSDIRECT)
$qlnx_cvsdirect = "--cvs-direct";

# Maps to --no-bkcvs (defaults to environment QLNX_BKCVS)
$qlnx_bkcvs = "--bkcvs";

# Maps to --hard-links (defaults to environment QLNX_USEHL)
$qlnx_usehl = 1;

The qlnx-psets.pl root directory (--qroot) is the place where the tool generate its own metadata to help and speedup the user command processing. Inside such directory, the following structure can be found:

caches   <dir>
patches  <dir>
xpatches <dir>
lastps
logfile
migfile
psets

The "caches" directory stores cached copies of the tree, to help cutting down the generation time of user configurations. Inside the "caches" directory, one or more directory will be stored by the user, with name equal to the patchset number they cache precedeed by the '#', or the version string they refer to. For example, my current "caches" contains only one cached copy "#18121", that corresponds to 2.6.6 (will see later how):

[davide@bigblue qlnx-psets]$ ll caches/
total 4
drwxrwxr-x 18 davide davide 4096 May 9 19:33 #18121

Inside cache directories we will find the typical content of the cached tree:

[davide@bigblue qlnx-psets]$ ll caches/#18121/
total 284
drwxrwxr-x 22 davide davide 4096 May 9 19:31 arch
-rw-r--r-- 1  davide davide 18691 May 9 19:32 COPYING
-rw-r--r-- 1  davide davide 85686 May 9 19:32 CREDITS
drwxrwxr-x 2  davide davide 4096 May 9 19:33 crypto
drwxrwxr-x 42 davide davide 4096 May 9 19:33 Documentation
drwxrwxr-x 44 davide davide 4096 May 9 19:33 drivers
drwxrwxr-x 53 davide davide 4096 May 9 19:33 fs
drwxrwxr-x 33 davide davide 4096 May 9 19:32 include
drwxrwxr-x 2  davide davide 4096 May 9 19:33 init
drwxrwxr-x 2  davide davide 4096 May 9 19:33 ipc
drwxrwxr-x 3  davide davide 4096 May 9 19:33 kernel
drwxrwxr-x 4  davide davide 4096 May 9 19:33 lib
-rw-r--r-- 1  davide davide 51590 May 9 19:32 MAINTAINERS
-rw-r--r-- 1  davide davide 36156 May 9 19:32 Makefile
drwxrwxr-x 2  davide davide 4096 May 9 19:33 mm
drwxrwxr-x 32 davide davide 4096 May 9 19:33 net
-rw-r--r-- 1  davide davide 13976 May 9 19:33 README
-rw-r--r-- 1  davide davide 2815 May 9 19:32 REPORTING-BUGS
drwxrwxr-x 7  davide davide 4096 May 9 19:33 scripts
drwxrwxr-x 3  davide davide 4096 May 9 19:33 security
drwxrwxr-x 15 davide davide 4096 May 9 19:33 sound
drwxrwxr-x 2  davide davide 4096 May 9 19:33 usr

I could rename the directory caches/#18121 to caches/2.6.6 and the effect would result to be the same. When used together with the bkcvs-generated data, caches should be retrieved directly from the CVS root, and not from kernel tarballs. This because the kernel tarballs is missing of some directories/files that patchsets wants to patch. Failing in doing so will result in errors from the patching operations. To do this you need to checkout from the CVS root by the date of the release you want to checkout. To do this, supposing we want to checkout release 2.6.5, we first search the logs for such release:

$ qlnx-psets.pl sl 'Tag: v2_6_5'

This will dump all log entries that contains such tag, and at the end you will see something like:

********************************
PatchSet 17248
Date: 2004/04/04 04:34:07
Author: torvalds
Branch: HEAD
Tag: v2_6_5
Log:
Linux 2.6.5

BKrev: 406f822foQTgmLYKWBzPlaKSfZ_q_g

Members:
        ChangeSet:1.17248->1.17249
        Makefile:1.423->1.424

This means that such tag has been created on 2004/04/04. So we can then use such date to checkout from CVS:

$ cd caches
$ cvs -d /usr/src/bkcvs co -D 2004-04-04 linux-2.5
$ mv linux-2.5 2.6.0

Other caches can simply be created with the qlnx-psets tool itself, by creating custom configuration file like:

$ cat 2.6.6.conf
@2.6.6

and by issuing:

$ qlnx-psets.pl cs 2.6.6.conf
$ mv linux-2.5 $QLNX_QROOT/caches/2.6.6

The qlnx-psets tool uses some heuristic to find out the best cache to use to generate a given user configuration. Care should be taken when using the --hard-links option because the generated copy uses hard link to the cached copy. The qlnx-psets tool automatically break the link when applies the patchsets, but the user should be careful when editing the built configuration. We will show later how to find mappings between patchset numbers and kernel revisions. The "patches" directory is the place where cvsps generated patch files wil be stored. File names inside the "patches" directory have the format:

PATCHSETNUM.patch

where PATCHSETNUM is the patchset that has to be used to migrate to version PATCHSETNUM. The "xpatches" directory stores patches that do not come from the CVS patch set, but that are instead coded by the user, and that the user wants to apply to his configuration. The "lastps" file is used by the tool to store the last patchset synced from the CVS repository, in order to cut update times. Only new patchset files will be generated when ussuing a 'cupdate' command. The "logfile" is a cached copy of the full cvsps log file, and it is used to speed up operations that might be time bound to the cvsps processing performance. The "migfile" is generated by the qlnx-psets tool, by parsing the "logfile" and detecting associations between patchset numbers and kernel versions. An example few lines of my "migfile" are:

2.6.5     17248
2.6.6-rc1 17705
2.6.6-rc2 17841
2.6.6-rc3 17985
2.6.6     18121
2.6.7-rc1 18806
2.6.7-rc2 19020
2.6.7-rc3 19355

We see here that, for example, the patchset 18121 correspond to the kernel version 2.6.6 (the one that is shown cached in the lines above). The "psets" file stores, one by line, all the patchsets contained inside the cvsps log file. To start using qlnx-psets, you can just edit the supplied configuration file and edit the variables to fit your setup. Typically you might want to set $qlnx_qroot, $qlnx_root, $qlnx_repo and $qlnx_outdir. As first-time caching speed up trick, you can edit (create actually) the "lastps" file containing the oldest patchset that you will ever request to remove. For example, if you know you configuration will never go back past version 2.6.2, you can set the "lastps" to 15325 (patchset version corresponding to 2.6.2). Then you can start issuing your first request to qlnx-psets, and this will be the one that will first create and sync its metadata with the CVS repository selected inside the configuration file:

$ ./qlnx-psets.pl --cfgfile qlnx.config cu

The command will take some time, because the qlnx-psets tool will ask cvsps to generate all patchsets starting from the "lastps" one up to the most current one (note that if you didn't set the "lastps" file, this can really take some time). The good news is, that new updates will always take a fraction of the time the first one took (depending on how many new patchsets have been generated inside the selected CVS repository).


Supported Commands

This is a list of commands supported by the qlnx-psets tool:

cupdate
cu

After a refresh of the CVS repository, it is necessary to resync the qlnx-psets metadata, to let the tool see the possible new patchsets. By issuing this command you will force qlnx-psets to regenerate many of its internal files, and to fetch the new (thanks to the "lastps" file) patchsets. The 'cupdate' command should be issued only when it is knows that new data has been fetched inside the CVS root, while all the following commands can run by only using the cached qlnx-psets metadata.

migs [VER] ...
mg [VER] ...

This command will print the migration patchset by parsing the "migfile" file that is generated during the 'cupdate' command. If no VER is specified, the command will emit all the versions, otherwise the output will be restricted to the selected ones. The output will have the kernel version in the first column, and the corresponding patchset number in the second column.

logs PSET ...
lg PSET ...

It dumps the log entries corresponding to the patchsets listed in the command line.

getpatch PSET ...
gp PSET ...

It dumps the patchsets (diff metadata) listed in the command line.

searchlogs REGEX
sl REGEX

It dumps the log entries that matches the regular expression listed in the command line.

ckstart CFGFILE
cs CFGFILE

This command reads the configuration file CFGFILE and builds the tree that matches the supplied configuration. The configuration file supports the folowing format (one rule per line):

ALL

Represent the whole-patchset, that basically matches the last tree version. The configuration patchset set is initialized with the whole "psets" file as a result of an ALL line in the CFGFILE.

@VER

Represent the kernel version VER, and the "migfile" file is used to lookup the corresponding patchset number from the kernel version KVER.

%PSET

Such directive request all patchsets <= PSET to be added as base configuration. It is worth to be noticed that ALL, @VER and %PSET are initializing commands, that have to be used to setup the baseline upon which to apply selective patchsets add/removal.

+PSET

Adds the patchset PSET to the configuration.

-PSET

Removes the patchset PSET to the configuration.

+PFNAME@ORDER

Adds the patch file stored inside xpatches/PFNAME, using the ORDER order in the patching sequence operation. Certain patches might need to be applied at a given sequence in the patchset order, to work correctly and not generate rejections. ORDER is allowed to be a floating point number, to let the user to place custom patches in between integer patchsets numbers.

-PFNAME@ORDER

Removes the patch file stored inside xpatches/PFNAME, using the ORDER order in the patching sequence operation.

Example configurations are:

ALL
-19406
-19405
-19400

and:

@2.6.6
+18122
+18123
+18124
+18125
+18126
-18000
+epoll-size-check.diff@18124.01
+epoll-use-rbtrees.diff@18124.02

The qlnx-psets automatically selects the best cache it can find to reduce the number of patchsets that will have to be applied to map the requested configuration.

ckresume
cr

The 'ckresume' command is used to resume the configuration driven tree build, when conflicts are found by qlnx-psets that need to be fixed. For example, it is possible that the configuration chosen will generate rejections (.rej files), and these are required to be fixed before proceeding. So it is really up to the user to fix rejections before invoking the next 'ckresume' command, since the qlnx-psets tool will start from the next patch that generated rejections.

patch [+-]PSET ...
pa [+-]PSET ...

Applies the patchsets specified in the command line, to the existing tree built with 'ckstart' (and possibly 'ckresume'). A plus (+) sign in from of the patch means APPLY, while a minus sign (-) means REMOVE. The PSET identifier can either be a patchset number (the patches/PSET.patch file will be used) or a patch file name inside "xpatches" (the xpatches/PSET file will be used).

genpsets FROM TO
ge FROM TO

Force generation of patch sets from patchset number FROM to patchset number TO.



Download

The current version of the qlnx-psets tool is available here:

qlnx-psets-0.5.tar.gz