Skip to contentSkip to conversation

kpatchup 0.02 kernel patching script

ANNOUNCE Matt Mackall 4 participants 8 messages Mar 3, 2004 – Mar 4, 2004

lore
Share & tools
lore
#0 lore raw

This is the first release of kpatchup, a script for managing switching between kernel releases via patches with some smarts:

- understands -pre and -rc version numbering
- aware of various external trees
- automatically patch between any tree in an x.y release
- automatically download and cache patches on demand
- automatically determine the latest patch in various series
- optionally print version strings or URLs for patches

Currently it knows about 2.4, 2.4-pre, 2.6, 2.6-pre, 2.6-bk, 2.6-mm, and 2.6-tiny.

Example usage:

 $ head Makefile
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 2
 EXTRAVERSION =-rc2
 [...]
 $ kpatchup 2.6-mm
 2.6.2-rc2 -> 2.6.4-rc1-mm1
 Applying patch-2.6.2-rc2.bz2 -R
 Applying patch-2.6.2.bz2
 Applying patch-2.6.3.bz2
 Downloading patch-2.6.4-rc1.bz2...
 Applying patch-2.6.4-rc1.bz2
 Downloading 2.6.4-rc1-mm1.bz2...
 Applying 2.6.4-rc1-mm1.bz2
 $ head Makefile
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 4
 EXTRAVERSION =-rc1-mm1
 NAME=Feisty Dunnart
 [...] 
 $ kpatchup -q 2.6.3-rc1
 $ head Makefile
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 3
 EXTRAVERSION =-rc1
 NAME=Feisty Dunnart
 [...]
 $ kpatchup -s 2.6-bk
 2.6.4-rc1-bk3
 $ kpatchup -u 2.4-pre
 http://www.kernel.org/pub/linux/kernel/v2.4/testing/patch-2.4.26-pre1.bz2

This is an alpha release for people to experiment with. Feedback and patches encouraged. Grab your copy today at:

http://selenic.com/kpatchup/

Signature · 1 line
Matt Mackall : http://www.selenic.com : Linux development and consulting
#1 lore raw
Quoted 10 lines · Matt Mackall

This is the first release of kpatchup, a script for managing switching between kernel releases via patches with some smarts:

- understands -pre and -rc version numbering
- aware of various external trees
- automatically patch between any tree in an x.y release
- automatically download and cache patches on demand
- automatically determine the latest patch in various series
- optionally print version strings or URLs for patches

Quoted 2 lines

Currently it knows about 2.4, 2.4-pre, 2.6, 2.6-pre, 2.6-bk, 2.6-mm, and 2.6-tiny.

Oh i definitely owe you one now, this is replacing the ugly shell script i had before, i'm mostly using this now to download and patch up trees before cvs import'ing them.

Thanks!

#2 lore raw
In reply to Matt Mackall
Quoted 2 lines · Matt Mackall

This is the first release of kpatchup, a script for managing switching between kernel releases via patches with some smarts:

Hi Matt!

Please find below my grab_kernel, latest-kernel-version and lkvercmp scripts, in case they are useful for you to merge. Kinda rough, like any unexposed code...

Rusty

Signature · 1 line
Anyone who quotes me in their signature is an idiot -- Rusty Russell

#! /bin/sh
# Compare multiple kernel versions

# Return more recent of two kernel versions. if [ $# -eq 0 ]; then
echo Usage: $0 kernelversion kernelversion ... >&2 echo Returns greatest of the versions. >&2 exit 1
fi

unknown()
{
# Does a -dj kernel come before a -ac kernel? Or a Linus kernel? # Undefined.
echo Can\'t compare extension \'"$1"\'. >&2
exit 1
}

firstpart()

{
    echo $1 | sed 's/[-.].*//'
}

lastpart()

{
    echo $1 | sed 's/.*[-.]//'
}

trimlastpart()

{
    echo $1 | sed 's/[-.][^-.]*$//'
}

trimfirstpart()

{
    echo $1 | sed 's/^[^-.]*[-.]//'
}

letters()

{
    echo $1 | tr -d '[0-9]'
}

numbers()

{
    echo $1 | tr -dc '[0-9]'
}

if [ $# -eq 1 ]; then

    echo "$1"
    exit 0

fi

if [ $# -gt 2 ]; then

    FIRST=$1
    shift
    set -- $FIRST $($0 "$@")

fi

VER1=$1
VER2=$2

# Vanity kernels (at end): handle 2.6.3-mm2 vs 2.6.3-rc1-mm3 by comparing

# base first.
TAIL1=$(lastpart $VER1)
TAIL2=$(lastpart $VER2)
if [ x"$(letters $TAIL1)" = x"$(letters $TAIL2)" ]; then
    case "$TAIL1" in
    bk*|rc*|pre*) : ;;
    *)
	BASE1=$(trimlastpart $VER1)
	BASE2=$(trimlastpart $VER2)
	if [ x"$BASE1" = x"$BASE2" ]; then
	    if [ $(numbers $TAIL1) -gt $(numbers $TAIL2) ]; then
		echo $1
	    else
		echo $2
	    fi
	else
	    if [ x$(lkvercmp $BASE1 $BASE2) = x$BASE1 ]; then
		echo $1
	    else
		echo $2
	    fi
	fi
	exit
	;;
   esac
fi

while true; do

    PART1=$(firstpart $VER1)
    PART2=$(firstpart $VER2)
    VER1=$(trimfirstpart $VER1)
    VER2=$(trimfirstpart $VER2)

    PART1LETTERS=$(letters $PART1)
    PART2LETTERS=$(letters $PART2)
    PART1NUMBERS=$(numbers $PART1)
    PART2NUMBERS=$(numbers $PART2)

    if [ "$PART1LETTERS" = "$PART2LETTERS" ]; then
	if [ "$PART1NUMBERS" = "$PART2NUMBERS" ]; then
	    continue;
	else
	    if [ "$PART1NUMBERS" -gt "$PART2NUMBERS" ]; then
		echo $1; exit
	    else
		echo $2; exit
	    fi
        fi
    fi

    case x"$PART1LETTERS" in
    xbk)
        # bk snapshots come after (ie. 2.6.1-bk1 is > 2.6.1).
	# So this always wins.
	case x"$PART2LETTERS" in
	x|xrc|xpre) echo $1; exit;;
	esac
	unknown $PART2LETTERS;;
    x)
	# Final beats everything except bk.
	case x"$PART2LETTERS" in
	xbk) echo $2; exit;;
	xrc|xpre) echo $1; exit;;
	esac
	unknown $PART2LETTERS;;
    xrc)
	# Only a final or bk can win.
	case x"$PART2LETTERS" in
        x|xbk) echo $2; exit;;
	xpre) echo $1; exit;;
        esac
	unknown $PART2LETTERS;;
    xpre)
        # Final, bk or rc can win.
	case x"$PART2LETTERS" in
        x|xbk|xrc) echo $2; exit;;
	xpre) echo $1; exit;;
        esac
	unknown $PART2LETTERS;;
    *)
	unknown $PART1LETTERS;;
    esac

done

#! /bin/sh -e
# Latest kernel version of this vintage.

KERNELDIR=~/devel/kernel

# Return the latest kernel versions.
get_kernel_vers()
{
# if [ x"$GK_KERNS" = x ]; then
# GK_KERNS=$(finger @finger.kernel.org | tail +2 | cut -s -d: -f2- | sed 's/[ ]*//')
# fi

    if [ x"$GK_KERNS" = x ]; then
	GK_KERNS=$(lynx -width=500 -dump http://www.kernel.org/kdist/finger_banner | tail +2 | cut -s -d: -f2- | sed 's/[ 	]*//')
    fi
    if [ x"$GK_KERNS" = x ]; then
	echo Failed to get kernel list from kernel.org >&2
	exit 1
    fi
    # Vanity are there too: only take recognized patches.
    # Ignore 2.4-bk versions: I don't know where they're coming from 8(
    echo "$GK_KERNS" # | egrep -- "$1|-rc|-bk|-pre|^[0-9]*\.[0-9]*\.[0-9]*\$" | grep -v '2\.4\.[0-9]*-bk[0-9]*'
}

PROBE=0
if [ x"$1" = x--probe ]; then

    PROBE=1
    shift

fi

case "$1" in --*)
echo "Usage: $0 [--probe] [kernel-base-version]" >&2
echo "Returns the latest version on the system, or with --probe," >&2 echo "returns the latest version on network." >&2
exit 1
;;
esac

if [ $# -eq 0 ]; then set .; fi

if [ $PROBE = 1 ]; then

    CANDIDATES=$(get_kernel_vers)
else
    CANDIDATES=$(cd $KERNELDIR && ls -d linux-* | fgrep -v '.tmp' | grep -v -- '-ppc' | sed 's/^linux-//')

fi

for w; do

    case "$w" in
	# By default, ignore any vanity kernels.
	.) FILTER="^[0-9.]*(-rc[0-9]*|-bk[0-9]*|-pre[0-9]*)*\$";;
	# eg -mm.
	-*) FILTER="^[0-9.]*(-rc[0-9]*|-bk[0-9]*|-pre[0-9]*)*$w[0-9]*\$";;
	# eg. 2.4
	*) FILTER="^$w[0-9.]*(-rc[0-9]*|-bk[0-9]*|-pre[0-9]*)*\$";;
    esac

    lkvercmp $(echo "$CANDIDATES" | egrep -- "$FILTER")

done

#! /bin/sh -e
# Script to grab a given kernel version (stolen from Virtual Anton).

barf()

{
    echo "$@" >&2
    exit 1
}

PATCHDIR=~/devel/kernel/kernel-patches
MIRRORS="rsync://kernel::kernel http://kernel.linux.pacific.net.au/linux/kernel/ rsync://master.kernel.org:/pub/linux/kernel" #MIRRORS="rsync.planetmirror.com::ftp.kernel.org/pub/linux/kernel master.kernel.org:/pub/linux/kernel"

# Takes destination directory, and filename, tries each mirror. get_file()

{
    get_file_dir=$1
    get_file_name=$2

    # Patch there already?
    if [ -f "$get_file_dir"/"$(basename "$get_file_name")" ]; then
	echo " Already have patch..."
	return 0
    fi

    for get_file_server in $MIRRORS; do
	echo Trying $get_file_server...
	case $get_file_server in
	rsync://*) get_file_s=`echo $get_file_server | cut -c9-`;
		  rsync $get_file_s/$get_file_name $get_file_dir && return 0
		  ;;
	http://*) get_file_s=`echo $get_file_server | cut -c8-`;
		wget -O $get_file_dir/`basename $get_file_name` $get_file_s/$get_file_name && return 0
		# Leaves a zero-len file on failure 8(
		rm -f $get_file_dir/`basename $get_file_name`
		;;
	esac
    done
    return 1
}

if [ $# -ne 2 ]; then
echo Usage: $0 version kerneldir >&2
exit 1
fi

VERSION=$1
KDIR=$2

if [ -d $KDIR/linux-$VERSION ]; then exit 0; fi
MAJOR=echo $VERSION | cut -d. -f1-2

# Given vanity kernel name, major version, and previous version, produce pathname vanity_source()

{
    case "$1" in
    *-ac*)
	echo people/alan/linux-$2/$3/patch-$1.bz2;;
    *-dj*)
	echo people/davej/patches/$2/$3/patch-$1.bz2;;
    *-mm*)
	echo people/akpm/patches/$2/$3/$1/$1.bz2;;
    *)
	barf "Unknown kernel series $1";;
    esac
}

# Given base name, new name and patch, do patch application. apply_patch()

{
    rm -rf $KDIR/.linux-$2
    cp -al $KDIR/linux-$1 $KDIR/.linux-$2 ||
	    barf "Couldn't clone $1 dir"
    bunzip2 -t $3 || barf "Corrupt patch $3"
    bzcat $3 | patch -s -d $KDIR/.linux-$2 -p1 ||
	    barf "Problem patching new kernel $2"
    mv $KDIR/.linux-$2 $KDIR/linux-$2
}

echo Grabbing $VERSION
case "$VERSION" in

    # Bitkeeper snapshot
    *-bk*)
	if [ -d $KDIR/linux-$VERSION ]; then exit 0; fi

	PREVIOUS=`echo "$VERSION" | sed 's/-bk.*//'`
	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/snapshots/patch-$VERSION.bz2  || barf "Couldn't get v$MAJOR/snapshots/patch-$VERSION.bz2"
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;

    # Various vanity patches
    *-mm[0-9]*|*-ac[0-9]*|*-dj[0-9]*)
	# First make sure we have base tree
	BASE_KERNEL=`echo "$VERSION" | sed -e 's/-[a-z][a-z][0-9]*$//'`
	$0 "$BASE_KERNEL" $KDIR	||
	    barf "Couldn't get $BASE_KERNEL to build $VERSION"
	VANITY=`vanity_source $VERSION $MAJOR $BASE_KERNEL`
	get_file $PATCHDIR $VANITY ||
		barf "Couldn't fetch $VANITY"

	apply_patch $BASE_KERNEL $VERSION $PATCHDIR/`basename $VANITY`
	;;

    # *-pre*: This refers to the Linus or Marcelo pre-patches.
    *-pre*|*-rc*)
	# Linus or Marcelo pre-patch.
	if [ -d $KDIR/linux-$VERSION ]; then exit 0; fi

	PREVIOUS=`echo "$VERSION" | sed 's/-\(pre\|rc\).*//'`
	LAST_VER=`echo "$PREVIOUS" | cut -d. -f3`
	PREVIOUS=`echo "$PREVIOUS" | cut -d. -f1,2`.`expr $LAST_VER - 1 || true`
	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/testing/patch-$VERSION.bz2 ||
	    get_file $PATCHDIR v$MAJOR/testing/old/patch-$VERSION.bz2 ||
	    barf "Couldn't get v$MAJOR/testing/patch-$VERSION.bz2"
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;

    # Linus 2.6.0 test patch.
    2.6.0-test*)
	VER=`echo "$VERSION" | sed 's/.*test//'`
	PREVIOUS=2.6.0-test`expr $VER - 1 || true`

	if [ $PREVIOUS = 2.6.0-test0 ]; then
	    PREVIOUS=2.5.75
	fi

	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/patch-$VERSION.bz2 ||
	    get_file $PATCHDIR v$MAJOR/old/patch-$VERSION.bz2 ||
	    barf "Couldn't get v$MAJOR/patch-$VERSION.bz2"
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;

    2.6.0)
	PREVIOUS=2.6.0-test11

	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/patch-$VERSION.bz2 ||
	    get_file $PATCHDIR v$MAJOR/old/patch-$VERSION.bz2 ||
	    barf "Couldn't get v$MAJOR/patch-$VERSION.bz2"
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;

    # *.0: This refers to the base of the Linus kernels, fetched whole.
    *.0)
	# Linus' root of all trees.
	get_file $PATCHDIR v$MAJOR/linux-$VERSION.tar.bz2 || barf "Couldn't get  v$MAJOR/linux-$VERSION.tar.bz2"
	bzcat $PATCHDIR/linux-$VERSION.tar.bz2 | (cd $KDIR && tar xf -) ||
	    barf "Problem unpacking linux-$VERSION.tar.bz2"
	mv $KDIR/linux $KDIR/linux-$VERSION
	;;

    #   [1-9].*.*: This refers to the any other Linus kernel.
    [1-9].*.*[0-9])
	# Linus standard ?
	if [ -d $KDIR/linux-$VERSION ]; then exit 0; fi

	LAST_VER=`echo "$VERSION" | cut -d. -f3`
	PREVIOUS=`echo "$VERSION" | cut -d. -f1,2`.`expr $LAST_VER - 1 || true`
	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/patch-$VERSION.bz2 || barf "Couldn't get v$MAJOR/patch-$VERSION.bz2"
	trap "rm -rf $KDIR/.linux-$VERSION" 0
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;
    *)
	barf "Unknown kernel version $VERSION"
	;;

esac

#3 lore raw
In reply to Matt Mackall
Quoted 2 lines · Matt Mackall

This is an alpha release for people to experiment with. Feedback and patches encouraged. Grab your copy today at:

First of all, very nice script.

But, it doesn't look like it properly handles empty directories. I tried this command, this morning, and it blew up. I think it's because this directory http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/ is empty because of last night's 2.6.4-rc2 release. I don't grok python very well but is the "return p[-1]" there just to cause a fault like this? Would it be better if it just returned a "no version of that patch right now" message and exited nicely?

[dave@nighthawk linux-2.6]$ kpatchup-0.02 2.6-bk "Traceback (most recent call last):
File "/home/dave/bin/kpatchup-0.02", line 283, in ?

    b = find_ver(args[0])
  File "/home/dave/bin/kpatchup-0.02", line 240, in find_ver
    return v[0](os.path.dirname(v[1]), v[2])
  File "/home/dave/bin/kpatchup-0.02", line 147, in latest_dir
    return p[-1]

IndexError: list index out of range

I think your script, combined with Rusty's latest-kernel-version could make me a very happy person.

-- dave

#4 lore raw
Quoted 6 lines · Dave Hansen
Quoted 2 lines · Matt Mackall

This is an alpha release for people to experiment with. Feedback and patches encouraged. Grab your copy today at:

First of all, very nice script.

Quoted 7 lines

But, it doesn't look like it properly handles empty directories. I tried this command, this morning, and it blew up. I think it's because this directory http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/ is empty because of last night's 2.6.4-rc2 release. I don't grok python very well but is the "return p[-1]" there just to cause a fault like this? Would it be better if it just returned a "no version of that patch right now" message and exited nicely?

Python does a good job at falling over loudly whenever anything unexpected happens. I hadn't noticed that the snapshot directory got purged. Hmmm. The right thing is to make it fall back to checking old/ where the most recent -bk is to be found. Like this:

$ kpatchup -s 2.6-pre
2.6.4-rc2
$ kpatchup -s 2.6-bk
2.6.4-rc1-bk4

New version at http://selenic.com/kpatchup/kpatchup-0.03

I've added a couple other niceties for scripting purposes: a -p option which will report the "base" version for a given version, -m which will parse Makefile and print the version therein.

Quoted 2 lines

I think your script, combined with Rusty's latest-kernel-version could make me a very happy person.

I skimmed latest-kernel-version, is it doing something my -s option doesn't do yet?

Signature · 1 line
Matt Mackall : http://www.selenic.com : Linux development and consulting
#5 lore raw
Quoted 2 lines · Matt Mackall

I skimmed latest-kernel-version, is it doing something my -s option doesn't do yet?

$ ./kpatchup-0.03 -s 2.6-bk
2.6.4-rc1-bk4
$ ./kpatchup-0.03 -s 2.6
2.6.3
$ ./kpatchup-0.03 -s 2.6-pre
2.6.4-rc2
$ latest-kernel-version --probe
2.6.4-rc2

I might be misunderstanding the options, but 'kpatchup -s' I think is limited to giving the latest version of a single tree, while 'latest-kernel-version' will look at several different trees. This is a tiny problem for kpatchup because it treats 2.6, 2.6-bk, and 2.6-pre as separate trees. For my use, I just want the latest snapshot, whether it's a bk snapshot, or one of the real point releases.

What I'm doing to work around it is this:
kpatchup latest-kernel-version --probe

-- dave

#6 lore raw
Quoted 6 lines · Dave Hansen
Quoted 2 lines · Matt Mackall

I skimmed latest-kernel-version, is it doing something my -s option doesn't do yet?

$ ./kpatchup-0.03 -s 2.6-bk
2.6.4-rc1-bk4

This is actually broken at the moment as it won't know how to fetch stuff from old/ when it's given a non-current bk version. I have a design assumption that I can go from a version name to a URL without any lookups, which the snapshot dir purging breaks.

ISTR Jeff maintains the snapshot scripts. Jeff, could you possibly have it put a copy of the each snapshot in old/ at creation time rather than moving stuff there at a later point? This will make URLs pointing to old/ for -bk patches stable but shouldn't break anything else.

Quoted 7 lines
$ ./kpatchup-0.03 -s 2.6
2.6.3
$ ./kpatchup-0.03 -s 2.6-pre
2.6.4-rc2
$ latest-kernel-version --probe
2.6.4-rc2
Quoted 6 lines

I might be misunderstanding the options, but 'kpatchup -s' I think is limited to giving the latest version of a single tree, while 'latest-kernel-version' will look at several different trees. This is a tiny problem for kpatchup because it treats 2.6, 2.6-bk, and 2.6-pre as separate trees. For my use, I just want the latest snapshot, whether it's a bk snapshot, or one of the real point releases.

I suppose I could add a 2.6-tip, which will return the greatest of 2.6, 2.6-pre, and 2.6-bk. Like this:

$ kpatchup -s 2.6-tip
2.6.4-rc2
$ kpatchup -u 2.6-tip
http://www.kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.4-rc2.bz2
Signature · 1 line
Matt Mackall : http://www.selenic.com : Linux development and consulting