Management of opendocument (openoffice.org) files in git
Quoted 4 lines
profile_unzip=PROFILE_UNZIP_${profile}
profile_zip=PROFILE_ZIP_${profile}
UNZIP_OPTS=${!profile_unzip}
ZIP_OPTS=${!profile_zip}Can be written (in pure bourne shell) as
eval UNZIP_OPTS=\$PROFILE_UNZIP_${profile}
eval ZIP_OPTS=\$PROFILE_ZIP_${profile}Quoted 2 lines
--unzip_opts)
UNZIP_OPTS=${UNZIP_OPTS} $2Missing quotes:
UNZIP_OPTS="${UNZIP_OPTS} $2"It could also be a good idea to do
UNZIP_OPTS="${UNZIP_OPTS} `echo $2 | sed 'y/,/ /' `"(compare with the -Wa/-Wl/-Wp options of gcc) so you can do
[filter "opendocument"]
clean = "rezip --unzip-opts -b,-qq,-X --zip-opts -q,-r,-D,-0"
smudge = "rezip --unzip-opts -b,-qq,-X --zip-opts -q,-r,-D,-6"And maybe -b,-qq,-X and -q,-r respectively could be added by default?
Anyway, nice script, thanks!
Paolo
Thanks for the fixes (particularly the missing quotation!) and the suggestions.
With regards to
Quoted 3 lines
And maybe -b,-qq,-X and -q,-r respectively could be added by default?
I would prefer not to do so: if you do you get something that is somehow "specialised", otherwise you have a totally generic "rezipper" that might also find other applications (who knows). BTW, that is why I added the profiles, so that there was no need to type repetitive stuff.
Sergio
Quoted 9 lines
With regards to
Quoted 3 lines
And maybe -b,-qq,-X and -q,-r respectively could be added by default?
I would prefer not to do so: if you do you get something that is somehow "specialised", otherwise you have a totally generic "rezipper" that might also find other applications (who knows).
Yeah, but regarding -b/-X, their effect can/should be undone with zip command line options (-l, -ll, -X). And for zip's -r option, a rezipper that by default only rezips the top directory does not seem very useful. :-)
You're right about letting the user specify -qq/-q. Or maybe you can have a -q/--quiet option to rezip that adds both -qq to unzip, and -q to zip. This way you can use the openoffice profile both in quiet mode (for git) and in non-quiet mode (for manual use).
Putting all of this together, it would make the filter look like this:
[filter "opendocument"]
clean = "rezip --quiet --zip-opts -D,-0"
smudge = "rezip --quiet --zip-opts -D,-6"or similarly with profiles:
[filter "opendocument"]
clean = "rezip --quiet -p ODF_UNCOMPRESS"
smudge = "rezip --quiet -p ODF_COMPRESS"After my signature you can find my attempt at making rezip more useful as a general program. It supports --quiet, multiple input files, and - for stdin. (And I learnt from you that >&$foo works).
Quoted 2 lines
BTW, that is why I added the profiles, so that there was no need to type repetitive stuff.
Understood.
Paolo
-----8<-----------------------
#! /bin/sh
#
# (c) 2008 Sergio Callegari
#
# Rewrites a zip archive, possibly changing the compression level
USAGE='Usage: rezip OPTIONS FILE...
with options:
-h, --help Gives help --unzip-opts OPTIONS Pass options to unzip helper to read zip file --zip-opts OPTIONS Pass options to zip helper to write zip file -p, --profile PROFILE Get options for helpers from profile -q, --quiet Make unzip and zip quiet
Rewrites a zip archive, possibily changing the compression level. If the archive name is unspecified or "-", then the command operates like a filter, reading from standard input and writing to standard output. Options (either space- or comma-separated) can be manually provided to the unzip process doing the read and to the zip process doing the write. Alternatively a profile can be used to set options automatically.'
PROFILES="ODF_UNCOMPRESS ODF_COMPRESS"
PROFILE_UNZIP_ODF_UNCOMPRESS=
PROFILE_ZIP_ODF_UNCOMPRESS=-D,-0
PROFILE_UNZIP_ODF_COMPRESS=
PROFILE_ZIP_ODF_COMPRESS=-D,-6
die()
{
echo "$3$USAGEAvailable profiles: ${PROFILES}" >&$1
exit $2 }
UNZIP_OPTS=""
ZIP_OPTS=""
while true ; do
case "$1" in
-h | --help)
die 1 0 ;;
# TODO: handle -p*, --profile=* and similarly for other options
-p | --profile)
eval UNZIP_OPTS=\$PROFILE_UNZIP_$2
eval ZIP_OPTS=\$PROFILE_ZIP_$2
shift ;;
--unzip-opts)
UNZIP_OPTS="${UNZIP_OPTS} $2"
shift ;;
--zip-opts)
ZIP_OPTS="${ZIP_OPTS} $2"
shift ;;
-q | --quiet)
UNZIP_QUIET=-qq
ZIP_QUIET=-q ;;
-*)
die 2 1 "Invalid option: $1
" ;;
*)
break ;;
esac
shiftdone
UNZIP_OPTS="$UNZIP_QUIET -b -X echo $UNZIP_OPTS | sed 'y/,/ /'" ZIP_OPTS="$ZIP_QUIET -r echo $ZIP_OPTS | sed 'y/,/ /'"
if [ $# = 0 ] ; then
set fnord -
shift
fi
redir=1
for filename
do
if [ "$filename" = - ]; then
redir=2
break
fidone
for filename
do
workdir=mktemp -d -t rezip.workdir.XXXXXX
if [ "$filename" = - ]; then
tmpcopy=:
filename=mktemp rezip.zip.XXXXXX
cat > $filename
else
tmpcopy=false
fi
(case $filename in
/*) ;;
*) filename=pwd/$filename ;;
esac
cd "$workdir"
unzip $UNZIP_OPTS "$filename" >&$redir
zip $ZIP_OPTS "$filename" . >&$redir)
rm -fr "$workdir"
if $tmpcopy ; then
cat "$filename"
rm "$filename"
fi
done
--------8<------------------------
Following up on the discussion about tracking oo files I conducted a minimalistic test. I simulated tracking an oo spreadsheat, where from one version to the next only a few cells would be entered in an existing spreadsheet. These are the sizes of the individual files:
48K 0.ods 48K 1.ods 60K 2.ods 60K 3.ods 56K 4.ods 64K 5.ods 68K 6.ods 64K 7.ods 64K 8.ods 68K 9.ods 600K total
I then tracked this in three different ways, each in a fresh repo:
"packed": copy $i.ods to t.ods as is, git add t.ods and commit. "unpacked": use the unzipped contents of $i.ods instead. "rezip": use the rezipped version (compression 0, using Sergio's script). "oofilter": use clean/smudge filters (calling Sergio's rezip)
Here are the resulting sizes: first ".git/objects" as is, then after repacking -adf, finally the total size of .git + the work tree (i.e. the last revision).
packed
708K .git/objects 492K .git/objects 692K .git + wt
unpacked
1,3M .git/objects 144K .git/objects 1,5M .git + wt
rezip
992K .git/objects 148K .git/objects 1,4M .git + wt
oofilter
984K .git/objects 148K .git/objects 352K .git + wt
Unsurprisingly, the total size is dominated by the work tree size if you
have few revisions. (Also, templates and such contribute.)
Note that git log --stat will report the sizes of packed files in the first case, but the sizes of unpacked files in all other cases. In particular, it reports a different size for the HEAD revision than you have in a HEAD checkout.
I tried rewriting "packed" after configuring the filters: filter-branch refuses to work with a dirty work-tree, even after "checkout -f HEAD" and "reset --hard". It seems that git status is permanently confused here. (Has anyone successfully rewritten existing oo files?)
I'm not sure about the lessons, but I wanted to share the numbers anyways. I think this (your script and its usage) is heading in a useful direction and should maybe made more known, if not made easier from the git side. Also I'm still looking for a good (deterministic) pdf recompressor.
Michael
git version 1.6.0.2.426.g2cfa6
Michael J Gruber:
Quoted 4 lines
I'm not sure about the lessons, but I wanted to share the numbers anyways. I think this (your script and its usage) is heading in a useful direction and should maybe made more known, if not made easier from the git side.
I had very positive experiences with the script for my use-case. I did post them to the list, but it seems as if they got lost. At least I can't seem to find them, did they show up?
I had some problems with the script when trying to run it under Windows, though. Running Windows-Git from a Cygwin prompt provides some confusion about some of the Unix tools' behaviour that I needed to work around (like removing "/cygdrive" prefixes).
Signature · 1 line
\\// Peter - http://www.softwolves.pp.se/