Add receive-pack server side support
cover Add receive-pack server side support
This is a simple implementation of the receive-pack part of the send-pack/receive-pack protocol used by push. Most of the logic the server implements can be customized by implementing your own PreReceiveHook implementation.
I'm actually going to use this within Gerrit 2 so users can push into the special ref name "refs/changes/new" and have an anonymous topic branch be created instead.
A basic "jgit daemon" for the git:// protocol is also included. This daemon still has some major limitations over "git daemon":
- Only "receive-pack" is supported. We do not have a Java version of "upload-pack" so we do not support it here.
- Virtual hosting is not supported; the "host=" header (if sent by the client) is discarded.
- User escapes ("git://host/~user/foo.git") are not honored.
- No receive timeouts. Evil clients can connect and do nothing, pinning the server thread indefinitely. This can cause the server to consume resources until it crashes.
- No connection limits. Evil clients can DOS the server by just opening a ton of connections.
- "jgit daemon" must be started within a git repository, even if it won't publish that repository over the network. This is due to a limitation in the way TextBuiltin initializes itself.
Providing "jgit receive-pack" should be trivial if the last item is fixed, permitting jgit commands to be executed from outside of a Git repository.
JSch does not have an SSH server implementation, so we cannot easily provide an SSH daemon for our new push support. Apache MINA SSHD has a nice daemon under development, and I plan on using it in Gerrit 2. Perhaps someone will eventually contribute an extension library for JGit which provides the binding glue necessary for MINA SSHD to invoke JGit.
JGit's ReceivePack class does not invoke the standard hooks honored by "git receive-pack", because doing so would require executing an external process. There's enough data available in memory that someone could easily contribute a PreReceiveHook and PostRecieveHook implementation that honored these.
The early part of the series has some code cleanups to ensure we are thread-safe, and to reduce the thrasing on the InflaterCache.
Shawn O. Pearce (13):
Fix invalid "double checked locking" in InflaterCache Cleanup stupid release of the cached Inflater in IndexPack Cache an Inflater inside a WindowCursor and reuse it as much as
possible
Make RefDatabase thread-safe
Make PackFile thread-safe
Make Repository thread-safe
Don't open a PackFile multiple times on scanForPacks Expose RepositoryConfig.getBoolean so applications can use it Add AnyObjectId.copyTo(StringBuilder)
Add compare-and-swap semantics to RefUpdate
Allow null new ObjectId during RefUpdate.delete Implement the git-receive-pack process in Java
Add basic git daemon support to publish receive-pack
.../services/org.spearce.jgit.pgm.TextBuiltin | 1 + .../src/org/spearce/jgit/pgm/Daemon.java | 125 +++ .../src/org/spearce/jgit/lib/AnyObjectId.java | 17 +- .../jgit/lib/DeltaRefPackedObjectLoader.java | 2 +- .../src/org/spearce/jgit/lib/InflaterCache.java | 31 +- .../src/org/spearce/jgit/lib/PackFile.java | 14 +- .../src/org/spearce/jgit/lib/RefDatabase.java | 29 +- .../src/org/spearce/jgit/lib/RefUpdate.java | 32 +- .../src/org/spearce/jgit/lib/Repository.java | 90 ++- .../src/org/spearce/jgit/lib/RepositoryConfig.java | 4 +- .../src/org/spearce/jgit/lib/WindowCursor.java | 33 +- .../src/org/spearce/jgit/lib/WindowedFile.java | 17 +- .../src/org/spearce/jgit/revwalk/RevWalk.java | 4 +- .../src/org/spearce/jgit/transport/Daemon.java | 309 ++++++++ .../org/spearce/jgit/transport/DaemonClient.java | 106 +++ .../org/spearce/jgit/transport/DaemonService.java | 120 +++ .../src/org/spearce/jgit/transport/IndexPack.java | 17 +- .../org/spearce/jgit/transport/PacketLineIn.java | 12 + .../spearce/jgit/transport/PostReceiveHook.java | 77 ++ .../org/spearce/jgit/transport/PreReceiveHook.java | 94 +++ .../org/spearce/jgit/transport/ReceiveCommand.java | 223 ++++++ .../org/spearce/jgit/transport/ReceivePack.java | 793 ++++++++++++++++++++ .../spearce/jgit/transport/TransportGitAnon.java | 3 +- 23 files changed, 2060 insertions(+), 93 deletions(-) create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/DaemonClient.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/DaemonService.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PostReceiveHook.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PreReceiveHook.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ReceiveCommand.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ReceivePack.java
PATCH 1/13 Fix invalid "double checked locking" in InflaterCache
We move the Inflater allocation out of the synchronized block as it requires JNI code to build the libz structure. This permits other threads to get into the Inflater allocation faster.
We also fix the release implementation to validate the array size only after we enter the lock, rather than before. This prevents us from releasing an inflater due to a stale copy of the open count being in our processor cache.
--- .../src/org/spearce/jgit/lib/InflaterCache.java | 31 ++++++++++--------- 1 files changed, 16 insertions(+), 15 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/InflaterCache.java +16 -15
@@ -59,13 +59,18 @@ * * @return an available inflater. Never null. */ - public synchronized static Inflater get() { + public static Inflater get() { + final Inflater r = getImpl(); + return r != null ? r : new Inflater(false); + } + + private synchronized static Inflater getImpl() { if (openInflaterCount > 0) { final Inflater r = inflaterCache[--openInflaterCount]; inflaterCache[openInflaterCount] = null; return r; } - return new Inflater(false); + return null; } /**
@@ -76,23 +81,19 @@ public synchronized static Inflater get() { * does nothing. */ public static void release(final Inflater i) { - if (i == null) - return; - - if (openInflaterCount == SZ) { - i.end(); - return; + if (i != null) { + i.reset(); + if (releaseImpl(i)) + i.end(); } - - i.reset(); - releaseImpl(i); } - private static synchronized void releaseImpl(final Inflater i) { - if (openInflaterCount == SZ) - i.end(); - else + private static synchronized boolean releaseImpl(final Inflater i) { + if (openInflaterCount < SZ) { inflaterCache[openInflaterCount++] = i; + return false; + } + return true; } private InflaterCache() {
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 2/13 Cleanup stupid release of the cached Inflater in IndexPack
The try/finally idiom avoids placing the Inflater on the stack, which means it goes unreachable slightly faster. Its also much less obtuse to read.
--- .../src/org/spearce/jgit/transport/IndexPack.java | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java +5 -3
@@ -320,9 +320,11 @@ public void index(final ProgressMonitor progress) throws IOException { writeIdx(); } finally { - final Inflater inf = inflater; - inflater = null; - InflaterCache.release(inf); + try { + InflaterCache.release(inflater); + } finally { + inflater = null; + } progress.endTask(); if (packOut != null)
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 3/13 Cache an Inflater inside a WindowCursor and reuse it as much as possible
By caching the Inflater within the WindowCursor we can improve performance associated with reading objects from the pack files. Each read can use the cached Inflater, especially when chasing down a delta chain. This avoids locking on the global cache.
--- .../jgit/lib/DeltaRefPackedObjectLoader.java | 2 +- .../src/org/spearce/jgit/lib/PackFile.java | 7 +++- .../src/org/spearce/jgit/lib/Repository.java | 7 +++- .../src/org/spearce/jgit/lib/WindowCursor.java | 33 ++++++++++++++------ .../src/org/spearce/jgit/lib/WindowedFile.java | 10 +----- .../src/org/spearce/jgit/revwalk/RevWalk.java | 4 +- .../src/org/spearce/jgit/transport/IndexPack.java | 9 ++++- 7 files changed, 47 insertions(+), 25 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/DeltaRefPackedObjectLoader.java +1 -1
@@ -54,7 +54,7 @@ DeltaRefPackedObjectLoader(final WindowCursor curs, final PackFile pr, } protected PackedObjectLoader getBaseLoader() throws IOException { - final PackedObjectLoader or = pack.get(deltaBase); + final PackedObjectLoader or = pack.get(curs, deltaBase); if (or == null) throw new MissingObjectException(deltaBase, "delta base"); return or;
org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java +6 -1
@@ -120,7 +120,12 @@ public boolean hasObject(final AnyObjectId id) { * the pack file or the index could not be read. */ public PackedObjectLoader get(final AnyObjectId id) throws IOException { - return get(new WindowCursor(), id); + final WindowCursor wc = new WindowCursor(); + try { + return get(wc, id); + } finally { + wc.release(); + } } /**
org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java +6 -1
@@ -251,7 +251,12 @@ public boolean hasObject(final AnyObjectId objectId) { */ public ObjectLoader openObject(final AnyObjectId id) throws IOException { - return openObject(new WindowCursor(),id); + final WindowCursor wc = new WindowCursor(); + try { + return openObject(wc, id); + } finally { + wc.release(); + } } /**
org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java +23 -10
@@ -46,6 +46,8 @@ /** Temporary buffer large enough for at least one raw object id. */ final byte[] tempId = new byte[Constants.OBJECT_ID_LENGTH]; + private Inflater inf; + ByteWindow window; Object handle;
@@ -98,16 +100,8 @@ int copy(final WindowedFile provider, long position, final byte[] dstbuf, * data to. * @param dstoff * current offset within <code>dstbuf</code> to inflate into. - * @param inf - * the inflater to feed input to. The caller is responsible for - * initializing the inflater as multiple windows may need to - * supply data to the same inflater to completely decompress - * something. * @return updated <code>dstoff</code> based on the number of bytes - * successfully copied into <code>dstbuf</code> by - * <code>inf</code>. If the inflater is not yet finished then - * another window's data must still be supplied as input to finish - * decompression. + * successfully inflated into <code>dstbuf</code>. * @throws IOException * this cursor does not match the provider or id and the proper * window could not be acquired through the provider's cache.
@@ -116,8 +110,12 @@ int copy(final WindowedFile provider, long position, final byte[] dstbuf, * stream corruption is likely. */ int inflate(final WindowedFile provider, long position, - final byte[] dstbuf, int dstoff, final Inflater inf) + final byte[] dstbuf, int dstoff) throws IOException, DataFormatException { + if (inf == null) + inf = InflaterCache.get(); + else + inf.reset(); for (;;) { pin(provider, position); dstoff = window.inflate(handle, position, dstbuf, dstoff, inf);
@@ -138,5 +136,20 @@ private void pin(final WindowedFile provider, final long position) public void release() { window = null; handle = null; + try { + InflaterCache.release(inf); + } finally { + inf = null; + } + } + + /** + * @param curs cursor to release; may be null. + * @return always null. + */ + public static WindowCursor release(final WindowCursor curs) { + if (curs != null) + curs.release(); + return null; } }
org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java +2 -8
@@ -45,7 +45,6 @@ import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel.MapMode; import java.util.zip.DataFormatException; -import java.util.zip.Inflater; /** * Read-only cached file access.
@@ -240,13 +239,8 @@ public void copyToStream(long position, final byte[] buf, long cnt, void readCompressed(final long position, final byte[] dstbuf, final WindowCursor curs) throws IOException, DataFormatException { - final Inflater inf = InflaterCache.get(); - try { - if (curs.inflate(this, position, dstbuf, 0, inf) != dstbuf.length) - throw new EOFException("Short compressed stream at " + position); - } finally { - InflaterCache.release(inf); - } + if (curs.inflate(this, position, dstbuf, 0) != dstbuf.length) + throw new EOFException("Short compressed stream at " + position); } /**
org.spearce.jgit/src/org/spearce/jgit/revwalk/RevWalk.java +2 -2
@@ -651,7 +651,7 @@ else if (!(c instanceof RevTree)) if ((t.flags & PARSED) != 0) return t; - final ObjectLoader ldr = db.openObject(t); + final ObjectLoader ldr = db.openObject(curs, t); if (ldr == null) throw new MissingObjectException(t, Constants.TYPE_TREE); if (ldr.getType() != Constants.OBJ_TREE)
@@ -680,7 +680,7 @@ public RevObject parseAny(final AnyObjectId id) throws MissingObjectException, IOException { RevObject r = objects.get(id); if (r == null) { - final ObjectLoader ldr = db.openObject(id); + final ObjectLoader ldr = db.openObject(curs, id); if (ldr == null) throw new MissingObjectException(id.toObjectId(), "unknown"); final byte[] data = ldr.getCachedBytes();
org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java +7 -2
@@ -68,6 +68,7 @@ import org.spearce.jgit.lib.PackIndexWriter; import org.spearce.jgit.lib.ProgressMonitor; import org.spearce.jgit.lib.Repository; +import org.spearce.jgit.lib.WindowCursor; import org.spearce.jgit.util.NB; /** Indexes Git pack files for local use. */
@@ -173,6 +174,8 @@ public static IndexPack create(final Repository db, final InputStream is) /** If {@link #fixThin} this is the last byte of the original checksum. */ private long originalEOF; + private WindowCursor readCurs; + /** * Create a new pack indexer utility. *
@@ -189,6 +192,7 @@ public IndexPack(final Repository db, final InputStream src, repo = db; in = src; inflater = InflaterCache.get(); + readCurs = new WindowCursor(); buf = new byte[BUFFER_SIZE]; objectData = new byte[BUFFER_SIZE]; objectDigest = Constants.newMessageDigest();
@@ -325,6 +329,7 @@ public void index(final ProgressMonitor progress) throws IOException { } finally { inflater = null; } + readCurs = WindowCursor.release(readCurs); progress.endTask(); if (packOut != null)
@@ -461,7 +466,7 @@ private void fixThinPack(final ProgressMonitor progress) throws IOException { final Deflater def = new Deflater(Deflater.DEFAULT_COMPRESSION, false); long end = originalEOF; for (final ObjectId baseId : new ArrayList<ObjectId>(baseById.keySet())) { - final ObjectLoader ldr = repo.openObject(baseId); + final ObjectLoader ldr = repo.openObject(readCurs, baseId); if (ldr == null) continue; final byte[] data = ldr.getBytes();
@@ -715,7 +720,7 @@ private void verifySafeObject(final AnyObjectId id, final int type, } } - final ObjectLoader ldr = repo.openObject(id); + final ObjectLoader ldr = repo.openObject(readCurs, id); if (ldr != null) { final byte[] existingData = ldr.getCachedBytes(); if (ldr.getType() != type || !Arrays.equals(data, existingData)) {
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 4/13 Make RefDatabase thread-safe
This is necessary to support a thread-safe Repository.
--- .../src/org/spearce/jgit/lib/RefDatabase.java | 29 +++++++++++--------- 1 files changed, 16 insertions(+), 13 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java +16 -13
@@ -93,7 +93,7 @@ RefDatabase(final Repository r) { clearCache(); } - void clearCache() { + synchronized void clearCache() { looseRefs = new HashMap<String, Ref>(); looseRefsMTime = new HashMap<String, Long>(); packedRefs = new HashMap<String, Ref>();
@@ -139,9 +139,11 @@ RefUpdate newUpdate(final String name) throws IOException { } void stored(final String origName, final String name, final ObjectId id, final long time) { - looseRefs.put(name, new Ref(Ref.Storage.LOOSE, origName, name, id)); - looseRefsMTime.put(name, time); - setModified(); + synchronized (this) { + looseRefs.put(name, new Ref(Ref.Storage.LOOSE, origName, name, id)); + looseRefsMTime.put(name, time); + setModified(); + } db.fireRefsMaybeChanged(); }
@@ -157,11 +159,13 @@ void stored(final String origName, final String name, final ObjectId id, final l void link(final String name, final String target) throws IOException { final byte[] content = Constants.encode("ref: " + target + "\n"); lockAndWriteFile(fileForRef(name), content); - setModified(); + synchronized (this) { + setModified(); + } db.fireRefsMaybeChanged(); } - void setModified() { + private void setModified() { lastRefModification = refModificationCounter++; }
@@ -210,7 +214,7 @@ Ref readRef(final String partialName) throws IOException { return avail; } - private void readPackedRefs(final Map<String, Ref> avail) { + private synchronized void readPackedRefs(final Map<String, Ref> avail) { refreshPackedRefs(); avail.putAll(packedRefs); }
@@ -229,7 +233,7 @@ private void readLooseRefs(final Map<String, Ref> avail, } } - private void readOneLooseRef(final Map<String, Ref> avail, + private synchronized void readOneLooseRef(final Map<String, Ref> avail, final String origName, final String refName, final File ent) { // Unchanged and cached? Don't read it again. //
@@ -323,8 +327,8 @@ private Ref readRefBasic(final String name, final int depth) throws IOException return readRefBasic(name, name, depth); } - private Ref readRefBasic(final String origName, final String name, final int depth) - throws IOException { + private synchronized Ref readRefBasic(final String origName, + final String name, final int depth) throws IOException { // Prefer loose ref to packed ref as the loose // file can be more up-to-date than a packed one. //
@@ -408,7 +412,7 @@ private Ref readRefBasic(final String origName, final String name, final int dep return ref; } - private void refreshPackedRefs() { + private synchronized void refreshPackedRefs() { final long currTime = packedRefsFile.lastModified(); final long currLen = currTime == 0 ? 0 : packedRefsFile.length(); if (currTime == packedRefsLastModified && currLen == packedRefsLength)
@@ -479,7 +483,7 @@ private void lockAndWriteFile(File file, byte[] content) throws IOException { throw new ObjectWritingException("Unable to write " + name); } - void removePackedRef(String name) throws IOException { + synchronized void removePackedRef(String name) throws IOException { packedRefs.remove(name); writePackedRefs(); }
@@ -508,5 +512,4 @@ private static BufferedReader openReader(final File fileLocation) return new BufferedReader(new InputStreamReader(new FileInputStream( fileLocation), Constants.CHARSET)); } - }
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 5/13 Make PackFile thread-safe
We really should try to avoid allocating the reverse index within the pack object itself, and instead make the applications that do use this data manage it themselves. This way the index could be released in memory when its no longer used, and the locking could be bypassed entirely.
Since most of PackFile is dependent upon the WindowedFile we can mostly ignore synchronization in this class.
--- .../src/org/spearce/jgit/lib/PackFile.java | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java +1 -1
@@ -341,7 +341,7 @@ private long findEndOffset(final long startOffset) return getReverseIdx().findNextOffset(startOffset, maxOffset); } - private PackReverseIndex getReverseIdx() { + private synchronized PackReverseIndex getReverseIdx() { if (reverseIdx == null) reverseIdx = new PackReverseIndex(idx); return reverseIdx;
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 6/13 Make Repository thread-safe
The Repository class itself is now thread-safe. Thread safety is ensured by copying values onto the caller stack prior to use.
--- .../src/org/spearce/jgit/lib/Repository.java | 75 ++++++++++++------- 1 files changed, 47 insertions(+), 28 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java +47 -28
@@ -46,6 +46,7 @@ import java.io.FilenameFilter; import java.io.IOException; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collection; import java.util.Collections; import java.util.HashMap;
@@ -79,20 +80,22 @@ * </ul> * </li> * </ul> - * + * <p> + * This class is thread-safe. + * <p> * This implementation only handles a subtly undocumented subset of git features. * */ public class Repository { private final File gitDir; - private final File[] objectsDirs; - private final RepositoryConfig config; private final RefDatabase refs; - private PackFile[] packs; + private File[] objectDirectoryList; + + private PackFile[] packFileList; private GitIndex index;
@@ -111,18 +114,19 @@ public Repository(final File d) throws IOException { gitDir = d.getAbsoluteFile(); try { - objectsDirs = readObjectsDirs(FS.resolve(gitDir, "objects"), - new ArrayList<File>()).toArray(new File[0]); + objectDirectoryList = readObjectsDirs( + FS.resolve(gitDir, "objects"), new ArrayList<File>()) + .toArray(new File[0]); } catch (IOException e) { IOException ex = new IOException("Cannot find all object dirs for " + gitDir); ex.initCause(e); throw ex; } refs = new RefDatabase(this); - packs = new PackFile[0]; + packFileList = new PackFile[0]; config = new RepositoryConfig(this); - final boolean isExisting = objectsDirs[0].exists(); + final boolean isExisting = objectDirectoryList[0].exists(); if (isExisting) { getConfig().load(); final String repositoryFormatVersion = getConfig().getString(
@@ -138,7 +142,8 @@ public Repository(final File d) throws IOException { scanForPacks(); } - private Collection<File> readObjectsDirs(File objectsDir, Collection<File> ret) throws IOException { + private static Collection<File> readObjectsDirs(File objectsDir, + Collection<File> ret) throws IOException { ret.add(objectsDir); final File altFile = FS.resolve(objectsDir, "info/alternates"); if (altFile.exists()) {
@@ -160,7 +165,7 @@ public Repository(final File d) throws IOException { * * @throws IOException */ - public void create() throws IOException { + public synchronized void create() throws IOException { if (gitDir.exists()) { throw new IllegalStateException("Repository already exists: " + gitDir);
@@ -169,9 +174,9 @@ public void create() throws IOException { gitDir.mkdirs(); refs.create(); - objectsDirs[0].mkdirs(); - new File(objectsDirs[0], "pack").mkdir(); - new File(objectsDirs[0], "info").mkdir(); + objectDirectoryList[0].mkdirs(); + new File(objectDirectoryList[0], "pack").mkdir(); + new File(objectDirectoryList[0], "info").mkdir(); new File(gitDir, "branches").mkdir(); new File(gitDir, "remotes").mkdir();
@@ -182,6 +187,14 @@ public void create() throws IOException { getConfig().save(); } + private synchronized File[] objectsDirs(){ + return objectDirectoryList; + } + + private synchronized PackFile[] packs(){ + return packFileList; + } + /** * @return GIT_DIR */
@@ -193,7 +206,7 @@ public File getDirectory() { * @return the directory containing the objects owned by this repository. */ public File getObjectsDirectory() { - return objectsDirs[0]; + return objectsDirs()[0]; } /**
@@ -217,6 +230,7 @@ public File toFile(final AnyObjectId objectId) { final String n = objectId.name(); String d=n.substring(0, 2); String f=n.substring(2); + final File[] objectsDirs = objectsDirs(); for (int i=0; i<objectsDirs.length; ++i) { File ret = new File(new File(objectsDirs[i], d), f); if (ret.exists())
@@ -231,6 +245,7 @@ public File toFile(final AnyObjectId objectId) { * known shared repositories. */ public boolean hasObject(final AnyObjectId objectId) { + final PackFile[] packs = packs(); int k = packs.length; if (k > 0) { do {
@@ -271,6 +286,7 @@ public ObjectLoader openObject(final AnyObjectId id) */ public ObjectLoader openObject(final WindowCursor curs, final AnyObjectId id) throws IOException { + final PackFile[] packs = packs(); int k = packs.length; if (k > 0) { do {
@@ -341,7 +357,7 @@ public ObjectLoader openObject(final WindowCursor curs, final AnyObjectId id) void openObjectInAllPacks(final AnyObjectId objectId, final Collection<PackedObjectLoader> resultLoaders, final WindowCursor curs) throws IOException { - for (PackFile pack : packs) { + for (PackFile pack : packs()) { final PackedObjectLoader loader = pack.get(curs, objectId); if (loader != null) resultLoaders.add(loader);
@@ -766,11 +782,10 @@ public void close() { closePacks(); } - void closePacks() { - for (int k = packs.length - 1; k >= 0; k--) { - packs[k].close(); - } - packs = new PackFile[0]; + synchronized void closePacks() { + for (int k = packFileList.length - 1; k >= 0; k--) + packFileList[k].close(); + packFileList = new PackFile[0]; } /**
@@ -795,11 +810,13 @@ public void openPack(final File pack, final File idx) throws IOException { throw new IllegalArgumentException("Pack " + pack + "does not match index " + idx); - final PackFile[] cur = packs; - final PackFile[] arr = new PackFile[cur.length + 1]; - System.arraycopy(cur, 0, arr, 1, cur.length); - arr[0] = new PackFile(this, idx, pack); - packs = arr; + synchronized (this) { + final PackFile[] cur = packFileList; + final PackFile[] arr = new PackFile[cur.length + 1]; + System.arraycopy(cur, 0, arr, 1, cur.length); + arr[0] = new PackFile(this, idx, pack); + packFileList = arr; + } } /**
@@ -808,11 +825,13 @@ public void openPack(final File pack, final File idx) throws IOException { */ public void scanForPacks() { final ArrayList<PackFile> p = new ArrayList<PackFile>(); - for (int i=0; i<objectsDirs.length; ++i) - scanForPacks(new File(objectsDirs[i], "pack"), p); + for (final File d : objectsDirs()) + scanForPacks(new File(d, "pack"), p); final PackFile[] arr = new PackFile[p.size()]; p.toArray(arr); - packs = arr; + synchronized (this) { + packFileList = arr; + } } private void scanForPacks(final File packDir, Collection<PackFile> packList) {
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 7/13 Don't open a PackFile multiple times on scanForPacks
If we already have a given PackFile open in this repository then we should reuse the existing PackFile handle.
--- .../src/org/spearce/jgit/lib/PackFile.java | 5 +++++ .../src/org/spearce/jgit/lib/Repository.java | 8 +++++++- .../src/org/spearce/jgit/lib/WindowedFile.java | 7 ++++++- 3 files changed, 18 insertions(+), 2 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java +5 -0
@@ -94,6 +94,11 @@ final PackedObjectLoader resolveBase(final WindowCursor curs, final long ofs) return reader(curs, ofs); } + /** @return the File object which locates this pack on disk. */ + public File getPackFile() { + return pack.getFile(); + } + /** * Determine if an object is contained within the pack file. * <p>
org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java +7 -1
@@ -825,6 +825,7 @@ synchronized (this) { */ public void scanForPacks() { final ArrayList<PackFile> p = new ArrayList<PackFile>(); + p.addAll(Arrays.asList(packs())); for (final File d : objectsDirs()) scanForPacks(new File(d, "pack"), p); final PackFile[] arr = new PackFile[p.size()];
@@ -843,7 +844,7 @@ public boolean accept(final File baseDir, final String n) { } }); if (idxList != null) { - for (final String indexName : idxList) { + SCAN: for (final String indexName : idxList) { final String n = indexName.substring(0, indexName.length() - 4); final File idxFile = new File(packDir, n + ".idx"); final File packFile = new File(packDir, n + ".pack");
@@ -856,6 +857,11 @@ public boolean accept(final File baseDir, final String n) { continue; } + for (final PackFile p : packList) { + if (packFile.equals(p.getPackFile())) + continue SCAN; + } + try { packList.add(new PackFile(this, idxFile, packFile)); } catch (IOException ioe) {
org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java +6 -1
@@ -101,13 +101,18 @@ public long length() { return length; } + /** @return the absolute file object this file reads from. */ + public File getFile() { + return fPath.getAbsoluteFile(); + } + /** * Get the path name of this file. * * @return the absolute path name of the file. */ public String getName() { - return fPath.getAbsolutePath(); + return getFile().getPath(); } /**
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 8/13 Expose RepositoryConfig.getBoolean so applications can use it
There's no reason for these to be hidden from application level code. We expose getInt because it is a popular type; getBoolean should also be exposed because it is also a popular type.
--- .../src/org/spearce/jgit/lib/RepositoryConfig.java | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java +2 -2
@@ -210,7 +210,7 @@ public int getInt(final String section, String subsection, * @return true if any value or defaultValue is true, false for missing or * explicit false */ - protected boolean getBoolean(final String section, final String name, + public boolean getBoolean(final String section, final String name, final boolean defaultValue) { return getBoolean(section, null, name, defaultValue); }
@@ -229,7 +229,7 @@ protected boolean getBoolean(final String section, final String name, * @return true if any value or defaultValue is true, false for missing or * explicit false */ - protected boolean getBoolean(final String section, String subsection, + public boolean getBoolean(final String section, String subsection, final String name, final boolean defaultValue) { String n = getRawString(section, subsection, name); if (n == null)
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 9/13 Add AnyObjectId.copyTo(StringBuilder)
This makes it easier to append an ObjectId onto a StringBuilder.
--- .../src/org/spearce/jgit/lib/AnyObjectId.java | 17 ++++++++++++++++- 1 files changed, 16 insertions(+), 1 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/AnyObjectId.java +16 -1
@@ -387,7 +387,22 @@ public void copyTo(final Writer w) throws IOException { */ public void copyTo(final char[] tmp, final Writer w) throws IOException { toHexCharArray(tmp); - w.write(tmp); + w.write(tmp, 0, STR_LEN); + } + + /** + * Copy this ObjectId to a StringBuilder in hex format. + * + * @param tmp + * temporary char array to buffer construct into before writing. + * Must be at least large enough to hold 2 digits for each byte + * of object id (40 characters or larger). + * @param w + * the string to append onto. + */ + public void copyTo(final char[] tmp, final StringBuilder w) { + toHexCharArray(tmp); + w.append(tmp, 0, STR_LEN); } private char[] toHexCharArray() {
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 10/13 Add compare-and-swap semantics to RefUpdate
This permits the caller to set the value it expects to find in the Ref upon obtaining the update lock. If the ref value doesn't match then the update is aborted with Result.LOCK_FAILURE, so the caller can recover gracefully without making unexpected changes.
--- .../src/org/spearce/jgit/lib/RefUpdate.java | 30 ++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java +30 -0
@@ -149,6 +149,9 @@ /** Old value of the ref, obtained after we lock it. */ private ObjectId oldValue; + /** If non-null, the value {@link #oldValue} must have to continue. */ + private ObjectId expValue; + /** Result of the update operation. */ private Result result = Result.NOT_ATTEMPTED;
@@ -190,6 +193,27 @@ public void setNewObjectId(final AnyObjectId id) { } /** + * @return the expected value of the ref after the lock is taken, but before + * update occurs. Null to avoid the compare and swap test. Use + * {@link ObjectId#zeroId()} to indicate expectation of a + * non-existant ref. + */ + public ObjectId getExpectedOldObjectId() { + return expValue; + } + + /** + * @param id + * the expected value of the ref after the lock is taken, but + * before update occurs. Null to avoid the compare and swap test. + * Use {@link ObjectId#zeroId()} to indicate expectation of a + * non-existant ref. + */ + public void setExpectedOldObjectId(final AnyObjectId id) { + expValue = id != null ? id.toObjectId() : null; + } + + /** * Check if this update wants to forcefully change the ref. * * @return true if this update should ignore merge tests.
@@ -370,6 +394,12 @@ private Result updateImpl(final RevWalk walk, final Store store) return Result.LOCK_FAILURE; try { oldValue = db.idOf(getName()); + if (expValue != null) { + final ObjectId o; + o = oldValue != null ? oldValue : ObjectId.zeroId(); + if (!expValue.equals(o)) + return Result.LOCK_FAILURE; + } if (oldValue == null) return store.store(lock, Result.NEW);
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 11/13 Allow null new ObjectId during RefUpdate.delete
If we are deleting a ref we really don't care about what the new ObjectId value should be; it just doesn't matter. If we didn't set the value we should consider it the same as if we had a MissingObjectException, which means treat the value as null and perform a force update test.
--- .../src/org/spearce/jgit/lib/RefUpdate.java | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java +1 -1
@@ -424,7 +424,7 @@ private Result updateImpl(final RevWalk walk, final Store store) private static RevObject safeParse(final RevWalk rw, final AnyObjectId id) throws IOException { try { - return rw.parseAny(id); + return id != null ? rw.parseAny(id) : null; } catch (MissingObjectException e) { // We can expect some objects to be missing, like if we are // trying to force a deletion of a branch and the object it
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 12/13 Implement the git-receive-pack process in Java
This implementation provides a basic git-receive-pack service within Java. Two hooks APIs are supported before and after commands are executed within the connection, allowing daemons to customize the behavior of the updates.
Logic to bind the ReceivePack class to a pipe or network socket is omitted, as it depends on the transport. SSH servers will need a pure Java SSH implementation such as Apache MINA SSHD. Anonymous push over git:// requires a basic git-daemon functionality. Local pipe access might use pure-Java pipes, or System.in/System.out.
--- .../org/spearce/jgit/transport/PacketLineIn.java | 12 + .../spearce/jgit/transport/PostReceiveHook.java | 77 ++ .../org/spearce/jgit/transport/PreReceiveHook.java | 94 +++ .../org/spearce/jgit/transport/ReceiveCommand.java | 223 ++++++ .../org/spearce/jgit/transport/ReceivePack.java | 793 ++++++++++++++++++++ 5 files changed, 1199 insertions(+), 0 deletions(-) create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PostReceiveHook.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PreReceiveHook.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ReceiveCommand.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ReceivePack.java
org.spearce.jgit/src/org/spearce/jgit/transport/PacketLineIn.java +12 -0
@@ -111,6 +111,18 @@ String readString() throws IOException { return RawParseUtils.decode(Constants.CHARSET, raw, 0, len); } + String readStringNoLF() throws IOException { + int len = readLength(); + if (len == 0) + return ""; + + len -= 4; // length header (4 bytes) + + final byte[] raw = new byte[len]; + NB.readFully(in, raw, 0, len); + return RawParseUtils.decode(Constants.CHARSET, raw, 0, len); + } + private void readLF() throws IOException { if (in.read() != '\n') throw new IOException("Protocol error: expected LF");
org.spearce.jgit/src/org/spearce/jgit/transport/PostReceiveHook.java new file +77 -0
77 lines not on this page · raw message
org.spearce.jgit/src/org/spearce/jgit/transport/PreReceiveHook.java new file +94 -0
94 lines not on this page · raw message
org.spearce.jgit/src/org/spearce/jgit/transport/ReceiveCommand.java new file +223 -0
223 lines not on this page · raw message
org.spearce.jgit/src/org/spearce/jgit/transport/ReceivePack.java new file +793 -0
793 lines not on this page · raw message
Signature · 1 line
1.6.1.rc4.301.g5497a
PATCH 13/13 Add basic git daemon support to publish receive-pack
The git:// daemon service receives anonymous TCP connections and runs commands as they are received.
Currently we only support the server portion of send-pack/receive-pack, so that is the only service registered in our Daemon class.
--- .../services/org.spearce.jgit.pgm.TextBuiltin | 1 + .../src/org/spearce/jgit/pgm/Daemon.java | 125 ++++++++ .../src/org/spearce/jgit/transport/Daemon.java | 309 ++++++++++++++++++++ .../org/spearce/jgit/transport/DaemonClient.java | 106 +++++++ .../org/spearce/jgit/transport/DaemonService.java | 120 ++++++++ .../spearce/jgit/transport/TransportGitAnon.java | 3 +- 6 files changed, 662 insertions(+), 2 deletions(-) create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/DaemonClient.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/DaemonService.java
org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin +1 -0
@@ -1,4 +1,5 @@ org.spearce.jgit.pgm.Branch +org.spearce.jgit.pgm.Daemon org.spearce.jgit.pgm.DiffTree org.spearce.jgit.pgm.Fetch org.spearce.jgit.pgm.Glog
org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java new file +125 -0
125 lines not on this page · raw message
org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java new file +309 -0
309 lines not on this page · raw message
org.spearce.jgit/src/org/spearce/jgit/transport/DaemonClient.java new file +106 -0
106 lines not on this page · raw message
org.spearce.jgit/src/org/spearce/jgit/transport/DaemonService.java new file +120 -0
120 lines not on this page · raw message
org.spearce.jgit/src/org/spearce/jgit/transport/TransportGitAnon.java +1 -2
@@ -56,8 +56,7 @@ * source projects, as there are no authentication or authorization overheads. */ class TransportGitAnon extends PackTransport { - /** IANA assigned port number for Git. */ - static final int GIT_PORT = 9418; + static final int GIT_PORT = Daemon.DEFAULT_PORT; static boolean canHandle(final URIish uri) { return "git".equals(uri.getScheme());
Signature · 1 line
1.6.1.rc4.301.g5497a
Re: [JGIT PATCH 13/13] Add basic git daemon support to publish receive-pack
tisdag 23 december 2008 01:27:23 skrev Shawn O. Pearce:
org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java line 65
Quoted 2 lines
org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java new file +2
-0+ private static final Pattern SAFE_REPOSITORY_NAME = Pattern + .compile("^[A-Za-z][A-Za-z0-9/_ -]+(\\.git)?$");
This restriction is too strict. Wouldn't any path not containing ".." be valid? In particular this did not work with my "EGIT.contrib" repo. I have a lot of repos with names llike "name.purpose". Just adding
'.' to the character set isn't really enough.
-- robin
PATCH 1/1 Permit a wider range of repository names in jgit daemon requests
The earlier restriction was too narrow for some applications, for example repositories named "jgit.dev" and "jgit.test" are perfectly valid Git repositories and should still be able to be served by the daemon.
By blocking out only uses of ".." as a path component and Windows UNC paths (by blocking "\") we can reasonably prevent the client from escaping the base dirctories configured in the daemon.
---
Quoted 7 lines · Robin Rosenberg
tisdag 23 december 2008 01:27:23 skrev Shawn O. Pearce:
Review
Quoted 2 lines
(quoted diff) +2
-0+ private static final Pattern SAFE_REPOSITORY_NAME = Pattern + .compile("^[A-Za-z][A-Za-z0-9/_ -]+(\\.git)?$");This restriction is too strict. Wouldn't any path not containing ".." be valid? In particular this did not work with my "EGIT.contrib" repo. I have a lot of repos with names llike "name.purpose". Just adding
'.' to the character set isn't really enough.
Yup.
.../src/org/spearce/jgit/transport/Daemon.java | 42 +++++++++---------- 1 files changed, 20 insertions(+), 22 deletions(-)
org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java +20 -22
@@ -51,7 +51,6 @@ import java.util.Collection; import java.util.HashMap; import java.util.Map; -import java.util.regex.Pattern; import org.spearce.jgit.lib.Repository;
@@ -62,9 +61,6 @@ private static final int BACKLOG = 5; - private static final Pattern SAFE_REPOSITORY_NAME = Pattern - .compile("^[A-Za-z][A-Za-z0-9/_ -]+(\\.git)?$"); - private InetSocketAddress myAddress; private final DaemonService[] services;
@@ -292,24 +288,26 @@ synchronized (exports) { return db; } - if (SAFE_REPOSITORY_NAME.matcher(name).matches()) { - final File[] search; - synchronized (exportBase) { - search = exportBase.toArray(new File[exportBase.size()]); - } - for (final File f : search) { - db = openRepository(new File(f, name)); - if (db != null) - return db; - - db = openRepository(new File(f, name + ".git")); - if (db != null) - return db; - - db = openRepository(new File(f, name + "/.git")); - if (db != null) - return db; - } + if (name.startsWith("../") || name.contains("/../") + || name.contains("\\")) + return null; + + final File[] search; + synchronized (exportBase) { + search = exportBase.toArray(new File[exportBase.size()]); + } + for (final File f : search) { + db = openRepository(new File(f, name)); + if (db != null) + return db; + + db = openRepository(new File(f, name + ".git")); + if (db != null) + return db; + + db = openRepository(new File(f, name + "/.git")); + if (db != null) + return db; } return null; }
Signature · 4 lines
1.6.1.94.g9388 -- Shawn.
Re: [PATCH] Permit a wider range of repository names in jgit daemon requests
måndag 05 januari 2009 03:46:22 skrev Shawn O. Pearce:
org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java synchronized (exports) { · line 292
Quoted 11 lines
The earlier restriction was too narrow for some applications, for example repositories named "jgit.dev" and "jgit.test" are perfectly valid Git repositories and should still be able to be served by the daemon.
By blocking out only uses of ".." as a path component and Windows UNC paths (by blocking "\") we can reasonably prevent the client from escaping the base dirctories configured in the daemon.
org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java +2
-0+ if (name.startsWith("../") || name.contains("/../") + || name.contains("\\"))
//host/share also works as UNC path (even the DOS commands support it, provided you quote the paths) and if you block // shuldn't '/', and '[A-Z]:' also be blocked? \\ is a UNC-prefix only at the beginning of a path so if / need not be filtered, nor does //. Inside a path \\ is the same as \ AFAIK (except directly after the drive letter. This should probablybe factored out into a utilty so we can have a simple unit test for it.
-- robin