Enhance credential helper protocol to include auth headers
Subsystems (2)
Re: [PATCH v4 8/8] t5556: add HTTP authentication tests
In reply to Victoria DyeQuoted 14 lines · Victoria Dye
Quoted 5 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Add a series of tests to exercise the HTTP authentication header parsing and the interop with credential helpers. Credential helpers will receive WWW-Authenticate information in credential requests.
A general comment about this series - the way you have the patches organized means that the "feature" content you're trying to integrate (the first two patches) is contextually separated from these tests. For people that learn/understand code via examples in tests, this makes it really difficult to understand what's going on. To avoid that, I think you could rearrange the patches pretty easily:
Quoted 12 lines
1. test-http-server: add stub HTTP server test helper (prev. patch 3)
- t5556 could be introduced here with the basic "anonymous" test in patch 6, but marked 'test_expect_failure'.
2. test-http-server: add HTTP error response function (prev. patch 4)
3. test-http-server: add HTTP request parsing (prev. patch 5)
4. test-http-server: pass Git requests to http-backend (prev. patch 6)
5. test-http-server: add simple authentication (prev. patch 7)
6. http: read HTTP WWW-Authenticate response headers (prev. patch 1)
7. credential: add WWW-Authenticate header to cred requests (prev patch 2)
- Some/all of the tests from the current patch (patch 8) could be squashed into this one so that the tests exist directly alongside the new functionality they're testing.
I think that order make sense - I'll rearrange for my next iteration. Thanks!
Quoted 18 lines
t/helper/test-credential-helper-replay.sh
Quoted 13 lines
Signed-off-by Matthew John Cheetham <[email protected]>--- t/helper/test-credential-helper-replay.sh | 14 +++ t/t5556-http-auth.sh | 120 +++++++++++++++++++++- 2 files changed, 133 insertions(+), 1 deletion(-) create mode 100755 t/helper/test-credential-helper-replay.sht/helper/test-credential-helper-replay.sh new file +0
-0I'm not sure a 't/helper' file is the right place for this - it's a pretty simple shell script, but it defines a lot of information (namely 'teefile', 'catfile') that is otherwise unexplained in 't5556'.
Quoted 5 lines
What about something like '
lib-rebase.sh' and its 'set_fake_editor()'? You could create a similar test lib ('lib-credential-helper.sh') and wrapper function (' that writes out a custom credential helper. Something like 'set_fake_credential_helper()' could also take 'teefile' and 'catfile' as arguments, making their names more transparent to 't5556'.
The lib-rebase.sh script sets the fake editor by setting an environment variable (from what I can see). Credential helpers can only be set via config or command-line arg. Would it be easier to move writing of the test credential helper script to the t5556 test script setup?
Quoted 30 lines
Quoted 20 lines
t/helper/test-credential-helper-replay.sh new file +14
-0@@ -0,0 +1,14 @@ +cmd=$1 +teefile=$cmd-actual.cred +catfile=$cmd-response.cred +rm -f $teefile +while read line; +do + if test -z "$line"; then + break; + fi + echo "$line" >> $teefile +done +if test "$cmd" = "get"; then + cat $catfile +fit/t5556-http-auth.sh +0
-0t/t5556-http-auth.sh PID_FILE="$(pwd)"/pid-file.pid · line 30
Quoted 5 lines
t/t5556-http-auth.sh +2
-0@@ -26,6 +26,8 @@ PID_FILE="$(pwd)"/pid-file.pid SERVER_LOG="$(pwd)"/OUT.server.log PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH +CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \ + && export CREDENTIAL_HELPERI see - this is how you connect the "test" credential helper to the HTTP server and header parsing (as implemented in patches 1 & 2), so that the results can be compared for correctness.
Quoted 3 lines
nit: you can just 'export CREDENTIAL_HELPER="..."', rather than breaking it into two lines. You also shouldn't need to 'export' at all - the value will be set in the context of the test.
I tried this originally, but got errors from one of the environments in CI that this was not portable.
Quoted 23 lines
Quoted 18 lines
t/t5556-http-auth.sh +3
-1test_expect_success 'setup repos' ' test_create_repo "$REPO_DIR" &&@@ -91,7 +93,8 @@ start_http_server () { per_test_cleanup () { stop_http_server && - rm -f OUT.* + rm -f OUT.* && + rm -f *.cred } test_expect_success 'http auth anonymous no challenge' '@@ -102,4 +105,119 @@ test_expect_success 'http auth anonymous no challenge' ' git ls-remote $ORIGIN_URL ' +test_expect_success 'http auth www-auth headers to credential helper basic valid' '...
Quoted 1 line
+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
Quoted 8 lines
...
Quoted 1 line
+test_expect_success 'http auth www-auth headers to credential helper invalid' '
These tests all look good. That said, is there any way to test more bizarre/edge cases (headers too long to fit on one line, headers that end with a long string of whitespace, etc.)?
Thanks,
Matthew
v5 cover Enhance credential helper protocol to include auth headers
In reply to Matthew John Cheetham via GitGitGadgetFollowing from my original RFC submission [0], this submission is considered ready for full review. This patch series is now based on top of current master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now separately submitted patches [1] to fix up the other credential helpers' behaviour.
In this patch series I update the existing credential helper design in order to allow for some new scenarios, and future evolution of auth methods that Git hosts may wish to provide. I outline the background, summary of changes and some challenges below.
Testing these new additions, I introduce a new test helper test-http-server that acts as a frontend to git-http-backend; a mini HTTP server sharing code with git-daemon, with simple authentication configurable by a config file.
Background
==========
Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the authentication capabilities of this protocol to address limitations (see below).
The Smart HTTP protocol in Git supports a few different types of HTTP authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478) [4]. Git uses a extensible model where credential helpers can provide credentials for protocols [5]. Several helpers support alternatives such as OAuth authentication (RFC 6749) [6], but this is typically done as an extension. For example, a helper might use basic auth and set the password to an OAuth Bearer access token. Git uses standard input and output to communicate with credential helpers.
After a HTTP 401 response, Git would call a credential helper with the following over standard input:
protocol=https
host=example.com
And then a credential helper would return over standard output:
protocol=https
host=example.com
[email protected]
password=<BEARER-TOKEN>
Git then the following request to the remote, including the standard HTTP Authorization header (RFC 7235 Section 4.2) [7]:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64([email protected]:<BEARER-TOKEN>)
Credential helpers are encouraged (see gitcredentials.txt) to return the minimum information necessary.
Limitations
===========
Because this credential model was built mostly for password based authentication systems, it's somewhat limited. In particular:
1. To generate valid credentials, additional information about the request (or indeed the requestee and their device) may be required. For example, OAuth is based around scopes. A scope, like "git.read", might be required to read data from the remote. However, the remote cannot tell the credential helper what scope is required for this request.
2. This system is not fully extensible. Each time a new type of authentication (like OAuth Bearer) is invented, Git needs updates before credential helpers can take advantage of it (or leverage a new capability in libcurl).
Goals
=====
* As a user with multiple federated cloud identities:
* Reach out to a remote and have my credential helper automatically prompt me for the correct identity.
* Allow credential helpers to differentiate between different authorities or authentication/authorization challenge types, even from the same DNS hostname (and without needing to use credential.useHttpPath).
* Leverage existing authentication systems built-in to many operating systems and devices to boost security and reduce reliance on passwords.
* As a Git host and/or cloud identity provider:
* Enforce security policies (like requiring two-factor authentication) dynamically.
* Allow integration with third party standard based identity providers in enterprises allowing customers to have a single plane of control for critical identities with access to source code.
Design Principles
=================
* Use the existing infrastructure. Git credential helpers are an already-working model.
* Follow widely-adopted time-proven open standards, avoid net new ideas in the authentication space.
* Minimize knowledge of authentication in Git; maintain modularity and extensibility.
Proposed Changes
================
1. Teach Git to read HTTP response headers, specifically the standard WWW-Authenticate (RFC 7235 Section 4.1) headers.
2. Teach Git to include extra information about HTTP responses that require authentication when calling credential helpers. Specifically the WWW-Authenticate header information.
Because the extra information forms an ordered list, and the existing credential helper I/O format only provides for simple key=value pairs, we introduce a new convention for transmitting an ordered list of values. Key names that are suffixed with a C-style array syntax should have values considered to form an order list, i.e. key[]=value, where the order of the key=value pairs in the stream specifies the order.
For the WWW-Authenticate header values we opt to use the key wwwauth[].
Handling the WWW-Authenticate header in detail
==============================================
RFC 6750 [8] envisions that OAuth Bearer resource servers would give responses that include WWW-Authenticate headers, for example:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
Specifically, a WWW-Authenticate header consists of a scheme and arbitrary attributes, depending on the scheme. This pattern enables generic OAuth or OpenID Connect [9] authorities. Note that it is possible to have several WWW-Authenticate challenges in a response.
First Git attempts to make a request, unauthenticated, which fails with a 401 response and includes WWW-Authenticate header(s).
Next, Git invokes a credential helper which may prompt the user. If the user approves, a credential helper can generate a token (or any auth challenge response) to be used for that request.
For example: with a remote that supports bearer tokens from an OpenID Connect [9] authority, a credential helper can use OpenID Connect's Discovery [10] and Dynamic Client Registration [11] to register a client and make a request with the correct permissions to access the remote. In this manner, a user can be dynamically sent to the right federated identity provider for a remote without any up-front configuration or manual processes.
Following from the principle of keeping authentication knowledge in Git to a minimum, we modify Git to add all WWW-Authenticate values to the credential helper call.
Git sends over standard input:
protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite" wwwauth[]=Basic realm="login.example"
A credential helper that understands the extra wwwauth[n] property can decide on the "best" or correct authentication scheme, generate credentials for the request, and interact with the user.
The credential helper would then return over standard output:
protocol=https
host=example.com
path=foo.git
[email protected]
password=<BEARER-TOKEN>
Note that WWW-Authenticate supports multiple challenges, either in one header:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
or in multiple headers:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the implementation, Git will not merge or split up any of these WWW-Authenticate headers, and instead pass each header line as one credential helper property. The credential helper is responsible for splitting, merging, and otherwise parsing these header values.
An alternative option to sending the header fields individually would be to merge the header values in to one key=value property, for example:
...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
Future work
===========
In the future we can further expand the protocol to allow credential helpers decide the best authentication scheme. Today credential helpers are still only expected to return a username/password pair to Git, meaning the other authentication schemes that may be offered still need challenge responses sent via a Basic Authorization header. The changes outlined above still permit helpers to select and configure an available authentication mode, but require the remote for example to unpack a bearer token from a basic challenge.
More careful consideration is required in the handling of custom authentication schemes which may not have a username, or may require arbitrary additional request header values be set.
For example imagine a new "FooBar" authentication scheme that is surfaced in the following response:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"
With support for arbitrary authentication schemes, Git would call credential helpers with the following over standard input:
protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"
And then an enlightened credential helper could return over standard output:
protocol=https
host=example.com
authtype=FooBar
[email protected]
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF
Git would be expected to attach this authorization header to the next request:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF
Why not SSH?
============
There's nothing wrong with SSH. However, Git's Smart HTTP transport is widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport sometimes requires less client setup than SSH transport, and works in environments when SSH ports may be blocked. As long as Git supports HTTP transport, it should support common and popular HTTP authentication methods.
References
==========
* [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers
https://lore.kernel.org/git/[email protected]/
* [1] [PATCH 0/3] Correct credential helper discrepancies handling input
https://lore.kernel.org/git/[email protected]/
* [2] Git on the Server - The Protocols
https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols
* [3] HTTP Authentication: Basic and Digest Access Authentication
https://datatracker.ietf.org/doc/html/rfc2617
* [4] The Simple and Protected GSS-API Negotiation Mechanism
https://datatracker.ietf.org/doc/html/rfc2478
* [5] Git Credentials - Custom Helpers
https://git-scm.com/docs/gitcredentials#_custom_helpers
* [6] The OAuth 2.0 Authorization Framework
https://datatracker.ietf.org/doc/html/rfc6749
* [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
https://datatracker.ietf.org/doc/html/rfc7235
* [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
https://datatracker.ietf.org/doc/html/rfc6750
* [9] OpenID Connect Core 1.0
https://openid.net/specs/openid-connect-core-1_0.html
* [10] OpenID Connect Discovery 1.0
https://openid.net/specs/openid-connect-discovery-1_0.html
* [11] OpenID Connect Dynamic Client Registration 1.0
https://openid.net/specs/openid-connect-registration-1_0.html
* [12] Hypertext Transfer Protocol (HTTP/1.1)
https://datatracker.ietf.org/doc/html/rfc2616
Updates from RFC
================
* Submitted first three patches as separate submission:
https://lore.kernel.org/git/[email protected]/
* Various style fixes and updates to- and addition of comments.
* Drop the explicit integer index in new 'array' style credential helper attrbiutes ("key[n]=value" becomes just "key[]=value").
* Added test helper; a mini HTTP server, and several tests.
Updates in v3
=============
* Split final patch that added the test-http-server in to several, easier to review patches.
* Updated wording in git-credential.txt to clarify which side of the credential helper protocol is sending/receiving the new wwwauth and authtype attributes.
Updates in v4
=============
* Drop authentication scheme selection authtype attribute patches to greatly simplify the series; auth scheme selection is punted to a future series. This series still allows credential helpers to generate credentials and intelligently select correct identities for a given auth challenge.
Updates in v5
=============
* Libify parts of daemon.c and share implementation with test-http-server.
* Clarify test-http-server Git request regex pattern and auth logic comments.
* Use STD*_FILENO in place of 'magic' file descriptor numbers.
* Use strbuf_* functions in continuation header parsing.
* Use configuration file to configure auth for test-http-server rather than command-line arguments. Add ability to specify arbitrary extra headers that is useful for testing 'malformed' server responses.
* Use st_mult over unchecked multiplication in http.c curl callback functions.
* Fix some documentation line break issues.
* Reorder some commits to bring in the tests and test-http-server helper first and, then the WWW-Authentication changes, alongside tests to cover.
* Expose previously static strvec_push_nodup function.
* Merge the two timeout args for test-http-server (--timeout and --init-timeout) that were a hang-over from the original daemon.c but are no longer required here.
* Be more careful around continuation headers where they may be empty strings. Add more tests to cover these header types.
* Include standard trace2 tracing calls at start of test-http-server helper.
Matthew John Cheetham (10):
daemon: libify socket setup and option functions
daemon: libify child process handling functions
daemon: rename some esoteric/laboured terminology
test-http-server: add stub HTTP server test helper
test-http-server: add HTTP error response function
test-http-server: add simple authentication
http: replace unsafe size_t multiplication with st_mult
strvec: expose strvec_push_nodup for external use
http: read HTTP WWW-Authenticate response headers
credential: add WWW-Authenticate header to cred requests
Documentation/git-credential.txt | 19 +- Makefile | 2 + contrib/buildsystems/CMakeLists.txt | 11 +- credential.c | 13 + credential.h | 15 + daemon-utils.c | 286 +++++++ daemon-utils.h | 38 + daemon.c | 306 +------ http.c | 102 ++- strvec.c | 2 +- strvec.h | 3 + t/helper/.gitignore | 1 + t/helper/test-credential-helper-replay.sh | 14 + t/helper/test-http-server.c | 920 ++++++++++++++++++++++ t/t5556-http-auth.sh | 372 +++++++++ 15 files changed, 1801 insertions(+), 303 deletions(-) create mode 100644 daemon-utils.c create mode 100644 daemon-utils.h create mode 100755 t/helper/test-credential-helper-replay.sh create mode 100644 t/helper/test-http-server.c create mode 100755 t/t5556-http-auth.sh
base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/1352
Range-diff vs v4:
-: ----------- > 1: 74b0de14185 daemon: libify socket setup and option functions
-: ----------- > 2: bc972fc8d3d daemon: libify child process handling functions
-: ----------- > 3: 8f176d5955d daemon: rename some esoteric/laboured terminology
3: 07a1845ea56 ! 4: 706fb3781bd test-http-server: add stub HTTP server test helper
@@ Commit message
Signed-off-by: Matthew John Cheetham <[email protected]>
## Makefile ##
-@@ Makefile: else
- endif
- BASIC_CFLAGS += $(CURL_CFLAGS)
+@@ Makefile: TEST_BUILTINS_OBJS += test-xml-encode.o
+ # Do not add more tests here unless they have extra dependencies. Add
+ # them in TEST_BUILTINS_OBJS above.
+ TEST_PROGRAMS_NEED_X += test-fake-ssh
++TEST_PROGRAMS_NEED_X += test-http-server
+ TEST_PROGRAMS_NEED_X += test-tool
-+ TEST_PROGRAMS_NEED_X += test-http-server
-+
- REMOTE_CURL_PRIMARY = git-remote-http$X
- REMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X
- REMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)
+ TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))
## contrib/buildsystems/CMakeLists.txt ##
+@@ contrib/buildsystems/CMakeLists.txt: if(BUILD_TESTING)
+ add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)
+ target_link_libraries(test-fake-ssh common-main)
+
++add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
++target_link_libraries(test-http-server common-main)
++
+ #reftable-tests
+ parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS")
+ list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/")
+@@ contrib/buildsystems/CMakeLists.txt: if(MSVC)
+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
+ set_target_properties(test-fake-ssh test-tool
+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
++
++ set_target_properties(test-http-server
++ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
++ set_target_properties(test-http-server
++ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
+ endif()
+
+ #wrapper scripts
@@ contrib/buildsystems/CMakeLists.txt: set(wrapper_scripts
- set(wrapper_test_scripts
- test-fake-ssh test-tool)
+ git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)
-+if(CURL_FOUND)
-+ list(APPEND wrapper_test_scripts test-http-server)
-+
-+ add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
-+ target_link_libraries(test-http-server common-main)
-+
-+ if(MSVC)
-+ set_target_properties(test-http-server
-+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
-+ set_target_properties(test-http-server
-+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
-+ endif()
-+endif()
+ set(wrapper_test_scripts
+- test-fake-ssh test-tool)
+-
++ test-http-server test-fake-ssh test-tool)
foreach(script ${wrapper_scripts})
file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
@@ t/helper/.gitignore
## t/helper/test-http-server.c (new) ##
@@
++#include "daemon-utils.h"
+#include "config.h"
+#include "run-command.h"
+#include "strbuf.h"
@@ t/helper/test-http-server.c (new)
+
+static const char test_http_auth_usage[] =
+"http-server [--verbose]\n"
-+" [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
++" [--timeout=<n>] [--max-connections=<n>]\n"
+" [--reuseaddr] [--pid-file=<file>]\n"
+" [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+;
+
-+/* Timeout, and initial timeout */
+static unsigned int timeout;
-+static unsigned int init_timeout;
+
+static void logreport(const char *label, const char *err, va_list params)
+{
@@ t/helper/test-http-server.c (new)
+ va_end(params);
+}
+
-+static void set_keep_alive(int sockfd)
-+{
-+ int ka = 1;
-+
-+ if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
-+ if (errno != ENOTSOCK)
-+ logerror("unable to set SO_KEEPALIVE on socket: %s",
-+ strerror(errno));
-+ }
-+}
-+
+/*
+ * The code in this section is used by "worker" instances to service
+ * a single connection from a client. The worker talks to the client
@@ t/helper/test-http-server.c (new)
+ * Close the socket and clean up. Does not imply an error.
+ */
+ WR_HANGUP = 1<<1,
-+
-+ WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
+};
+
+static enum worker_result worker(void)
@@ t/helper/test-http-server.c (new)
+ if (client_addr)
+ loginfo("Connection from %s:%s", client_addr, client_port);
+
-+ set_keep_alive(0);
++ set_keep_alive(0, logerror);
+
+ while (1) {
-+ if (write_in_full(1, response, strlen(response)) < 0) {
++ if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
+ logerror("unable to write response");
+ wr = WR_IO_ERROR;
+ }
+
-+ if (wr & WR_STOP_THE_MUSIC)
++ if (wr != WR_OK)
+ break;
+ }
+
-+ close(0);
-+ close(1);
++ close(STDIN_FILENO);
++ close(STDOUT_FILENO);
+
+ return !!(wr & WR_IO_ERROR);
+}
+
-+/*
-+ * This section contains the listener and child-process management
-+ * code used by the primary instance to accept incoming connections
-+ * and dispatch them to async child process "worker" instances.
-+ */
-+
-+static int addrcmp(const struct sockaddr_storage *s1,
-+ const struct sockaddr_storage *s2)
-+{
-+ const struct sockaddr *sa1 = (const struct sockaddr*) s1;
-+ const struct sockaddr *sa2 = (const struct sockaddr*) s2;
-+
-+ if (sa1->sa_family != sa2->sa_family)
-+ return sa1->sa_family - sa2->sa_family;
-+ if (sa1->sa_family == AF_INET)
-+ return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
-+ &((struct sockaddr_in *)s2)->sin_addr,
-+ sizeof(struct in_addr));
-+#ifndef NO_IPV6
-+ if (sa1->sa_family == AF_INET6)
-+ return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
-+ &((struct sockaddr_in6 *)s2)->sin6_addr,
-+ sizeof(struct in6_addr));
-+#endif
-+ return 0;
-+}
-+
+static int max_connections = 32;
+
+static unsigned int live_children;
+
-+static struct child {
-+ struct child *next;
-+ struct child_process cld;
-+ struct sockaddr_storage address;
-+} *firstborn;
-+
-+static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
-+{
-+ struct child *newborn, **cradle;
-+
-+ newborn = xcalloc(1, sizeof(*newborn));
-+ live_children++;
-+ memcpy(&newborn->cld, cld, sizeof(*cld));
-+ memcpy(&newborn->address, addr, addrlen);
-+ for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
-+ if (!addrcmp(&(*cradle)->address, &newborn->address))
-+ break;
-+ newborn->next = *cradle;
-+ *cradle = newborn;
-+}
-+
-+/*
-+ * This gets called if the number of connections grows
-+ * past "max_connections".
-+ *
-+ * We kill the newest connection from a duplicate IP.
-+ */
-+static void kill_some_child(void)
-+{
-+ const struct child *blanket, *next;
-+
-+ if (!(blanket = firstborn))
-+ return;
-+
-+ for (; (next = blanket->next); blanket = next)
-+ if (!addrcmp(&blanket->address, &next->address)) {
-+ kill(blanket->cld.pid, SIGTERM);
-+ break;
-+ }
-+}
-+
-+static void check_dead_children(void)
-+{
-+ int status;
-+ pid_t pid;
-+
-+ struct child **cradle, *blanket;
-+ for (cradle = &firstborn; (blanket = *cradle);)
-+ if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
-+ const char *dead = "";
-+ if (status)
-+ dead = " (with error)";
-+ loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead);
-+
-+ /* remove the child */
-+ *cradle = blanket->next;
-+ live_children--;
-+ child_process_clear(&blanket->cld);
-+ free(blanket);
-+ } else
-+ cradle = &blanket->next;
-+}
++static struct child *first_child;
+
+static struct strvec cld_argv = STRVEC_INIT;
+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
@@ t/helper/test-http-server.c (new)
+ struct child_process cld = CHILD_PROCESS_INIT;
+
+ if (max_connections && live_children >= max_connections) {
-+ kill_some_child();
++ kill_some_child(first_child);
+ sleep(1); /* give it some time to die */
-+ check_dead_children();
++ check_dead_children(first_child, &live_children, loginfo);
+ if (live_children >= max_connections) {
+ close(incoming);
+ logerror("Too many children, dropping connection");
@@ t/helper/test-http-server.c (new)
+ else if (start_command(&cld))
+ logerror("unable to fork");
+ else
-+ add_child(&cld, addr, addrlen);
++ add_child(&cld, addr, addrlen, first_child, &live_children);
+}
+
+static void child_handler(int signo)
@@ t/helper/test-http-server.c (new)
+ signal(SIGCHLD, child_handler);
+}
+
-+static int set_reuse_addr(int sockfd)
-+{
-+ int on = 1;
-+
-+ if (!reuseaddr)
-+ return 0;
-+ return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
-+ &on, sizeof(on));
-+}
-+
-+struct socketlist {
-+ int *list;
-+ size_t nr;
-+ size_t alloc;
-+};
-+
-+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
-+{
-+#ifdef NO_IPV6
-+ static char ip[INET_ADDRSTRLEN];
-+#else
-+ static char ip[INET6_ADDRSTRLEN];
-+#endif
-+
-+ switch (family) {
-+#ifndef NO_IPV6
-+ case AF_INET6:
-+ inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
-+ break;
-+#endif
-+ case AF_INET:
-+ inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
-+ break;
-+ default:
-+ xsnprintf(ip, sizeof(ip), "<unknown>");
-+ }
-+ return ip;
-+}
-+
-+#ifndef NO_IPV6
-+
-+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-+{
-+ int socknum = 0;
-+ char pbuf[NI_MAXSERV];
-+ struct addrinfo hints, *ai0, *ai;
-+ int gai;
-+ long flags;
-+
-+ xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
-+ memset(&hints, 0, sizeof(hints));
-+ hints.ai_family = AF_UNSPEC;
-+ hints.ai_socktype = SOCK_STREAM;
-+ hints.ai_protocol = IPPROTO_TCP;
-+ hints.ai_flags = AI_PASSIVE;
-+
-+ gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
-+ if (gai) {
-+ logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
-+ return 0;
-+ }
-+
-+ for (ai = ai0; ai; ai = ai->ai_next) {
-+ int sockfd;
-+
-+ sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
-+ if (sockfd < 0)
-+ continue;
-+ if (sockfd >= FD_SETSIZE) {
-+ logerror("Socket descriptor too large");
-+ close(sockfd);
-+ continue;
-+ }
-+
-+#ifdef IPV6_V6ONLY
-+ if (ai->ai_family == AF_INET6) {
-+ int on = 1;
-+ setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
-+ &on, sizeof(on));
-+ /* Note: error is not fatal */
-+ }
-+#endif
-+
-+ if (set_reuse_addr(sockfd)) {
-+ logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
-+ close(sockfd);
-+ continue;
-+ }
-+
-+ set_keep_alive(sockfd);
-+
-+ if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
-+ logerror("Could not bind to %s: %s",
-+ ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
-+ strerror(errno));
-+ close(sockfd);
-+ continue; /* not fatal */
-+ }
-+ if (listen(sockfd, 5) < 0) {
-+ logerror("Could not listen to %s: %s",
-+ ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
-+ strerror(errno));
-+ close(sockfd);
-+ continue; /* not fatal */
-+ }
-+
-+ flags = fcntl(sockfd, F_GETFD, 0);
-+ if (flags >= 0)
-+ fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-+
-+ ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
-+ socklist->list[socklist->nr++] = sockfd;
-+ socknum++;
-+ }
-+
-+ freeaddrinfo(ai0);
-+
-+ return socknum;
-+}
-+
-+#else /* NO_IPV6 */
-+
-+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-+{
-+ struct sockaddr_in sin;
-+ int sockfd;
-+ long flags;
-+
-+ memset(&sin, 0, sizeof sin);
-+ sin.sin_family = AF_INET;
-+ sin.sin_port = htons(listen_port);
-+
-+ if (listen_addr) {
-+ /* Well, host better be an IP address here. */
-+ if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
-+ return 0;
-+ } else {
-+ sin.sin_addr.s_addr = htonl(INADDR_ANY);
-+ }
-+
-+ sockfd = socket(AF_INET, SOCK_STREAM, 0);
-+ if (sockfd < 0)
-+ return 0;
-+
-+ if (set_reuse_addr(sockfd)) {
-+ logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
-+ close(sockfd);
-+ return 0;
-+ }
-+
-+ set_keep_alive(sockfd);
-+
-+ if (bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0) {
-+ logerror("Could not bind to %s: %s",
-+ ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
-+ strerror(errno));
-+ close(sockfd);
-+ return 0;
-+ }
-+
-+ if (listen(sockfd, 5) < 0) {
-+ logerror("Could not listen to %s: %s",
-+ ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
-+ strerror(errno));
-+ close(sockfd);
-+ return 0;
-+ }
-+
-+ flags = fcntl(sockfd, F_GETFD, 0);
-+ if (flags >= 0)
-+ fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-+
-+ ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
-+ socklist->list[socklist->nr++] = sockfd;
-+ return 1;
-+}
-+
-+#endif
-+
-+static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
-+{
-+ if (!listen_addr->nr)
-+ setup_named_sock("127.0.0.1", listen_port, socklist);
-+ else {
-+ int i, socknum;
-+ for (i = 0; i < listen_addr->nr; i++) {
-+ socknum = setup_named_sock(listen_addr->items[i].string,
-+ listen_port, socklist);
-+
-+ if (socknum == 0)
-+ logerror("unable to allocate any listen sockets for host %s on port %u",
-+ listen_addr->items[i].string, listen_port);
-+ }
-+ }
-+}
-+
+static int service_loop(struct socketlist *socklist)
+{
+ struct pollfd *pfd;
@@ t/helper/test-http-server.c (new)
+ int nr_ready;
+ int timeout = (pid_file ? 100 : -1);
+
-+ check_dead_children();
++ check_dead_children(first_child, &live_children, loginfo);
+
+ nr_ready = poll(pfd, socklist->nr, timeout);
+ if (nr_ready < 0) {
@@ t/helper/test-http-server.c (new)
+{
+ struct socketlist socklist = { NULL, 0, 0 };
+
-+ socksetup(listen_addr, listen_port, &socklist);
++ socksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);
+ if (socklist.nr == 0)
+ die("unable to allocate any listen sockets on port %u",
+ listen_port);
@@ t/helper/test-http-server.c (new)
+ int i;
+
+ trace2_cmd_name("test-http-server");
++ trace2_cmd_list_config();
++ trace2_cmd_list_env_vars();
+ setup_git_directory_gently(NULL);
+
+ for (i = 1; i < argc; i++) {
@@ t/helper/test-http-server.c (new)
+ timeout = atoi(v);
+ continue;
+ }
-+ if (skip_prefix(arg, "--init-timeout=", &v)) {
-+ init_timeout = atoi(v);
-+ continue;
-+ }
+ if (skip_prefix(arg, "--max-connections=", &v)) {
+ max_connections = atoi(v);
+ if (max_connections < 0)
5: 5c4e36e23ee ! 5: 6f66bf146b4 test-http-server: add HTTP request parsing
@@ Metadata
Author: Matthew John Cheetham <[email protected]>
## Commit message ##
- test-http-server: add HTTP request parsing
+ test-http-server: add HTTP error response function
- Add ability to parse HTTP requests to the test-http-server test helper.
+ Introduce a function to the test-http-server test helper to write more
+ full and valid HTTP error responses, including all the standard response
+ headers like `Server` and `Date`.
Signed-off-by: Matthew John Cheetham <[email protected]>
## t/helper/test-http-server.c ##
@@ t/helper/test-http-server.c: enum worker_result {
- WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
+ WR_HANGUP = 1<<1,
};
+/*
@@ t/helper/test-http-server.c: enum worker_result {
+ string_list_clear(&req->header_list, 0);
+}
+
- static enum worker_result send_http_error(
- int fd,
- int http_code, const char *http_code_name,
-@@ t/helper/test-http-server.c: done:
- return wr;
- }
-
++static enum worker_result send_http_error(
++ int fd,
++ int http_code, const char *http_code_name,
++ int retry_after_seconds, struct string_list *response_headers,
++ enum worker_result wr_in)
++{
++ struct strbuf response_header = STRBUF_INIT;
++ struct strbuf response_content = STRBUF_INIT;
++ struct string_list_item *h;
++ enum worker_result wr;
++
++ strbuf_addf(&response_content, "Error: %d %s\r\n",
++ http_code, http_code_name);
++ if (retry_after_seconds > 0)
++ strbuf_addf(&response_content, "Retry-After: %d\r\n",
++ retry_after_seconds);
++
++ strbuf_addf (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
++ strbuf_addstr(&response_header, "Cache-Control: private\r\n");
++ strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
++ strbuf_addf (&response_header, "Content-Length: %d\r\n", (int)response_content.len);
++ if (retry_after_seconds > 0)
++ strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
++ strbuf_addf( &response_header, "Server: test-http-server/%s\r\n", git_version_string);
++ strbuf_addf( &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
++ if (response_headers)
++ for_each_string_list_item(h, response_headers)
++ strbuf_addf(&response_header, "%s\r\n", h->string);
++ strbuf_addstr(&response_header, "\r\n");
++
++ if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
++ logerror("unable to write response header");
++ wr = WR_IO_ERROR;
++ goto done;
++ }
++
++ if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
++ logerror("unable to write response content body");
++ wr = WR_IO_ERROR;
++ goto done;
++ }
++
++ wr = wr_in;
++
++done:
++ strbuf_release(&response_header);
++ strbuf_release(&response_content);
++
++ return wr;
++}
++
+/*
+ * Read the HTTP request up to the start of the optional message-body.
+ * We do this byte-by-byte because we have keep-alive turned on and
@@ t/helper/test-http-server.c: done:
+ hp = strbuf_detach(&h, NULL);
+ string_list_append(&req->header_list, hp);
+
-+ /* store common request headers separately */
++ /* also store common request headers as struct req members */
+ if (skip_prefix(hp, "Content-Type: ", &hv)) {
+ req->content_type = hv;
+ } else if (skip_prefix(hp, "Content-Length: ", &hv)) {
@@ t/helper/test-http-server.c: done:
+ return result;
+}
+
++static int is_git_request(struct req *req)
++{
++ static regex_t *smart_http_regex;
++ static int initialized;
++
++ if (!initialized) {
++ smart_http_regex = xmalloc(sizeof(*smart_http_regex));
++ /*
++ * This regular expression matches all dumb and smart HTTP
++ * requests that are currently in use, and defined in
++ * Documentation/gitprotocol-http.txt.
++ *
++ */
++ if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
++ "objects/info/[^/]+|git-(upload|receive)-pack)$",
++ REG_EXTENDED)) {
++ warning("could not compile smart HTTP regex");
++ smart_http_regex = NULL;
++ }
++ initialized = 1;
++ }
++
++ return smart_http_regex &&
++ !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
++}
++
++static enum worker_result do__git(struct req *req)
++{
++ const char *ok = "HTTP/1.1 200 OK\r\n";
++ struct child_process cp = CHILD_PROCESS_INIT;
++ int res;
++
++ /*
++ * Note that we always respond with a 200 OK response even if the
++ * http-backend process exits with an error. This helper is intended
++ * only to be used to exercise the HTTP auth handling in the Git client,
++ * and specifically around authentication (not handled by http-backend).
++ *
++ * If we wanted to respond with a more 'valid' HTTP response status then
++ * we'd need to buffer the output of http-backend, wait for and grok the
++ * exit status of the process, then write the HTTP status line followed
++ * by the http-backend output. This is outside of the scope of this test
++ * helper's use at time of writing.
++ *
++ * The important auth responses (401) we are handling prior to getting
++ * to this point.
++ */
++ if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
++ return error(_("could not send '%s'"), ok);
++
++ strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
++ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
++ req->uri_path.buf);
++ strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
++ if (req->query_args.len)
++ strvec_pushf(&cp.env, "QUERY_STRING=%s",
++ req->query_args.buf);
++ if (req->content_type)
++ strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
++ req->content_type);
++ if (req->content_length >= 0)
++ strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
++ (intmax_t)req->content_length);
++ cp.git_cmd = 1;
++ strvec_push(&cp.args, "http-backend");
++ res = run_command(&cp);
++ close(STDOUT_FILENO);
++ close(STDIN_FILENO);
++ return !!res;
++}
++
+static enum worker_result dispatch(struct req *req)
+{
-+ return send_http_error(1, 501, "Not Implemented", -1, NULL,
++ if (is_git_request(req))
++ return do__git(req);
++
++ return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
+ WR_OK | WR_HANGUP);
+}
+
static enum worker_result worker(void)
{
+- const char *response = "HTTP/1.1 501 Not Implemented\r\n";
+ struct req req = REQ__INIT;
char *client_addr = getenv("REMOTE_ADDR");
char *client_port = getenv("REMOTE_PORT");
enum worker_result wr = WR_OK;
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
- set_keep_alive(0);
+ set_keep_alive(0, logerror);
while (1) {
-- wr = send_http_error(1, 501, "Not Implemented", -1, NULL,
-- WR_OK | WR_HANGUP);
+- if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
+- logerror("unable to write response");
+- wr = WR_IO_ERROR;
+- }
+ req__release(&req);
+
-+ alarm(init_timeout ? init_timeout : timeout);
++ alarm(timeout);
+ wr = req__read(&req, 0);
+ alarm(0);
+
-+ if (wr & WR_STOP_THE_MUSIC)
++ if (wr != WR_OK)
+ break;
-+
+
+ wr = dispatch(&req);
- if (wr & WR_STOP_THE_MUSIC)
+ if (wr != WR_OK)
break;
}
+
+ ## t/t5556-http-auth.sh (new) ##
+@@
++#!/bin/sh
++
++test_description='test http auth header and credential helper interop'
++
++TEST_NO_CREATE_REPO=1
++. ./test-lib.sh
++
++test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
++
++# Setup a repository
++#
++REPO_DIR="$TRASH_DIRECTORY"/repo
++
++# Setup some lookback URLs where test-http-server will be listening.
++# We will spawn it directly inside the repo directory, so we avoid
++# any need to configure directory mappings etc - we only serve this
++# repository from the root '/' of the server.
++#
++HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
++ORIGIN_URL=http://$HOST_PORT/
++
++# The pid-file is created by test-http-server when it starts.
++# The server will shutdown if/when we delete it (this is easier than
++# killing it by PID).
++#
++PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
++SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
++
++PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
++
++test_expect_success 'setup repos' '
++ test_create_repo "$REPO_DIR" &&
++ git -C "$REPO_DIR" branch -M main
++'
++
++stop_http_server () {
++ if ! test -f "$PID_FILE"
++ then
++ return 0
++ fi
++ #
++ # The server will shutdown automatically when we delete the pid-file.
++ #
++ rm -f "$PID_FILE"
++ #
++ # Give it a few seconds to shutdown (mainly to completely release the
++ # port before the next test start another instance and it attempts to
++ # bind to it).
++ #
++ for k in 0 1 2 3 4
++ do
++ if grep -q "Starting graceful shutdown" "$SERVER_LOG"
++ then
++ return 0
++ fi
++ sleep 1
++ done
++
++ echo "stop_http_server: timeout waiting for server shutdown"
++ return 1
++}
++
++start_http_server () {
++ #
++ # Launch our server into the background in repo_dir.
++ #
++ (
++ cd "$REPO_DIR"
++ test-http-server --verbose \
++ --listen=127.0.0.1 \
++ --port=$GIT_TEST_HTTP_PROTOCOL_PORT \
++ --reuseaddr \
++ --pid-file="$PID_FILE" \
++ "$@" \
++ 2>"$SERVER_LOG" &
++ )
++ #
++ # Give it a few seconds to get started.
++ #
++ for k in 0 1 2 3 4
++ do
++ if test -f "$PID_FILE"
++ then
++ return 0
++ fi
++ sleep 1
++ done
++
++ echo "start_http_server: timeout waiting for server startup"
++ return 1
++}
++
++per_test_cleanup () {
++ stop_http_server &&
++ rm -f OUT.*
++}
++
++test_expect_success 'http auth anonymous no challenge' '
++ test_when_finished "per_test_cleanup" &&
++ start_http_server &&
++
++ # Attempt to read from a protected repository
++ git ls-remote $ORIGIN_URL
++'
++
++test_done
7: 794256754c1 ! 6: c3c3d17a688 test-http-server: add simple authentication
@@ Commit message
Add simple authentication to the test-http-server test helper.
Authentication schemes and sets of valid tokens can be specified via
- command-line arguments. Incoming requests are compared against the set
- of valid schemes and tokens and only approved if a matching token is
- found, or if no auth was provided and anonymous auth is enabled.
+ a configuration file (in the normal gitconfig file format).
+ Incoming requests are compared against the set of valid schemes and
+ tokens and only approved if a matching token is found, or if no auth
+ was provided and anonymous auth is enabled.
Signed-off-by: Matthew John Cheetham <[email protected]>
## t/helper/test-http-server.c ##
+@@
+ #include "version.h"
+ #include "dir.h"
+ #include "date.h"
++#include "config.h"
+
+ #define TR2_CAT "test-http-server"
+
@@ t/helper/test-http-server.c: static const char test_http_auth_usage[] =
- " [--timeout=<n>] [--init-timeout=<n>] [--max-connections=<n>]\n"
+ " [--timeout=<n>] [--max-connections=<n>]\n"
" [--reuseaddr] [--pid-file=<file>]\n"
" [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
-+" [--anonymous-allowed]\n"
-+" [--auth=<scheme>[:<params>] [--auth-token=<scheme>:<token>]]*\n"
++" [--auth-config=<file>]\n"
;
- /* Timeout, and initial timeout */
-@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req, const char *user)
+ static unsigned int timeout;
+@@ t/helper/test-http-server.c: static int is_git_request(struct req *req)
+ !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+ }
+
+-static enum worker_result do__git(struct req *req)
++static enum worker_result do__git(struct req *req, const char *user)
+ {
+ const char *ok = "HTTP/1.1 200 OK\r\n";
+ struct child_process cp = CHILD_PROCESS_INIT;
+@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
+ return error(_("could not send '%s'"), ok);
+
++ if (user)
++ strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
++
+ strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
+ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
+ req->uri_path.buf);
+@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
return !!res;
}
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
+static struct auth_module **auth_modules = NULL;
+static size_t auth_modules_nr = 0;
+static size_t auth_modules_alloc = 0;
++static struct strvec extra_headers = STRVEC_INIT;
++
++static struct auth_module *create_auth_module(const char *scheme,
++ const char *challenge)
++{
++ struct auth_module *mod = xmalloc(sizeof(struct auth_module));
++ mod->scheme = xstrdup(scheme);
++ mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
++ CALLOC_ARRAY(mod->tokens, 1);
++ string_list_init_dup(mod->tokens);
++ return mod;
++}
+
+static struct auth_module *get_auth_module(const char *scheme)
+{
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
+ return NULL;
+}
+
-+static void add_auth_module(struct auth_module *mod)
++static int add_auth_module(struct auth_module *mod)
+{
++ if (get_auth_module(mod->scheme))
++ return error("duplicate auth scheme '%s'\n", mod->scheme);
++
+ ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
+ auth_modules[auth_modules_nr++] = mod;
++
++ return 0;
+}
+
+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
+
+ /*
+ * Check all auth modules and try to validate the request.
-+ * The first module that matches a valid token approves the request.
++ * The first Authorization header that matches a known auth module
++ * scheme will be consulted to either approve or deny the request.
+ * If no module is found, or if there is no valid token, then 401 error.
+ * Otherwise, only permit the request if anonymous auth is enabled.
++ * It's atypical for user agents/clients to send multiple Authorization
++ * headers, but not explicitly forbidden or defined.
+ */
+ for_each_string_list_item(hdr, &req->header_list) {
+ if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
+ case AUTH_UNKNOWN:
+ if (result != AUTH_DENY && allow_anonymous)
+ break;
++
+ for (i = 0; i < auth_modules_nr; i++) {
+ mod = auth_modules[i];
+ if (mod->challenge_params)
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
+ mod->scheme);
+ string_list_append(&hdrs, challenge);
+ }
-+ *wr = send_http_error(1, 401, "Unauthorized", -1, &hdrs, *wr);
++
++ for (i = 0; i < extra_headers.nr; i++)
++ string_list_append(&hdrs, extra_headers.v[i]);
++
++ *wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
++ &hdrs, *wr);
+ }
+
+ strbuf_list_free(split);
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
+ return result == AUTH_ALLOW ||
+ (result == AUTH_UNKNOWN && allow_anonymous);
+}
++
++static int split_auth_param(const char *str, char **scheme, char **val, int required_val)
++{
++ struct strbuf **p = strbuf_split_str(str, ':', 2);
++
++ if (!p[0])
++ return -1;
++
++ /* trim trailing ':' */
++ if (p[1])
++ strbuf_setlen(p[0], p[0]->len - 1);
++
++ if (required_val && !p[1])
++ return -1;
++
++ *scheme = strbuf_detach(p[0], NULL);
++
++ if (p[1])
++ *val = strbuf_detach(p[1], NULL);
++
++ strbuf_list_free(p);
++ return 0;
++}
++
++static int read_auth_config(const char *name, const char *val, void *data)
++{
++ int ret = 0;
++ char *scheme = NULL;
++ char *token = NULL;
++ char *challenge = NULL;
++ struct auth_module *mod = NULL;
++
++ if (!strcmp(name, "auth.challenge")) {
++ if (split_auth_param(val, &scheme, &challenge, 0)) {
++ ret = error("invalid auth challenge '%s'", val);
++ goto cleanup;
++ }
++
++ mod = create_auth_module(scheme, challenge);
++ if (add_auth_module(mod)) {
++ ret = error("failed to add auth module '%s'", val);
++ goto cleanup;
++ }
++ }
++ if (!strcmp(name, "auth.token")) {
++ if (split_auth_param(val, &scheme, &token, 1)) {
++ ret = error("invalid auth token '%s'", val);
++ goto cleanup;
++ }
++
++ mod = get_auth_module(scheme);
++ if (!mod) {
++ ret = error("auth scheme not defined '%s'\n", scheme);
++ goto cleanup;
++ }
++
++ string_list_append(mod->tokens, token);
++ }
++ if (!strcmp(name, "auth.allowanonymous")) {
++ allow_anonymous = git_config_bool(name, val);
++ }
++ if (!strcmp(name, "auth.extraheader")) {
++ strvec_push(&extra_headers, val);
++ }
++
++cleanup:
++ free(scheme);
++ free(token);
++ free(challenge);
++
++ return ret;
++}
+
static enum worker_result dispatch(struct req *req)
{
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,
+ return wr;
+
if (is_git_request(req))
-- return do__git(req, NULL);
+- return do__git(req);
+ return do__git(req, user);
- return send_http_error(1, 501, "Not Implemented", -1, NULL,
+ return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
WR_OK | WR_HANGUP);
-@@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
- struct string_list listen_addr = STRING_LIST_INIT_NODUP;
- int worker_mode = 0;
- int i;
-+ struct auth_module *mod = NULL;
-
- trace2_cmd_name("test-http-server");
- setup_git_directory_gently(NULL);
@@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
pid_file = v;
continue;
}
-+ if (skip_prefix(arg, "--allow-anonymous", &v)) {
-+ allow_anonymous = 1;
-+ continue;
-+ }
-+ if (skip_prefix(arg, "--auth=", &v)) {
-+ struct strbuf **p = strbuf_split_str(v, ':', 2);
-+
-+ if (!p[0]) {
-+ error("invalid argument '%s'", v);
++ if (skip_prefix(arg, "--auth-config=", &v)) {
++ if (!strlen(v)) {
++ error("invalid argument - missing file path");
+ usage(test_http_auth_usage);
+ }
+
-+ /* trim trailing ':' */
-+ if (p[1])
-+ strbuf_setlen(p[0], p[0]->len - 1);
-+
-+ if (get_auth_module(p[0]->buf)) {
-+ error("duplicate auth scheme '%s'\n", p[0]->buf);
-+ usage(test_http_auth_usage);
-+ }
-+
-+ mod = xmalloc(sizeof(struct auth_module));
-+ mod->scheme = xstrdup(p[0]->buf);
-+ mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;
-+ CALLOC_ARRAY(mod->tokens, 1);
-+ string_list_init_dup(mod->tokens);
-+
-+ add_auth_module(mod);
-+
-+ strbuf_list_free(p);
-+ continue;
-+ }
-+ if (skip_prefix(arg, "--auth-token=", &v)) {
-+ struct strbuf **p = strbuf_split_str(v, ':', 2);
-+ if (!p[0]) {
-+ error("invalid argument '%s'", v);
-+ usage(test_http_auth_usage);
-+ }
-+
-+ if (!p[1]) {
-+ error("missing token value '%s'\n", v);
-+ usage(test_http_auth_usage);
-+ }
-+
-+ /* trim trailing ':' */
-+ strbuf_setlen(p[0], p[0]->len - 1);
-+
-+ mod = get_auth_module(p[0]->buf);
-+ if (!mod) {
-+ error("auth scheme not defined '%s'\n", p[0]->buf);
++ if (git_config_from_file(read_auth_config, v, NULL)) {
++ error("failed to read auth config file '%s'", v);
+ usage(test_http_auth_usage);
+ }
+
-+ string_list_append(mod->tokens, p[1]->buf);
-+ strbuf_list_free(p);
+ continue;
+ } -: ----------- > 7: 9c4d25945dd http: replace unsafe size_t multiplication with st_mult
-: ----------- > 8: 65a620b08ef strvec: expose strvec_push_nodup for external use
1: b5b56ccd941 ! 9: bcfec529d95 http: read HTTP WWW-Authenticate response headers
@@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
-+ size_t size = eltsize * nmemb;
++ size_t size = st_mult(eltsize, nmemb);
+ struct strvec *values = &http_auth.wwwauth_headers;
+ struct strbuf buf = STRBUF_INIT;
+ const char *val;
-+ const char *z = NULL;
+
+ /*
+ * Header lines may not come NULL-terminated from libcurl so we must
@@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
+ * This line could be a continuation of the previously matched header
+ * field. If this is the case then we should append this value to the
+ * end of the previously consumed value.
++ * Continuation lines start with at least one whitespace, maybe more,
++ * so we should collapse these down to a single SP (valid per the spec).
+ */
+ if (http_auth.header_is_last_match && isspace(*buf.buf)) {
-+ const char **v = values->v + values->nr - 1;
-+ char *append = xstrfmt("%s%.*s", *v, (int)(size - 1), ptr + 1);
++ /* Trim leading whitespace from this continuation hdr line. */
++ strbuf_ltrim(&buf);
+
-+ free((void*)*v);
-+ *v = append;
++ /*
++ * At this point we should always have at least one existing
++ * value, even if it is empty. Do not bother appending the new
++ * value if this continuation header is itself empty.
++ */
++ if (!values->nr) {
++ BUG("should have at least one existing header value");
++ } else if (buf.len) {
++ const char *prev = values->v[values->nr - 1];
++ struct strbuf append = STRBUF_INIT;
++ strbuf_addstr(&append, prev);
++
++ /* Join two non-empty values with a single space. */
++ if (append.len)
++ strbuf_addch(&append, ' ');
++
++ strbuf_addbuf(&append, &buf);
++
++ strvec_pop(values);
++ strvec_push_nodup(values, strbuf_detach(&append, NULL));
++ }
+
+ goto exit;
+ }
@@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
+ * We only care about the last HTTP request response's headers so clear
+ * the existing array.
+ */
-+ if (skip_iprefix(buf.buf, "http/", &z))
++ if (istarts_with(buf.buf, "http/"))
+ strvec_clear(values);
+
+exit:
2: d02875dda7c < -: ----------- credential: add WWW-Authenticate header to cred requests
4: 98dd286db7c < -: ----------- test-http-server: add HTTP error response function
6: 0a0f4fd10c8 < -: ----------- test-http-server: pass Git requests to http-backend
8: 8ecf6383522 ! 10: af66d2d2ede t5556: add HTTP authentication tests
@@ Metadata
Author: Matthew John Cheetham <[email protected]>
## Commit message ##
- t5556: add HTTP authentication tests
+ credential: add WWW-Authenticate header to cred requests
- Add a series of tests to exercise the HTTP authentication header parsing
+ Add the value of the WWW-Authenticate response header to credential
+ requests. Credential helpers that understand and support HTTP
+ authentication and authorization can use this standard header (RFC 2616
+ Section 14.47 [1]) to generate valid credentials.
+
+ WWW-Authenticate headers can contain information pertaining to the
+ authority, authentication mechanism, or extra parameters/scopes that are
+ required.
+
+ The current I/O format for credential helpers only allows for unique
+ names for properties/attributes, so in order to transmit multiple header
+ values (with a specific order) we introduce a new convention whereby a
+ C-style array syntax is used in the property name to denote multiple
+ ordered values for the same property.
+
+ In this case we send multiple `wwwauth[]` properties where the order
+ that the repeated attributes appear in the conversation reflects the
+ order that the WWW-Authenticate headers appeared in the HTTP response.
+
+ Add a set of tests to exercise the HTTP authentication header parsing
and the interop with credential helpers. Credential helpers will receive
WWW-Authenticate information in credential requests.
+ [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47
+
Signed-off-by: Matthew John Cheetham <[email protected]>
+ ## Documentation/git-credential.txt ##
+@@ Documentation/git-credential.txt: separated by an `=` (equals) sign, followed by a newline.
+ The key may contain any bytes except `=`, newline, or NUL. The value may
+ contain any bytes except newline or NUL.
+
+-In both cases, all bytes are treated as-is (i.e., there is no quoting,
++Attributes with keys that end with C-style array brackets `[]` can have
++multiple values. Each instance of a multi-valued attribute forms an
++ordered list of values - the order of the repeated attributes defines
++the order of the values. An empty multi-valued attribute (`key[]=\n`)
++acts to clear any previous entries and reset the list.
++
++In all cases, all bytes are treated as-is (i.e., there is no quoting,
+ and one cannot transmit a value with newline or NUL in it). The list of
+ attributes is terminated by a blank line or end-of-file.
+
+@@ Documentation/git-credential.txt: empty string.
+ Components which are missing from the URL (e.g., there is no
+ username in the example above) will be left unset.
+
++`wwwauth[]`::
++
++ When an HTTP response is received by Git that includes one or more
++ 'WWW-Authenticate' authentication headers, these will be passed by Git
++ to credential helpers.
+++
++Each 'WWW-Authenticate' header value is passed as a multi-valued
++attribute 'wwwauth[]', where the order of the attributes is the same as
++they appear in the HTTP response. This attribute is 'one-way' from Git
++to pass additional information to credential helpers.
++
+ Unrecognised attributes are silently discarded.
+
+ GIT
+
+ ## credential.c ##
+@@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,
+ fprintf(fp, "%s=%s\n", key, value);
+ }
+
++static void credential_write_strvec(FILE *fp, const char *key,
++ const struct strvec *vec)
++{
++ int i = 0;
++ const char *full_key = xstrfmt("%s[]", key);
++ for (; i < vec->nr; i++) {
++ credential_write_item(fp, full_key, vec->v[i], 0);
++ }
++ free((void*)full_key);
++}
++
+ void credential_write(const struct credential *c, FILE *fp)
+ {
+ credential_write_item(fp, "protocol", c->protocol, 1);
+@@ credential.c: void credential_write(const struct credential *c, FILE *fp)
+ credential_write_item(fp, "path", c->path, 0);
+ credential_write_item(fp, "username", c->username, 0);
+ credential_write_item(fp, "password", c->password, 0);
++ credential_write_strvec(fp, "wwwauth", &c->wwwauth_headers);
+ }
+
+ static int run_credential_helper(struct credential *c,
+
## t/helper/test-credential-helper-replay.sh (new) ##
@@
+cmd=$1
@@ t/helper/test-credential-helper-replay.sh (new)
+fi
## t/t5556-http-auth.sh ##
-@@ t/t5556-http-auth.sh: PID_FILE="$(pwd)"/pid-file.pid
- SERVER_LOG="$(pwd)"/OUT.server.log
+@@ t/t5556-http-auth.sh: PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
+ SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
@@ t/t5556-http-auth.sh: start_http_server () {
stop_http_server &&
- rm -f OUT.*
+ rm -f OUT.* &&
-+ rm -f *.cred
++ rm -f *.cred &&
++ rm -f auth.config
}
test_expect_success 'http auth anonymous no challenge' '
-@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
+ test_when_finished "per_test_cleanup" &&
+- start_http_server &&
++
++ cat >auth.config <<-EOF &&
++ [auth]
++ allowAnonymous = true
++ EOF
++
++ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ # Attempt to read from a protected repository
git ls-remote $ORIGIN_URL
'
@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
+ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+ export USERPASS64 &&
+
-+ start_http_server \
-+ --auth=basic:realm=\"example.com\" \
-+ --auth-token=basic:$USERPASS64 &&
++ cat >auth.config <<-EOF &&
++ [auth]
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
++ EOF
++
++ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ cat >get-expected.cred <<-EOF &&
+ protocol=http
@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
+ test_cmp store-expected.cred store-actual.cred
+'
+
++test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '
++ test_when_finished "per_test_cleanup" &&
++ # base64("alice:secret-passwd")
++ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
++ export USERPASS64 &&
++
++ cat >auth.config <<-EOF &&
++ [auth]
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
++ extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
++ EOF
++
++ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
++
++ cat >get-expected.cred <<-EOF &&
++ protocol=http
++ host=$HOST_PORT
++ wwwauth[]=basic realm="example.com"
++ wwwauth[]=bEaRer auThoRiTy="id.example.com"
++ EOF
++
++ cat >store-expected.cred <<-EOF &&
++ protocol=http
++ host=$HOST_PORT
++ username=alice
++ password=secret-passwd
++ EOF
++
++ cat >get-response.cred <<-EOF &&
++ protocol=http
++ host=$HOST_PORT
++ username=alice
++ password=secret-passwd
++ EOF
++
++ git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
++
++ test_cmp get-expected.cred get-actual.cred &&
++ test_cmp store-expected.cred store-actual.cred
++'
++
++test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '
++ test_when_finished "per_test_cleanup" &&
++ # base64("alice:secret-passwd")
++ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
++ export USERPASS64 &&
++
++ cat >auth.config <<-EOF &&
++ [auth]
++ challenge = "bearer:authority=\"id.example.com\"\\n q=1\\n \\t p=0"
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
++ EOF
++
++ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
++
++ cat >get-expected.cred <<-EOF &&
++ protocol=http
++ host=$HOST_PORT
++ wwwauth[]=bearer authority="id.example.com" q=1 p=0
++ wwwauth[]=basic realm="example.com"
++ EOF
++
++ cat >store-expected.cred <<-EOF &&
++ protocol=http
++ host=$HOST_PORT
++ username=alice
++ password=secret-passwd
++ EOF
++
++ cat >get-response.cred <<-EOF &&
++ protocol=http
++ host=$HOST_PORT
++ username=alice
++ password=secret-passwd
++ EOF
++
++ git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
++
++ test_cmp get-expected.cred get-actual.cred &&
++ test_cmp store-expected.cred store-actual.cred
++'
++
++test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '
++ test_when_finished "per_test_cleanup" &&
++ # base64("alice:secret-passwd")
++ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
++ export USERPASS64 &&
++
++ cat >auth.config <<-EOF &&
++ [auth]
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
++ extraheader = "WWW-Authenticate:"
++ extraheader = " "
++ extraheader = " bearer authority=\"id.example.com\""
++ EOF
++
++ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
++
++ cat >get-expected.cred <<-EOF &&
++ protocol=http
++ host=$HOST_PORT
++ wwwauth[]=basic realm="example.com"
++ wwwauth[]=bearer authority="id.example.com"
++ EOF
++
++ cat >store-expected.cred <<-EOF &&
++ protocol=http
++ host=$HOST_PORT
++ username=alice
++ password=secret-passwd
++ EOF
++
++ cat >get-response.cred <<-EOF &&
++ protocol=http
++ host=$HOST_PORT
++ username=alice
++ password=secret-passwd
++ EOF
++
++ git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
++
++ test_cmp get-expected.cred get-actual.cred &&
++ test_cmp store-expected.cred store-actual.cred
++'
++
+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
+ test_when_finished "per_test_cleanup" &&
+ # base64("alice:secret-passwd")
+ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+ export USERPASS64 &&
+
-+ start_http_server \
-+ --auth=foobar:alg=test\ widget=1 \
-+ --auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
-+ --auth=basic:realm=\"example.com\" \
-+ --auth-token=basic:$USERPASS64 &&
++ cat >auth.config <<-EOF &&
++ [auth]
++ challenge = "foobar:alg=test widget=1"
++ challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
++ EOF
++
++ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ cat >get-expected.cred <<-EOF &&
+ protocol=http
@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
+ # base64("alice:secret-passwd")
+ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
+ export USERPASS64 &&
-+ start_http_server \
-+ --auth=bearer:authority=\"id.example.com\"\ q=1\ p=0 \
-+ --auth=basic:realm=\"example.com\" \
-+ --auth-token=basic:$USERPASS64 &&
++
++ cat >auth.config <<-EOF &&
++ [auth]
++ challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
++ EOF
++
++ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ cat >get-expected.cred <<-EOF &&
+ protocol=httpSignature · 1 line
gitgitgadget
PATCH v5 1/10 daemon: libify socket setup and option functions
From: Matthew John Cheetham <[email protected]>
Extract functions for setting up listening sockets and keep-alive options from daemon.c to new daemon-utils.{c,h} files. Remove direct dependencies on global state by inlining the behaviour at the callsites for all libified functions.
--- Makefile | 1 + daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 23 ++++++ daemon.c | 214 +------------------------------------------------ 4 files changed, 237 insertions(+), 210 deletions(-) create mode 100644 daemon-utils.c create mode 100644 daemon-utils.h
Makefile +1 -0
@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o LIB_OBJS += csum-file.o LIB_OBJS += ctype.o LIB_OBJS += date.o +LIB_OBJS += daemon-utils.o LIB_OBJS += decorate.o LIB_OBJS += delta-islands.o LIB_OBJS += diagnose.o
daemon-utils.c new file +209 -0
209 lines not on this page · raw message
daemon-utils.h new file +23 -0
@@ -0,0 +1,23 @@ +#ifndef DAEMON_UTILS_H +#define DAEMON_UTILS_H + +#include "git-compat-util.h" +#include "string-list.h" + +typedef void (*log_fn )(const char *msg, ...); + +struct socketlist { +int *list; +size_t nr; +size_t alloc; +}; + +/* Enable sending of keep-alive messages on the socket. */ +void set_keep_alive (int sockfd,log_fn logerror); + +/* Setup a number of sockets to listen on the provided addresses. */ +void socksetup (struct string_list *listen_addr,int listen_port, +struct socketlist *socklist,int reuseaddr, +log_fn logerror); + +#endif
daemon.c +4 -210
246 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v5 01/10] daemon: libify socket setup and option functions
Quoted 6 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Extract functions for setting up listening sockets and keep-alive options from
daemon.cto newdaemon-utils.{c,h}files. Remove direct dependencies on global state by inlining the behaviour at the callsites for all libified functions.
Thanks for making this change, the reduced code duplication should make the common daemon-related code more maintainable.
For reference, I used
'git blame -s -b -C -C -C master..<this patch> -- daemon-utils.c'
to help identify which lines in 'daemon-utils.c' were changed from their original implementation in 'daemon.c'. I'll try to rearrange the diff to show those differences more directly.
The first main change I see is that 'logerror' and 'reuseaddr' are changed from global references to arguments in 'set_keep_alive()', 'setup_named_sock()' (same for 'NO_IPV6' defined and undefined), and 'socksetup()':
Review
Quoted 12 lines
(quoted diff) +7
-3-static void set_keep_alive(int sockfd) +void set_keep_alive(int sockfd, log_fn logerror) -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist) +static int setup_named_sock(char *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror) -static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist) +void socksetup(struct string_list *listen_addr, int listen_port, + struct socketlist *socklist, int reuseaddr, + log_fn logerror)
The external calls in 'daemon.c' to 'set_keep_alive()' and 'socksetup()' are updated to pass the 'logerror()' function and global 'reuseaddr' as arguments, so there isn't any change in behavior.
Quoted 10 lines
daemon.c +1
-1@@ -759,7 +748,7 @@ static int execute(void)if (addr)loginfo ("Connection from %s:%s" , addr, port); -set_keep_alive (0 ); +set_keep_alive (0 , logerror);alarm (init_timeout ? init_timeout : timeout); pktlen =packet_read (0 , packet_buffer,sizeof (packet_buffer),0 );alarm (0 );
daemon.c static int serve(struct string_list *listen_addr, int listen_port, · line 1046
Quoted 9 lines
daemon.c +2
-1@@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port, {struct socketlist socklist = {NULL ,0 ,0 }; -socksetup (listen_addr, listen_port, &socklist); +socksetup (listen_addr, listen_port, &socklist, reuseaddr, + logerror);if (socklist.nr ==0 )die ("unable to allocate any listen sockets on port %u" , listen_port);
The other notable change is moving the 'if (!reusaddr) return 0' block in 'set_reuse_addr()' to its callers in both 'setup_named_sock()'s:
Quoted 6 lines
daemon-utils.c new file +6
-0+#ifndef NO_IPV6 + +static int setup_named_sock (char *listen_addr,int listen_port, +struct socketlist *socklist,int reuseaddr, +log_fn logerror) +{
...
Quoted 5 lines
daemon-utils.c new file +5
-0+if (reuseaddr &&set_reuse_addr (sockfd)) { +logerror ("Could not set SO_REUSEADDR: %s" ,strerror (errno)); +close (sockfd); +continue ; + }
...
Quoted 8 lines
daemon-utils.c new file +8
-0+} + +#else /* NO_IPV6 */ + +static int setup_named_sock (char *listen_addr,int listen_port, +struct socketlist *socklist,int reuseaddr, +log_fn logerror) +{
...
Quoted 5 lines
daemon-utils.c new file +5
-0+if (reuseaddr &&set_reuse_addr (sockfd)) { +logerror ("Could not set SO_REUSEADDR: %s" ,strerror (errno)); +close (sockfd); +return 0 ; + }
...
daemon-utils.c line 29
Quoted 3 lines
daemon-utils.c new file +3
-0+} + +#endif
Where, previously, that region looked like:
Quoted 4 lines
daemon.c +0
-4-#ifndef NO_IPV6 - -static int setup_named_sock (char *listen_addr,int listen_port,struct socketlist *socklist) -{
...
Quoted 5 lines
daemon.c +0
-5-if (set_reuse_addr (sockfd)) { -logerror ("Could not set SO_REUSEADDR: %s" ,strerror (errno)); -close (sockfd); -continue ; - }
...
Quoted 6 lines
daemon.c +0
-6-} - -#else /* NO_IPV6 */ - -static int setup_named_sock (char *listen_addr,int listen_port,struct socketlist *socklist) -{
...
Quoted 5 lines
daemon.c +0
-5-if (set_reuse_addr (sockfd)) { -logerror ("Could not set SO_REUSEADDR: %s" ,strerror (errno)); -close (sockfd); -return 0 ; - }
...
daemon.c static void child_handler(int signo) · line 963
Quoted 3 lines
daemon.c +0
-3-} - -#endif
'reuseaddr' is passed into 'setup_named_sock()' from 'socksetup()' calls in 'daemon.c', so this also won't result in changed behavior.
Otherwise, you only expose functions & types that are called in 'daemon.c' (the rest are still static), and everything else is a verbatim copy. Looks good to me!
Re: [PATCH v5 01/10] daemon: libify socket setup and option functions
Quoted 11 lines · Victoria Dye
Quoted 6 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Extract functions for setting up listening sockets and keep-alive options from
daemon.cto newdaemon-utils.{c,h}files. Remove direct dependencies on global state by inlining the behaviour at the callsites for all libified functions.Thanks for making this change, the reduced code duplication should make the common daemon-related code more maintainable.
Quoted 6 lines
For reference, I used
'git blame -s -b -C -C -C master..<this patch> --
daemon-utils.c'to help identify which lines in '
daemon-utils.c' were changed from their original implementation in 'daemon.c'.
Neat trick! Thanks for sharing. Using --color-moved was giving similar results, but with a lot more tracking back-and-forth to see what the differences were.
I agree with your assessment on this patch that the differences are valid, safe, and desired.
Thanks,
-Stolee
PATCH v5 2/10 daemon: libify child process handling functions
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Extract functions and structures for managing child processes started from the parent daemon-like process from daemon.c to the new shared daemon-utils.{c,h} files.
--- daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 15 ++++++++ daemon.c | 92 +++----------------------------------------------- 3 files changed, 97 insertions(+), 87 deletions(-)
daemon-utils.c +77 -0
@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port, } } } + +static int addrcmp (const struct sockaddr_storage *s1, +const struct sockaddr_storage *s2) +{ +const struct sockaddr *sa1 = (const struct sockaddr *) s1; +const struct sockaddr *sa2 = (const struct sockaddr *) s2; + +if (sa1->sa_family != sa2->sa_family) +return sa1->sa_family - sa2->sa_family; +if (sa1->sa_family ==AF_INET ) +return memcmp (&((struct sockaddr_in *)s1)->sin_addr, + &((struct sockaddr_in *)s2)->sin_addr, +sizeof (struct in_addr )); +#ifndef NO_IPV6 +if (sa1->sa_family ==AF_INET6 ) +return memcmp (&((struct sockaddr_in6 *)s1)->sin6_addr, + &((struct sockaddr_in6 *)s2)->sin6_addr, +sizeof (struct in6_addr )); +#endif +return 0 ; +} + +void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, +struct child *firstborn ,unsigned int *live_children) +{ +struct child *newborn, **cradle; + +CALLOC_ARRAY (newborn,1 ); + live_children++; +memcpy (&newborn->cld, cld,sizeof (*cld)); +memcpy (&newborn->address, addr, addrlen); +for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) +if (!addrcmp (&(*cradle)->address, &newborn->address)) +break ; + newborn->next = *cradle; + *cradle = newborn; +} + +void kill_some_child (struct child *firstborn) +{ +const struct child *blanket, *next; + +if (!(blanket = firstborn)) +return ; + +for (; (next = blanket->next); blanket = next) +if (!addrcmp (&blanket->address, &next->address)) { +kill (blanket->cld.pid,SIGTERM ); +break ; + } +} + +void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo) +{ +int status; +pid_t pid; + +struct child **cradle, *blanket; +for (cradle = &firstborn; (blanket = *cradle);) +if ((pid =waitpid (blanket->cld.pid, &status,WNOHANG )) >1 ) { +if (loginfo) { +const char *dead ="" ; +if (status) + dead =" (with error)" ; +loginfo ("[%" PRIuMAX"] Disconnected%s" , + (uintmax_t )pid, dead); + } + +/* remove the child */ + *cradle = blanket->next; + live_children--; +child_process_clear (&blanket->cld); +free (blanket); + }else + cradle = &blanket->next; +}
daemon-utils.h +15 -0
@@ -2,6 +2,7 @@#define DAEMON_UTILS_H #include "git-compat-util.h" +#include "run-command.h" #include "string-list.h" typedef void (*log_fn )(const char *msg, ...);
@@ -20,4 +21,18 @@ void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist, int reuseaddr, log_fn logerror); +struct child { +struct child *next; +struct child_process cld; +struct sockaddr_storage address; +}; + +void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, +struct child *firstborn,unsigned int *live_children); + +void kill_some_child (struct child *firstborn); + +void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo); +#endif
daemon.c +5 -87
121 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v5 02/10] daemon: libify child process handling functions
Quoted 5 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Extract functions and structures for managing child processes started from the parent daemon-like process from
daemon.cto the new shareddaemon-utils.{c,h}files.
As with patch 1, it looks like the main changes here are changing global references to function arguments. Specifically, those variables are 'firstborn', 'live_children', and 'loginfo':
daemon-utils.c void socksetup(struct string_list *listen_addr, int listen_port, · line 263
Quoted 10 lines
daemon-utils.c +5
-3-static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen) +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, + struct child *firstborn , unsigned int *live_children) -static void kill_some_child(void) +void kill_some_child(struct child *firstborn) -static void check_dead_children(void) +void check_dead_children(struct child *firstborn, unsigned int *live_children, + log_fn loginfo)
Those values are provided by the callers in 'daemon.c'. The major change here is that 'live_children' is passed as a pointer, since its value is updated by difference is passing 'live_children' as a pointer, since its value is updated by 'check_dead_children()' and 'add_child()':
Quoted 22 lines
daemon.c +3
-3@@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)struct child_process cld =CHILD_PROCESS_INIT ;if (max_connections && live_children >= max_connections) { -kill_some_child (); +kill_some_child (firstborn);sleep (1 );/* give it some time to die */ -check_dead_children (); +check_dead_children (firstborn, &live_children, loginfo);if (live_children >= max_connections) {close (incoming);logerror ("Too many children, dropping connection" );@@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)if (start_command (&cld))logerror ("unable to fork" );else -add_child (&cld, addr, addrlen); +add_child (&cld, addr, addrlen, firstborn, &live_children); }static void child_handler (int signo)
daemon.c static int service_loop(struct socketlist *socklist) · line 868
Quoted 8 lines
daemon.c +1
-1@@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)for (;;) {int i; -check_dead_children (); +check_dead_children (firstborn, &live_children, loginfo);if (poll (pfd, socklist->nr,-1 ) <0 ) {if (errno !=EINTR ) {
However, I think that change to 'live_children' may have caused a bug. In 'check_dead_children()', you decrement the 'live_children' *pointer*. That changes its address, not its value:
Quoted 3 lines
daemon-utils.c +3
-0+void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo) +{
...
Quoted 1 line
daemon-utils.c +1
-0+ live_children--;
...
Quoted 1 line
+}
Same thing in 'add_child()':
Quoted 3 lines
daemon-utils.c +3
-0+void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, +struct child *firstborn ,unsigned int *live_children) +{
...
Quoted 1 line
daemon-utils.c +1
-0+ live_children++;
...
Quoted 1 line
+}
These should be changed to '(*live_children)--' and '(*live_children)++', respectively.
There's also one minor functional change in 'check_dead_children()', where an 'if (loginfo)' check is added guarding the call to 'loginfo()':
Quoted 3 lines
daemon-utils.c +3
-0+void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo) +{
...
Quoted 7 lines
daemon-utils.c +7
-0+if (loginfo) { +const char *dead ="" ; +if (status) + dead =" (with error)" ; +loginfo ("[%" PRIuMAX"] Disconnected%s" , + (uintmax_t )pid, dead); + }
...
Quoted 1 line
+}
I'm guessing this is done because a caller later in the series won't provide a 'loginfo', but if that's the case, it would help to note that in this patch's commit message.
The one other thing I noticed is that you removed the function documentation for 'kill_some_child()':
Quoted 6 lines
-/*
- * This gets called if the number of connections grows
- * past "max_connections".
- *
- * We kill the newest connection from a duplicate IP.
- */
Is there a reason why you removed it? Otherwise, it should be added back in
- probably in 'daemon-utils.h'?
Everything else here looks good.
Re: [PATCH v5 02/10] daemon: libify child process handling functions
Quoted 76 lines · Victoria Dye
Quoted 5 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Extract functions and structures for managing child processes started from the parent daemon-like process from
daemon.cto the new shareddaemon-utils.{c,h}files.As with patch 1, it looks like the main changes here are changing global references to function arguments. Specifically, those variables are 'firstborn', '
live_children', and 'loginfo':daemon-utils.c void socksetup(struct string_list *listen_addr, int listen_port, · line 263
Quoted 10 lines
daemon-utils.c +5
-3-static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen) +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen, + struct child *firstborn , unsigned int *live_children) -static void kill_some_child(void) +void kill_some_child(struct child *firstborn) -static void check_dead_children(void) +void check_dead_children(struct child *firstborn, unsigned int *live_children, + log_fn loginfo)Those values are provided by the callers in '
daemon.c'. The major change here is that 'live_children' is passed as a pointer, since its value is updated by difference is passing 'live_children' as a pointer, since its value is updated by 'check_dead_children()' and 'add_child()':Quoted 22 lines
daemon.c +3
-3@@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)struct child_process cld =CHILD_PROCESS_INIT ;if (max_connections && live_children >= max_connections) { -kill_some_child (); +kill_some_child (firstborn);sleep (1 );/* give it some time to die */ -check_dead_children (); +check_dead_children (firstborn, &live_children, loginfo);if (live_children >= max_connections) {close (incoming);logerror ("Too many children, dropping connection" );@@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)if (start_command (&cld))logerror ("unable to fork" );else -add_child (&cld, addr, addrlen); +add_child (&cld, addr, addrlen, firstborn, &live_children); }static void child_handler (int signo)daemon.c static int service_loop(struct socketlist *socklist) · line 868
Quoted 8 lines
daemon.c +1
-1@@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)for (;;) {int i; -check_dead_children (); +check_dead_children (firstborn, &live_children, loginfo);if (poll (pfd, socklist->nr,-1 ) <0 ) {if (errno !=EINTR ) {However, I think that change to '
live_children' may have caused a bug. In 'check_dead_children()', you decrement the 'live_children' *pointer*. That changes its address, not its value:Quoted 3 lines
daemon-utils.c +3
-0+void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo) +{...
Quoted 1 line
daemon-utils.c +1
-0+ live_children--;...
Quoted 1 line
+}
Same thing in '
add_child()':Quoted 3 lines
daemon-utils.c +3
-0+void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, +struct child *firstborn ,unsigned int *live_children) +{...
Quoted 6 lines
Quoted 1 line
daemon-utils.c +1
-0+ live_children++;...
Quoted 1 line
+}
These should be changed to '(*live_children)--' and '(*live_children)++', respectively.
Ah! You are correct; my bad. Will correct this in v6.
Quoted 15 lines
There's also one minor functional change in '
check_dead_children()', where an 'if (loginfo)' check is added guarding the call to 'loginfo()':Quoted 3 lines
daemon-utils.c +3
-0+void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo) +{...
Quoted 7 lines
daemon-utils.c +7
-0+if (loginfo) { +const char *dead ="" ; +if (status) + dead =" (with error)" ; +loginfo ("[%" PRIuMAX"] Disconnected%s" , + (uintmax_t )pid, dead); + }...
Quoted 5 lines
Quoted 1 line
+}
I'm guessing this is done because a caller later in the series won't provide a 'loginfo', but if that's the case, it would help to note that in this patch's commit message.
Will call this out in the commit message in v6.
Quoted 10 lines
The one other thing I noticed is that you removed the function documentation for '
kill_some_child()':Quoted 6 lines
-/*
- * This gets called if the number of connections grows
- * past "max_connections".
- *
- * We kill the newest connection from a duplicate IP.
- */
Quoted 2 lines
Is there a reason why you removed it? Otherwise, it should be added back in
- probably in 'daemon-utils.h'?
I removed it initially as it was referencing things like max_connections which no longer existed in the context of daemon-utils.{c,h}.
Next iteration I can restore the spirit of the comment, that this should be called when the maximimum number of connections has been reached, in order to kill the newest connection from a duplicate IP.
Quoted 1 line
Everything else here looks good.
Thanks,
Matthew
PATCH v5 3/10 daemon: rename some esoteric/laboured terminology
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Rename some of the variables and function arguments used to manage child processes. The existing names are esoteric; stretching an analogy too far to the point of being confusing to understand.
Rename "firstborn" to simply "first", "newborn" to "new_cld", "blanket" to "current" and "cradle" to "ptr".
--- daemon-utils.c | 46 +++++++++++++++++++++++----------------------- daemon-utils.h | 6 +++--- daemon.c | 10 +++++----- 3 files changed, 31 insertions(+), 31 deletions(-)
daemon-utils.c +23 -23
@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1, }void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, -struct child *firstborn ,unsigned int *live_children) +struct child *first,unsigned int *live_children) { -struct child *newborn, **cradle; +struct child *new_cld, **current; -CALLOC_ARRAY (newborn,1 ); +CALLOC_ARRAY (new_cld,1 ); live_children++; -memcpy (&newborn->cld, cld,sizeof (*cld)); -memcpy (&newborn->address, addr, addrlen); -for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) -if (!addrcmp (&(*cradle)->address, &newborn->address)) +memcpy (&new_cld->cld, cld,sizeof (*cld)); +memcpy (&new_cld->address, addr, addrlen); +for (current = &first; *current; current = &(*current)->next) +if (!addrcmp (&(*current)->address, &new_cld->address))break ; - newborn->next = *cradle; - *cradle = newborn; + new_cld->next = *current; + *current = new_cld; } -void kill_some_child (struct child *firstborn) +void kill_some_child (struct child *first) { -const struct child *blanket, *next; +const struct child *current, *next; -if (!(blanket = firstborn)) +if (!(current = first))return ; -for (; (next = blanket->next); blanket = next) -if (!addrcmp (&blanket->address, &next->address)) { -kill (blanket->cld.pid,SIGTERM ); +for (; (next = current->next); current = next) +if (!addrcmp (¤t->address, &next->address)) { +kill (current->cld.pid,SIGTERM );break ; } } -void check_dead_children (struct child *firstborn,unsigned int *live_children, +void check_dead_children (struct child *first,unsigned int *live_children,log_fn loginfo) {int status;pid_t pid; -struct child **cradle, *blanket; -for (cradle = &firstborn; (blanket = *cradle);) -if ((pid =waitpid (blanket->cld.pid, &status,WNOHANG )) >1 ) { +struct child **ptr, *current; +for (ptr = &first; (current = *ptr);) +if ((pid =waitpid (current->cld.pid, &status,WNOHANG )) >1 ) {if (loginfo) {const char *dead ="" ;if (status)
@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children, }/* remove the child */ - *cradle = blanket->next; + *ptr = current->next; live_children--; -child_process_clear (&blanket->cld); -free (blanket); +child_process_clear (¤t->cld); +free (current); }else - cradle = &blanket->next; + ptr = ¤t->next; }
daemon-utils.h +3 -3
@@ -28,11 +28,11 @@ struct child { };void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, -struct child *firstborn,unsigned int *live_children); +struct child *first,unsigned int *live_children); -void kill_some_child (struct child *firstborn); +void kill_some_child (struct child *first); -void check_dead_children (struct child *firstborn,unsigned int *live_children, +void check_dead_children (struct child *first,unsigned int *live_children,log_fn loginfo);#endif
daemon.c +5 -5
@@ -789,7 +789,7 @@ static int max_connections = 32;static unsigned int live_children; -static struct child *firstborn; +static struct child *first_child;static struct strvec cld_argv =STRVEC_INIT ;static void handle (int incoming,struct sockaddr *addr,socklen_t addrlen)
@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)struct child_process cld =CHILD_PROCESS_INIT ;if (max_connections && live_children >= max_connections) { -kill_some_child (firstborn); +kill_some_child (first_child);sleep (1 );/* give it some time to die */ -check_dead_children (firstborn, &live_children, loginfo); +check_dead_children (first_child, &live_children, loginfo);if (live_children >= max_connections) {close (incoming);logerror ("Too many children, dropping connection" );
@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)if (start_command (&cld))logerror ("unable to fork" );else -add_child (&cld, addr, addrlen, firstborn, &live_children); +add_child (&cld, addr, addrlen, first_child, &live_children); }static void child_handler (int signo)
@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)for (;;) {int i; -check_dead_children (firstborn, &live_children, loginfo); +check_dead_children (first_child, &live_children, loginfo);if (poll (pfd, socklist->nr,-1 ) <0 ) {if (errno !=EINTR ) {
Signature · 1 line
gitgitgadget
Re: [PATCH v5 03/10] daemon: rename some esoteric/laboured terminology
Quoted 8 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Rename some of the variables and function arguments used to manage child processes. The existing names are esoteric; stretching an analogy too far to the point of being confusing to understand.
Rename "firstborn" to simply "first", "newborn" to "
new_cld", "blanket" to "current" and "cradle" to "ptr".
Thanks for this, I agree that the new names make the code much easier to read.
daemon.c static int max_connections = 32; · line 792
Quoted 10 lines
daemon.c +1
-1@@ -789,7 +789,7 @@ static int max_connections = 32;static unsigned int live_children; -static struct child *firstborn; +static struct child *first_child;
minor nit: you changed "firstborn" to "first" in 'daemon-utils.c' (aligning with the commit message), but it's "first_child" here. If you end up re-rolling, it would be nice to make the names consistent across both files (could be 'first', 'first_child', 'first_cld', or anything really).
Re: [PATCH v5 03/10] daemon: rename some esoteric/laboured terminology
Quoted 21 lines · Victoria Dye
Quoted 8 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Rename some of the variables and function arguments used to manage child processes. The existing names are esoteric; stretching an analogy too far to the point of being confusing to understand.
Rename "firstborn" to simply "first", "newborn" to "
new_cld", "blanket" to "current" and "cradle" to "ptr".Thanks for this, I agree that the new names make the code much easier to read.
Quoted 8 lines
daemon.c +0
-0@@ -789,7 +789,7 @@ static int max_connections = 32;static unsigned int live_children;
Quoted 8 lines
daemon.c static int max_connections = 32; · line 792
Quoted 2 lines
daemon.c +1
-1-static struct child *firstborn; +static struct child *first_child;minor nit: you changed "firstborn" to "first" in '
daemon-utils.c' (aligning with the commit message), but it's "first_child" here. If you end up re-rolling, it would be nice to make the names consistent across both files (could be 'first', 'first_child', 'first_cld', or anything really).
Fair point. There's no technical reason to keep them named differently between the 'libified' functions, and the actual variables for the callers. I shall align these to first_child in the next iteration.
Thanks,
Matthew
PATCH v5 4/10 test-http-server: add stub HTTP server test helper
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Introduce a mini HTTP server helper that in the future will be enhanced to provide a frontend for the git-http-backend, with support for arbitrary authentication schemes.
Right now, test-http-server is a pared-down copy of the git-daemon that always returns a 501 Not Implemented response to all callers.
--- Makefile | 1 + contrib/buildsystems/CMakeLists.txt | 11 +- t/helper/.gitignore | 1 + t/helper/test-http-server.c | 385 ++++++++++++++++++++++++++++ 4 files changed, 396 insertions(+), 2 deletions(-) create mode 100644 t/helper/test-http-server.c
Makefile +1 -0
@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o # Do not add more tests here unless they have extra dependencies. Add # them in TEST_BUILTINS_OBJS above. TEST_PROGRAMS_NEED_X += test-fake-ssh +TEST_PROGRAMS_NEED_X += test-http-server TEST_PROGRAMS_NEED_X += test-tool TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))
contrib/buildsystems/CMakeLists.txt +9 -2
@@ -961,6 +961,9 @@ if(BUILD_TESTING) add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c) target_link_libraries(test-fake-ssh common-main) +add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c) +target_link_libraries(test-http-server common-main) + #reftable-tests parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS") list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/")
@@ -980,6 +983,11 @@ if(MSVC) PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) set_target_properties(test-fake-ssh test-tool PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) + + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) endif() #wrapper scripts
@@ -987,8 +995,7 @@ set(wrapper_scripts git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar) set(wrapper_test_scripts - test-fake-ssh test-tool) - + test-http-server test-fake-ssh test-tool) foreach(script ${wrapper_scripts}) file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
t/helper/.gitignore +1 -0
@@ -1,2 +1,3 @@ /test-tool /test-fake-ssh +/test-http-server
t/helper/test-http-server.c new file +385 -0
385 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v5 04/10] test-http-server: add stub HTTP server test helper
Quoted 8 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Introduce a mini HTTP server helper that in the future will be enhanced to provide a frontend for the git-http-backend, with support for arbitrary authentication schemes.
Right now, test-http-server is a pared-down copy of the git-daemon that always returns a 501 Not Implemented response to all callers.
Between your earlier response [1] and this iteration of the patch, all of the comments from my previous review [2] have been addressed. The changes to drop the dependency on cURL also look correct to me. Thanks!
[1] https://lore.kernel.org/git/AS2PR03MB98150C33F9704D2CA10A2EF9C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com/
[2] https://lore.kernel.org/git/[email protected]/
Quoted 9 lines
Signed-off-by Matthew John Cheetham <[email protected]>--- Makefile | 1 + contrib/buildsystems/CMakeLists.txt | 11 +- t/helper/.gitignore | 1 + t/helper/test-http-server.c | 385 ++++++++++++++++++++++++++++ 4 files changed, 396 insertions(+), 2 deletions(-) create mode 100644 t/helper/test-http-server.c
PATCH v5 5/10 test-http-server: add HTTP error response function
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Introduce a function to the test-http-server test helper to write more full and valid HTTP error responses, including all the standard response headers like Server and Date.
--- t/helper/test-http-server.c | 303 +++++++++++++++++++++++++++++++++++- t/t5556-http-auth.sh | 106 +++++++++++++ 2 files changed, 404 insertions(+), 5 deletions(-) create mode 100755 t/t5556-http-auth.sh
t/helper/test-http-server.c +298 -5
@@ -83,9 +83,297 @@ enum worker_result { WR_HANGUP = 1<<1, }; +/* +* Fields from a parsed HTTP request. +*/ +struct req { +struct strbuf start_line; + +const char *method; +const char *http_version; + +struct strbuf uri_path; +struct strbuf query_args; + +struct string_list header_list; +const char *content_type; +ssize_t content_length; +}; + +#define REQ__INIT { \ + .start_line = STRBUF_INIT, \ + .uri_path = STRBUF_INIT, \ + .query_args = STRBUF_INIT, \ + .header_list = STRING_LIST_INIT_NODUP, \ + .content_type = NULL, \ + .content_length = -1 \ + } + +static void req__release (struct req *req) +{ +strbuf_release (&req->start_line); + +strbuf_release (&req->uri_path); +strbuf_release (&req->query_args); + +string_list_clear (&req->header_list,0 ); +} + +static enum worker_result send_http_error ( +int fd, +int http_code,const char *http_code_name, +int retry_after_seconds,struct string_list *response_headers, +enum worker_result wr_in) +{ +struct strbuf response_header =STRBUF_INIT ; +struct strbuf response_content =STRBUF_INIT ; +struct string_list_item *h; +enum worker_result wr; + +strbuf_addf (&response_content,"Error: %d %s\r\n" , + http_code, http_code_name); +if (retry_after_seconds >0 ) +strbuf_addf (&response_content,"Retry-After: %d\r\n" , + retry_after_seconds); + +strbuf_addf (&response_header,"HTTP/1.1 %d %s\r\n" , http_code, http_code_name); +strbuf_addstr (&response_header,"Cache-Control: private\r\n" ); +strbuf_addstr (&response_header,"Content-Type: text/plain\r\n" ); +strbuf_addf (&response_header,"Content-Length: %d\r\n" , (int )response_content.len); +if (retry_after_seconds >0 ) +strbuf_addf (&response_header,"Retry-After: %d\r\n" , retry_after_seconds); +strbuf_addf ( &response_header,"Server: test-http-server/%s\r\n" , git_version_string); +strbuf_addf ( &response_header,"Date: %s\r\n" ,show_date (time (NULL ),0 ,DATE_MODE (RFC2822 ))); +if (response_headers) +for_each_string_list_item (h, response_headers) +strbuf_addf (&response_header,"%s\r\n" , h->string); +strbuf_addstr (&response_header,"\r\n" ); + +if (write_in_full (fd, response_header.buf, response_header.len) <0 ) { +logerror ("unable to write response header" ); + wr =WR_IO_ERROR ; + goto done; + } + +if (write_in_full (fd, response_content.buf, response_content.len) <0 ) { +logerror ("unable to write response content body" ); + wr =WR_IO_ERROR ; + goto done; + } + + wr = wr_in; + +done: +strbuf_release (&response_header); +strbuf_release (&response_content); + +return wr; +} + +/* +* Read the HTTP request up to the start of the optional message-body. +* We do this byte-by-byte because we have keep-alive turned on and +* cannot rely on an EOF. +* +* https://tools.ietf.org/html/rfc7230 +* +* We cannot call die() here because our caller needs to properly +* respond to the client and/or close the socket before this +* child exits so that the client doesn't get a connection reset +* by peer error. +*/ +static enum worker_result req__read (struct req *req,int fd) +{ +struct strbuf h =STRBUF_INIT ; +struct string_list start_line_fields =STRING_LIST_INIT_DUP ; +int nr_start_line_fields; +const char *uri_target; +const char *query; +char *hp; +const char *hv; + +enum worker_result result =WR_OK ; + +/* +* Read line 0 of the request and split it into component parts: +* +* <method> SP <uri-target> SP <HTTP-version> CRLF +* +*/ +if (strbuf_getwholeline_fd (&req->start_line, fd,'\n' ) ==EOF ) { + result =WR_OK |WR_HANGUP ; + goto done; + } + +strbuf_trim_trailing_newline (&req->start_line); + + nr_start_line_fields =string_list_split (&start_line_fields, + req->start_line.buf, +' ' ,-1 ); +if (nr_start_line_fields !=3 ) { +logerror ("could not parse request start-line '%s'" , + req->start_line.buf); + result =WR_IO_ERROR ; + goto done; + } + + req->method =xstrdup (start_line_fields.items[0 ].string); + req->http_version =xstrdup (start_line_fields.items[2 ].string); + + uri_target = start_line_fields.items[1 ].string; + +if (strcmp (req->http_version,"HTTP/1.1" )) { +logerror ("unsupported version '%s' (expecting HTTP/1.1)" , + req->http_version); + result =WR_IO_ERROR ; + goto done; + } + + query =strchr (uri_target,'?' ); + +if (query) { +strbuf_add (&req->uri_path, uri_target, (query - uri_target)); +strbuf_trim_trailing_dir_sep (&req->uri_path); +strbuf_addstr (&req->query_args, query +1 ); + }else { +strbuf_addstr (&req->uri_path, uri_target); +strbuf_trim_trailing_dir_sep (&req->uri_path); + } + +/* +* Read the set of HTTP headers into a string-list. +*/ +while (1 ) { +if (strbuf_getwholeline_fd (&h, fd,'\n' ) ==EOF ) + goto done; +strbuf_trim_trailing_newline (&h); + +if (!h.len) + goto done;/* a blank line ends the header */ + + hp =strbuf_detach (&h,NULL ); +string_list_append (&req->header_list, hp); + +/* also store common request headers as struct req members */ +if (skip_prefix (hp,"Content-Type: " , &hv)) { + req->content_type = hv; + }else if (skip_prefix (hp,"Content-Length: " , &hv)) { + req->content_length =strtol (hv, &hp,10 ); + } + } + +/* +* We do not attempt to read the <message-body>, if it exists. +* We let our caller read/chunk it in as appropriate. +*/ + +done: +string_list_clear (&start_line_fields,0 ); + +/* +* This is useful for debugging the request, but very noisy. +*/ +if (trace2_is_enabled ()) { +struct string_list_item *item; +trace2_printf ("%s: %s" ,TR2_CAT , req->start_line.buf); +trace2_printf ("%s: hver: %s" ,TR2_CAT , req->http_version); +trace2_printf ("%s: hmth: %s" ,TR2_CAT , req->method); +trace2_printf ("%s: path: %s" ,TR2_CAT , req->uri_path.buf); +trace2_printf ("%s: qury: %s" ,TR2_CAT , req->query_args.buf); +if (req->content_length >=0 ) +trace2_printf ("%s: clen: %d" ,TR2_CAT , req->content_length); +if (req->content_type) +trace2_printf ("%s: ctyp: %s" ,TR2_CAT , req->content_type); +for_each_string_list_item (item, &req->header_list) +trace2_printf ("%s: hdrs: %s" ,TR2_CAT , item->string); + } + +return result; +} + +static int is_git_request (struct req *req) +{ +static regex_t *smart_http_regex; +static int initialized; + +if (!initialized) { + smart_http_regex =xmalloc (sizeof (*smart_http_regex)); +/* +* This regular expression matches all dumb and smart HTTP +* requests that are currently in use, and defined in +* Documentation/gitprotocol-http.txt. +* +*/ +if (regcomp (smart_http_regex,"^/(HEAD|info/refs|" +"objects/info/[^/]+|git-(upload|receive)-pack)$" , +REG_EXTENDED )) { +warning ("could not compile smart HTTP regex" ); + smart_http_regex =NULL ; + } + initialized =1 ; + } + +return smart_http_regex && + !regexec (smart_http_regex, req->uri_path.buf,0 ,NULL ,0 ); +} + +static enum worker_result do__git (struct req *req) +{ +const char *ok ="HTTP/1.1 200 OK\r\n" ; +struct child_process cp =CHILD_PROCESS_INIT ; +int res; + +/* +* Note that we always respond with a 200 OK response even if the +* http-backend process exits with an error. This helper is intended +* only to be used to exercise the HTTP auth handling in the Git client, +* and specifically around authentication (not handled by http-backend). +* +* If we wanted to respond with a more 'valid' HTTP response status then +* we'd need to buffer the output of http-backend, wait for and grok the +* exit status of the process, then write the HTTP status line followed +* by the http-backend output. This is outside of the scope of this test +* helper's use at time of writing. +* +* The important auth responses (401) we are handling prior to getting +* to this point. +*/ +if (write (STDOUT_FILENO , ok,strlen (ok)) <0 ) +return error (_ ("could not send '%s'" ), ok); + +strvec_pushf (&cp.env,"REQUEST_METHOD=%s" , req->method); +strvec_pushf (&cp.env,"PATH_TRANSLATED=%s" , + req->uri_path.buf); +strvec_push (&cp.env,"SERVER_PROTOCOL=HTTP/1.1" ); +if (req->query_args.len) +strvec_pushf (&cp.env,"QUERY_STRING=%s" , + req->query_args.buf); +if (req->content_type) +strvec_pushf (&cp.env,"CONTENT_TYPE=%s" , + req->content_type); +if (req->content_length >=0 ) +strvec_pushf (&cp.env,"CONTENT_LENGTH=%" PRIdMAX, + (intmax_t )req->content_length); + cp.git_cmd =1 ; +strvec_push (&cp.args,"http-backend" ); + res =run_command (&cp); +close (STDOUT_FILENO ); +close (STDIN_FILENO ); +return !!res; +} + +static enum worker_result dispatch (struct req *req) +{ +if (is_git_request (req)) +return do__git (req); + +return send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 ,NULL , +WR_OK |WR_HANGUP ); +} +static enum worker_result worker (void ) { -const char *response ="HTTP/1.1 501 Not Implemented\r\n" ; +struct req req =REQ__INIT ;char *client_addr =getenv ("REMOTE_ADDR" );char *client_port =getenv ("REMOTE_PORT" );enum worker_result wr =WR_OK ;
@@ -96,11 +384,16 @@ static enum worker_result worker(void)set_keep_alive (0 , logerror);while (1 ) { -if (write_in_full (STDOUT_FILENO , response,strlen (response)) <0 ) { -logerror ("unable to write response" ); - wr =WR_IO_ERROR ; - } +req__release (&req); + +alarm (timeout); + wr =req__read (&req,0 ); +alarm (0 ); + +if (wr !=WR_OK ) +break ; + wr =dispatch (&req);if (wr !=WR_OK )break ; }
t/t5556-http-auth.sh new file +106 -0
106 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v5 05/10] test-http-server: add HTTP error response function
Quoted 5 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Introduce a function to the test-http-server test helper to write more full and valid HTTP error responses, including all the standard response headers like
ServerandDate.
It took me a second to figure out, but this patch combines the content of patches 4, 5, and 6 from the last iteration. After squashing those three patches from v4 together locally, the range-diff is actually pretty simple (see below).
Out of curiosity, why did you combine those patches? I don't feel strongly about changing it, but the smaller, incremental patches in the previous version were a bit easier to review.
In any case, this version addresses my feedback from [1], [2], and [3] - the explanatory comments are particularly helpful. Thanks!
[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/[email protected]/
[3] https://lore.kernel.org/git/[email protected]/
Range diff v4 (patches 4-6, squashed) vs. v5 (this patch)
4: 127827637e ! 5: 6f66bf146b test-http-server: add HTTP error response function
@@ Commit message
## t/helper/test-http-server.c ##
@@ t/helper/test-http-server.c: enum worker_result {
- WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),
+ WR_HANGUP = 1<<1,
};
+/*
@@ t/helper/test-http-server.c: enum worker_result {
+ hp = strbuf_detach(&h, NULL);
+ string_list_append(&req->header_list, hp);
+
-+ /* store common request headers separately */
++ /* also store common request headers as struct req members */
+ if (skip_prefix(hp, "Content-Type: ", &hv)) {
+ req->content_type = hv;
+ } else if (skip_prefix(hp, "Content-Length: ", &hv)) {
@@ t/helper/test-http-server.c: enum worker_result {
+
+ if (!initialized) {
+ smart_http_regex = xmalloc(sizeof(*smart_http_regex));
++ /*
++ * This regular expression matches all dumb and smart HTTP
++ * requests that are currently in use, and defined in
++ * Documentation/gitprotocol-http.txt.
++ *
++ */
+ if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
+ "objects/info/[^/]+|git-(upload|receive)-pack)$",
+ REG_EXTENDED)) {
@@ t/helper/test-http-server.c: enum worker_result {
+ !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+}
+
-+static enum worker_result do__git(struct req *req, const char *user)
++static enum worker_result do__git(struct req *req)
+{
+ const char *ok = "HTTP/1.1 200 OK\r\n";
+ struct child_process cp = CHILD_PROCESS_INIT;
+ int res;
+
-+ if (write(1, ok, strlen(ok)) < 0)
++ /*
++ * Note that we always respond with a 200 OK response even if the
++ * http-backend process exits with an error. This helper is intended
++ * only to be used to exercise the HTTP auth handling in the Git client,
++ * and specifically around authentication (not handled by http-backend).
++ *
++ * If we wanted to respond with a more 'valid' HTTP response status then
++ * we'd need to buffer the output of http-backend, wait for and grok the
++ * exit status of the process, then write the HTTP status line followed
++ * by the http-backend output. This is outside of the scope of this test
++ * helper's use at time of writing.
++ *
++ * The important auth responses (401) we are handling prior to getting
++ * to this point.
++ */
++ if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
+ return error(_("could not send '%s'"), ok);
+
-+ if (user)
-+ strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
-+
+ strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
+ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
+ req->uri_path.buf);
@@ t/helper/test-http-server.c: enum worker_result {
+ cp.git_cmd = 1;
+ strvec_push(&cp.args, "http-backend");
+ res = run_command(&cp);
-+ close(1);
-+ close(0);
++ close(STDOUT_FILENO);
++ close(STDIN_FILENO);
+ return !!res;
+}
+
+static enum worker_result dispatch(struct req *req)
+{
+ if (is_git_request(req))
-+ return do__git(req, NULL);
++ return do__git(req);
+
-+ return send_http_error(1, 501, "Not Implemented", -1, NULL,
++ return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
+ WR_OK | WR_HANGUP);
+}
+
@@ t/helper/test-http-server.c: enum worker_result {
char *client_port = getenv("REMOTE_PORT");
enum worker_result wr = WR_OK;
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
- set_keep_alive(0);
+ set_keep_alive(0, logerror);
while (1) {
-- if (write_in_full(1, response, strlen(response)) < 0) {
+- if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
- logerror("unable to write response");
- wr = WR_IO_ERROR;
- }
+ req__release(&req);
+
-+ alarm(init_timeout ? init_timeout : timeout);
++ alarm(timeout);
+ wr = req__read(&req, 0);
+ alarm(0);
+
-+ if (wr & WR_STOP_THE_MUSIC)
++ if (wr != WR_OK)
+ break;
+ wr = dispatch(&req);
- if (wr & WR_STOP_THE_MUSIC)
+ if (wr != WR_OK)
break;
}
@@ t/t5556-http-auth.sh (new)
+
+test_description='test http auth header and credential helper interop'
+
++TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
+
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
+# Setup a repository
+#
-+REPO_DIR="$(pwd)"/repo
++REPO_DIR="$TRASH_DIRECTORY"/repo
+
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
@@ t/t5556-http-auth.sh (new)
+# The server will shutdown if/when we delete it (this is easier than
+# killing it by PID).
+#
-+PID_FILE="$(pwd)"/pid-file.pid
-+SERVER_LOG="$(pwd)"/OUT.server.log
++PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
++SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
+
+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+
@@ t/t5556-http-auth.sh (new)
+
+test_expect_success 'http auth anonymous no challenge' '
+ test_when_finished "per_test_cleanup" &&
-+ start_http_server --allow-anonymous &&
++ start_http_server &&
+
+ # Attempt to read from a protected repository
+ git ls-remote $ORIGIN_URLRe: [PATCH v5 05/10] test-http-server: add HTTP error response function
Quoted 7 lines · Victoria Dye
Quoted 5 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Introduce a function to the test-http-server test helper to write more full and valid HTTP error responses, including all the standard response headers like
ServerandDate.
Quoted 8 lines
It took me a second to figure out, but this patch combines the content of patches 4, 5, and 6 from the last iteration. After squashing those three patches from v4 together locally, the range-diff is actually pretty simple (see below).
Out of curiosity, why did you combine those patches? I don't feel strongly about changing it, but the smaller, incremental patches in the previous version were a bit easier to review.
This is my mistake. I didn't intend to do this, and agree splitting them is easier to grok. I will restore this! My apologies! :-(
Quoted 166 lines
In any case, this version addresses my feedback from [1], [2], and [3] - the explanatory comments are particularly helpful. Thanks!
[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/[email protected]/
[3] https://lore.kernel.org/git/[email protected]/Range diff v4 (patches 4-6, squashed) vs. v5 (this patch)
4:
127827637e! 5:6f66bf146btest-http-server: add HTTP error response function@@ Commit message ## t/helper/test-http-server.c ## @@ t/helper/test-http-server.c: enum worker_result { - WR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP), + WR_HANGUP = 1<<1, }; +/* @@ t/helper/test-http-server.c: enum worker_result { + hp = strbuf_detach(&h, NULL); + string_list_append(&req->header_list, hp); + -+ /* store common request headers separately */ ++ /* also store common request headers as struct req members */ + if (skip_prefix(hp, "Content-Type: ", &hv)) { + req->content_type = hv; + } else if (skip_prefix(hp, "Content-Length: ", &hv)) { @@ t/helper/test-http-server.c: enum worker_result { + + if (!initialized) { + smart_http_regex = xmalloc(sizeof(*smart_http_regex)); ++ /* ++ * This regular expression matches all dumb and smart HTTP ++ * requests that are currently in use, and defined in ++ * Documentation/gitprotocol-http.txt. ++ * ++ */ + if (regcomp(smart_http_regex, "^/(HEAD|info/refs|" + "objects/info/[^/]+|git-(upload|receive)-pack)$", + REG_EXTENDED)) { @@ t/helper/test-http-server.c: enum worker_result { + !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0); +} + -+static enum worker_result do__git(struct req *req, const char *user) ++static enum worker_result do__git(struct req *req) +{ + const char *ok = "HTTP/1.1 200 OK\r\n"; + struct child_process cp = CHILD_PROCESS_INIT; + int res; + -+ if (write(1, ok, strlen(ok)) < 0) ++ /* ++ * Note that we always respond with a 200 OK response even if the ++ * http-backend process exits with an error. This helper is intended ++ * only to be used to exercise the HTTP auth handling in the Git client, ++ * and specifically around authentication (not handled by http-backend). ++ * ++ * If we wanted to respond with a more 'valid' HTTP response status then ++ * we'd need to buffer the output of http-backend, wait for and grok the ++ * exit status of the process, then write the HTTP status line followed ++ * by the http-backend output. This is outside of the scope of this test ++ * helper's use at time of writing. ++ * ++ * The important auth responses (401) we are handling prior to getting ++ * to this point. ++ */ ++ if (write(STDOUT_FILENO, ok, strlen(ok)) < 0) + return error(_("could not send '%s'"), ok); + -+ if (user) -+ strvec_pushf(&cp.env, "REMOTE_USER=%s", user); -+ + strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method); + strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", + req->uri_path.buf); @@ t/helper/test-http-server.c: enum worker_result { + cp.git_cmd = 1; + strvec_push(&cp.args, "http-backend"); + res = run_command(&cp); -+ close(1); -+ close(0); ++ close(STDOUT_FILENO); ++ close(STDIN_FILENO); + return !!res; +} + +static enum worker_result dispatch(struct req *req) +{ + if (is_git_request(req)) -+ return do__git(req, NULL); ++ return do__git(req); + -+ return send_http_error(1, 501, "Not Implemented", -1, NULL, ++ return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL, + WR_OK | WR_HANGUP); +} + @@ t/helper/test-http-server.c: enum worker_result { char *client_port = getenv("REMOTE_PORT"); enum worker_result wr = WR_OK; @@ t/helper/test-http-server.c: static enum worker_result worker(void) - set_keep_alive(0); + set_keep_alive(0, logerror); while (1) { -- if (write_in_full(1, response, strlen(response)) < 0) { +- if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) { - logerror("unable to write response"); - wr = WR_IO_ERROR; - } + req__release(&req); + -+ alarm(init_timeout ? init_timeout : timeout); ++ alarm(timeout); + wr = req__read(&req, 0); + alarm(0); + -+ if (wr & WR_STOP_THE_MUSIC) ++ if (wr != WR_OK) + break; + wr = dispatch(&req); - if (wr & WR_STOP_THE_MUSIC) + if (wr != WR_OK) break; } @@ t/t5556-http-auth.sh (new) + +test_description='test http auth header and credential helper interop' + ++TEST_NO_CREATE_REPO=1 +. ./test-lib.sh + +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT + +# Setup a repository +# -+REPO_DIR="$(pwd)"/repo ++REPO_DIR="$TRASH_DIRECTORY"/repo + +# Setup some lookback URLs where test-http-server will be listening. +# We will spawn it directly inside the repo directory, so we avoid @@ t/t5556-http-auth.sh (new) +# The server will shutdown if/when we delete it (this is easier than +# killing it by PID). +# -+PID_FILE="$(pwd)"/pid-file.pid -+SERVER_LOG="$(pwd)"/OUT.server.log ++PID_FILE="$TRASH_DIRECTORY"/pid-file.pid ++SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log + +PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH + @@ t/t5556-http-auth.sh (new) + +test_expect_success 'http auth anonymous no challenge' ' + test_when_finished "per_test_cleanup" && -+ start_http_server --allow-anonymous && ++ start_http_server && + + # Attempt to read from a protected repository + git ls-remote $ORIGIN_URL
PATCH v5 6/10 test-http-server: add simple authentication
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add simple authentication to the test-http-server test helper. Authentication schemes and sets of valid tokens can be specified via a configuration file (in the normal gitconfig file format). Incoming requests are compared against the set of valid schemes and tokens and only approved if a matching token is found, or if no auth was provided and anonymous auth is enabled.
--- t/helper/test-http-server.c | 246 +++++++++++++++++++++++++++++++++++- 1 file changed, 244 insertions(+), 2 deletions(-)
t/helper/test-http-server.c +244 -2
@@ -7,6 +7,7 @@#include "version.h" #include "dir.h" #include "date.h" +#include "config.h" #define TR2_CAT "test-http-server"
@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =" [--timeout=<n>] [--max-connections=<n>]\n" " [--reuseaddr] [--pid-file=<file>]\n" " [--listen=<host_or_ipaddr>]* [--port=<n>]\n" +" [--auth-config=<file>]\n" ;static unsigned int timeout;
@@ -317,7 +319,7 @@ static int is_git_request(struct req *req) !regexec (smart_http_regex, req->uri_path.buf,0 ,NULL ,0 ); } -static enum worker_result do__git (struct req *req) +static enum worker_result do__git (struct req *req,const char *user) {const char *ok ="HTTP/1.1 200 OK\r\n" ;struct child_process cp =CHILD_PROCESS_INIT ;
@@ -341,6 +343,9 @@ static enum worker_result do__git(struct req *req)if (write (STDOUT_FILENO , ok,strlen (ok)) <0 )return error (_ ("could not send '%s'" ), ok); +if (user) +strvec_pushf (&cp.env,"REMOTE_USER=%s" , user); +strvec_pushf (&cp.env,"REQUEST_METHOD=%s" , req->method);strvec_pushf (&cp.env,"PATH_TRANSLATED=%s" , req->uri_path.buf);
@@ -362,10 +367,234 @@ static enum worker_result do__git(struct req *req)return !!res; } +enum auth_result { +/* No auth module matches the request. */ +AUTH_UNKNOWN =0 , + +/* Auth module denied the request. */ +AUTH_DENY =1 , + +/* Auth module successfully validated the request. */ +AUTH_ALLOW =2 , +}; + +struct auth_module { +char *scheme; +char *challenge_params; +struct string_list *tokens; +}; + +static int allow_anonymous; +static struct auth_module **auth_modules =NULL ; +static size_t auth_modules_nr =0 ; +static size_t auth_modules_alloc =0 ; +static struct strvec extra_headers =STRVEC_INIT ; + +static struct auth_module *create_auth_module (const char *scheme, +const char *challenge) +{ +struct auth_module *mod =xmalloc (sizeof (struct auth_module )); + mod->scheme =xstrdup (scheme); + mod->challenge_params = challenge ?xstrdup (challenge) :NULL ; +CALLOC_ARRAY (mod->tokens,1 ); +string_list_init_dup (mod->tokens); +return mod; +} + +static struct auth_module *get_auth_module (const char *scheme) +{ +int i; +struct auth_module *mod; +for (i =0 ; i < auth_modules_nr; i++) { + mod = auth_modules[i]; +if (!strcasecmp (mod->scheme, scheme)) +return mod; + } + +return NULL ; +} + +static int add_auth_module (struct auth_module *mod) +{ +if (get_auth_module (mod->scheme)) +return error ("duplicate auth scheme '%s'\n" , mod->scheme); + +ALLOC_GROW (auth_modules, auth_modules_nr +1 , auth_modules_alloc); + auth_modules[auth_modules_nr++] = mod; + +return 0 ; +} + +static int is_authed (struct req *req,const char **user,enum worker_result *wr) +{ +enum auth_result result =AUTH_UNKNOWN ; +struct string_list hdrs =STRING_LIST_INIT_NODUP ; +struct auth_module *mod; + +struct string_list_item *hdr; +struct string_list_item *token; +const char *v; +struct strbuf **split =NULL ; +int i; +char *challenge; + +/* +* Check all auth modules and try to validate the request. +* The first Authorization header that matches a known auth module +* scheme will be consulted to either approve or deny the request. +* If no module is found, or if there is no valid token, then 401 error. +* Otherwise, only permit the request if anonymous auth is enabled. +* It's atypical for user agents/clients to send multiple Authorization +* headers, but not explicitly forbidden or defined. +*/ +for_each_string_list_item (hdr, &req->header_list) { +if (skip_iprefix (hdr->string,"Authorization: " , &v)) { + split =strbuf_split_str (v,' ' ,2 ); +if (!split[0 ] || !split[1 ])continue ; + +/* trim trailing space ' ' */ +strbuf_setlen (split[0 ], split[0 ]->len -1 ); + + mod =get_auth_module (split[0 ]->buf); +if (mod) { + result =AUTH_DENY ; + +for_each_string_list_item (token, mod->tokens) { +if (!strcmp (split[1 ]->buf, token->string)) { + result =AUTH_ALLOW ; +break ; + } + } + + goto done; + } + } + } + +done: +switch (result) { +case AUTH_ALLOW : +trace2_printf ("%s: auth '%s' ALLOW" ,TR2_CAT , mod->scheme); + *user ="VALID_TEST_USER" ; + *wr =WR_OK ; +break ; + +case AUTH_DENY : +trace2_printf ("%s: auth '%s' DENY" ,TR2_CAT , mod->scheme); +/* fall-through */ + +case AUTH_UNKNOWN : +if (result !=AUTH_DENY && allow_anonymous) +break ; + +for (i =0 ; i < auth_modules_nr; i++) { + mod = auth_modules[i]; +if (mod->challenge_params) + challenge =xstrfmt ("WWW-Authenticate: %s %s" , + mod->scheme, + mod->challenge_params); +else + challenge =xstrfmt ("WWW-Authenticate: %s" , + mod->scheme); +string_list_append (&hdrs, challenge); + } + +for (i =0 ; i < extra_headers.nr; i++) +string_list_append (&hdrs, extra_headers.v[i]); + + *wr =send_http_error (STDOUT_FILENO ,401 ,"Unauthorized" ,-1 , + &hdrs, *wr); + } + +strbuf_list_free (split); +string_list_clear (&hdrs,0 ); + +return result ==AUTH_ALLOW || + (result ==AUTH_UNKNOWN && allow_anonymous); +} + +static int split_auth_param (const char *str,char **scheme,char **val,int required_val) +{ +struct strbuf **p =strbuf_split_str (str,':' ,2 ); + +if (!p[0 ]) +return -1 ; + +/* trim trailing ':' */ +if (p[1 ]) +strbuf_setlen (p[0 ], p[0 ]->len -1 ); + +if (required_val && !p[1 ]) +return -1 ; + + *scheme =strbuf_detach (p[0 ],NULL ); + +if (p[1 ]) + *val =strbuf_detach (p[1 ],NULL ); + +strbuf_list_free (p); +return 0 ; +} + +static int read_auth_config (const char *name,const char *val,void *data) +{ +int ret =0 ; +char *scheme =NULL ; +char *token =NULL ; +char *challenge =NULL ; +struct auth_module *mod =NULL ; + +if (!strcmp (name,"auth.challenge" )) { +if (split_auth_param (val, &scheme, &challenge,0 )) { + ret =error ("invalid auth challenge '%s'" , val); + goto cleanup; + } + + mod =create_auth_module (scheme, challenge); +if (add_auth_module (mod)) { + ret =error ("failed to add auth module '%s'" , val); + goto cleanup; + } + } +if (!strcmp (name,"auth.token" )) { +if (split_auth_param (val, &scheme, &token,1 )) { + ret =error ("invalid auth token '%s'" , val); + goto cleanup; + } + + mod =get_auth_module (scheme); +if (!mod) { + ret =error ("auth scheme not defined '%s'\n" , scheme); + goto cleanup; + } + +string_list_append (mod->tokens, token); + } +if (!strcmp (name,"auth.allowanonymous" )) { + allow_anonymous =git_config_bool (name, val); + } +if (!strcmp (name,"auth.extraheader" )) { +strvec_push (&extra_headers, val); + } + +cleanup: +free (scheme); +free (token); +free (challenge); + +return ret; +} +static enum worker_result dispatch (struct req *req) { +enum worker_result wr =WR_OK ; +const char *user =NULL ; + +if (!is_authed (req, &user, &wr)) +return wr; +if (is_git_request (req)) -return do__git (req); +return do__git (req, user);return send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 ,NULL ,WR_OK |WR_HANGUP );
@@ -624,6 +853,19 @@ int cmd_main(int argc, const char **argv) pid_file = v;continue ; } +if (skip_prefix (arg,"--auth-config=" , &v)) { +if (!strlen (v)) { +error ("invalid argument - missing file path" ); +usage (test_http_auth_usage); + } + +if (git_config_from_file (read_auth_config, v,NULL )) { +error ("failed to read auth config file '%s'" , v); +usage (test_http_auth_usage); + } + +continue ; + }fprintf (stderr,"error: unknown argument '%s'\n" , arg);usage (test_http_auth_usage);
Signature · 1 line
gitgitgadget
Re: [PATCH v5 06/10] test-http-server: add simple authentication
Quoted 35 lines · Matthew John Cheetham via GitGitGadget
t/helper/test-http-server.c +33
-0+static struct auth_module *create_auth_module (const char *scheme, +const char *challenge) +{ +struct auth_module *mod =xmalloc (sizeof (struct auth_module )); + mod->scheme =xstrdup (scheme); + mod->challenge_params = challenge ?xstrdup (challenge) :NULL ; +CALLOC_ARRAY (mod->tokens,1 ); +string_list_init_dup (mod->tokens); +return mod; +} + +static int add_auth_module (struct auth_module *mod) +{ +if (get_auth_module (mod->scheme)) +return error ("duplicate auth scheme '%s'\n" , mod->scheme); + +ALLOC_GROW (auth_modules, auth_modules_nr +1 , auth_modules_alloc); + auth_modules[auth_modules_nr++] = mod; + +return 0 ; +} +static int split_auth_param (const char *str,char **scheme,char **val,int required_val) +{ +struct strbuf **p =strbuf_split_str (str,':' ,2 ); + +if (!p[0 ]) +return -1 ; + +/* trim trailing ':' */ +if (p[1 ]) +strbuf_setlen (p[0 ], p[0 ]->len -1 ); + +if (required_val && !p[1 ])
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 537
Quoted 10 lines
t/helper/test-http-server.c +10
-0+return -1 ; + + *scheme =strbuf_detach (p[0 ],NULL ); + +if (p[1 ]) + *val =strbuf_detach (p[1 ],NULL ); + +strbuf_list_free (p); +return 0 ; +}
There's nothing really *new* in these functions in this iteration, just code moved from the option parsing/handling in 'cmd_main()' into dedicated functions. Looks good!
Quoted 4 lines
t/helper/test-http-server.c +4
-0+switch (result) { +case AUTH_ALLOW : +trace2_printf ("%s: auth '%s' ALLOW" ,TR2_CAT , mod->scheme); + *user ="VALID_TEST_USER" ;
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 488
Quoted 10 lines
t/helper/test-http-server.c +10
-0+ *wr =WR_OK ; +break ; + +case AUTH_DENY : +trace2_printf ("%s: auth '%s' DENY" ,TR2_CAT , mod->scheme); +/* fall-through */ + +case AUTH_UNKNOWN : +if (result !=AUTH_DENY && allow_anonymous) +break ;
I completely missed the "fall-through" comment in my last review [1], as you kindly pointed out [2]. ;) Given that, this makes sense to me.
[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/AS2PR03MB981593EB3382F9738D2CA3D7C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com/
Quoted 8 lines
t/helper/test-http-server.c +8
-0+ +for (i =0 ; i < auth_modules_nr; i++) { + mod = auth_modules[i]; +if (mod->challenge_params) + challenge =xstrfmt ("WWW-Authenticate: %s %s" , + mod->scheme, + mod->challenge_params); +else
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 506
Quoted 10 lines
t/helper/test-http-server.c +10
-0+ challenge =xstrfmt ("WWW-Authenticate: %s" , + mod->scheme); +string_list_append (&hdrs, challenge); + } + +for (i =0 ; i < extra_headers.nr; i++) +string_list_append (&hdrs, extra_headers.v[i]); + + *wr =send_http_error (STDOUT_FILENO ,401 ,"Unauthorized" ,-1 , + &hdrs, *wr);
The "extra_headers" configuration is new, and helps make the test server more flexible.
Quoted 24 lines
t/helper/test-http-server.c +24
-0+static int read_auth_config (const char *name,const char *val,void *data) +{ +int ret =0 ; +char *scheme =NULL ; +char *token =NULL ; +char *challenge =NULL ; +struct auth_module *mod =NULL ; + +if (!strcmp (name,"auth.challenge" )) { +if (split_auth_param (val, &scheme, &challenge,0 )) { + ret =error ("invalid auth challenge '%s'" , val); + goto cleanup; + } + + mod =create_auth_module (scheme, challenge); +if (add_auth_module (mod)) { + ret =error ("failed to add auth module '%s'" , val); + goto cleanup; + } + } +if (!strcmp (name,"auth.token" )) { +if (split_auth_param (val, &scheme, &token,1 )) { + ret =error ("invalid auth token '%s'" , val); + goto cleanup;
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 572
Quoted 10 lines
t/helper/test-http-server.c +10
-0+ } + + mod =get_auth_module (scheme); +if (!mod) { + ret =error ("auth scheme not defined '%s'\n" , scheme); + goto cleanup; + } + +string_list_append (mod->tokens, token); + }
I don't think this addresses the implicit option ordering requirement noted in [3]; instead of needing '--auth' before '--auth-token', this now needs 'auth.challenge' before 'auth.token' in the config file. While I'd prefer it if this could be rearranged so that the auth setup happens after all config parsing (so the order doesn't matter), if you want to leave it as-is please add a comment somewhere in this file explaining that requirement and/or add a note to the "auth scheme not defined" error message.
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 578
Quoted 6 lines
t/helper/test-http-server.c +6
-0+if (!strcmp (name,"auth.allowanonymous" )) { + allow_anonymous =git_config_bool (name, val); + } +if (!strcmp (name,"auth.extraheader" )) { +strvec_push (&extra_headers, val); + }
Is it worth printing a warning if the option found isn't any of the above? Something like "ignoring <config option>". This is a test helper, so user-friendliness isn't quite as important as it is for builtins, but the warning might be helpful to developers trying to use it in the future.
Quoted 43 lines
t/helper/test-http-server.c +29
-1+ +cleanup: +free (scheme); +free (token); +free (challenge); + +return ret; +} +static enum worker_result dispatch (struct req *req) { +enum worker_result wr =WR_OK ; +const char *user =NULL ; + +if (!is_authed (req, &user, &wr)) +return wr; +if (is_git_request (req)) -return do__git (req); +return do__git (req, user);return send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 ,NULL ,WR_OK |WR_HANGUP );@@ -624,6 +853,19 @@ int cmd_main(int argc, const char **argv) pid_file = v;continue ; } +if (skip_prefix (arg,"--auth-config=" , &v)) { +if (!strlen (v)) { +error ("invalid argument - missing file path" ); +usage (test_http_auth_usage); + } + +if (git_config_from_file (read_auth_config, v,NULL )) { +error ("failed to read auth config file '%s'" , v); +usage (test_http_auth_usage); + } + +continue ; + }fprintf (stderr,"error: unknown argument '%s'\n" , arg);usage (test_http_auth_usage);
Re: [PATCH v5 06/10] test-http-server: add simple authentication
Quoted 11 lines · Victoria Dye
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 537
Quoted 4 lines
t/helper/test-http-server.c +3
-0+static int split_auth_param (const char *str,char **scheme,char **val,int required_val) +{ +}There's nothing really *new* in these functions in this iteration, just code moved from the option parsing/handling in '
cmd_main()' into dedicated functions. Looks good!...
Quoted 8 lines
I completely missed the "fall-through" comment in my last review [1], as you kindly pointed out [2]. ;) Given that, this makes sense to me.
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 506
Quoted 2 lines
t/helper/test-http-server.c +2
-0+ *wr =send_http_error (STDOUT_FILENO ,401 ,"Unauthorized" ,-1 , + &hdrs, *wr);The "
extra_headers" configuration is new, and helps make the test server more flexible.
This is not limited to this single review message, but it is good to see "this part of the patch is good because ..." explicitly stated. I wish more people did so, in addition to pointing out what needs to be improved.
Thanks.
Re: [PATCH v5 06/10] test-http-server: add simple authentication
In reply to Victoria DyeQuoted 46 lines · Victoria Dye
Quoted 24 lines · Matthew John Cheetham via GitGitGadget
t/helper/test-http-server.c +24
-0+static int read_auth_config (const char *name,const char *val,void *data) +{ +int ret =0 ; +char *scheme =NULL ; +char *token =NULL ; +char *challenge =NULL ; +struct auth_module *mod =NULL ; + +if (!strcmp (name,"auth.challenge" )) { +if (split_auth_param (val, &scheme, &challenge,0 )) { + ret =error ("invalid auth challenge '%s'" , val); + goto cleanup; + } + + mod =create_auth_module (scheme, challenge); +if (add_auth_module (mod)) { + ret =error ("failed to add auth module '%s'" , val); + goto cleanup; + } + } +if (!strcmp (name,"auth.token" )) { +if (split_auth_param (val, &scheme, &token,1 )) { + ret =error ("invalid auth token '%s'" , val); + goto cleanup;t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 572
Quoted 10 lines
t/helper/test-http-server.c +10
-0+ } + + mod =get_auth_module (scheme); +if (!mod) { + ret =error ("auth scheme not defined '%s'\n" , scheme); + goto cleanup; + } + +string_list_append (mod->tokens, token); + }I don't think this addresses the implicit option ordering requirement noted in [3]; instead of needing '--auth' before '--auth-token', this now needs 'auth.challenge' before 'auth.token' in the config file. While I'd prefer it if this could be rearranged so that the auth setup happens after all config parsing (so the order doesn't matter), if you want to leave it as-is please add a comment somewhere in this file explaining that requirement and/or add a note to the "auth scheme not defined" error message.
Quoted 11 lines
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 578
Quoted 6 lines
t/helper/test-http-server.c +6
-0+if (!strcmp (name,"auth.allowanonymous" )) { + allow_anonymous =git_config_bool (name, val); + } +if (!strcmp (name,"auth.extraheader" )) { +strvec_push (&extra_headers, val); + }Is it worth printing a warning if the option found isn't any of the above? Something like "ignoring <config option>". This is a test helper, so user-friendliness isn't quite as important as it is for builtins, but the warning might be helpful to developers trying to use it in the future.
I tried this suggestion of adding a warning, but it felt wrong. You are correct in the first instance that it should really "just work" when specified in any order. Watch for the next iteration where I'll make it such you can specify them in any order :-)
Thanks,
Matthew
PATCH v5 7/10 http: replace unsafe size_t multiplication with st_mult
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Replace direct multiplication of two size_t parameters in curl response stream handling callback functions with st_mult to guard against overflows.
--- http.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
http.c +2 -2
@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;size_t fread_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct buffer *buffer = buffer_;if (size > buffer->buf.len - buffer->posn)
@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)size_t fwrite_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct strbuf *buffer = buffer_;strbuf_add (buffer, ptr, size);
Signature · 1 line
gitgitgadget
PATCH v5 8/10 strvec: expose strvec_push_nodup for external use
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Remove the static modifier from the existing strvec_push_nodup function and define the function is strvec.h to make it available for other callers, making it now possible to append to a struct strvec array without duplication.
--- strvec.c | 2 +- strvec.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-)
strvec.c +1 -1
@@ -10,7 +10,7 @@ void strvec_init(struct strvec *array)memcpy (array, &blank,sizeof (*array)); } -static void strvec_push_nodup (struct strvec *array,const char *value) +void strvec_push_nodup (struct strvec *array,const char *value) {if (array->v == empty_strvec) array->v =NULL ;
strvec.h +3 -0
@@ -43,6 +43,9 @@ struct strvec {*/ void strvec_init (struct strvec *); +/* Push a string onto the end of the array without copying. */ +void strvec_push_nodup (struct strvec *array,const char *value); +/* Push a copy of a string onto the end of the array. */ const char *strvec_push (struct strvec *,const char *);
Signature · 1 line
gitgitgadget
PATCH v5 9/10 http: read HTTP WWW-Authenticate response headers
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Read and store the HTTP WWW-Authenticate response headers made for a particular request.
This will allow us to pass important authentication challenge information to credential helpers or others that would otherwise have been lost.
According to RFC2616 Section 4.2 [1], header field names are not case-sensitive meaning when collecting multiple values for the same field name, we can just use the case of the first observed instance of each field name and no normalisation is required.
libcurl only provides us with the ability to read all headers recieved for a particular request, including any intermediate redirect requests or proxies. The lines returned by libcurl include HTTP status lines delinating any intermediate requests such as "HTTP/1.1 200". We use these lines to reset the strvec of WWW-Authenticate header values as we encounter them in order to only capture the final response headers.
The collection of all header values matching the WWW-Authenticate header is complicated by the fact that it is legal for header fields to be continued over multiple lines, but libcurl only gives us one line at a time.
In the future [2] we may be able to leverage functions to read headers from libcurl itself, but as of today we must do this ourselves.
[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/
--- credential.c | 1 + credential.h | 15 ++++++++ http.c | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 114 insertions(+)
credential.c +1 -0
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)free (c->username);free (c->password);string_list_clear (&c->helpers,0 ); +strvec_clear (&c->wwwauth_headers);credential_init (c); }
credential.h +15 -0
@@ -2,6 +2,7 @@#define CREDENTIAL_H #include "string-list.h" +#include "strvec.h" /** * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {*/ struct string_list helpers; +/** +* A `strvec` of WWW-Authenticate header values. Each string +* is the value of a WWW-Authenticate header in an HTTP response, +* in the order they were received in the response. +*/ +struct strvec wwwauth_headers; + +/** +* Internal use only. Used to keep track of split header fields +* in order to fold multiple lines into one value. +*/ + unsigned header_is_last_match:1; + unsigned approved:1, configured:1, quit:1,
@@ -130,6 +144,7 @@ struct credential {#define CREDENTIAL_INIT { \ .helpers = STRING_LIST_INIT_DUP, \ + .wwwauth_headers = STRVEC_INIT, \ }/* Initialize a credential structure, setting all fields to empty. */
http.c +98 -0
110 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v5 09/10] http: read HTTP WWW-Authenticate response headers
Quoted 7 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]> [...]
http.c +5
-0+ }else if (buf.len) { +const char *prev = values->v[values->nr -1 ]; +struct strbuf append =STRBUF_INIT ; +strbuf_addstr (&append, prev); +
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 260
Quoted 10 lines
http.c +10
-0+/* Join two non-empty values with a single space. */ +if (append.len) +strbuf_addch (&append,' ' ); + +strbuf_addbuf (&append, &buf); + +strvec_pop (values); +strvec_push_nodup (values,strbuf_detach (&append,NULL )); + } +
I've written something like the strvec_push_nodup() patch that preceded this myself for similar reasons, and as recently noted in [1] I think such a thing (although I implemented a different interface) might be useful in general.
But this really doesn't seem like a good justification for adding this new API. Let's instead do:
} else if (buf.len) {
const char *prev = values->v[values->nr - 1];
/* Join two non-empty values with a single space. */
const char *const sp = *prev ? " " : ""
strvec_pop(values);
strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
}There may be cases where a public strvec_push_nodup() simplifies things, but this doesn't seem like such a case, just use strvec_pushf() directly instead, and skip the strbuf & strbuf_detach().
I haven't compiled/tested the above, so there may e.g. be a typo in there. But I think the general concept should work in this case.
Re: [PATCH v5 09/10] http: read HTTP WWW-Authenticate response headers
Quoted 17 lines · Ævar Arnfjörð Bjarmason
Quoted 14 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]> [...]
http.c +12
-0+ }else if (buf.len) { +const char *prev = values->v[values->nr -1 ]; +struct strbuf append =STRBUF_INIT ; +strbuf_addstr (&append, prev); + +/* Join two non-empty values with a single space. */ +if (append.len) +strbuf_addch (&append,' ' ); + +strbuf_addbuf (&append, &buf); + +strvec_pop (values);
Quoted 8 lines
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 260
Quoted 3 lines
http.c +3
-0+strvec_push_nodup (values,strbuf_detach (&append,NULL )); + } +I've written something like the
strvec_push_nodup()patch that preceded this myself for similar reasons, and as recently noted in [1] I think such a thing (although I implemented a different interface) might be useful in general.
A fair point, and reading [1] I see there's some concerns about making the strvec interface more complicated w.r.t. ownership vs saving a xstrdup. In light of this, I'll drop the commit to add strvec_push_nodup.
Quoted 12 lines
But this really doesn't seem like a good justification for adding this new API. Let's instead do:
} else if (buf.len) { const char *prev = values->v[values->nr - 1]; /* Join two non-empty values with a single space. */ const char *const sp = *prev ? " " : "" strvec_pop(values); strvec_pushf(values, "%s%s%s", prev, sp, buf.buf); }
Quoted 8 lines
There may be cases where a public
strvec_push_nodup()simplifies things, but this doesn't seem like such a case, just usestrvec_pushf()directly instead, and skip the strbuf &strbuf_detach().I haven't compiled/tested the above, so there may e.g. be a typo in there. But I think the general concept should work in this case.
There's a bug in your suggestion. We're strvec_pop-ing from the array which also frees the previous value that we want to use to append to in the next call to strvec_pushf. We need to keep a copy of the previous header value around.
This should work instead (adding an xstrdup and free):
} else if (buf.len) {
char *prev = xstrdup(values->v[values->nr - 1]);
/* Join two non-empty values with a single space. */
const char *const sp = *prev ? " " : "";
strvec_pop(values);
strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
free(prev);
}Thanks,
Matthew
PATCH v5 10/10 credential: add WWW-Authenticate header to cred requests
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add the value of the WWW-Authenticate response header to credential requests. Credential helpers that understand and support HTTP authentication and authorization can use this standard header (RFC 2616 Section 14.47 [1]) to generate valid credentials.
WWW-Authenticate headers can contain information pertaining to the authority, authentication mechanism, or extra parameters/scopes that are required.
The current I/O format for credential helpers only allows for unique names for properties/attributes, so in order to transmit multiple header values (with a specific order) we introduce a new convention whereby a C-style array syntax is used in the property name to denote multiple ordered values for the same property.
In this case we send multiple wwwauth[] properties where the order that the repeated attributes appear in the conversation reflects the order that the WWW-Authenticate headers appeared in the HTTP response.
Add a set of tests to exercise the HTTP authentication header parsing and the interop with credential helpers. Credential helpers will receive WWW-Authenticate information in credential requests.
[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47
--- Documentation/git-credential.txt | 19 +- credential.c | 12 + t/helper/test-credential-helper-replay.sh | 14 ++ t/t5556-http-auth.sh | 270 +++++++++++++++++++++- 4 files changed, 312 insertions(+), 3 deletions(-) create mode 100755 t/helper/test-credential-helper-replay.sh
Documentation/git-credential.txt +18 -1
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline. The key may contain any bytes except `=`, newline, or NUL. The value may contain any bytes except newline or NUL. -In both cases, all bytes are treated as-is (i.e., there is no quoting, +Attributes with keys that end with C-style array brackets `[]` can have +multiple values. Each instance of a multi-valued attribute forms an +ordered list of values - the order of the repeated attributes defines +the order of the values. An empty multi-valued attribute (`key[]=\n`) +acts to clear any previous entries and reset the list. + +In all cases, all bytes are treated as-is (i.e., there is no quoting, and one cannot transmit a value with newline or NUL in it). The list of attributes is terminated by a blank line or end-of-file.
@@ -160,6 +166,17 @@ empty string. Components which are missing from the URL (e.g., there is no username in the example above) will be left unset. +`wwwauth[]`:: + + When an HTTP response is received by Git that includes one or more + 'WWW-Authenticate' authentication headers, these will be passed by Git + to credential helpers. ++ +Each 'WWW-Authenticate' header value is passed as a multi-valued +attribute 'wwwauth[]', where the order of the attributes is the same as +they appear in the HTTP response. This attribute is 'one-way' from Git +to pass additional information to credential helpers. + Unrecognised attributes are silently discarded. GIT
credential.c +12 -0
@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,fprintf (fp,"%s=%s\n" , key, value); } +static void credential_write_strvec (FILE *fp,const char *key, +const struct strvec *vec) +{ +int i =0 ; +const char *full_key =xstrfmt ("%s[]" , key); +for (; i < vec->nr; i++) { +credential_write_item (fp, full_key, vec->v[i],0 ); + } +free ((void *)full_key); +} +void credential_write (const struct credential *c,FILE *fp) {credential_write_item (fp,"protocol" , c->protocol,1 );
@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)credential_write_item (fp,"path" , c->path,0 );credential_write_item (fp,"username" , c->username,0 );credential_write_item (fp,"password" , c->password,0 ); +credential_write_strvec (fp,"wwwauth" , &c->wwwauth_headers); } static int run_credential_helper(struct credential *c,
t/helper/test-credential-helper-replay.sh new file +14 -0
@@ -0,0 +1,14 @@ +cmd=$1 +teefile=$cmd-actual.cred +catfile=$cmd-response.cred +rm -f $teefile +while read line; +do + if test -z "$line"; then + break; + fi + echo "$line" >> $teefile +done +if test "$cmd" = "get"; then + cat $catfile +fi
t/t5556-http-auth.sh +268 -2
289 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests
credential.c static void credential_write_item(FILE *fp, const char *key, const char *value, · line 272
Quoted 9 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]> [...]
credential.c +7
-0+static void credential_write_strvec (FILE *fp,const char *key, +const struct strvec *vec) +{ +int i =0 ; +const char *full_key =xstrfmt ("%s[]" , key); +for (; i < vec->nr; i++) { +credential_write_item (fp, full_key, vec->v[i],0 );
Style: Don't mismatch types if there's no good reason. Use "size_t i" here, also let's do:
for (size_t i = 0; ....
I.e. no reason to declare it earlier.
credential.c static void credential_write_item(FILE *fp, const char *key, const char *value, · line 274
Quoted 2 lines
credential.c +2
-0+ } +free ((void *)full_key);
Just don't add a "const" to that "full_key" and skip the cast with free() here.
Quoted 1 line
+++
b/t/helper/test-credential-helper-replay.sh
I see to my surprise that we have one existing *.sh helper in that directory, but in any case...
t/helper/test-credential-helper-replay.sh line 11
Quoted 12 lines
t/helper/test-credential-helper-replay.sh new file +11
-0@@ -0,0 +1,14 @@ +cmd=$1 +teefile=$cmd-actual.cred +catfile=$cmd-response.cred +rm -f $teefile +while read line; +do + if test -z "$line"; then + break; + fi + echo "$line" >> $teefile +done
It looks like you're re-inventing "sed" here, isn't this whole loop just
sed -n -e '/^$/q' -n 'p'
And then you can skip the "rm" before, as you could just clobber the thing.
t/t5556-http-auth.sh PID_FILE="$TRASH_DIRECTORY"/pid-file.pid · line 31
Quoted 13 lines
t/helper/test-credential-helper-replay.sh new file +3
-0+if test "$cmd" = "get"; then + cat $catfile +fit/t5556-http-auth.sh +2
-0@@ -27,6 +27,8 @@ PID_FILE="$TRASH_DIRECTORY"/pid-file.pid SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH +CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \ + && export CREDENTIAL_HELPER
...(continued from above): Let's just use write_script() here or whatever, i.e. no reason to make this a global script, it's just used in this one test, so it can set it up.
Quoted 10 lines
t/t5556-http-auth.sh +2
-1test_expect_success 'setup repos' ' test_create_repo "$REPO_DIR" &&@@ -92,15 +94,279 @@ start_http_server () { per_test_cleanup () { stop_http_server && - rm -f OUT.* + rm -f OUT.* && + rm -f *.cred &&
t/t5556-http-auth.sh start_http_server () { · line 107
Quoted 10 lines
t/t5556-http-auth.sh +5
-1+ rm -f auth.config } test_expect_success 'http auth anonymous no challenge' ' test_when_finished "per_test_cleanup" && - start_http_server && + + cat >auth.config <<-EOF && + [auth] + allowAnonymous = true
Mixed tab/space. Use "\t" not 4x " " (ditto below).
Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests
Quoted 17 lines · Ævar Arnfjörð Bjarmason
credential.c static void credential_write_item(FILE *fp, const char *key, const char *value, · line 272
Quoted 9 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]> [...]
credential.c +7
-0+static void credential_write_strvec (FILE *fp,const char *key, +const struct strvec *vec) +{ +int i =0 ; +const char *full_key =xstrfmt ("%s[]" , key); +for (; i < vec->nr; i++) { +credential_write_item (fp, full_key, vec->v[i],0 );Style: Don't mismatch types if there's no good reason. Use "
size_ti" here, also let's do:for (size_t i = 0; ....
Quoted 7 lines
I.e. no reason to declare it earlier.
credential.c static void credential_write_item(FILE *fp, const char *key, const char *value, · line 274
Quoted 2 lines
credential.c +2
-0+ } +free ((void *)full_key);Just don't add a "const" to that "
full_key" and skip the cast withfree()here.
Both good points! Thanks - will take this onboard in next iteration.
Quoted 20 lines
Quoted 1 line
+++
b/t/helper/test-credential-helper-replay.shI see to my surprise that we have one existing *.sh helper in that directory, but in any case...
t/helper/test-credential-helper-replay.sh line 11
Quoted 12 lines
t/helper/test-credential-helper-replay.sh new file +11
-0@@ -0,0 +1,14 @@ +cmd=$1 +teefile=$cmd-actual.cred +catfile=$cmd-response.cred +rm -f $teefile +while read line; +do + if test -z "$line"; then + break; + fi + echo "$line" >> $teefile +doneIt looks like you're re-inventing "sed" here, isn't this whole loop just
Quoted 1 line
sed -n -e '/^$/q' -n 'p'
True; sed -n -e '/^$/q' -e 'p' is equivalent here.
Quoted 13 lines
And then you can skip the "rm" before, as you could just clobber the thing.
Quoted 10 lines
t/helper/test-credential-helper-replay.sh new file +3
-0+if test "$cmd" = "get"; then + cat $catfile +fit/t5556-http-auth.sh +0
-0@@ -27,6 +27,8 @@ PID_FILE="$TRASH_DIRECTORY"/pid-file.pid SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
Quoted 7 lines
t/t5556-http-auth.sh PID_FILE="$TRASH_DIRECTORY"/pid-file.pid · line 31
Quoted 3 lines
t/t5556-http-auth.sh +2
-0PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH +CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \ + && export CREDENTIAL_HELPER...(continued from above): Let's just use
write_script()here or whatever, i.e. no reason to make this a global script, it's just used in this one test, so it can set it up.
In the next iteration I will move to using write_script; thanks!
Quoted 12 lines
Quoted 12 lines
t/t5556-http-auth.sh +3
-1test_expect_success 'setup repos' ' test_create_repo "$REPO_DIR" &&@@ -92,15 +94,279 @@ start_http_server () { per_test_cleanup () { stop_http_server && - rm -f OUT.* + rm -f OUT.* && + rm -f *.cred && + rm -f auth.config }
Quoted 9 lines
t/t5556-http-auth.sh start_http_server () { · line 107
Quoted 7 lines
t/t5556-http-auth.sh +4
-1test_expect_success 'http auth anonymous no challenge' ' test_when_finished "per_test_cleanup" && - start_http_server && + + cat >auth.config <<-EOF && + [auth] + allowAnonymous = trueMixed tab/space. Use "\t" not 4x " " (ditto below).
Sure!
Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests
In reply to Matthew John Cheetham via GitGitGadgetcredential.c static void credential_write_item(FILE *fp, const char *key, const char *value, · line 271
Quoted 6 lines · Matthew John Cheetham via GitGitGadget
credential.c +6
-0+static void credential_write_strvec (FILE *fp,const char *key, +const struct strvec *vec) +{ +int i =0 ; +const char *full_key =xstrfmt ("%s[]" , key); +for (; i < vec->nr; i++) {
style nit: use "int i;" and "for (i = 0; ..."
t/t5556-http-auth.sh start_http_server () { · line 110
Quoted 10 lines
test_expect_success'http auth anonymous no challenge' 't/t5556-http-auth.sh +7
-1test_when_finished "per_test_cleanup" && - start_http_server && + + cat >auth.config <<-EOF && + [auth] + allowAnonymous = true + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
I see that you added auth.allowAnonymous and --auth-config options in Patch 6, so perhaps this test change could move to that patch.
Thanks,
-Stolee
Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests
Quoted 10 lines · Derrick Stolee
credential.c static void credential_write_item(FILE *fp, const char *key, const char *value, · line 271
Quoted 6 lines · Matthew John Cheetham via GitGitGadget
credential.c +6
-0+static void credential_write_strvec (FILE *fp,const char *key, +const struct strvec *vec) +{ +int i =0 ; +const char *full_key =xstrfmt ("%s[]" , key); +for (; i < vec->nr; i++) {style nit: use "int i;" and "for (i = 0; ..."
Thanks for pointing this out; I missed that C99 style for-loops were allowed now. As Ævar pointed out, this should also be size_t and not int.
Quoted 5 lines
Quoted 5 lines
test_expect_success'http auth anonymous no challenge' 't/t5556-http-auth.sh +2
-1test_when_finished "per_test_cleanup" && - start_http_server && + + cat >auth.config <<-EOF &&
Quoted 8 lines
t/t5556-http-auth.sh start_http_server () { · line 110
Quoted 5 lines
t/t5556-http-auth.sh +5
-0+ [auth] + allowAnonymous = true + EOF + + start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&I see that you added auth.allowAnonymous and --auth-config options in Patch 6, so perhaps this test change could move to that patch.
Good point; will update on reroll.
Quoted 2 lines
Thanks,
-Stolee
v6 cover Enhance credential helper protocol to include auth headers
In reply to Matthew John Cheetham via GitGitGadgetFollowing from my original RFC submission [0], this submission is considered ready for full review. This patch series is now based on top of current master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now separately submitted patches [1] to fix up the other credential helpers' behaviour.
In this patch series I update the existing credential helper design in order to allow for some new scenarios, and future evolution of auth methods that Git hosts may wish to provide. I outline the background, summary of changes and some challenges below.
Testing these new additions, I introduce a new test helper test-http-server that acts as a frontend to git-http-backend; a mini HTTP server sharing code with git-daemon, with simple authentication configurable by a config file.
Background
==========
Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the authentication capabilities of this protocol to address limitations (see below).
The Smart HTTP protocol in Git supports a few different types of HTTP authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478) [4]. Git uses a extensible model where credential helpers can provide credentials for protocols [5]. Several helpers support alternatives such as OAuth authentication (RFC 6749) [6], but this is typically done as an extension. For example, a helper might use basic auth and set the password to an OAuth Bearer access token. Git uses standard input and output to communicate with credential helpers.
After a HTTP 401 response, Git would call a credential helper with the following over standard input:
protocol=https
host=example.com
And then a credential helper would return over standard output:
protocol=https
host=example.com
[email protected]
password=<BEARER-TOKEN>
Git then the following request to the remote, including the standard HTTP Authorization header (RFC 7235 Section 4.2) [7]:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64([email protected]:<BEARER-TOKEN>)
Credential helpers are encouraged (see gitcredentials.txt) to return the minimum information necessary.
Limitations
===========
Because this credential model was built mostly for password based authentication systems, it's somewhat limited. In particular:
1. To generate valid credentials, additional information about the request (or indeed the requestee and their device) may be required. For example, OAuth is based around scopes. A scope, like "git.read", might be required to read data from the remote. However, the remote cannot tell the credential helper what scope is required for this request.
2. This system is not fully extensible. Each time a new type of authentication (like OAuth Bearer) is invented, Git needs updates before credential helpers can take advantage of it (or leverage a new capability in libcurl).
Goals
=====
* As a user with multiple federated cloud identities:
* Reach out to a remote and have my credential helper automatically prompt me for the correct identity.
* Allow credential helpers to differentiate between different authorities or authentication/authorization challenge types, even from the same DNS hostname (and without needing to use credential.useHttpPath).
* Leverage existing authentication systems built-in to many operating systems and devices to boost security and reduce reliance on passwords.
* As a Git host and/or cloud identity provider:
* Enforce security policies (like requiring two-factor authentication) dynamically.
* Allow integration with third party standard based identity providers in enterprises allowing customers to have a single plane of control for critical identities with access to source code.
Design Principles
=================
* Use the existing infrastructure. Git credential helpers are an already-working model.
* Follow widely-adopted time-proven open standards, avoid net new ideas in the authentication space.
* Minimize knowledge of authentication in Git; maintain modularity and extensibility.
Proposed Changes
================
1. Teach Git to read HTTP response headers, specifically the standard WWW-Authenticate (RFC 7235 Section 4.1) headers.
2. Teach Git to include extra information about HTTP responses that require authentication when calling credential helpers. Specifically the WWW-Authenticate header information.
Because the extra information forms an ordered list, and the existing credential helper I/O format only provides for simple key=value pairs, we introduce a new convention for transmitting an ordered list of values. Key names that are suffixed with a C-style array syntax should have values considered to form an order list, i.e. key[]=value, where the order of the key=value pairs in the stream specifies the order.
For the WWW-Authenticate header values we opt to use the key wwwauth[].
Handling the WWW-Authenticate header in detail
==============================================
RFC 6750 [8] envisions that OAuth Bearer resource servers would give responses that include WWW-Authenticate headers, for example:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
Specifically, a WWW-Authenticate header consists of a scheme and arbitrary attributes, depending on the scheme. This pattern enables generic OAuth or OpenID Connect [9] authorities. Note that it is possible to have several WWW-Authenticate challenges in a response.
First Git attempts to make a request, unauthenticated, which fails with a 401 response and includes WWW-Authenticate header(s).
Next, Git invokes a credential helper which may prompt the user. If the user approves, a credential helper can generate a token (or any auth challenge response) to be used for that request.
For example: with a remote that supports bearer tokens from an OpenID Connect [9] authority, a credential helper can use OpenID Connect's Discovery [10] and Dynamic Client Registration [11] to register a client and make a request with the correct permissions to access the remote. In this manner, a user can be dynamically sent to the right federated identity provider for a remote without any up-front configuration or manual processes.
Following from the principle of keeping authentication knowledge in Git to a minimum, we modify Git to add all WWW-Authenticate values to the credential helper call.
Git sends over standard input:
protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite" wwwauth[]=Basic realm="login.example"
A credential helper that understands the extra wwwauth[n] property can decide on the "best" or correct authentication scheme, generate credentials for the request, and interact with the user.
The credential helper would then return over standard output:
protocol=https
host=example.com
path=foo.git
[email protected]
password=<BEARER-TOKEN>
Note that WWW-Authenticate supports multiple challenges, either in one header:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
or in multiple headers:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the implementation, Git will not merge or split up any of these WWW-Authenticate headers, and instead pass each header line as one credential helper property. The credential helper is responsible for splitting, merging, and otherwise parsing these header values.
An alternative option to sending the header fields individually would be to merge the header values in to one key=value property, for example:
...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
Future work
===========
In the future we can further expand the protocol to allow credential helpers decide the best authentication scheme. Today credential helpers are still only expected to return a username/password pair to Git, meaning the other authentication schemes that may be offered still need challenge responses sent via a Basic Authorization header. The changes outlined above still permit helpers to select and configure an available authentication mode, but require the remote for example to unpack a bearer token from a basic challenge.
More careful consideration is required in the handling of custom authentication schemes which may not have a username, or may require arbitrary additional request header values be set.
For example imagine a new "FooBar" authentication scheme that is surfaced in the following response:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"
With support for arbitrary authentication schemes, Git would call credential helpers with the following over standard input:
protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"
And then an enlightened credential helper could return over standard output:
protocol=https
host=example.com
authtype=FooBar
[email protected]
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF
Git would be expected to attach this authorization header to the next request:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF
Why not SSH?
============
There's nothing wrong with SSH. However, Git's Smart HTTP transport is widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport sometimes requires less client setup than SSH transport, and works in environments when SSH ports may be blocked. As long as Git supports HTTP transport, it should support common and popular HTTP authentication methods.
References
==========
* [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers
https://lore.kernel.org/git/[email protected]/
* [1] [PATCH 0/3] Correct credential helper discrepancies handling input
https://lore.kernel.org/git/[email protected]/
* [2] Git on the Server - The Protocols
https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols
* [3] HTTP Authentication: Basic and Digest Access Authentication
https://datatracker.ietf.org/doc/html/rfc2617
* [4] The Simple and Protected GSS-API Negotiation Mechanism
https://datatracker.ietf.org/doc/html/rfc2478
* [5] Git Credentials - Custom Helpers
https://git-scm.com/docs/gitcredentials#_custom_helpers
* [6] The OAuth 2.0 Authorization Framework
https://datatracker.ietf.org/doc/html/rfc6749
* [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
https://datatracker.ietf.org/doc/html/rfc7235
* [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
https://datatracker.ietf.org/doc/html/rfc6750
* [9] OpenID Connect Core 1.0
https://openid.net/specs/openid-connect-core-1_0.html
* [10] OpenID Connect Discovery 1.0
https://openid.net/specs/openid-connect-discovery-1_0.html
* [11] OpenID Connect Dynamic Client Registration 1.0
https://openid.net/specs/openid-connect-registration-1_0.html
* [12] Hypertext Transfer Protocol (HTTP/1.1)
https://datatracker.ietf.org/doc/html/rfc2616
Updates from RFC
================
* Submitted first three patches as separate submission:
https://lore.kernel.org/git/[email protected]/
* Various style fixes and updates to- and addition of comments.
* Drop the explicit integer index in new 'array' style credential helper attrbiutes ("key[n]=value" becomes just "key[]=value").
* Added test helper; a mini HTTP server, and several tests.
Updates in v3
=============
* Split final patch that added the test-http-server in to several, easier to review patches.
* Updated wording in git-credential.txt to clarify which side of the credential helper protocol is sending/receiving the new wwwauth and authtype attributes.
Updates in v4
=============
* Drop authentication scheme selection authtype attribute patches to greatly simplify the series; auth scheme selection is punted to a future series. This series still allows credential helpers to generate credentials and intelligently select correct identities for a given auth challenge.
Updates in v5
=============
* Libify parts of daemon.c and share implementation with test-http-server.
* Clarify test-http-server Git request regex pattern and auth logic comments.
* Use STD*_FILENO in place of 'magic' file descriptor numbers.
* Use strbuf_* functions in continuation header parsing.
* Use configuration file to configure auth for test-http-server rather than command-line arguments. Add ability to specify arbitrary extra headers that is useful for testing 'malformed' server responses.
* Use st_mult over unchecked multiplication in http.c curl callback functions.
* Fix some documentation line break issues.
* Reorder some commits to bring in the tests and test-http-server helper first and, then the WWW-Authentication changes, alongside tests to cover.
* Expose previously static strvec_push_nodup function.
* Merge the two timeout args for test-http-server (--timeout and --init-timeout) that were a hang-over from the original daemon.c but are no longer required here.
* Be more careful around continuation headers where they may be empty strings. Add more tests to cover these header types.
* Include standard trace2 tracing calls at start of test-http-server helper.
Updates in v6
=============
* Clarify the change to make logging optional in the check_dead_children() function during libification of daemon.c.
* Fix missing pointer dereference bugs identified in libification of child process handling functions for daemon.c.
* Add doc comments to child process handling function declarations in the daemon-utils.h header.
* Align function parameter names with variable names at callsites for libified daemon functions.
* Re-split out the test-http-server test helper commits in to smaller
patches: error response handling, request parsing, http-backend pass-through, simple authentication, arbitrary header support.
* Call out auth configuration file format for test-http-server test helper and supported options in commit messages, as well as a test to exercise and demonstrate these options.
* Permit auth.token and auth.challenge to appear in any order; create the struct auth_module just-in-time as options for that scheme are read. This simplifies the configuration authoring of the test-http-server test helper.
* Update tests to use auth.allowAnoymous in the patch that introduces the new test helper option.
* Drop the strvec_push_nodup() commit and update the implementation of HTTP request header line folding to use xstrdup and strvec_pop and _pushf.
* Use size_t instead of int in credential.c when iterating over the struct strvec credential members. Also drop the not required const and cast from the full_key definition and free.
* Replace in-tree test-credential-helper-reply.sh test cred helper script with the lib-credential-helper.sh reusable 'lib' test script and shell functions to configure the helper behaviour.
* Leverage sed over the while read $line loop in the test credential helper script.
Matthew John Cheetham (12):
daemon: libify socket setup and option functions
daemon: libify child process handling functions
daemon: rename some esoteric/laboured terminology
test-http-server: add stub HTTP server test helper
test-http-server: add HTTP error response function
test-http-server: add HTTP request parsing
test-http-server: pass Git requests to http-backend
test-http-server: add simple authentication
test-http-server: add sending of arbitrary headers
http: replace unsafe size_t multiplication with st_mult
http: read HTTP WWW-Authenticate response headers
credential: add WWW-Authenticate header to cred requests
Documentation/git-credential.txt | 19 +- Makefile | 2 + contrib/buildsystems/CMakeLists.txt | 11 +- credential.c | 12 + credential.h | 15 + daemon-utils.c | 286 +++++++++ daemon-utils.h | 55 ++ daemon.c | 306 +--------- http.c | 98 ++- t/helper/.gitignore | 1 + t/helper/test-http-server.c | 910 ++++++++++++++++++++++++++++ t/lib-credential-helper.sh | 27 + t/t5556-http-auth.sh | 398 ++++++++++++ 13 files changed, 1838 insertions(+), 302 deletions(-) create mode 100644 daemon-utils.c create mode 100644 daemon-utils.h create mode 100644 t/helper/test-http-server.c create mode 100644 t/lib-credential-helper.sh create mode 100755 t/t5556-http-auth.sh
base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v6
Pull-Request: https://github.com/gitgitgadget/git/pull/1352
Range-diff vs v5:
1: 74b0de14185 = 1: 74b0de14185 daemon: libify socket setup and option functions
2: bc972fc8d3d ! 2: b6ba344a671 daemon: libify child process handling functions
@@ Commit message
from the parent daemon-like process from `daemon.c` to the new shared
`daemon-utils.{c,h}` files.
+ One minor functional change is introduced to `check_dead_children()`
+ where the logging of a dead/disconnected child is now optional. With the
+ 'libification' of these functions we extract the call to `loginfo` to a
+ call to a function pointer, and guard the log message creation and
+ logging behind a `NULL` check. Callers can now skip logging by passing
+ `NULL` as the `log_fn loginfo` argument.
+ The behaviour of callers in `daemon.c` remains the same (save one extra
+ NULL check) however as a pointer to `loginfo` is always passed.
+
Signed-off-by: Matthew John Cheetham <[email protected]>
## daemon-utils.c ##
@@ daemon-utils.c: void socksetup(struct string_list *listen_addr, int listen_port,
+ struct child *newborn, **cradle;
+
+ CALLOC_ARRAY(newborn, 1);
-+ live_children++;
++ (*live_children)++;
+ memcpy(&newborn->cld, cld, sizeof(*cld));
+ memcpy(&newborn->address, addr, addrlen);
+ for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
@@ daemon-utils.c: void socksetup(struct string_list *listen_addr, int listen_port,
+
+ /* remove the child */
+ *cradle = blanket->next;
-+ live_children--;
++ (*live_children)--;
+ child_process_clear(&blanket->cld);
+ free(blanket);
+ } else
@@ daemon-utils.h: void socksetup(struct string_list *listen_addr, int listen_port,
+ struct sockaddr_storage address;
+};
+
++/*
++ * Add the child_process to the set of children and increment the number of
++ * live children.
++ */
+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
+ struct child *firstborn, unsigned int *live_children);
+
++/*
++ * Kill the newest connection from a duplicate IP.
++ *
++ * This function should be called if the number of connections grows
++ * past the maximum number of allowed connections.
++ */
+void kill_some_child(struct child *firstborn);
+
++/*
++ * Check for children that have disconnected and remove them from the
++ * active set, decrementing the number of live children.
++ *
++ * Optionally log the child PID that disconnected by passing a loginfo
++ * function.
++ */
+void check_dead_children(struct child *firstborn, unsigned int *live_children,
+ log_fn loginfo);
+
3: 8f176d5955d ! 3: 9967401c972 daemon: rename some esoteric/laboured terminology
@@ Commit message
processes. The existing names are esoteric; stretching an analogy too
far to the point of being confusing to understand.
- Rename "firstborn" to simply "first", "newborn" to "new_cld", "blanket"
+ Rename "firstborn" to "first_child", "newborn" to "new_cld", "blanket"
to "current" and "cradle" to "ptr".
Signed-off-by: Matthew John Cheetham <[email protected]>
@@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,
void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
- struct child *firstborn , unsigned int *live_children)
-+ struct child *first, unsigned int *live_children)
++ struct child *first_child, unsigned int *live_children)
{
- struct child *newborn, **cradle;
+ struct child *new_cld, **current;
- CALLOC_ARRAY(newborn, 1);
+ CALLOC_ARRAY(new_cld, 1);
- live_children++;
+ (*live_children)++;
- memcpy(&newborn->cld, cld, sizeof(*cld));
- memcpy(&newborn->address, addr, addrlen);
- for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
- if (!addrcmp(&(*cradle)->address, &newborn->address))
+ memcpy(&new_cld->cld, cld, sizeof(*cld));
+ memcpy(&new_cld->address, addr, addrlen);
-+ for (current = &first; *current; current = &(*current)->next)
++ for (current = &first_child; *current; current = &(*current)->next)
+ if (!addrcmp(&(*current)->address, &new_cld->address))
break;
- newborn->next = *cradle;
@@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,
}
-void kill_some_child(struct child *firstborn)
-+void kill_some_child(struct child *first)
++void kill_some_child(struct child *first_child)
{
- const struct child *blanket, *next;
+ const struct child *current, *next;
- if (!(blanket = firstborn))
-+ if (!(current = first))
++ if (!(current = first_child))
return;
- for (; (next = blanket->next); blanket = next)
@@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,
}
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
-+void check_dead_children(struct child *first, unsigned int *live_children,
++void check_dead_children(struct child *first_child, unsigned int *live_children,
log_fn loginfo)
{
int status;
@@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,
- for (cradle = &firstborn; (blanket = *cradle);)
- if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+ struct child **ptr, *current;
-+ for (ptr = &first; (current = *ptr);)
++ for (ptr = &first_child; (current = *ptr);)
+ if ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {
if (loginfo) {
const char *dead = "";
@@ daemon-utils.c: void check_dead_children(struct child *firstborn, unsigned int *
/* remove the child */
- *cradle = blanket->next;
+ *ptr = current->next;
- live_children--;
+ (*live_children)--;
- child_process_clear(&blanket->cld);
- free(blanket);
+ child_process_clear(¤t->cld);
@@ daemon-utils.c: void check_dead_children(struct child *firstborn, unsigned int *
## daemon-utils.h ##
@@ daemon-utils.h: struct child {
- };
-
+ * live children.
+ */
void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
- struct child *firstborn, unsigned int *live_children);
-+ struct child *first, unsigned int *live_children);
-
++ struct child *first_child, unsigned int *live_children);
+
+ /*
+ * Kill the newest connection from a duplicate IP.
+@@ daemon-utils.h: void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl
+ * This function should be called if the number of connections grows
+ * past the maximum number of allowed connections.
+ */
-void kill_some_child(struct child *firstborn);
-+void kill_some_child(struct child *first);
-
++void kill_some_child(struct child *first_child);
+
+ /*
+ * Check for children that have disconnected and remove them from the
+@@ daemon-utils.h: void kill_some_child(struct child *firstborn);
+ * Optionally log the child PID that disconnected by passing a loginfo
+ * function.
+ */
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
-+void check_dead_children(struct child *first, unsigned int *live_children,
++void check_dead_children(struct child *first_child, unsigned int *live_children,
log_fn loginfo);
#endif
4: 706fb3781bd = 4: d6e5e8825e8 test-http-server: add stub HTTP server test helper
-: ----------- > 5: 79805f042b9 test-http-server: add HTTP error response function
5: 6f66bf146b4 ! 6: 252098db219 test-http-server: add HTTP error response function
@@ Metadata
Author: Matthew John Cheetham <[email protected]>
## Commit message ##
- test-http-server: add HTTP error response function
+ test-http-server: add HTTP request parsing
- Introduce a function to the test-http-server test helper to write more
- full and valid HTTP error responses, including all the standard response
- headers like `Server` and `Date`.
+ Add ability to parse HTTP requests to the test-http-server test helper.
Signed-off-by: Matthew John Cheetham <[email protected]>
@@ t/helper/test-http-server.c: enum worker_result {
+ string_list_clear(&req->header_list, 0);
+}
+
-+static enum worker_result send_http_error(
-+ int fd,
-+ int http_code, const char *http_code_name,
-+ int retry_after_seconds, struct string_list *response_headers,
-+ enum worker_result wr_in)
-+{
-+ struct strbuf response_header = STRBUF_INIT;
-+ struct strbuf response_content = STRBUF_INIT;
-+ struct string_list_item *h;
-+ enum worker_result wr;
-+
-+ strbuf_addf(&response_content, "Error: %d %s\r\n",
-+ http_code, http_code_name);
-+ if (retry_after_seconds > 0)
-+ strbuf_addf(&response_content, "Retry-After: %d\r\n",
-+ retry_after_seconds);
-+
-+ strbuf_addf (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
-+ strbuf_addstr(&response_header, "Cache-Control: private\r\n");
-+ strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
-+ strbuf_addf (&response_header, "Content-Length: %d\r\n", (int)response_content.len);
-+ if (retry_after_seconds > 0)
-+ strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
-+ strbuf_addf( &response_header, "Server: test-http-server/%s\r\n", git_version_string);
-+ strbuf_addf( &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
-+ if (response_headers)
-+ for_each_string_list_item(h, response_headers)
-+ strbuf_addf(&response_header, "%s\r\n", h->string);
-+ strbuf_addstr(&response_header, "\r\n");
-+
-+ if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
-+ logerror("unable to write response header");
-+ wr = WR_IO_ERROR;
-+ goto done;
-+ }
-+
-+ if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
-+ logerror("unable to write response content body");
-+ wr = WR_IO_ERROR;
-+ goto done;
-+ }
-+
-+ wr = wr_in;
-+
-+done:
-+ strbuf_release(&response_header);
-+ strbuf_release(&response_content);
-+
-+ return wr;
-+}
-+
+ static enum worker_result send_http_error(
+ int fd,
+ int http_code, const char *http_code_name,
+@@ t/helper/test-http-server.c: done:
+ return wr;
+ }
+
+/*
+ * Read the HTTP request up to the start of the optional message-body.
+ * We do this byte-by-byte because we have keep-alive turned on and
@@ t/helper/test-http-server.c: enum worker_result {
+ return result;
+}
+
-+static int is_git_request(struct req *req)
-+{
-+ static regex_t *smart_http_regex;
-+ static int initialized;
-+
-+ if (!initialized) {
-+ smart_http_regex = xmalloc(sizeof(*smart_http_regex));
-+ /*
-+ * This regular expression matches all dumb and smart HTTP
-+ * requests that are currently in use, and defined in
-+ * Documentation/gitprotocol-http.txt.
-+ *
-+ */
-+ if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
-+ "objects/info/[^/]+|git-(upload|receive)-pack)$",
-+ REG_EXTENDED)) {
-+ warning("could not compile smart HTTP regex");
-+ smart_http_regex = NULL;
-+ }
-+ initialized = 1;
-+ }
-+
-+ return smart_http_regex &&
-+ !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
-+}
-+
-+static enum worker_result do__git(struct req *req)
-+{
-+ const char *ok = "HTTP/1.1 200 OK\r\n";
-+ struct child_process cp = CHILD_PROCESS_INIT;
-+ int res;
-+
-+ /*
-+ * Note that we always respond with a 200 OK response even if the
-+ * http-backend process exits with an error. This helper is intended
-+ * only to be used to exercise the HTTP auth handling in the Git client,
-+ * and specifically around authentication (not handled by http-backend).
-+ *
-+ * If we wanted to respond with a more 'valid' HTTP response status then
-+ * we'd need to buffer the output of http-backend, wait for and grok the
-+ * exit status of the process, then write the HTTP status line followed
-+ * by the http-backend output. This is outside of the scope of this test
-+ * helper's use at time of writing.
-+ *
-+ * The important auth responses (401) we are handling prior to getting
-+ * to this point.
-+ */
-+ if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
-+ return error(_("could not send '%s'"), ok);
-+
-+ strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
-+ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
-+ req->uri_path.buf);
-+ strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
-+ if (req->query_args.len)
-+ strvec_pushf(&cp.env, "QUERY_STRING=%s",
-+ req->query_args.buf);
-+ if (req->content_type)
-+ strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
-+ req->content_type);
-+ if (req->content_length >= 0)
-+ strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
-+ (intmax_t)req->content_length);
-+ cp.git_cmd = 1;
-+ strvec_push(&cp.args, "http-backend");
-+ res = run_command(&cp);
-+ close(STDOUT_FILENO);
-+ close(STDIN_FILENO);
-+ return !!res;
-+}
-+
+static enum worker_result dispatch(struct req *req)
+{
-+ if (is_git_request(req))
-+ return do__git(req);
-+
+ return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
+ WR_OK | WR_HANGUP);
+}
+
static enum worker_result worker(void)
{
-- const char *response = "HTTP/1.1 501 Not Implemented\r\n";
+ struct req req = REQ__INIT;
char *client_addr = getenv("REMOTE_ADDR");
char *client_port = getenv("REMOTE_PORT");
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
set_keep_alive(0, logerror);
while (1) {
-- if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
-- logerror("unable to write response");
-- wr = WR_IO_ERROR;
-- }
+- wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
+- NULL, WR_OK | WR_HANGUP);
+ req__release(&req);
+
+ alarm(timeout);
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
if (wr != WR_OK)
break;
}
-
- ## t/t5556-http-auth.sh (new) ##
-@@
-+#!/bin/sh
-+
-+test_description='test http auth header and credential helper interop'
-+
-+TEST_NO_CREATE_REPO=1
-+. ./test-lib.sh
-+
-+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
-+
-+# Setup a repository
-+#
-+REPO_DIR="$TRASH_DIRECTORY"/repo
-+
-+# Setup some lookback URLs where test-http-server will be listening.
-+# We will spawn it directly inside the repo directory, so we avoid
-+# any need to configure directory mappings etc - we only serve this
-+# repository from the root '/' of the server.
-+#
-+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
-+ORIGIN_URL=http://$HOST_PORT/
-+
-+# The pid-file is created by test-http-server when it starts.
-+# The server will shutdown if/when we delete it (this is easier than
-+# killing it by PID).
-+#
-+PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
-+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
-+
-+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
-+
-+test_expect_success 'setup repos' '
-+ test_create_repo "$REPO_DIR" &&
-+ git -C "$REPO_DIR" branch -M main
-+'
-+
-+stop_http_server () {
-+ if ! test -f "$PID_FILE"
-+ then
-+ return 0
-+ fi
-+ #
-+ # The server will shutdown automatically when we delete the pid-file.
-+ #
-+ rm -f "$PID_FILE"
-+ #
-+ # Give it a few seconds to shutdown (mainly to completely release the
-+ # port before the next test start another instance and it attempts to
-+ # bind to it).
-+ #
-+ for k in 0 1 2 3 4
-+ do
-+ if grep -q "Starting graceful shutdown" "$SERVER_LOG"
-+ then
-+ return 0
-+ fi
-+ sleep 1
-+ done
-+
-+ echo "stop_http_server: timeout waiting for server shutdown"
-+ return 1
-+}
-+
-+start_http_server () {
-+ #
-+ # Launch our server into the background in repo_dir.
-+ #
-+ (
-+ cd "$REPO_DIR"
-+ test-http-server --verbose \
-+ --listen=127.0.0.1 \
-+ --port=$GIT_TEST_HTTP_PROTOCOL_PORT \
-+ --reuseaddr \
-+ --pid-file="$PID_FILE" \
-+ "$@" \
-+ 2>"$SERVER_LOG" &
-+ )
-+ #
-+ # Give it a few seconds to get started.
-+ #
-+ for k in 0 1 2 3 4
-+ do
-+ if test -f "$PID_FILE"
-+ then
-+ return 0
-+ fi
-+ sleep 1
-+ done
-+
-+ echo "start_http_server: timeout waiting for server startup"
-+ return 1
-+}
-+
-+per_test_cleanup () {
-+ stop_http_server &&
-+ rm -f OUT.*
-+}
-+
-+test_expect_success 'http auth anonymous no challenge' '
-+ test_when_finished "per_test_cleanup" &&
-+ start_http_server &&
-+
-+ # Attempt to read from a protected repository
-+ git ls-remote $ORIGIN_URL
-+'
-+
-+test_done
-: ----------- > 7: ab06ac9b965 test-http-server: pass Git requests to http-backend
6: c3c3d17a688 ! 8: a1ff55dd6e2 test-http-server: add simple authentication
@@ Commit message
tokens and only approved if a matching token is found, or if no auth
was provided and anonymous auth is enabled.
+ Configuration for auth includes a simple set of three options:
+
+ [auth]
+ challenge = <scheme>[:<challenge_params>]
+ token = <scheme>:[<token>]*
+ allowAnonymous = <bool>
+
+ `auth.challenge` allows you define what authentication schemes, and
+ optional challenge parameters the server should use. Scheme names are
+ unique and subsequently specified challenge parameters in the config
+ file will replace previously specified ones.
+
+ `auth.token` allows you to define the set of value token values for an
+ authentication scheme. This is a multi-var and each entry in the
+ config file will append to the set of valid tokens for that scheme.
+ Specifying an empty token value will clear the list of tokens so far for
+ that scheme, i.e. `token = <scheme>:`.
+
+ `auth.allowAnonymous` controls whether or not unauthenticated requests
+ (those without any `Authorization` headers) should succeed or not, and
+ trigger a 401 Unauthorized response.
+
Signed-off-by: Matthew John Cheetham <[email protected]>
## t/helper/test-http-server.c ##
@@ t/helper/test-http-server.c: static int is_git_request(struct req *req)
const char *ok = "HTTP/1.1 200 OK\r\n";
struct child_process cp = CHILD_PROCESS_INIT;
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ * exit status of the process, then write the HTTP status line followed
+ * by the http-backend output. This is outside of the scope of this test
+ * helper's use at time of writing.
++ *
++ * The important auth responses (401) we are handling prior to getting
++ * to this point.
+ */
if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
return error(_("could not send '%s'"), ok);
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+static struct auth_module **auth_modules = NULL;
+static size_t auth_modules_nr = 0;
+static size_t auth_modules_alloc = 0;
-+static struct strvec extra_headers = STRVEC_INIT;
+
-+static struct auth_module *create_auth_module(const char *scheme,
-+ const char *challenge)
-+{
-+ struct auth_module *mod = xmalloc(sizeof(struct auth_module));
-+ mod->scheme = xstrdup(scheme);
-+ mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
-+ CALLOC_ARRAY(mod->tokens, 1);
-+ string_list_init_dup(mod->tokens);
-+ return mod;
-+}
-+
-+static struct auth_module *get_auth_module(const char *scheme)
++static struct auth_module *get_auth_module(const char *scheme, int create)
+{
+ int i;
+ struct auth_module *mod;
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ return mod;
+ }
+
-+ return NULL;
-+}
++ if (create) {
++ struct auth_module *mod = xmalloc(sizeof(struct auth_module));
++ mod->scheme = xstrdup(scheme);
++ mod->challenge_params = NULL;
++ CALLOC_ARRAY(mod->tokens, 1);
++ string_list_init_dup(mod->tokens);
+
-+static int add_auth_module(struct auth_module *mod)
-+{
-+ if (get_auth_module(mod->scheme))
-+ return error("duplicate auth scheme '%s'\n", mod->scheme);
++ ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
++ auth_modules[auth_modules_nr++] = mod;
+
-+ ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
-+ auth_modules[auth_modules_nr++] = mod;
++ return mod;
++ }
+
-+ return 0;
++ return NULL;
+}
+
+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ /* trim trailing space ' ' */
+ strbuf_setlen(split[0], split[0]->len - 1);
+
-+ mod = get_auth_module(split[0]->buf);
++ mod = get_auth_module(split[0]->buf, 0);
+ if (mod) {
+ result = AUTH_DENY;
+
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ string_list_append(&hdrs, challenge);
+ }
+
-+ for (i = 0; i < extra_headers.nr; i++)
-+ string_list_append(&hdrs, extra_headers.v[i]);
-+
+ *wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
+ &hdrs, *wr);
+ }
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ (result == AUTH_UNKNOWN && allow_anonymous);
+}
+
-+static int split_auth_param(const char *str, char **scheme, char **val, int required_val)
++static int split_auth_param(const char *str, char **scheme, char **val)
+{
+ struct strbuf **p = strbuf_split_str(str, ':', 2);
+
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ return -1;
+
+ /* trim trailing ':' */
-+ if (p[1])
++ if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')
+ strbuf_setlen(p[0], p[0]->len - 1);
+
-+ if (required_val && !p[1])
-+ return -1;
-+
+ *scheme = strbuf_detach(p[0], NULL);
+
+ if (p[1])
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ struct auth_module *mod = NULL;
+
+ if (!strcmp(name, "auth.challenge")) {
-+ if (split_auth_param(val, &scheme, &challenge, 0)) {
++ if (split_auth_param(val, &scheme, &challenge)) {
+ ret = error("invalid auth challenge '%s'", val);
+ goto cleanup;
+ }
+
-+ mod = create_auth_module(scheme, challenge);
-+ if (add_auth_module(mod)) {
-+ ret = error("failed to add auth module '%s'", val);
-+ goto cleanup;
-+ }
-+ }
-+ if (!strcmp(name, "auth.token")) {
-+ if (split_auth_param(val, &scheme, &token, 1)) {
-+ ret = error("invalid auth token '%s'", val);
-+ goto cleanup;
-+ }
++ mod = get_auth_module(scheme, 1);
+
-+ mod = get_auth_module(scheme);
-+ if (!mod) {
-+ ret = error("auth scheme not defined '%s'\n", scheme);
++ /* Replace any existing challenge parameters */
++ free(mod->challenge_params);
++ mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
++ } else if (!strcmp(name, "auth.token")) {
++ if (split_auth_param(val, &scheme, &token)) {
++ ret = error("invalid auth token '%s'", val);
+ goto cleanup;
+ }
+
-+ string_list_append(mod->tokens, token);
-+ }
-+ if (!strcmp(name, "auth.allowanonymous")) {
++ mod = get_auth_module(scheme, 1);
++
++ /*
++ * Append to set of valid tokens unless an empty token value
++ * is provided, then clear the existing list.
++ */
++ if (token)
++ string_list_append(mod->tokens, token);
++ else
++ string_list_clear(mod->tokens, 1);
++ } else if (!strcmp(name, "auth.allowanonymous")) {
+ allow_anonymous = git_config_bool(name, val);
-+ }
-+ if (!strcmp(name, "auth.extraheader")) {
-+ strvec_push(&extra_headers, val);
++ } else {
++ warning("unknown auth config '%s'", name);
+ }
+
+cleanup:
@@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
fprintf(stderr, "error: unknown argument '%s'\n", arg);
usage(test_http_auth_usage);
+
+ ## t/t5556-http-auth.sh ##
+@@ t/t5556-http-auth.sh: per_test_cleanup () {
+ rm -f OUT.*
+ }
+
++test_expect_success CURL 'http auth server auth config' '
++ #test_when_finished "per_test_cleanup" &&
++
++ cat >auth.config <<-EOF &&
++ [auth]
++ challenge = no-params
++ challenge = with-params:foo=\"bar\" p=1
++ challenge = with-params:foo=\"replaced\" q=1
++
++ token = no-explicit-challenge:valid-token
++ token = no-explicit-challenge:also-valid
++ token = reset-tokens:these-tokens
++ token = reset-tokens:will-be-reset
++ token = reset-tokens:
++ token = reset-tokens:the-only-valid-one
++
++ allowAnonymous = false
++ EOF
++
++ cat >OUT.expected <<-EOF &&
++ WWW-Authenticate: no-params
++ WWW-Authenticate: with-params foo="replaced" q=1
++ WWW-Authenticate: no-explicit-challenge
++ WWW-Authenticate: reset-tokens
++
++ Error: 401 Unauthorized
++ EOF
++
++ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
++
++ curl --include $ORIGIN_URL >OUT.curl &&
++ tr -d "\r" <OUT.curl | sed -n "/WWW-Authenticate/,\$p" >OUT.actual &&
++
++ test_cmp OUT.expected OUT.actual
++'
++
+ test_expect_success 'http auth anonymous no challenge' '
+ test_when_finished "per_test_cleanup" &&
+
+- start_http_server &&
++ cat >auth.config <<-EOF &&
++ [auth]
++ allowAnonymous = true
++ EOF
++
++ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ # Attempt to read from a protected repository
+ git ls-remote $ORIGIN_URL
-: ----------- > 9: 76125cdf239 test-http-server: add sending of arbitrary headers
7: 9c4d25945dd = 10: cc9a220ed1f http: replace unsafe size_t multiplication with st_mult
8: 65a620b08ef < -: ----------- strvec: expose strvec_push_nodup for external use
9: bcfec529d95 ! 11: bc1ac8d3eb3 http: read HTTP WWW-Authenticate response headers
@@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
+ if (!values->nr) {
+ BUG("should have at least one existing header value");
+ } else if (buf.len) {
-+ const char *prev = values->v[values->nr - 1];
-+ struct strbuf append = STRBUF_INIT;
-+ strbuf_addstr(&append, prev);
++ char *prev = xstrdup(values->v[values->nr - 1]);
+
+ /* Join two non-empty values with a single space. */
-+ if (append.len)
-+ strbuf_addch(&append, ' ');
-+
-+ strbuf_addbuf(&append, &buf);
++ const char *const sp = *prev ? " " : "";
+
+ strvec_pop(values);
-+ strvec_push_nodup(values, strbuf_detach(&append, NULL));
++ strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
++ free(prev);
+ }
+
+ goto exit;
10: af66d2d2ede ! 12: 7c8229f0b11 credential: add WWW-Authenticate header to cred requests
@@ credential.c: static void credential_write_item(FILE *fp, const char *key, const
+static void credential_write_strvec(FILE *fp, const char *key,
+ const struct strvec *vec)
+{
-+ int i = 0;
-+ const char *full_key = xstrfmt("%s[]", key);
-+ for (; i < vec->nr; i++) {
++ char *full_key = xstrfmt("%s[]", key);
++ for (size_t i = 0; i < vec->nr; i++) {
+ credential_write_item(fp, full_key, vec->v[i], 0);
+ }
-+ free((void*)full_key);
++ free(full_key);
+}
+
void credential_write(const struct credential *c, FILE *fp)
@@ credential.c: void credential_write(const struct credential *c, FILE *fp)
static int run_credential_helper(struct credential *c,
- ## t/helper/test-credential-helper-replay.sh (new) ##
+ ## t/lib-credential-helper.sh (new) ##
@@
-+cmd=$1
-+teefile=$cmd-actual.cred
-+catfile=$cmd-response.cred
-+rm -f $teefile
-+while read line;
-+do
-+ if test -z "$line"; then
-+ break;
-+ fi
-+ echo "$line" >> $teefile
-+done
-+if test "$cmd" = "get"; then
-+ cat $catfile
-+fi
++setup_credential_helper() {
++ test_expect_success 'setup credential helper' '
++ CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" &&
++ export CREDENTIAL_HELPER &&
++ echo $CREDENTIAL_HELPER &&
++
++ write_script "$CREDENTIAL_HELPER" <<-\EOF
++ cmd=$1
++ teefile=$cmd-query.cred
++ catfile=$cmd-reply.cred
++ sed -n -e "/^$/q" -e "p" >> $teefile
++ if test "$cmd" = "get"; then
++ cat $catfile
++ fi
++ EOF
++ '
++}
++
++set_credential_reply() {
++ cat >"$TRASH_DIRECTORY/$1-reply.cred"
++}
++
++expect_credential_query() {
++ cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
++ test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
++ "$TRASH_DIRECTORY/$1-query.cred"
++}
## t/t5556-http-auth.sh ##
-@@ t/t5556-http-auth.sh: PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
- SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
+@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'
+
+ TEST_NO_CREATE_REPO=1
+ . ./test-lib.sh
++. "$TEST_DIRECTORY"/lib-credential-helper.sh
- PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
-+CREDENTIAL_HELPER="$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh" \
-+ && export CREDENTIAL_HELPER
+ test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
- test_expect_success 'setup repos' '
- test_create_repo "$REPO_DIR" &&
+@@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '
+ git -C "$REPO_DIR" branch -M main
+ '
+
++setup_credential_helper
++
+ stop_http_server () {
+ if ! test -f "$PID_FILE"
+ then
@@ t/t5556-http-auth.sh: start_http_server () {
per_test_cleanup () {
@@ t/t5556-http-auth.sh: start_http_server () {
+ rm -f auth.config
}
- test_expect_success 'http auth anonymous no challenge' '
- test_when_finished "per_test_cleanup" &&
-- start_http_server &&
-+
-+ cat >auth.config <<-EOF &&
-+ [auth]
-+ allowAnonymous = true
-+ EOF
-+
-+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
-
- # Attempt to read from a protected repository
+ test_expect_success CURL 'http auth server auth config' '
+@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
git ls-remote $ORIGIN_URL
'
@@ t/t5556-http-auth.sh: start_http_server () {
+
+ cat >auth.config <<-EOF &&
+ [auth]
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
+ EOF
+
+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
-+ cat >get-expected.cred <<-EOF &&
++ set_credential_reply get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ wwwauth[]=basic realm="example.com"
++ username=alice
++ password=secret-passwd
+ EOF
+
-+ cat >store-expected.cred <<-EOF &&
++ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++
++ expect_credential_query get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ username=alice
-+ password=secret-passwd
++ wwwauth[]=basic realm="example.com"
+ EOF
+
-+ cat >get-response.cred <<-EOF &&
++ expect_credential_query store <<-EOF
+ protocol=http
+ host=$HOST_PORT
+ username=alice
+ password=secret-passwd
+ EOF
-+
-+ git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
-+
-+ test_cmp get-expected.cred get-actual.cred &&
-+ test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '
@@ t/t5556-http-auth.sh: start_http_server () {
+
+ cat >auth.config <<-EOF &&
+ [auth]
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
-+ extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
++ extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
+ EOF
+
+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
-+ cat >get-expected.cred <<-EOF &&
++ set_credential_reply get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ wwwauth[]=basic realm="example.com"
-+ wwwauth[]=bEaRer auThoRiTy="id.example.com"
++ username=alice
++ password=secret-passwd
+ EOF
+
-+ cat >store-expected.cred <<-EOF &&
++ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++
++ expect_credential_query get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ username=alice
-+ password=secret-passwd
++ wwwauth[]=basic realm="example.com"
++ wwwauth[]=bEaRer auThoRiTy="id.example.com"
+ EOF
+
-+ cat >get-response.cred <<-EOF &&
++ expect_credential_query store <<-EOF
+ protocol=http
+ host=$HOST_PORT
+ username=alice
+ password=secret-passwd
+ EOF
-+
-+ git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
-+
-+ test_cmp get-expected.cred get-actual.cred &&
-+ test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '
@@ t/t5556-http-auth.sh: start_http_server () {
+
+ cat >auth.config <<-EOF &&
+ [auth]
-+ challenge = "bearer:authority=\"id.example.com\"\\n q=1\\n \\t p=0"
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
++ challenge = "bearer:authority=\"id.example.com\"\\n q=1\\n \\t p=0"
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
+ EOF
+
+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
-+ cat >get-expected.cred <<-EOF &&
++ set_credential_reply get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ wwwauth[]=bearer authority="id.example.com" q=1 p=0
-+ wwwauth[]=basic realm="example.com"
++ username=alice
++ password=secret-passwd
+ EOF
+
-+ cat >store-expected.cred <<-EOF &&
++ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++
++ expect_credential_query get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ username=alice
-+ password=secret-passwd
++ wwwauth[]=bearer authority="id.example.com" q=1 p=0
++ wwwauth[]=basic realm="example.com"
+ EOF
+
-+ cat >get-response.cred <<-EOF &&
++ expect_credential_query store <<-EOF
+ protocol=http
+ host=$HOST_PORT
+ username=alice
+ password=secret-passwd
+ EOF
-+
-+ git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
-+
-+ test_cmp get-expected.cred get-actual.cred &&
-+ test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '
@@ t/t5556-http-auth.sh: start_http_server () {
+
+ cat >auth.config <<-EOF &&
+ [auth]
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
-+ extraheader = "WWW-Authenticate:"
-+ extraheader = " "
-+ extraheader = " bearer authority=\"id.example.com\""
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
++ extraheader = "WWW-Authenticate:"
++ extraheader = " "
++ extraheader = " bearer authority=\"id.example.com\""
+ EOF
+
+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
-+ cat >get-expected.cred <<-EOF &&
++ set_credential_reply get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ wwwauth[]=basic realm="example.com"
-+ wwwauth[]=bearer authority="id.example.com"
++ username=alice
++ password=secret-passwd
+ EOF
+
-+ cat >store-expected.cred <<-EOF &&
++ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++
++ expect_credential_query get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ username=alice
-+ password=secret-passwd
++ wwwauth[]=basic realm="example.com"
++ wwwauth[]=bearer authority="id.example.com"
+ EOF
+
-+ cat >get-response.cred <<-EOF &&
++ expect_credential_query store <<-EOF
+ protocol=http
+ host=$HOST_PORT
+ username=alice
+ password=secret-passwd
+ EOF
-+
-+ git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
-+
-+ test_cmp get-expected.cred get-actual.cred &&
-+ test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
@@ t/t5556-http-auth.sh: start_http_server () {
+
+ cat >auth.config <<-EOF &&
+ [auth]
-+ challenge = "foobar:alg=test widget=1"
-+ challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
++ challenge = "foobar:alg=test widget=1"
++ challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
+ EOF
+
+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
-+ cat >get-expected.cred <<-EOF &&
++ set_credential_reply get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ wwwauth[]=foobar alg=test widget=1
-+ wwwauth[]=bearer authority="id.example.com" q=1 p=0
-+ wwwauth[]=basic realm="example.com"
++ username=alice
++ password=secret-passwd
+ EOF
+
-+ cat >store-expected.cred <<-EOF &&
++ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++
++ expect_credential_query get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ username=alice
-+ password=secret-passwd
++ wwwauth[]=foobar alg=test widget=1
++ wwwauth[]=bearer authority="id.example.com" q=1 p=0
++ wwwauth[]=basic realm="example.com"
+ EOF
+
-+ cat >get-response.cred <<-EOF &&
++ expect_credential_query store <<-EOF
+ protocol=http
+ host=$HOST_PORT
+ username=alice
+ password=secret-passwd
+ EOF
-+
-+ git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
-+
-+ test_cmp get-expected.cred get-actual.cred &&
-+ test_cmp store-expected.cred store-actual.cred
+'
+
+test_expect_success 'http auth www-auth headers to credential helper invalid' '
@@ t/t5556-http-auth.sh: start_http_server () {
+
+ cat >auth.config <<-EOF &&
+ [auth]
-+ challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
++ challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
++ challenge = basic:realm=\"example.com\"
++ token = basic:$USERPASS64
+ EOF
+
+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
-+ cat >get-expected.cred <<-EOF &&
++ set_credential_reply get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ wwwauth[]=bearer authority="id.example.com" q=1 p=0
-+ wwwauth[]=basic realm="example.com"
++ username=alice
++ password=invalid-passwd
+ EOF
+
-+ cat >erase-expected.cred <<-EOF &&
++ test_must_fail git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++
++ expect_credential_query get <<-EOF &&
+ protocol=http
+ host=$HOST_PORT
-+ username=alice
-+ password=invalid-passwd
+ wwwauth[]=bearer authority="id.example.com" q=1 p=0
+ wwwauth[]=basic realm="example.com"
+ EOF
+
-+ cat >get-response.cred <<-EOF &&
++ expect_credential_query erase <<-EOF
+ protocol=http
+ host=$HOST_PORT
+ username=alice
+ password=invalid-passwd
++ wwwauth[]=bearer authority="id.example.com" q=1 p=0
++ wwwauth[]=basic realm="example.com"
+ EOF
-+
-+ test_must_fail git -c credential.helper="$CREDENTIAL_HELPER" ls-remote $ORIGIN_URL &&
-+
-+ test_cmp get-expected.cred get-actual.cred &&
-+ test_cmp erase-expected.cred erase-actual.cred
+'
+
test_doneSignature · 1 line
gitgitgadget
PATCH v6 1/12 daemon: libify socket setup and option functions
From: Matthew John Cheetham <[email protected]>
Extract functions for setting up listening sockets and keep-alive options from daemon.c to new daemon-utils.{c,h} files. Remove direct dependencies on global state by inlining the behaviour at the callsites for all libified functions.
--- Makefile | 1 + daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 23 ++++++ daemon.c | 214 +------------------------------------------------ 4 files changed, 237 insertions(+), 210 deletions(-) create mode 100644 daemon-utils.c create mode 100644 daemon-utils.h
Makefile +1 -0
@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o LIB_OBJS += csum-file.o LIB_OBJS += ctype.o LIB_OBJS += date.o +LIB_OBJS += daemon-utils.o LIB_OBJS += decorate.o LIB_OBJS += delta-islands.o LIB_OBJS += diagnose.o
daemon-utils.c new file +209 -0
209 lines not on this page · raw message
daemon-utils.h new file +23 -0
@@ -0,0 +1,23 @@ +#ifndef DAEMON_UTILS_H +#define DAEMON_UTILS_H + +#include "git-compat-util.h" +#include "string-list.h" + +typedef void (*log_fn )(const char *msg, ...); + +struct socketlist { +int *list; +size_t nr; +size_t alloc; +}; + +/* Enable sending of keep-alive messages on the socket. */ +void set_keep_alive (int sockfd,log_fn logerror); + +/* Setup a number of sockets to listen on the provided addresses. */ +void socksetup (struct string_list *listen_addr,int listen_port, +struct socketlist *socklist,int reuseaddr, +log_fn logerror); + +#endif
daemon.c +4 -210
246 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v6 2/12 daemon: libify child process handling functions
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Extract functions and structures for managing child processes started from the parent daemon-like process from daemon.c to the new shared daemon-utils.{c,h} files.
One minor functional change is introduced to check_dead_children() where the logging of a dead/disconnected child is now optional. With the 'libification' of these functions we extract the call to loginfo to a call to a function pointer, and guard the log message creation and logging behind a NULL check. Callers can now skip logging by passing NULL as the log_fn loginfo argument.
The behaviour of callers in daemon.c remains the same (save one extra NULL check) however as a pointer to loginfo is always passed.
--- daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 32 ++++++++++++++++++ daemon.c | 92 +++----------------------------------------------- 3 files changed, 114 insertions(+), 87 deletions(-)
daemon-utils.c +77 -0
@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port, } } } + +static int addrcmp (const struct sockaddr_storage *s1, +const struct sockaddr_storage *s2) +{ +const struct sockaddr *sa1 = (const struct sockaddr *) s1; +const struct sockaddr *sa2 = (const struct sockaddr *) s2; + +if (sa1->sa_family != sa2->sa_family) +return sa1->sa_family - sa2->sa_family; +if (sa1->sa_family ==AF_INET ) +return memcmp (&((struct sockaddr_in *)s1)->sin_addr, + &((struct sockaddr_in *)s2)->sin_addr, +sizeof (struct in_addr )); +#ifndef NO_IPV6 +if (sa1->sa_family ==AF_INET6 ) +return memcmp (&((struct sockaddr_in6 *)s1)->sin6_addr, + &((struct sockaddr_in6 *)s2)->sin6_addr, +sizeof (struct in6_addr )); +#endif +return 0 ; +} + +void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, +struct child *firstborn ,unsigned int *live_children) +{ +struct child *newborn, **cradle; + +CALLOC_ARRAY (newborn,1 ); + (*live_children)++; +memcpy (&newborn->cld, cld,sizeof (*cld)); +memcpy (&newborn->address, addr, addrlen); +for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) +if (!addrcmp (&(*cradle)->address, &newborn->address)) +break ; + newborn->next = *cradle; + *cradle = newborn; +} + +void kill_some_child (struct child *firstborn) +{ +const struct child *blanket, *next; + +if (!(blanket = firstborn)) +return ; + +for (; (next = blanket->next); blanket = next) +if (!addrcmp (&blanket->address, &next->address)) { +kill (blanket->cld.pid,SIGTERM ); +break ; + } +} + +void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo) +{ +int status; +pid_t pid; + +struct child **cradle, *blanket; +for (cradle = &firstborn; (blanket = *cradle);) +if ((pid =waitpid (blanket->cld.pid, &status,WNOHANG )) >1 ) { +if (loginfo) { +const char *dead ="" ; +if (status) + dead =" (with error)" ; +loginfo ("[%" PRIuMAX"] Disconnected%s" , + (uintmax_t )pid, dead); + } + +/* remove the child */ + *cradle = blanket->next; + (*live_children)--; +child_process_clear (&blanket->cld); +free (blanket); + }else + cradle = &blanket->next; +}
daemon-utils.h +32 -0
@@ -2,6 +2,7 @@#define DAEMON_UTILS_H #include "git-compat-util.h" +#include "run-command.h" #include "string-list.h" typedef void (*log_fn )(const char *msg, ...);
@@ -20,4 +21,35 @@ void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist, int reuseaddr, log_fn logerror); +struct child { +struct child *next; +struct child_process cld; +struct sockaddr_storage address; +}; + +/* +* Add the child_process to the set of children and increment the number of +* live children. +*/ +void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, +struct child *firstborn,unsigned int *live_children); + +/* +* Kill the newest connection from a duplicate IP. +* +* This function should be called if the number of connections grows +* past the maximum number of allowed connections. +*/ +void kill_some_child (struct child *firstborn); + +/* +* Check for children that have disconnected and remove them from the +* active set, decrementing the number of live children. +* +* Optionally log the child PID that disconnected by passing a loginfo +* function. +*/ +void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo); +#endif
daemon.c +5 -87
121 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v6 3/12 daemon: rename some esoteric/laboured terminology
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Rename some of the variables and function arguments used to manage child processes. The existing names are esoteric; stretching an analogy too far to the point of being confusing to understand.
Rename "firstborn" to "first_child", "newborn" to "new_cld", "blanket" to "current" and "cradle" to "ptr".
--- daemon-utils.c | 46 +++++++++++++++++++++++----------------------- daemon-utils.h | 6 +++--- daemon.c | 10 +++++----- 3 files changed, 31 insertions(+), 31 deletions(-)
daemon-utils.c +23 -23
@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1, }void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, -struct child *firstborn ,unsigned int *live_children) +struct child *first_child,unsigned int *live_children) { -struct child *newborn, **cradle; +struct child *new_cld, **current; -CALLOC_ARRAY (newborn,1 ); +CALLOC_ARRAY (new_cld,1 ); (*live_children)++; -memcpy (&newborn->cld, cld,sizeof (*cld)); -memcpy (&newborn->address, addr, addrlen); -for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) -if (!addrcmp (&(*cradle)->address, &newborn->address)) +memcpy (&new_cld->cld, cld,sizeof (*cld)); +memcpy (&new_cld->address, addr, addrlen); +for (current = &first_child; *current; current = &(*current)->next) +if (!addrcmp (&(*current)->address, &new_cld->address))break ; - newborn->next = *cradle; - *cradle = newborn; + new_cld->next = *current; + *current = new_cld; } -void kill_some_child (struct child *firstborn) +void kill_some_child (struct child *first_child) { -const struct child *blanket, *next; +const struct child *current, *next; -if (!(blanket = firstborn)) +if (!(current = first_child))return ; -for (; (next = blanket->next); blanket = next) -if (!addrcmp (&blanket->address, &next->address)) { -kill (blanket->cld.pid,SIGTERM ); +for (; (next = current->next); current = next) +if (!addrcmp (¤t->address, &next->address)) { +kill (current->cld.pid,SIGTERM );break ; } } -void check_dead_children (struct child *firstborn,unsigned int *live_children, +void check_dead_children (struct child *first_child,unsigned int *live_children,log_fn loginfo) {int status;pid_t pid; -struct child **cradle, *blanket; -for (cradle = &firstborn; (blanket = *cradle);) -if ((pid =waitpid (blanket->cld.pid, &status,WNOHANG )) >1 ) { +struct child **ptr, *current; +for (ptr = &first_child; (current = *ptr);) +if ((pid =waitpid (current->cld.pid, &status,WNOHANG )) >1 ) {if (loginfo) {const char *dead ="" ;if (status)
@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children, }/* remove the child */ - *cradle = blanket->next; + *ptr = current->next; (*live_children)--; -child_process_clear (&blanket->cld); -free (blanket); +child_process_clear (¤t->cld); +free (current); }else - cradle = &blanket->next; + ptr = ¤t->next; }
daemon-utils.h +3 -3
@@ -32,7 +32,7 @@ struct child {* live children. */ void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, -struct child *firstborn,unsigned int *live_children); +struct child *first_child,unsigned int *live_children);/* * Kill the newest connection from a duplicate IP.
@@ -40,7 +40,7 @@ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl* This function should be called if the number of connections grows * past the maximum number of allowed connections. */ -void kill_some_child (struct child *firstborn); +void kill_some_child (struct child *first_child);/* * Check for children that have disconnected and remove them from the
@@ -49,7 +49,7 @@ void kill_some_child(struct child *firstborn);* Optionally log the child PID that disconnected by passing a loginfo * function. */ -void check_dead_children (struct child *firstborn,unsigned int *live_children, +void check_dead_children (struct child *first_child,unsigned int *live_children,log_fn loginfo);#endif
daemon.c +5 -5
@@ -789,7 +789,7 @@ static int max_connections = 32;static unsigned int live_children; -static struct child *firstborn; +static struct child *first_child;static struct strvec cld_argv =STRVEC_INIT ;static void handle (int incoming,struct sockaddr *addr,socklen_t addrlen)
@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)struct child_process cld =CHILD_PROCESS_INIT ;if (max_connections && live_children >= max_connections) { -kill_some_child (firstborn); +kill_some_child (first_child);sleep (1 );/* give it some time to die */ -check_dead_children (firstborn, &live_children, loginfo); +check_dead_children (first_child, &live_children, loginfo);if (live_children >= max_connections) {close (incoming);logerror ("Too many children, dropping connection" );
@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)if (start_command (&cld))logerror ("unable to fork" );else -add_child (&cld, addr, addrlen, firstborn, &live_children); +add_child (&cld, addr, addrlen, first_child, &live_children); }static void child_handler (int signo)
@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)for (;;) {int i; -check_dead_children (firstborn, &live_children, loginfo); +check_dead_children (first_child, &live_children, loginfo);if (poll (pfd, socklist->nr,-1 ) <0 ) {if (errno !=EINTR ) {
Signature · 1 line
gitgitgadget
PATCH v6 4/12 test-http-server: add stub HTTP server test helper
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Introduce a mini HTTP server helper that in the future will be enhanced to provide a frontend for the git-http-backend, with support for arbitrary authentication schemes.
Right now, test-http-server is a pared-down copy of the git-daemon that always returns a 501 Not Implemented response to all callers.
--- Makefile | 1 + contrib/buildsystems/CMakeLists.txt | 11 +- t/helper/.gitignore | 1 + t/helper/test-http-server.c | 385 ++++++++++++++++++++++++++++ 4 files changed, 396 insertions(+), 2 deletions(-) create mode 100644 t/helper/test-http-server.c
Makefile +1 -0
@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o # Do not add more tests here unless they have extra dependencies. Add # them in TEST_BUILTINS_OBJS above. TEST_PROGRAMS_NEED_X += test-fake-ssh +TEST_PROGRAMS_NEED_X += test-http-server TEST_PROGRAMS_NEED_X += test-tool TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))
contrib/buildsystems/CMakeLists.txt +9 -2
@@ -961,6 +961,9 @@ if(BUILD_TESTING) add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c) target_link_libraries(test-fake-ssh common-main) +add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c) +target_link_libraries(test-http-server common-main) + #reftable-tests parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS") list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/")
@@ -980,6 +983,11 @@ if(MSVC) PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) set_target_properties(test-fake-ssh test-tool PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) + + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) endif() #wrapper scripts
@@ -987,8 +995,7 @@ set(wrapper_scripts git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar) set(wrapper_test_scripts - test-fake-ssh test-tool) - + test-http-server test-fake-ssh test-tool) foreach(script ${wrapper_scripts}) file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
t/helper/.gitignore +1 -0
@@ -1,2 +1,3 @@ /test-tool /test-fake-ssh +/test-http-server
t/helper/test-http-server.c new file +385 -0
385 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v6 04/12] test-http-server: add stub HTTP server test helper
Quoted 21 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]> [...]
t/helper/test-http-server.c new file +18
-0+enum worker_result { +/* +* Operation successful. +* Caller *might* keep the socket open and allow keep-alive. +*/ +WR_OK =0 , + enum worker_result wr = WR_OK; + +if (client_addr) +loginfo ("Connection from %s:%s" , client_addr, client_port); + +set_keep_alive (0 , logerror); + +while (1 ) { +if (write_in_full (STDOUT_FILENO , response,strlen (response)) <0 ) { +logerror ("unable to write response" ); + wr =WR_IO_ERROR ; + }
t/helper/test-http-server.c line 112
Quoted 10 lines
t/helper/test-http-server.c new file +10
-0+ +if (wr !=WR_OK ) +break ; + } + +close (STDIN_FILENO ); +close (STDOUT_FILENO ); + +return !!(wr &WR_IO_ERROR ); +}
We have cases where we assign "0" to a bitfield-looking structure like this, but only in cases where we're planning to use it as a boolean too.
Or, in other cases where we want some to be explicitly <-1.
Here though we're adding a mixed "OK" and error use, which seems a bit odd. Shouldn't we pick one or the other?
So far (maybe in later commits?) nothing uses WR_HANGUP, and oddly we also use the bitfield-looking thing as a return value from main()....
Re: [PATCH v6 04/12] test-http-server: add stub HTTP server test helper
Quoted 38 lines · Ævar Arnfjörð Bjarmason
Quoted 21 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]> [...]
t/helper/test-http-server.c new file +18
-0+enum worker_result { +/* +* Operation successful. +* Caller *might* keep the socket open and allow keep-alive. +*/ +WR_OK =0 , + enum worker_result wr = WR_OK; + +if (client_addr) +loginfo ("Connection from %s:%s" , client_addr, client_port); + +set_keep_alive (0 , logerror); + +while (1 ) { +if (write_in_full (STDOUT_FILENO , response,strlen (response)) <0 ) { +logerror ("unable to write response" ); + wr =WR_IO_ERROR ; + }t/helper/test-http-server.c line 112
Quoted 10 lines
t/helper/test-http-server.c new file +10
-0+ +if (wr !=WR_OK ) +break ; + } + +close (STDIN_FILENO ); +close (STDOUT_FILENO ); + +return !!(wr &WR_IO_ERROR ); +}We have cases where we assign "0" to a bitfield-looking structure like this, but only in cases where we're planning to use it as a boolean too.
Quoted 4 lines
Or, in other cases where we want some to be explicitly <-1.
Here though we're adding a mixed "OK" and error use, which seems a bit odd. Shouldn't we pick one or the other?
You make a fair point about bitfields vs simple integer values. This was a holdover from previous early hacking on this work where I had the bitfield serve as a way to communicate the aspects of "does this count as an error?" and "should we close the connection?".
Upon second thought, I think just simple integer values would be fine as really only an "OK" and "HANGUP" are non-errors (the latter being the case that the client gracefully ended the connection without an error and we should exit).
Check for my next iteration for a rework on these worker_result values.
Quoted 2 lines
So far (maybe in later commits?) nothing uses
WR_HANGUP, and oddly we also use the bitfield-looking thing as a return value frommain()....
We don't use the enum worker_result values as a return from main. We only ever return 0 or 1 as we return worker() from main, and the only return from worker() is !!(wr & WR_IO_ERROR) - 1 if we have WR_IO_ERROR set, otherwise 0.
Thanks,
Matthew
PATCH v6 5/12 test-http-server: add HTTP error response function
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Introduce a function to the test-http-server test helper to write more full and valid HTTP error responses, including all the standard response headers like Server and Date.
--- t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 5 deletions(-)
t/helper/test-http-server.c +53 -5
@@ -83,9 +83,59 @@ enum worker_result { WR_HANGUP = 1<<1, }; +static enum worker_result send_http_error ( +int fd, +int http_code,const char *http_code_name, +int retry_after_seconds,struct string_list *response_headers, +enum worker_result wr_in) +{ +struct strbuf response_header =STRBUF_INIT ; +struct strbuf response_content =STRBUF_INIT ; +struct string_list_item *h; +enum worker_result wr; + +strbuf_addf (&response_content,"Error: %d %s\r\n" , + http_code, http_code_name); +if (retry_after_seconds >0 ) +strbuf_addf (&response_content,"Retry-After: %d\r\n" , + retry_after_seconds); + +strbuf_addf (&response_header,"HTTP/1.1 %d %s\r\n" , http_code, http_code_name); +strbuf_addstr (&response_header,"Cache-Control: private\r\n" ); +strbuf_addstr (&response_header,"Content-Type: text/plain\r\n" ); +strbuf_addf (&response_header,"Content-Length: %d\r\n" , (int )response_content.len); +if (retry_after_seconds >0 ) +strbuf_addf (&response_header,"Retry-After: %d\r\n" , retry_after_seconds); +strbuf_addf ( &response_header,"Server: test-http-server/%s\r\n" , git_version_string); +strbuf_addf ( &response_header,"Date: %s\r\n" ,show_date (time (NULL ),0 ,DATE_MODE (RFC2822 ))); +if (response_headers) +for_each_string_list_item (h, response_headers) +strbuf_addf (&response_header,"%s\r\n" , h->string); +strbuf_addstr (&response_header,"\r\n" ); + +if (write_in_full (fd, response_header.buf, response_header.len) <0 ) { +logerror ("unable to write response header" ); + wr =WR_IO_ERROR ; + goto done; + } + +if (write_in_full (fd, response_content.buf, response_content.len) <0 ) { +logerror ("unable to write response content body" ); + wr =WR_IO_ERROR ; + goto done; + } + + wr = wr_in; + +done: +strbuf_release (&response_header); +strbuf_release (&response_content); + +return wr; +} +static enum worker_result worker (void ) { -const char *response ="HTTP/1.1 501 Not Implemented\r\n" ;char *client_addr =getenv ("REMOTE_ADDR" );char *client_port =getenv ("REMOTE_PORT" );enum worker_result wr =WR_OK ;
@@ -96,10 +146,8 @@ static enum worker_result worker(void)set_keep_alive (0 , logerror);while (1 ) { -if (write_in_full (STDOUT_FILENO , response,strlen (response)) <0 ) { -logerror ("unable to write response" ); - wr =WR_IO_ERROR ; - } + wr =send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 , +NULL ,WR_OK |WR_HANGUP );if (wr !=WR_OK )break ;
Signature · 1 line
gitgitgadget
Re: [PATCH v6 05/12] test-http-server: add HTTP error response function
Quoted 16 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Introduce a function to the test-http-server test helper to write more full and valid HTTP error responses, including all the standard response headers like
ServerandDate.Signed-off-by Matthew John Cheetham <[email protected]>--- t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 5 deletions(-)t/helper/test-http-server.c +0
-0
t/helper/test-http-server.c enum worker_result { · line 84
Quoted 2 lines
t/helper/test-http-server.c +0
-0@@ -83,9 +83,59 @@ enum worker_result { WR_HANGUP = 1<<1, };
...okey, this is the commit that makes use of WR_HANGUP. Whatever else we do, let's then squash that addition into this change.
t/helper/test-http-server.c enum worker_result { · line 90
Quoted 5 lines
t/helper/test-http-server.c +5
-0+static enum worker_result send_http_error ( +int fd, +int http_code,const char *http_code_name, +int retry_after_seconds,struct string_list *response_headers, +enum worker_result wr_in)
In general in this series you are mis-indenting argument lists. Our usual style is to wrap at 79 characters, then to align (with tabs and spaces) with the "(".
So in this case:
static enum worker_result send_http_error(int fd, int http_code,
const char *http_code_name, int retry_after_seconds, struct string_list *response_headers, enum worker_result wr_in)
t/helper/test-http-server.c enum worker_result { · line 98
Quoted 8 lines
t/helper/test-http-server.c +8
-0+{ +struct strbuf response_header =STRBUF_INIT ; +struct strbuf response_content =STRBUF_INIT ; +struct string_list_item *h; +enum worker_result wr; + +strbuf_addf (&response_content,"Error: %d %s\r\n" , + http_code, http_code_name);
Ditto here, where "http_code" should go on the preceding line...
t/helper/test-http-server.c enum worker_result { · line 103
Quoted 5 lines
t/helper/test-http-server.c +5
-0+if (retry_after_seconds >0 ) +strbuf_addf (&response_content,"Retry-After: %d\r\n" , + retry_after_seconds); + +strbuf_addf (&response_header,"HTTP/1.1 %d %s\r\n" , http_code, http_code_name);
...and here there's a lack of such wrapping...
t/helper/test-http-server.c enum worker_result { · line 110
Quoted 7 lines
t/helper/test-http-server.c +7
-0+strbuf_addstr (&response_header,"Cache-Control: private\r\n" ); +strbuf_addstr (&response_header,"Content-Type: text/plain\r\n" ); +strbuf_addf (&response_header,"Content-Length: %d\r\n" , (int )response_content.len); +if (retry_after_seconds >0 ) +strbuf_addf (&response_header,"Retry-After: %d\r\n" , retry_after_seconds); +strbuf_addf ( &response_header,"Server: test-http-server/%s\r\n" , git_version_string); +strbuf_addf ( &response_header,"Date: %s\r\n" ,show_date (time (NULL ),0 ,DATE_MODE (RFC2822 )));
...here you're adding strange whitespace at the start of an argument list...
t/helper/test-http-server.c enum worker_result { · line 114
Quoted 4 lines
t/helper/test-http-server.c +4
-0+if (response_headers) +for_each_string_list_item (h, response_headers) +strbuf_addf (&response_header,"%s\r\n" , h->string); +strbuf_addstr (&response_header,"\r\n" );
To comment on the code a bit, this whole thing would be more readable IMO if your own headers were also a "struct string_list". Yes we'd waste a bit more memory, but in this case that's fine..
I.e. don't add the "\r\n" every time, just:
string_list_append(&headers, "Cache-Control: private");
etc.
Then at the end you'd do e.g.:
add_headers(&buf, &headers); if (response_headers) add_headers(&buf, response_headers);
Where the add_headers() is a trivial "static" helper which does that for_each_string_list_item() loop above.
t/helper/test-http-server.c static enum worker_result worker(void) · line 150
Quoted 7 lines
t/helper/test-http-server.c +2
-4while (1 ) { -if (write_in_full (STDOUT_FILENO , response,strlen (response)) <0 ) { -logerror ("unable to write response" ); - wr =WR_IO_ERROR ; - } + wr =send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 , +NULL ,WR_OK |WR_HANGUP );
This *does* use correct wrapping & indenation for a continuing argument list.
Re: [PATCH v6 05/12] test-http-server: add HTTP error response function
Quoted 73 lines · Ævar Arnfjörð Bjarmason
Quoted 16 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Introduce a function to the test-http-server test helper to write more full and valid HTTP error responses, including all the standard response headers like
ServerandDate.Signed-off-by Matthew John Cheetham <[email protected]>--- t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 5 deletions(-)t/helper/test-http-server.c +0
-0t/helper/test-http-server.c enum worker_result { · line 84
Quoted 2 lines
t/helper/test-http-server.c +0
-0@@ -83,9 +83,59 @@ enum worker_result { WR_HANGUP = 1<<1, };...okey, this is the commit that makes use of
WR_HANGUP. Whatever else we do, let's then squash that addition into this change.t/helper/test-http-server.c enum worker_result { · line 90
Quoted 5 lines
t/helper/test-http-server.c +5
-0+static enum worker_result send_http_error ( +int fd, +int http_code,const char *http_code_name, +int retry_after_seconds,struct string_list *response_headers, +enum worker_result wr_in)In general in this series you are mis-indenting argument lists. Our usual style is to wrap at 79 characters, then to align (with tabs and spaces) with the "(".
So in this case:
static enum
worker_resultsend_http_error(int fd, inthttp_code,
const char*http_code_name, intretry_after_seconds, structstring_list*response_headers, enumworker_resultwr_in)t/helper/test-http-server.c enum worker_result { · line 98
Quoted 8 lines
t/helper/test-http-server.c +8
-0+{ +struct strbuf response_header =STRBUF_INIT ; +struct strbuf response_content =STRBUF_INIT ; +struct string_list_item *h; +enum worker_result wr; + +strbuf_addf (&response_content,"Error: %d %s\r\n" , + http_code, http_code_name);Ditto here, where "
http_code" should go on the preceding line...t/helper/test-http-server.c enum worker_result { · line 103
Quoted 5 lines
t/helper/test-http-server.c +5
-0+if (retry_after_seconds >0 ) +strbuf_addf (&response_content,"Retry-After: %d\r\n" , + retry_after_seconds); + +strbuf_addf (&response_header,"HTTP/1.1 %d %s\r\n" , http_code, http_code_name);...and here there's a lack of such wrapping...
t/helper/test-http-server.c enum worker_result { · line 110
Quoted 7 lines
t/helper/test-http-server.c +7
-0+strbuf_addstr (&response_header,"Cache-Control: private\r\n" ); +strbuf_addstr (&response_header,"Content-Type: text/plain\r\n" ); +strbuf_addf (&response_header,"Content-Length: %d\r\n" , (int )response_content.len); +if (retry_after_seconds >0 ) +strbuf_addf (&response_header,"Retry-After: %d\r\n" , retry_after_seconds); +strbuf_addf ( &response_header,"Server: test-http-server/%s\r\n" , git_version_string); +strbuf_addf ( &response_header,"Date: %s\r\n" ,show_date (time (NULL ),0 ,DATE_MODE (RFC2822 )));...here you're adding strange whitespace at the start of an argument list...
Quoted 4 lines
t/helper/test-http-server.c enum worker_result { · line 114
Quoted 4 lines
t/helper/test-http-server.c +4
-0+if (response_headers) +for_each_string_list_item (h, response_headers) +strbuf_addf (&response_header,"%s\r\n" , h->string); +strbuf_addstr (&response_header,"\r\n" );
Argh! Thanks again for catching these. I shall address them.
Quoted 10 lines
To comment on the code a bit, this whole thing would be more readable IMO if your own headers were also a "struct
string_list". Yes we'd waste a bit more memory, but in this case that's fine..I.e. don't add the "\r\n" every time, just:
string_list_append(&headers, "Cache-Control: private");etc.
Quoted 8 lines
Then at the end you'd do e.g.:
add_headers(&buf, &headers); if (response_headers) add_headers(&buf, response_headers);Where the
add_headers()is a trivial "static" helper which does thatfor_each_string_list_item()loop above.
In reality this only helps simplify the code in the case of a simple static header like "Cache-Control: private". There's no string_list_appendf or similar where I need to append a header that contains dynamic information (date, content length, etc).
Building the strbuf directly, and specifying the CRLF seems a lot easier IMO.
Quoted 10 lines
t/helper/test-http-server.c static enum worker_result worker(void) · line 150
Quoted 7 lines
t/helper/test-http-server.c +2
-4while (1 ) { -if (write_in_full (STDOUT_FILENO , response,strlen (response)) <0 ) { -logerror ("unable to write response" ); - wr =WR_IO_ERROR ; - } + wr =send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 , +NULL ,WR_OK |WR_HANGUP );This *does* use correct wrapping & indenation for a continuing argument list.
Thanks,
Matthew
PATCH v6 6/12 test-http-server: add HTTP request parsing
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add ability to parse HTTP requests to the test-http-server test helper.
--- t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++- 1 file changed, 173 insertions(+), 2 deletions(-)
t/helper/test-http-server.c +173 -2
@@ -83,6 +83,42 @@ enum worker_result { WR_HANGUP = 1<<1, }; +/* +* Fields from a parsed HTTP request. +*/ +struct req { +struct strbuf start_line; + +const char *method; +const char *http_version; + +struct strbuf uri_path; +struct strbuf query_args; + +struct string_list header_list; +const char *content_type; +ssize_t content_length; +}; + +#define REQ__INIT { \ + .start_line = STRBUF_INIT, \ + .uri_path = STRBUF_INIT, \ + .query_args = STRBUF_INIT, \ + .header_list = STRING_LIST_INIT_NODUP, \ + .content_type = NULL, \ + .content_length = -1 \ + } + +static void req__release (struct req *req) +{ +strbuf_release (&req->start_line); + +strbuf_release (&req->uri_path); +strbuf_release (&req->query_args); + +string_list_clear (&req->header_list,0 ); +} + static enum worker_result send_http_error( int fd, int http_code, const char *http_code_name,
@@ -134,8 +170,136 @@ done:return wr; } +/* +* Read the HTTP request up to the start of the optional message-body. +* We do this byte-by-byte because we have keep-alive turned on and +* cannot rely on an EOF. +* +* https://tools.ietf.org/html/rfc7230 +* +* We cannot call die() here because our caller needs to properly +* respond to the client and/or close the socket before this +* child exits so that the client doesn't get a connection reset +* by peer error. +*/ +static enum worker_result req__read (struct req *req,int fd) +{ +struct strbuf h =STRBUF_INIT ; +struct string_list start_line_fields =STRING_LIST_INIT_DUP ; +int nr_start_line_fields; +const char *uri_target; +const char *query; +char *hp; +const char *hv; + +enum worker_result result =WR_OK ; + +/* +* Read line 0 of the request and split it into component parts: +* +* <method> SP <uri-target> SP <HTTP-version> CRLF +* +*/ +if (strbuf_getwholeline_fd (&req->start_line, fd,'\n' ) ==EOF ) { + result =WR_OK |WR_HANGUP ; + goto done; + } + +strbuf_trim_trailing_newline (&req->start_line); + + nr_start_line_fields =string_list_split (&start_line_fields, + req->start_line.buf, +' ' ,-1 ); +if (nr_start_line_fields !=3 ) { +logerror ("could not parse request start-line '%s'" , + req->start_line.buf); + result =WR_IO_ERROR ; + goto done; + } + + req->method =xstrdup (start_line_fields.items[0 ].string); + req->http_version =xstrdup (start_line_fields.items[2 ].string); + + uri_target = start_line_fields.items[1 ].string; + +if (strcmp (req->http_version,"HTTP/1.1" )) { +logerror ("unsupported version '%s' (expecting HTTP/1.1)" , + req->http_version); + result =WR_IO_ERROR ; + goto done; + } + + query =strchr (uri_target,'?' ); + +if (query) { +strbuf_add (&req->uri_path, uri_target, (query - uri_target)); +strbuf_trim_trailing_dir_sep (&req->uri_path); +strbuf_addstr (&req->query_args, query +1 ); + }else { +strbuf_addstr (&req->uri_path, uri_target); +strbuf_trim_trailing_dir_sep (&req->uri_path); + } + +/* +* Read the set of HTTP headers into a string-list. +*/ +while (1 ) { +if (strbuf_getwholeline_fd (&h, fd,'\n' ) ==EOF ) + goto done; +strbuf_trim_trailing_newline (&h); + +if (!h.len) + goto done;/* a blank line ends the header */ + + hp =strbuf_detach (&h,NULL ); +string_list_append (&req->header_list, hp); + +/* also store common request headers as struct req members */ +if (skip_prefix (hp,"Content-Type: " , &hv)) { + req->content_type = hv; + }else if (skip_prefix (hp,"Content-Length: " , &hv)) { + req->content_length =strtol (hv, &hp,10 ); + } + } + +/* +* We do not attempt to read the <message-body>, if it exists. +* We let our caller read/chunk it in as appropriate. +*/ + +done: +string_list_clear (&start_line_fields,0 ); + +/* +* This is useful for debugging the request, but very noisy. +*/ +if (trace2_is_enabled ()) { +struct string_list_item *item; +trace2_printf ("%s: %s" ,TR2_CAT , req->start_line.buf); +trace2_printf ("%s: hver: %s" ,TR2_CAT , req->http_version); +trace2_printf ("%s: hmth: %s" ,TR2_CAT , req->method); +trace2_printf ("%s: path: %s" ,TR2_CAT , req->uri_path.buf); +trace2_printf ("%s: qury: %s" ,TR2_CAT , req->query_args.buf); +if (req->content_length >=0 ) +trace2_printf ("%s: clen: %d" ,TR2_CAT , req->content_length); +if (req->content_type) +trace2_printf ("%s: ctyp: %s" ,TR2_CAT , req->content_type); +for_each_string_list_item (item, &req->header_list) +trace2_printf ("%s: hdrs: %s" ,TR2_CAT , item->string); + } + +return result; +} + +static enum worker_result dispatch (struct req *req) +{ +return send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 ,NULL , +WR_OK |WR_HANGUP ); +} +static enum worker_result worker (void ) { +struct req req =REQ__INIT ;char *client_addr =getenv ("REMOTE_ADDR" );char *client_port =getenv ("REMOTE_PORT" );enum worker_result wr =WR_OK ;
@@ -146,9 +310,16 @@ static enum worker_result worker(void)set_keep_alive (0 , logerror);while (1 ) { - wr =send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 , -NULL ,WR_OK |WR_HANGUP ); +req__release (&req); + +alarm (timeout); + wr =req__read (&req,0 ); +alarm (0 ); + +if (wr !=WR_OK ) +break ; + wr =dispatch (&req);if (wr !=WR_OK )break ; }
Signature · 1 line
gitgitgadget
Re: [PATCH v6 06/12] test-http-server: add HTTP request parsing
Quoted 32 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Add ability to parse HTTP requests to the test-http-server test helper.
Signed-off-by Matthew John Cheetham <[email protected]>--- t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++- 1 file changed, 173 insertions(+), 2 deletions(-)t/helper/test-http-server.c +15
-0@@ -83,6 +83,42 @@ enum worker_result { WR_HANGUP = 1<<1, }; +/* +* Fields from a parsed HTTP request. +*/ +struct req { +struct strbuf start_line; + +const char *method; +const char *http_version; + +struct strbuf uri_path; +struct strbuf query_args; + +struct string_list header_list; +const char *content_type; +ssize_t content_length;
t/helper/test-http-server.c enum worker_result { · line 110
Quoted 10 lines
t/helper/test-http-server.c +10
-0+}; + +#define REQ__INIT { \ + .start_line = STRBUF_INIT, \ + .uri_path = STRBUF_INIT, \ + .query_args = STRBUF_INIT, \ + .header_list = STRING_LIST_INIT_NODUP, \ + .content_type = NULL, \ + .content_length = -1 \ + }
Style nit: Don't indent the trailing "}", and add a "," after the last "content_length" item.
We omit the comma by convention when there really should not be another item, such as when we have a "NULL" terminator, here though we might add a struct element at the end, so...
Quoted 67 lines
t/helper/test-http-server.c +67
-0+static enum worker_result req__read (struct req *req,int fd) +{ +struct strbuf h =STRBUF_INIT ; +struct string_list start_line_fields =STRING_LIST_INIT_DUP ; +int nr_start_line_fields; +const char *uri_target; +const char *query; +char *hp; +const char *hv; + +enum worker_result result =WR_OK ; + +/* +* Read line 0 of the request and split it into component parts: +* +* <method> SP <uri-target> SP <HTTP-version> CRLF +* +*/ +if (strbuf_getwholeline_fd (&req->start_line, fd,'\n' ) ==EOF ) { + result =WR_OK |WR_HANGUP ; + goto done; + } + +strbuf_trim_trailing_newline (&req->start_line); + + nr_start_line_fields =string_list_split (&start_line_fields, + req->start_line.buf, +' ' ,-1 ); +if (nr_start_line_fields !=3 ) { +logerror ("could not parse request start-line '%s'" , + req->start_line.buf); + result =WR_IO_ERROR ; + goto done; + } + + req->method =xstrdup (start_line_fields.items[0 ].string); + req->http_version =xstrdup (start_line_fields.items[2 ].string); + + uri_target = start_line_fields.items[1 ].string; + +if (strcmp (req->http_version,"HTTP/1.1" )) { +logerror ("unsupported version '%s' (expecting HTTP/1.1)" , + req->http_version); + result =WR_IO_ERROR ; + goto done; + } + + query =strchr (uri_target,'?' ); + +if (query) { +strbuf_add (&req->uri_path, uri_target, (query - uri_target)); +strbuf_trim_trailing_dir_sep (&req->uri_path); +strbuf_addstr (&req->query_args, query +1 ); + }else { +strbuf_addstr (&req->uri_path, uri_target); +strbuf_trim_trailing_dir_sep (&req->uri_path); + } + +/* +* Read the set of HTTP headers into a string-list. +*/ +while (1 ) { +if (strbuf_getwholeline_fd (&h, fd,'\n' ) ==EOF ) + goto done; +strbuf_trim_trailing_newline (&h); + +if (!h.len)
t/helper/test-http-server.c done: · line 261
Quoted 10 lines
t/helper/test-http-server.c +10
-0+ goto done;/* a blank line ends the header */ + + hp =strbuf_detach (&h,NULL ); +string_list_append (&req->header_list, hp); + +/* also store common request headers as struct req members */ +if (skip_prefix (hp,"Content-Type: " , &hv)) { + req->content_type = hv; + }else if (skip_prefix (hp,"Content-Length: " , &hv)) { + req->content_length =strtol (hv, &hp,10 );
In POSIX the "ssize_t" is not a "this is the unsigned size_t", but can be a much smaller integer type (although in practice it tends to be the signed version of "size_t".
But this seems like a potential overflow trap as a result, but sometimes we need to live with "ssize_t".
However, in this case it seems like we don't, as it seems the only reason you init'd this to -1 and then...
t/helper/test-http-server.c done: · line 284
Quoted 9 lines
t/helper/test-http-server.c +9
-0+if (trace2_is_enabled ()) { +struct string_list_item *item; +trace2_printf ("%s: %s" ,TR2_CAT , req->start_line.buf); +trace2_printf ("%s: hver: %s" ,TR2_CAT , req->http_version); +trace2_printf ("%s: hmth: %s" ,TR2_CAT , req->method); +trace2_printf ("%s: path: %s" ,TR2_CAT , req->uri_path.buf); +trace2_printf ("%s: qury: %s" ,TR2_CAT , req->query_args.buf); +if (req->content_length >=0 ) +trace2_printf ("%s: clen: %d" ,TR2_CAT , req->content_length);
...use that ">= 0" is to keep the state of "did I assign to this above?
So firstly, shouldn't we error or something on a "Content-Length: 0", and aside from that wouldn't we just have a "int have_content_length = 0" in this function that we'd then flip to 1?
Re: [PATCH v6 06/12] test-http-server: add HTTP request parsing
Quoted 49 lines · Ævar Arnfjörð Bjarmason
Quoted 32 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Add ability to parse HTTP requests to the test-http-server test helper.
Signed-off-by Matthew John Cheetham <[email protected]>--- t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++- 1 file changed, 173 insertions(+), 2 deletions(-)t/helper/test-http-server.c +15
-0@@ -83,6 +83,42 @@ enum worker_result { WR_HANGUP = 1<<1, }; +/* +* Fields from a parsed HTTP request. +*/ +struct req { +struct strbuf start_line; + +const char *method; +const char *http_version; + +struct strbuf uri_path; +struct strbuf query_args; + +struct string_list header_list; +const char *content_type; +ssize_t content_length;t/helper/test-http-server.c enum worker_result { · line 110
Quoted 10 lines
t/helper/test-http-server.c +10
-0+}; + +#define REQ__INIT { \ + .start_line = STRBUF_INIT, \ + .uri_path = STRBUF_INIT, \ + .query_args = STRBUF_INIT, \ + .header_list = STRING_LIST_INIT_NODUP, \ + .content_type = NULL, \ + .content_length = -1 \ + }Style nit: Don't indent the trailing "}", and add a "," after the last "
content_length" item.
Quoted 3 lines
We omit the comma by convention when there really should not be another item, such as when we have a "NULL" terminator, here though we might add a struct element at the end, so...
Sure.
Quoted 100 lines
Quoted 67 lines
t/helper/test-http-server.c +67
-0+static enum worker_result req__read (struct req *req,int fd) +{ +struct strbuf h =STRBUF_INIT ; +struct string_list start_line_fields =STRING_LIST_INIT_DUP ; +int nr_start_line_fields; +const char *uri_target; +const char *query; +char *hp; +const char *hv; + +enum worker_result result =WR_OK ; + +/* +* Read line 0 of the request and split it into component parts: +* +* <method> SP <uri-target> SP <HTTP-version> CRLF +* +*/ +if (strbuf_getwholeline_fd (&req->start_line, fd,'\n' ) ==EOF ) { + result =WR_OK |WR_HANGUP ; + goto done; + } + +strbuf_trim_trailing_newline (&req->start_line); + + nr_start_line_fields =string_list_split (&start_line_fields, + req->start_line.buf, +' ' ,-1 ); +if (nr_start_line_fields !=3 ) { +logerror ("could not parse request start-line '%s'" , + req->start_line.buf); + result =WR_IO_ERROR ; + goto done; + } + + req->method =xstrdup (start_line_fields.items[0 ].string); + req->http_version =xstrdup (start_line_fields.items[2 ].string); + + uri_target = start_line_fields.items[1 ].string; + +if (strcmp (req->http_version,"HTTP/1.1" )) { +logerror ("unsupported version '%s' (expecting HTTP/1.1)" , + req->http_version); + result =WR_IO_ERROR ; + goto done; + } + + query =strchr (uri_target,'?' ); + +if (query) { +strbuf_add (&req->uri_path, uri_target, (query - uri_target)); +strbuf_trim_trailing_dir_sep (&req->uri_path); +strbuf_addstr (&req->query_args, query +1 ); + }else { +strbuf_addstr (&req->uri_path, uri_target); +strbuf_trim_trailing_dir_sep (&req->uri_path); + } + +/* +* Read the set of HTTP headers into a string-list. +*/ +while (1 ) { +if (strbuf_getwholeline_fd (&h, fd,'\n' ) ==EOF ) + goto done; +strbuf_trim_trailing_newline (&h); + +if (!h.len)t/helper/test-http-server.c done: · line 261
Quoted 10 lines
t/helper/test-http-server.c +10
-0+ goto done;/* a blank line ends the header */ + + hp =strbuf_detach (&h,NULL ); +string_list_append (&req->header_list, hp); + +/* also store common request headers as struct req members */ +if (skip_prefix (hp,"Content-Type: " , &hv)) { + req->content_type = hv; + }else if (skip_prefix (hp,"Content-Length: " , &hv)) { + req->content_length =strtol (hv, &hp,10 );In POSIX the "
ssize_t" is not a "this is the unsignedsize_t", but can be a much smaller integer type (although in practice it tends to be the signed version of "size_t".But this seems like a potential overflow trap as a result, but sometimes we need to live with "
ssize_t".However, in this case it seems like we don't, as it seems the only reason you init'd this to -1 and then...
t/helper/test-http-server.c done: · line 284
Quoted 9 lines
t/helper/test-http-server.c +9
-0+if (trace2_is_enabled ()) { +struct string_list_item *item; +trace2_printf ("%s: %s" ,TR2_CAT , req->start_line.buf); +trace2_printf ("%s: hver: %s" ,TR2_CAT , req->http_version); +trace2_printf ("%s: hmth: %s" ,TR2_CAT , req->method); +trace2_printf ("%s: path: %s" ,TR2_CAT , req->uri_path.buf); +trace2_printf ("%s: qury: %s" ,TR2_CAT , req->query_args.buf); +if (req->content_length >=0 ) +trace2_printf ("%s: clen: %d" ,TR2_CAT , req->content_length);...use that ">= 0" is to keep the state of "did I assign to this above?
Quoted 3 lines
So firstly, shouldn't we error or something on a "Content-Length: 0", and aside from that wouldn't we just have a "int
have_content_length= 0" in this function that we'd then flip to 1?
It seems like the perfect type for such a non-zero-or-error size value; from POSIX specifications[1]:
Quoted 6 lines
...
size_t
Used for sizes of objects.ssize_t
Used for a count of bytes or an error indication.
...
But you're probably right here that ssize_t isn't that suitable in practice due to the comically low minimum size of the SSIZE_MAX (2^15 I believe).
RFC 9110 §8.6 [2] addresses the Content-Length HTTP header and says that its value should be non-negative, but also have no upper bound; we're gonna have to set at least some practical limit.
Libcurl handles this by writing it's own parsing function that's good up to a max 64-bit integer value [3][4].
Given this is for a test helper and only going to be receiving data from tests, I propose just using something like uintmax_t and storing a bit with unsigned has_content_length:1; to show if we actually got a header in the request or not.
Thanks,
Matthew
[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_types.h.html
[2] https://www.rfc-editor.org/rfc/rfc9110.html#section-8.6
[3] https://github.com/curl/curl/blob/6113dec2a829d4ab766428ccca9535b7a5efd012/lib/http.c#L3348-L3349
[4] https://github.com/curl/curl/blob/6113dec2a829d4ab766428ccca9535b7a5efd012/lib/strtoofft.c#L214-L218
PATCH v6 7/12 test-http-server: pass Git requests to http-backend
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Teach the test-http-sever test helper to forward Git requests to the git-http-backend.
Introduce a new test script t5556-http-auth.sh that spins up the test HTTP server and attempts an ls-remote on the served repository, without any authentication.
--- t/helper/test-http-server.c | 71 ++++++++++++++++++++++++ t/t5556-http-auth.sh | 107 ++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+) create mode 100755 t/t5556-http-auth.sh
t/helper/test-http-server.c +71 -0
@@ -291,8 +291,79 @@ done:return result; } +static int is_git_request (struct req *req) +{ +static regex_t *smart_http_regex; +static int initialized; + +if (!initialized) { + smart_http_regex =xmalloc (sizeof (*smart_http_regex)); +/* +* This regular expression matches all dumb and smart HTTP +* requests that are currently in use, and defined in +* Documentation/gitprotocol-http.txt. +* +*/ +if (regcomp (smart_http_regex,"^/(HEAD|info/refs|" +"objects/info/[^/]+|git-(upload|receive)-pack)$" , +REG_EXTENDED )) { +warning ("could not compile smart HTTP regex" ); + smart_http_regex =NULL ; + } + initialized =1 ; + } + +return smart_http_regex && + !regexec (smart_http_regex, req->uri_path.buf,0 ,NULL ,0 ); +} + +static enum worker_result do__git (struct req *req) +{ +const char *ok ="HTTP/1.1 200 OK\r\n" ; +struct child_process cp =CHILD_PROCESS_INIT ; +int res; + +/* +* Note that we always respond with a 200 OK response even if the +* http-backend process exits with an error. This helper is intended +* only to be used to exercise the HTTP auth handling in the Git client, +* and specifically around authentication (not handled by http-backend). +* +* If we wanted to respond with a more 'valid' HTTP response status then +* we'd need to buffer the output of http-backend, wait for and grok the +* exit status of the process, then write the HTTP status line followed +* by the http-backend output. This is outside of the scope of this test +* helper's use at time of writing. +*/ +if (write (STDOUT_FILENO , ok,strlen (ok)) <0 ) +return error (_ ("could not send '%s'" ), ok); + +strvec_pushf (&cp.env,"REQUEST_METHOD=%s" , req->method); +strvec_pushf (&cp.env,"PATH_TRANSLATED=%s" , + req->uri_path.buf); +strvec_push (&cp.env,"SERVER_PROTOCOL=HTTP/1.1" ); +if (req->query_args.len) +strvec_pushf (&cp.env,"QUERY_STRING=%s" , + req->query_args.buf); +if (req->content_type) +strvec_pushf (&cp.env,"CONTENT_TYPE=%s" , + req->content_type); +if (req->content_length >=0 ) +strvec_pushf (&cp.env,"CONTENT_LENGTH=%" PRIdMAX, + (intmax_t )req->content_length); + cp.git_cmd =1 ; +strvec_push (&cp.args,"http-backend" ); + res =run_command (&cp); +close (STDOUT_FILENO ); +close (STDIN_FILENO ); +return !!res; +} +static enum worker_result dispatch (struct req *req) { +if (is_git_request (req)) +return do__git (req); +return send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 ,NULL ,WR_OK |WR_HANGUP ); }
t/t5556-http-auth.sh new file +107 -0
107 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v6 8/12 test-http-server: add simple authentication
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add simple authentication to the test-http-server test helper. Authentication schemes and sets of valid tokens can be specified via a configuration file (in the normal gitconfig file format). Incoming requests are compared against the set of valid schemes and tokens and only approved if a matching token is found, or if no auth was provided and anonymous auth is enabled.
Configuration for auth includes a simple set of three options:
[auth]
challenge = <scheme>[:<challenge_params>]
token = <scheme>:[<token>]*
allowAnonymous = <bool>
auth.challenge allows you define what authentication schemes, and optional challenge parameters the server should use. Scheme names are unique and subsequently specified challenge parameters in the config file will replace previously specified ones.
auth.token allows you to define the set of value token values for an authentication scheme. This is a multi-var and each entry in the config file will append to the set of valid tokens for that scheme. Specifying an empty token value will clear the list of tokens so far for that scheme, i.e. token = <scheme>:.
auth.allowAnonymous controls whether or not unauthenticated requests (those without any Authorization headers) should succeed or not, and trigger a 401 Unauthorized response.
--- t/helper/test-http-server.c | 233 +++++++++++++++++++++++++++++++++++- t/t5556-http-auth.sh | 43 ++++++- 2 files changed, 273 insertions(+), 3 deletions(-)
t/helper/test-http-server.c +231 -2
@@ -7,6 +7,7 @@#include "version.h" #include "dir.h" #include "date.h" +#include "config.h" #define TR2_CAT "test-http-server"
@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =" [--timeout=<n>] [--max-connections=<n>]\n" " [--reuseaddr] [--pid-file=<file>]\n" " [--listen=<host_or_ipaddr>]* [--port=<n>]\n" +" [--auth-config=<file>]\n" ;static unsigned int timeout;
@@ -317,7 +319,7 @@ static int is_git_request(struct req *req) !regexec (smart_http_regex, req->uri_path.buf,0 ,NULL ,0 ); } -static enum worker_result do__git (struct req *req) +static enum worker_result do__git (struct req *req,const char *user) {const char *ok ="HTTP/1.1 200 OK\r\n" ;struct child_process cp =CHILD_PROCESS_INIT ;
@@ -334,10 +336,16 @@ static enum worker_result do__git(struct req *req)* exit status of the process, then write the HTTP status line followed * by the http-backend output. This is outside of the scope of this test * helper's use at time of writing. +* +* The important auth responses (401) we are handling prior to getting +* to this point. */ if (write (STDOUT_FILENO , ok,strlen (ok)) <0 )return error (_ ("could not send '%s'" ), ok); +if (user) +strvec_pushf (&cp.env,"REMOTE_USER=%s" , user); +strvec_pushf (&cp.env,"REQUEST_METHOD=%s" , req->method);strvec_pushf (&cp.env,"PATH_TRANSLATED=%s" , req->uri_path.buf);
@@ -359,10 +367,218 @@ static enum worker_result do__git(struct req *req)return !!res; } +enum auth_result { +/* No auth module matches the request. */ +AUTH_UNKNOWN =0 , + +/* Auth module denied the request. */ +AUTH_DENY =1 , + +/* Auth module successfully validated the request. */ +AUTH_ALLOW =2 , +}; + +struct auth_module { +char *scheme; +char *challenge_params; +struct string_list *tokens; +}; + +static int allow_anonymous; +static struct auth_module **auth_modules =NULL ; +static size_t auth_modules_nr =0 ; +static size_t auth_modules_alloc =0 ; + +static struct auth_module *get_auth_module (const char *scheme,int create) +{ +int i; +struct auth_module *mod; +for (i =0 ; i < auth_modules_nr; i++) { + mod = auth_modules[i]; +if (!strcasecmp (mod->scheme, scheme)) +return mod; + } + +if (create) { +struct auth_module *mod =xmalloc (sizeof (struct auth_module )); + mod->scheme =xstrdup (scheme); + mod->challenge_params =NULL ; +CALLOC_ARRAY (mod->tokens,1 ); +string_list_init_dup (mod->tokens); + +ALLOC_GROW (auth_modules, auth_modules_nr +1 , auth_modules_alloc); + auth_modules[auth_modules_nr++] = mod; + +return mod; + } + +return NULL ; +} + +static int is_authed (struct req *req,const char **user,enum worker_result *wr) +{ +enum auth_result result =AUTH_UNKNOWN ; +struct string_list hdrs =STRING_LIST_INIT_NODUP ; +struct auth_module *mod; + +struct string_list_item *hdr; +struct string_list_item *token; +const char *v; +struct strbuf **split =NULL ; +int i; +char *challenge; + +/* +* Check all auth modules and try to validate the request. +* The first Authorization header that matches a known auth module +* scheme will be consulted to either approve or deny the request. +* If no module is found, or if there is no valid token, then 401 error. +* Otherwise, only permit the request if anonymous auth is enabled. +* It's atypical for user agents/clients to send multiple Authorization +* headers, but not explicitly forbidden or defined. +*/ +for_each_string_list_item (hdr, &req->header_list) { +if (skip_iprefix (hdr->string,"Authorization: " , &v)) { + split =strbuf_split_str (v,' ' ,2 ); +if (!split[0 ] || !split[1 ])continue ; + +/* trim trailing space ' ' */ +strbuf_setlen (split[0 ], split[0 ]->len -1 ); + + mod =get_auth_module (split[0 ]->buf,0 ); +if (mod) { + result =AUTH_DENY ; + +for_each_string_list_item (token, mod->tokens) { +if (!strcmp (split[1 ]->buf, token->string)) { + result =AUTH_ALLOW ; +break ; + } + } + + goto done; + } + } + } + +done: +switch (result) { +case AUTH_ALLOW : +trace2_printf ("%s: auth '%s' ALLOW" ,TR2_CAT , mod->scheme); + *user ="VALID_TEST_USER" ; + *wr =WR_OK ; +break ; + +case AUTH_DENY : +trace2_printf ("%s: auth '%s' DENY" ,TR2_CAT , mod->scheme); +/* fall-through */ + +case AUTH_UNKNOWN : +if (result !=AUTH_DENY && allow_anonymous) +break ; + +for (i =0 ; i < auth_modules_nr; i++) { + mod = auth_modules[i]; +if (mod->challenge_params) + challenge =xstrfmt ("WWW-Authenticate: %s %s" , + mod->scheme, + mod->challenge_params); +else + challenge =xstrfmt ("WWW-Authenticate: %s" , + mod->scheme); +string_list_append (&hdrs, challenge); + } + + *wr =send_http_error (STDOUT_FILENO ,401 ,"Unauthorized" ,-1 , + &hdrs, *wr); + } + +strbuf_list_free (split); +string_list_clear (&hdrs,0 ); + +return result ==AUTH_ALLOW || + (result ==AUTH_UNKNOWN && allow_anonymous); +} + +static int split_auth_param (const char *str,char **scheme,char **val) +{ +struct strbuf **p =strbuf_split_str (str,':' ,2 ); + +if (!p[0 ]) +return -1 ; + +/* trim trailing ':' */ +if (p[0 ]->len >0 && p[0 ]->buf[p[0 ]->len -1 ] ==':' ) +strbuf_setlen (p[0 ], p[0 ]->len -1 ); + + *scheme =strbuf_detach (p[0 ],NULL ); + +if (p[1 ]) + *val =strbuf_detach (p[1 ],NULL ); + +strbuf_list_free (p); +return 0 ; +} + +static int read_auth_config (const char *name,const char *val,void *data) +{ +int ret =0 ; +char *scheme =NULL ; +char *token =NULL ; +char *challenge =NULL ; +struct auth_module *mod =NULL ; + +if (!strcmp (name,"auth.challenge" )) { +if (split_auth_param (val, &scheme, &challenge)) { + ret =error ("invalid auth challenge '%s'" , val); + goto cleanup; + } + + mod =get_auth_module (scheme,1 ); + +/* Replace any existing challenge parameters */ +free (mod->challenge_params); + mod->challenge_params = challenge ?xstrdup (challenge) :NULL ; + }else if (!strcmp (name,"auth.token" )) { +if (split_auth_param (val, &scheme, &token)) { + ret =error ("invalid auth token '%s'" , val); + goto cleanup; + } + + mod =get_auth_module (scheme,1 ); + +/* +* Append to set of valid tokens unless an empty token value +* is provided, then clear the existing list. +*/ +if (token) +string_list_append (mod->tokens, token); +else +string_list_clear (mod->tokens,1 ); + }else if (!strcmp (name,"auth.allowanonymous" )) { + allow_anonymous =git_config_bool (name, val); + }else { +warning ("unknown auth config '%s'" , name); + } + +cleanup: +free (scheme); +free (token); +free (challenge); + +return ret; +} +static enum worker_result dispatch (struct req *req) { +enum worker_result wr =WR_OK ; +const char *user =NULL ; + +if (!is_authed (req, &user, &wr)) +return wr; +if (is_git_request (req)) -return do__git (req); +return do__git (req, user);return send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 ,NULL ,WR_OK |WR_HANGUP );
@@ -621,6 +837,19 @@ int cmd_main(int argc, const char **argv) pid_file = v;continue ; } +if (skip_prefix (arg,"--auth-config=" , &v)) { +if (!strlen (v)) { +error ("invalid argument - missing file path" ); +usage (test_http_auth_usage); + } + +if (git_config_from_file (read_auth_config, v,NULL )) { +error ("failed to read auth config file '%s'" , v); +usage (test_http_auth_usage); + } + +continue ; + }fprintf (stderr,"error: unknown argument '%s'\n" , arg);usage (test_http_auth_usage);
t/t5556-http-auth.sh +42 -1
52 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v6 08/12] test-http-server: add simple authentication
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 396
Quoted 5 lines · Matthew John Cheetham via GitGitGadget
t/helper/test-http-server.c +5
-0+static struct auth_module *get_auth_module (const char *scheme,int create) +{ +int i; +struct auth_module *mod; +for (i =0 ; i < auth_modules_nr; i++) {
We can use "for (size_t i = 0" syntax now, let's do that here to not mix "size_t" and "int" types needlessly.
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 407
Quoted 6 lines
t/helper/test-http-server.c +6
-0+if (create) { +struct auth_module *mod =xmalloc (sizeof (struct auth_module )); + mod->scheme =xstrdup (scheme); + mod->challenge_params =NULL ; +CALLOC_ARRAY (mod->tokens,1 ); +string_list_init_dup (mod->tokens);
Don't use CALLOC_ARRAY() if you're then going to use string_list_init_dup() (which is good!), use ALLOC_ARRAY() instead. We don't need to set the memory to 0, only to overwrite it entirely again.
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 410
Quoted 2 lines
t/helper/test-http-server.c +2
-0+ALLOC_GROW (auth_modules, auth_modules_nr +1 , auth_modules_alloc); + auth_modules[auth_modules_nr++] = mod;
I have not looked at the whole context here, but instead of:
struct auth_module {
char *scheme;
char *challenge_params;
struct string_list *tokens;
};Why not:
struct auth_module {
char *challenge_params;
struct string_list *tokens;
};Then you could use a "struct string_list" for this, make the "scheme" be the "string" member, and stick the remaining two fields in the "util", and thus save yourself the manual memory management etc.
Quoted 32 lines
t/helper/test-http-server.c +32
-0+static int is_authed (struct req *req,const char **user,enum worker_result *wr) +{ +enum auth_result result =AUTH_UNKNOWN ; +struct string_list hdrs =STRING_LIST_INIT_NODUP ; +struct auth_module *mod; + +struct string_list_item *hdr; +struct string_list_item *token; +const char *v; +struct strbuf **split =NULL ; +int i; +char *challenge; + +/* +* Check all auth modules and try to validate the request. +* The first Authorization header that matches a known auth module +* scheme will be consulted to either approve or deny the request. +* If no module is found, or if there is no valid token, then 401 error. +* Otherwise, only permit the request if anonymous auth is enabled. +* It's atypical for user agents/clients to send multiple Authorization +* headers, but not explicitly forbidden or defined. +*/ +for_each_string_list_item (hdr, &req->header_list) { +if (skip_iprefix (hdr->string,"Authorization: " , &v)) { + split =strbuf_split_str (v,' ' ,2 ); +if (!split[0 ] || !split[1 ])continue ; + +/* trim trailing space ' ' */ +strbuf_setlen (split[0 ], split[0 ]->len -1 ); + + mod =get_auth_module (split[0 ]->buf,0 ); +if (mod) {
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 459
Quoted 10 lines
t/helper/test-http-server.c +10
-0+ result =AUTH_DENY ; + +for_each_string_list_item (token, mod->tokens) { +if (!strcmp (split[1 ]->buf, token->string)) { + result =AUTH_ALLOW ; +break ; + } + } + + goto done;
Sometimes we need a strbuf_split_str, but in this case couldn't you use the in-place "struct string_list" variant of that instead, and just carry a "size_t len" here for it, which you'd then pass to get_auth_module() (which this commit adds)?
Also, you "split" in the loop, but...
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 496
Quoted 1 line
t/helper/test-http-server.c +1
-0+strbuf_list_free (split);
...only free() the last one here, isn't this leaking?
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 511
Quoted 9 lines
t/helper/test-http-server.c +9
-0+static int split_auth_param (const char *str,char **scheme,char **val) +{ +struct strbuf **p =strbuf_split_str (str,':' ,2 ); + +if (!p[0 ]) +return -1 ; + +/* trim trailing ':' */ +if (p[0 ]->len >0 && p[0 ]->buf[p[0 ]->len -1 ] ==':' )
Don't compare unsigned length fields to "> 0", just do "if (len && ....)".
Also, maybe I'm just groggy today, but how do we have a trailing ":" if we just split on ":", and with a limit such that...
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 517
Quoted 2 lines
t/helper/test-http-server.c +2
-0+if (p[1 ]) + *val =strbuf_detach (p[1 ],NULL );
...we have an item after that?
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 526
Quoted 4 lines
t/helper/test-http-server.c +4
-0+static int read_auth_config (const char *name,const char *val,void *data) +{ +int ret =0 ; +char *scheme =NULL ;
Don't init this to NULL, instead the split_auth_param() return value should be trusted, the compiler will then help us catch errors, no?
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 528
Quoted 2 lines
t/helper/test-http-server.c +2
-0+char *token =NULL ; +char *challenge =NULL ;
In this case it *is* needed though, as the function will return non-errors, but *maybe* give us the second out parameter.
For such a function though, isn't just assigning "*second_param = NULL" at the start of it less of a "running with scissors" pattern?
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 529
Quoted 1 line
t/helper/test-http-server.c +1
-0+struct auth_module *mod =NULL ;
This NULL assignment can be dropped, we assign to it below unconditionally before using it.
Quoted 42 lines
t/helper/test-http-server.c +42
-0+ +if (!strcmp (name,"auth.challenge" )) { +if (split_auth_param (val, &scheme, &challenge)) { + ret =error ("invalid auth challenge '%s'" , val); + goto cleanup; + } + + mod =get_auth_module (scheme,1 ); + +/* Replace any existing challenge parameters */ +free (mod->challenge_params); + mod->challenge_params = challenge ?xstrdup (challenge) :NULL ; + }else if (!strcmp (name,"auth.token" )) { +if (split_auth_param (val, &scheme, &token)) { + ret =error ("invalid auth token '%s'" , val); + goto cleanup; + } + + mod =get_auth_module (scheme,1 ); + +/* +* Append to set of valid tokens unless an empty token value +* is provided, then clear the existing list. +*/ +if (token) +string_list_append (mod->tokens, token); +else +string_list_clear (mod->tokens,1 ); + }else if (!strcmp (name,"auth.allowanonymous" )) { + allow_anonymous =git_config_bool (name, val); + }else { +warning ("unknown auth config '%s'" , name); + } + +cleanup: +free (scheme); +free (token); +free (challenge); + +return ret; +} +
Re: [PATCH v6 08/12] test-http-server: add simple authentication
Quoted 4 lines · Ævar Arnfjörð Bjarmason
On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
Quoted 8 lines
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 396
Quoted 5 lines
t/helper/test-http-server.c +5
-0+static struct auth_module *get_auth_module (const char *scheme,int create) +{ +int i; +struct auth_module *mod; +for (i =0 ; i < auth_modules_nr; i++) {We can use "for (
size_ti = 0" syntax now, let's do that here to not mix "size_t" and "int" types needlessly.
Yep!
Quoted 10 lines
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 407
Quoted 6 lines
t/helper/test-http-server.c +6
-0+if (create) { +struct auth_module *mod =xmalloc (sizeof (struct auth_module )); + mod->scheme =xstrdup (scheme); + mod->challenge_params =NULL ; +CALLOC_ARRAY (mod->tokens,1 ); +string_list_init_dup (mod->tokens);Don't use
CALLOC_ARRAY()if you're then going to usestring_list_init_dup()(which is good!), useALLOC_ARRAY()instead. We don't need to set the memory to 0, only to overwrite it entirely again.
Sure.
Quoted 13 lines
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 410
Quoted 2 lines
t/helper/test-http-server.c +2
-0+ALLOC_GROW (auth_modules, auth_modules_nr +1 , auth_modules_alloc); + auth_modules[auth_modules_nr++] = mod;I have not looked at the whole context here, but instead of:
struct auth_module { char *scheme; char *challenge_params; struct string_list *tokens; };Why not:
Quoted 8 lines
struct auth_module { char *challenge_params; struct string_list *tokens; };Then you could use a "struct
string_list" for this, make the "scheme" be the "string" member, and stick the remaining two fields in the "util", and thus save yourself the manual memory management etc.
I looked at this, but this then means being more careful when looping over different struct auth_modules to keep the current 'scheme' and *mod in sync/together. Just feels like overkill right now.
Quoted 39 lines
Quoted 39 lines
t/helper/test-http-server.c +39
-0+static int is_authed (struct req *req,const char **user,enum worker_result *wr) +{ +enum auth_result result =AUTH_UNKNOWN ; +struct string_list hdrs =STRING_LIST_INIT_NODUP ; +struct auth_module *mod; + +struct string_list_item *hdr; +struct string_list_item *token; +const char *v; +struct strbuf **split =NULL ; +int i; +char *challenge; + +/* +* Check all auth modules and try to validate the request. +* The first Authorization header that matches a known auth module +* scheme will be consulted to either approve or deny the request. +* If no module is found, or if there is no valid token, then 401 error. +* Otherwise, only permit the request if anonymous auth is enabled. +* It's atypical for user agents/clients to send multiple Authorization +* headers, but not explicitly forbidden or defined. +*/ +for_each_string_list_item (hdr, &req->header_list) { +if (skip_iprefix (hdr->string,"Authorization: " , &v)) { + split =strbuf_split_str (v,' ' ,2 ); +if (!split[0 ] || !split[1 ])continue ; + +/* trim trailing space ' ' */ +strbuf_setlen (split[0 ], split[0 ]->len -1 ); + + mod =get_auth_module (split[0 ]->buf,0 ); +if (mod) { + result =AUTH_DENY ; + +for_each_string_list_item (token, mod->tokens) { +if (!strcmp (split[1 ]->buf, token->string)) { + result =AUTH_ALLOW ; +break ; + }
Quoted 8 lines
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 459
Quoted 3 lines
t/helper/test-http-server.c +3
-0+ } + + goto done;Sometimes we need a
strbuf_split_str, but in this case couldn't you use the in-place "structstring_list" variant of that instead, and just carry a "size_tlen" here for it, which you'd then pass toget_auth_module()(which this commit adds)?
get_auth_module taking a scheme name as parameter is a more sensible, IMO, than a string_list or string_list_item and an offset. Given this is a test helper, performance also isn't a priority. Readability wins here I think.
Quoted 4 lines
Also, you "split" in the loop, but...
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 496
Quoted 1 line
t/helper/test-http-server.c +1
-0+strbuf_list_free (split);...only
free()the last one here, isn't this leaking?
Yes, it is. Will fix in next iteration.
Quoted 4 lines
Quoted 4 lines
t/helper/test-http-server.c +4
-0+static int split_auth_param (const char *str,char **scheme,char **val) +{ +struct strbuf **p =strbuf_split_str (str,':' ,2 ); +
Quoted 8 lines
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 511
Quoted 5 lines
t/helper/test-http-server.c +5
-0+if (!p[0 ]) +return -1 ; + +/* trim trailing ':' */ +if (p[0 ]->len >0 && p[0 ]->buf[p[0 ]->len -1 ] ==':' )Don't compare unsigned length fields to "> 0", just do "if (len && ....)".
Sure!
Quoted 7 lines
Also, maybe I'm just groggy today, but how do we have a trailing ":" if we just split on ":", and with a limit such that...
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 517
Quoted 2 lines
t/helper/test-http-server.c +2
-0+if (p[1 ]) + *val =strbuf_detach (p[1 ],NULL );...we have an item after that?
Because that's how the strbuf_split_str function works. The comments in the header file even call that out. "The substrings include the terminator". From strbuf.h:
/**
* Split str (of length slen) at the specified terminator character.
* Return a null-terminated array of pointers to strbuf objects
* holding the substrings. The substrings include the terminator,
* except for the last substring, which might be unterminated if the
* original string did not end with a terminator. [cut] ...
...
*/
struct strbuf **strbuf_split_buf(const char *str, size_t len,
int terminator, int max);
Quoted 14 lines
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 526
Quoted 4 lines
t/helper/test-http-server.c +4
-0+static int read_auth_config (const char *name,const char *val,void *data) +{ +int ret =0 ; +char *scheme =NULL ;Don't init this to NULL, instead the
split_auth_param()return value should be trusted, the compiler will then help us catch errors, no?t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 528
Quoted 2 lines
t/helper/test-http-server.c +2
-0+char *token =NULL ; +char *challenge =NULL ;In this case it *is* needed though, as the function will return non-errors, but *maybe* give us the second out parameter.
Quoted 7 lines
For such a function though, isn't just assigning "
*second_param= NULL" at the start of it less of a "running with scissors" pattern?t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 529
Quoted 1 line
t/helper/test-http-server.c +1
-0+struct auth_module *mod =NULL ;This NULL assignment can be dropped, we assign to it below unconditionally before using it.
All of these variables need to be initialised to NULL because not all arms of the if-elseif chain assign to all of these variables, but we always free all of them at the function exit.
For example,
char *scheme = NULL;
char *token = NULL;
char *challenge = NULL;
...
} else if (!strcmp(name, "auth.allowanonymous")) {
allow_anonymous = git_config_bool(name, val);
} else {...
free(scheme);
free(token);
free(challenge);
Quoted 34 lines
Quoted 34 lines
t/helper/test-http-server.c +34
-0+ +if (!strcmp (name,"auth.challenge" )) { +if (split_auth_param (val, &scheme, &challenge)) { + ret =error ("invalid auth challenge '%s'" , val); + goto cleanup; + } + + mod =get_auth_module (scheme,1 ); + +/* Replace any existing challenge parameters */ +free (mod->challenge_params); + mod->challenge_params = challenge ?xstrdup (challenge) :NULL ; + }else if (!strcmp (name,"auth.token" )) { +if (split_auth_param (val, &scheme, &token)) { + ret =error ("invalid auth token '%s'" , val); + goto cleanup; + } + + mod =get_auth_module (scheme,1 ); + +/* +* Append to set of valid tokens unless an empty token value +* is provided, then clear the existing list. +*/ +if (token) +string_list_append (mod->tokens, token); +else +string_list_clear (mod->tokens,1 ); + }else if (!strcmp (name,"auth.allowanonymous" )) { + allow_anonymous =git_config_bool (name, val); + }else { +warning ("unknown auth config '%s'" , name); + } +
Quoted 8 lines
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 571
Quoted 8 lines
t/helper/test-http-server.c +8
-0+cleanup: +free (scheme); +free (token); +free (challenge); + +return ret; +} +
Thanks,
Matthew
PATCH v6 9/12 test-http-server: add sending of arbitrary headers
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add the ability to send arbitrary headers in HTTP responses from the test-http-server. This is useful when we want to test 'malformed' response message handling.
Add the following option to the server auth config file:
[auth]
extraHeader = [<value>]*
Each auth.extraHeader value will be appended to the response headers verbatim.
--- t/helper/test-http-server.c | 6 ++++++ t/t5556-http-auth.sh | 7 +++++++ 2 files changed, 13 insertions(+)
t/helper/test-http-server.c +6 -0
@@ -388,6 +388,7 @@ static int allow_anonymous;static struct auth_module **auth_modules =NULL ;static size_t auth_modules_nr =0 ;static size_t auth_modules_alloc =0 ; +static struct strvec extra_headers =STRVEC_INIT ;static struct auth_module *get_auth_module (const char *scheme,int create) {
@@ -489,6 +490,9 @@ done:string_list_append (&hdrs, challenge); } +for (i =0 ; i < extra_headers.nr; i++) +string_list_append (&hdrs, extra_headers.v[i]); + *wr =send_http_error (STDOUT_FILENO ,401 ,"Unauthorized" ,-1 , &hdrs, *wr); }
@@ -557,6 +561,8 @@ static int read_auth_config(const char *name, const char *val, void *data)string_list_clear (mod->tokens,1 ); }else if (!strcmp (name,"auth.allowanonymous" )) { allow_anonymous =git_config_bool (name, val); + }else if (!strcmp (name,"auth.extraheader" )) { +strvec_push (&extra_headers, val); }else {warning ("unknown auth config '%s'" , name); }
t/t5556-http-auth.sh +7 -0
@@ -112,6 +112,10 @@ test_expect_success CURL 'http auth server auth config' ' token = reset-tokens:the-only-valid-one allowAnonymous = false + + extraHeader = X-Extra-Header: abc + extraHeader = X-Extra-Header: 123 + extraHeader = X-Another: header\twith\twhitespace! EOF cat >OUT.expected <<-EOF &&
@@ -119,6 +123,9 @@ test_expect_success CURL 'http auth server auth config' ' WWW-Authenticate: with-params foo="replaced" q=1 WWW-Authenticate: no-explicit-challenge WWW-Authenticate: reset-tokens + X-Extra-Header: abc + X-Extra-Header: 123 + X-Another: header with whitespace! Error: 401 Unauthorized EOF
Signature · 1 line
gitgitgadget
PATCH v6 10/12 http: replace unsafe size_t multiplication with st_mult
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Replace direct multiplication of two size_t parameters in curl response stream handling callback functions with st_mult to guard against overflows.
--- http.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
http.c +2 -2
@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;size_t fread_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct buffer *buffer = buffer_;if (size > buffer->buf.len - buffer->posn)
@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)size_t fwrite_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct strbuf *buffer = buffer_;strbuf_add (buffer, ptr, size);
Signature · 1 line
gitgitgadget
Re: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult
Quoted 25 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Replace direct multiplication of two
size_tparameters in curl response stream handling callback functions withst_multto guard against overflows.Signed-off-by Matthew John Cheetham <[email protected]>--- http.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)http.c +1
-1@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;size_t fread_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct buffer *buffer = buffer_;if (size > buffer->buf.len - buffer->posn)
http.c curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp) · line 182
Quoted 8 lines
http.c +1
-1@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)size_t fwrite_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct strbuf *buffer = buffer_;strbuf_add (buffer, ptr, size);
This is a really worthwhile fix, but shouldn't this be split into its own stand-alone patch? It applies on "master", and seems like something that's a good idea outside of this "test-http-server" topic.
Re: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult
Quoted 35 lines · Ævar Arnfjörð Bjarmason
Quoted 32 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Replace direct multiplication of two
size_tparameters in curl response stream handling callback functions withst_multto guard against overflows.Signed-off-by Matthew John Cheetham <[email protected]>--- http.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)http.c +2
-2@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;size_t fread_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct buffer *buffer = buffer_;if (size > buffer->buf.len - buffer->posn)@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)size_t fwrite_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct strbuf *buffer = buffer_;
Quoted 5 lines
http.c curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp) · line 182
Quoted 1 line
http.c +0
-0strbuf_add (buffer, ptr, size);This is a really worthwhile fix, but shouldn't this be split into its own stand-alone patch? It applies on "master", and seems like something that's a good idea outside of this "test-http-server" topic.
While it's this change *can* stand alone, please keep in mind that suggestions like this (recommending a series be split and resubmitted) can be highly disruptive to the in-flight topic and the original contributor.
Monitoring and iterating on multiple series at once is time-consuming for the contributor and reviewers, and often (although not in this case) it creates a dependency of one series on another, which comes with a cost to the maintainer's time. Not to say those recommendations should never be made (e.g. in a clearly too-long series early in its review cycle, or when certain patches lead to excessive context switching while reviewing), just that they should be made more carefully, with consideration for the time of other contributors.
So, with that in mind, I don't think this patch is critical enough to separate into an independent submission, and (subjectively) it does not disrupt the flow of this series.
Re: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult
Quoted 55 lines · Victoria Dye
Quoted 35 lines · Ævar Arnfjörð Bjarmason
Quoted 32 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Replace direct multiplication of two
size_tparameters in curl response stream handling callback functions withst_multto guard against overflows.Signed-off-by Matthew John Cheetham <[email protected]>--- http.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)http.c +2
-2@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;size_t fread_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct buffer *buffer = buffer_;if (size > buffer->buf.len - buffer->posn)@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)size_t fwrite_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct strbuf *buffer = buffer_;Quoted 5 lines
http.c curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp) · line 182
Quoted 1 line
http.c +0
-0strbuf_add (buffer, ptr, size);This is a really worthwhile fix, but shouldn't this be split into its own stand-alone patch? It applies on "master", and seems like something that's a good idea outside of this "test-http-server" topic.
While it's this change *can* stand alone, please keep in mind that suggestions like this (recommending a series be split and resubmitted) can be highly disruptive to the in-flight topic and the original contributor.
Monitoring and iterating on multiple series at once is time-consuming for the contributor and reviewers, and often (although not in this case) it creates a dependency of one series on another, which comes with a cost to the maintainer's time. Not to say those recommendations should never be made (e.g. in a clearly too-long series early in its review cycle, or when certain patches lead to excessive context switching while reviewing), just that they should be made more carefully, with consideration for the time of other contributors.
Quoted 3 lines
So, with that in mind, I don't think this patch is critical enough to separate into an independent submission, and (subjectively) it does not disrupt the flow of this series.
Yes, I take your general point, it's not always the right thing, sometimes a while-at-it cleanup is better than a split-out etc.
In this case the split-out seemed like it wouldn't create a dependency between topics, as the rest of the series didn't rely on the overflow sanity check being added, it's just a good idea to do it in general.
PATCH v6 11/12 http: read HTTP WWW-Authenticate response headers
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Read and store the HTTP WWW-Authenticate response headers made for a particular request.
This will allow us to pass important authentication challenge information to credential helpers or others that would otherwise have been lost.
According to RFC2616 Section 4.2 [1], header field names are not case-sensitive meaning when collecting multiple values for the same field name, we can just use the case of the first observed instance of each field name and no normalisation is required.
libcurl only provides us with the ability to read all headers recieved for a particular request, including any intermediate redirect requests or proxies. The lines returned by libcurl include HTTP status lines delinating any intermediate requests such as "HTTP/1.1 200". We use these lines to reset the strvec of WWW-Authenticate header values as we encounter them in order to only capture the final response headers.
The collection of all header values matching the WWW-Authenticate header is complicated by the fact that it is legal for header fields to be continued over multiple lines, but libcurl only gives us one line at a time.
In the future [2] we may be able to leverage functions to read headers from libcurl itself, but as of today we must do this ourselves.
[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/
--- credential.c | 1 + credential.h | 15 +++++++++ http.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+)
credential.c +1 -0
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)free (c->username);free (c->password);string_list_clear (&c->helpers,0 ); +strvec_clear (&c->wwwauth_headers);credential_init (c); }
credential.h +15 -0
@@ -2,6 +2,7 @@#define CREDENTIAL_H #include "string-list.h" +#include "strvec.h" /** * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {*/ struct string_list helpers; +/** +* A `strvec` of WWW-Authenticate header values. Each string +* is the value of a WWW-Authenticate header in an HTTP response, +* in the order they were received in the response. +*/ +struct strvec wwwauth_headers; + +/** +* Internal use only. Used to keep track of split header fields +* in order to fold multiple lines into one value. +*/ + unsigned header_is_last_match:1; + unsigned approved:1, configured:1, quit:1,
@@ -130,6 +144,7 @@ struct credential {#define CREDENTIAL_INIT { \ .helpers = STRING_LIST_INIT_DUP, \ + .wwwauth_headers = STRVEC_INIT, \ }/* Initialize a credential structure, setting all fields to empty. */
http.c +94 -0
106 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers
Quoted 9 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
http.c +7
-0+strbuf_add (&buf, ptr, size); + +/* Strip the CRLF that should be present at the end of each field */ +strbuf_trim_trailing_newline (&buf); + +/* Start of a new WWW-Authenticate header */ +if (skip_iprefix (buf.buf,"www-authenticate:" , &val)) {
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 237
Quoted 10 lines
http.c +9
-0+while (isspace (*val)) + val++; + +strvec_push (values, val); + http_auth.header_is_last_match =1 ; + goto exit; +if (http_auth.header_is_last_match &&isspace (*buf.buf)) { +/* Trim leading whitespace from this continuation hdr line. */ +strbuf_ltrim (&buf);
The mixture of this isspace() loop and then strbuf_ltrim() seems odd, why not stick with the strbuf API?
I.e. after skip_iprefix() strbuf_splice() the start of the string away, then use strbuf_ltrim() in the first "if" branch here?
Likewise this is open-coding the "isspace" in strbuf_ltrim() for the second "if". Maybe run the strbuf_ltrim() unconditionally, save away the length before, and then:
if (http_auth.header_is_last_match && prev_len != buf.len) { ...?
Re: [PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers
Quoted 27 lines · Ævar Arnfjörð Bjarmason
Quoted 9 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
http.c +7
-0+strbuf_add (&buf, ptr, size); + +/* Strip the CRLF that should be present at the end of each field */ +strbuf_trim_trailing_newline (&buf); + +/* Start of a new WWW-Authenticate header */ +if (skip_iprefix (buf.buf,"www-authenticate:" , &val)) {http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 237
Quoted 10 lines
http.c +9
-0+while (isspace (*val)) + val++; + +strvec_push (values, val); + http_auth.header_is_last_match =1 ; + goto exit; +if (http_auth.header_is_last_match &&isspace (*buf.buf)) { +/* Trim leading whitespace from this continuation hdr line. */ +strbuf_ltrim (&buf);The mixture of this
isspace()loop and thenstrbuf_ltrim()seems odd, why not stick with the strbuf API?
Quoted 2 lines
I.e. after
skip_iprefix()strbuf_splice()the start of the string away, then usestrbuf_ltrim()in the first "if" branch here?
You mean like this?
size_t size = st_mult(eltsize, nmemb);
struct strvec *values = &http_auth.wwwauth_headers;
struct strbuf buf = STRBUF_INIT;- const char *val;
/*
* Header lines may not come NULL-terminated from libcurl so we must
@@ -216,11 +215,11 @@ static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
strbuf_trim_trailing_newline(&buf);
/* Start of a new WWW-Authenticate header */- if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
- while (isspace(*val))
- val++;
+ if (istarts_with(buf.buf, "www-authenticate:")) {
+ strbuf_splice(&buf, 0, 17, NULL, 0);
+ strbuf_ltrim(&buf);
- strvec_push(values, val);
+ strvec_push(values, buf.buf);
http_auth.header_is_last_match = 1;
goto exit;
}I don't particularly like this given we're now introducing the 'magic' number 17 that's the length of www-authenticate:, plus strbuf_splice is doing a lot more work moving pieces of memory around rather than just producing a new starting pointer to the start of the value (skipping leading whitespace).
Quoted 7 lines
Likewise this is open-coding the "isspace" in
strbuf_ltrim()for the second "if". Maybe run thestrbuf_ltrim()unconditionally, save away the length before, and then:if (http_auth.header_is_last_match && prev_len != buf.len) { ...?
The suggestion of trimming and comparing lengths just makes a piece of code handling a little-known edge case less immediately obvious in its intent in my opinion. The current implementation of "if starts with a single space" matches the definition of continuation header lines, rather than re-piecing together this from "why are we trimming and comparing lengths?" Perf-wise the current implementation is only adding one extra isspace call which we're potentially about to do in a loop inside of strbuf_ltrim anyway. Plus, the common case will be a single space anyway.
Thanks,
Matthew
PATCH v6 12/12 credential: add WWW-Authenticate header to cred requests
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add the value of the WWW-Authenticate response header to credential requests. Credential helpers that understand and support HTTP authentication and authorization can use this standard header (RFC 2616 Section 14.47 [1]) to generate valid credentials.
WWW-Authenticate headers can contain information pertaining to the authority, authentication mechanism, or extra parameters/scopes that are required.
The current I/O format for credential helpers only allows for unique names for properties/attributes, so in order to transmit multiple header values (with a specific order) we introduce a new convention whereby a C-style array syntax is used in the property name to denote multiple ordered values for the same property.
In this case we send multiple wwwauth[] properties where the order that the repeated attributes appear in the conversation reflects the order that the WWW-Authenticate headers appeared in the HTTP response.
Add a set of tests to exercise the HTTP authentication header parsing and the interop with credential helpers. Credential helpers will receive WWW-Authenticate information in credential requests.
[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47
--- Documentation/git-credential.txt | 19 ++- credential.c | 11 ++ t/lib-credential-helper.sh | 27 ++++ t/t5556-http-auth.sh | 245 ++++++++++++++++++++++++++++++- 4 files changed, 300 insertions(+), 2 deletions(-) create mode 100644 t/lib-credential-helper.sh
Documentation/git-credential.txt +18 -1
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline. The key may contain any bytes except `=`, newline, or NUL. The value may contain any bytes except newline or NUL. -In both cases, all bytes are treated as-is (i.e., there is no quoting, +Attributes with keys that end with C-style array brackets `[]` can have +multiple values. Each instance of a multi-valued attribute forms an +ordered list of values - the order of the repeated attributes defines +the order of the values. An empty multi-valued attribute (`key[]=\n`) +acts to clear any previous entries and reset the list. + +In all cases, all bytes are treated as-is (i.e., there is no quoting, and one cannot transmit a value with newline or NUL in it). The list of attributes is terminated by a blank line or end-of-file.
@@ -160,6 +166,17 @@ empty string. Components which are missing from the URL (e.g., there is no username in the example above) will be left unset. +`wwwauth[]`:: + + When an HTTP response is received by Git that includes one or more + 'WWW-Authenticate' authentication headers, these will be passed by Git + to credential helpers. ++ +Each 'WWW-Authenticate' header value is passed as a multi-valued +attribute 'wwwauth[]', where the order of the attributes is the same as +they appear in the HTTP response. This attribute is 'one-way' from Git +to pass additional information to credential helpers. + Unrecognised attributes are silently discarded. GIT
credential.c +11 -0
@@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,fprintf (fp,"%s=%s\n" , key, value); } +static void credential_write_strvec (FILE *fp,const char *key, +const struct strvec *vec) +{ +char *full_key =xstrfmt ("%s[]" , key); +for (size_t i =0 ; i < vec->nr; i++) { +credential_write_item (fp, full_key, vec->v[i],0 ); + } +free (full_key); +} +void credential_write (const struct credential *c,FILE *fp) {credential_write_item (fp,"protocol" , c->protocol,1 );
@@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp)credential_write_item (fp,"path" , c->path,0 );credential_write_item (fp,"username" , c->username,0 );credential_write_item (fp,"password" , c->password,0 ); +credential_write_strvec (fp,"wwwauth" , &c->wwwauth_headers); } static int run_credential_helper(struct credential *c,
t/lib-credential-helper.sh new file +27 -0
@@ -0,0 +1,27 @@ +setup_credential_helper() { + test_expect_success 'setup credential helper' ' + CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" && + export CREDENTIAL_HELPER && + echo $CREDENTIAL_HELPER && + + write_script "$CREDENTIAL_HELPER" <<-\EOF + cmd=$1 + teefile=$cmd-query.cred + catfile=$cmd-reply.cred + sed -n -e "/^$/q" -e "p" >> $teefile + if test "$cmd" = "get"; then + cat $catfile + fi + EOF + ' +} + +set_credential_reply() { + cat >"$TRASH_DIRECTORY/$1-reply.cred" +} + +expect_credential_query() { + cat >"$TRASH_DIRECTORY/$1-expect.cred" && + test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \ + "$TRASH_DIRECTORY/$1-query.cred" +}
t/t5556-http-auth.sh +244 -1
267 lines not on this page · raw message
Signature · 1 line
gitgitgadget
v7 cover Enhance credential helper protocol to include auth headers
In reply to Matthew John Cheetham via GitGitGadgetFollowing from my original RFC submission [0], this submission is considered ready for full review. This patch series is now based on top of current master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now separately submitted patches [1] to fix up the other credential helpers' behaviour.
In this patch series I update the existing credential helper design in order to allow for some new scenarios, and future evolution of auth methods that Git hosts may wish to provide. I outline the background, summary of changes and some challenges below.
Testing these new additions, I introduce a new test helper test-http-server that acts as a frontend to git-http-backend; a mini HTTP server sharing code with git-daemon, with simple authentication configurable by a config file.
Background
==========
Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the authentication capabilities of this protocol to address limitations (see below).
The Smart HTTP protocol in Git supports a few different types of HTTP authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478) [4]. Git uses a extensible model where credential helpers can provide credentials for protocols [5]. Several helpers support alternatives such as OAuth authentication (RFC 6749) [6], but this is typically done as an extension. For example, a helper might use basic auth and set the password to an OAuth Bearer access token. Git uses standard input and output to communicate with credential helpers.
After a HTTP 401 response, Git would call a credential helper with the following over standard input:
protocol=https
host=example.com
And then a credential helper would return over standard output:
protocol=https
host=example.com
[email protected]
password=<BEARER-TOKEN>
Git then the following request to the remote, including the standard HTTP Authorization header (RFC 7235 Section 4.2) [7]:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64([email protected]:<BEARER-TOKEN>)
Credential helpers are encouraged (see gitcredentials.txt) to return the minimum information necessary.
Limitations
===========
Because this credential model was built mostly for password based authentication systems, it's somewhat limited. In particular:
1. To generate valid credentials, additional information about the request (or indeed the requestee and their device) may be required. For example, OAuth is based around scopes. A scope, like "git.read", might be required to read data from the remote. However, the remote cannot tell the credential helper what scope is required for this request.
2. This system is not fully extensible. Each time a new type of authentication (like OAuth Bearer) is invented, Git needs updates before credential helpers can take advantage of it (or leverage a new capability in libcurl).
Goals
=====
* As a user with multiple federated cloud identities:
* Reach out to a remote and have my credential helper automatically prompt me for the correct identity.
* Allow credential helpers to differentiate between different authorities or authentication/authorization challenge types, even from the same DNS hostname (and without needing to use credential.useHttpPath).
* Leverage existing authentication systems built-in to many operating systems and devices to boost security and reduce reliance on passwords.
* As a Git host and/or cloud identity provider:
* Enforce security policies (like requiring two-factor authentication) dynamically.
* Allow integration with third party standard based identity providers in enterprises allowing customers to have a single plane of control for critical identities with access to source code.
Design Principles
=================
* Use the existing infrastructure. Git credential helpers are an already-working model.
* Follow widely-adopted time-proven open standards, avoid net new ideas in the authentication space.
* Minimize knowledge of authentication in Git; maintain modularity and extensibility.
Proposed Changes
================
1. Teach Git to read HTTP response headers, specifically the standard WWW-Authenticate (RFC 7235 Section 4.1) headers.
2. Teach Git to include extra information about HTTP responses that require authentication when calling credential helpers. Specifically the WWW-Authenticate header information.
Because the extra information forms an ordered list, and the existing credential helper I/O format only provides for simple key=value pairs, we introduce a new convention for transmitting an ordered list of values. Key names that are suffixed with a C-style array syntax should have values considered to form an order list, i.e. key[]=value, where the order of the key=value pairs in the stream specifies the order.
For the WWW-Authenticate header values we opt to use the key wwwauth[].
Handling the WWW-Authenticate header in detail
==============================================
RFC 6750 [8] envisions that OAuth Bearer resource servers would give responses that include WWW-Authenticate headers, for example:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
Specifically, a WWW-Authenticate header consists of a scheme and arbitrary attributes, depending on the scheme. This pattern enables generic OAuth or OpenID Connect [9] authorities. Note that it is possible to have several WWW-Authenticate challenges in a response.
First Git attempts to make a request, unauthenticated, which fails with a 401 response and includes WWW-Authenticate header(s).
Next, Git invokes a credential helper which may prompt the user. If the user approves, a credential helper can generate a token (or any auth challenge response) to be used for that request.
For example: with a remote that supports bearer tokens from an OpenID Connect [9] authority, a credential helper can use OpenID Connect's Discovery [10] and Dynamic Client Registration [11] to register a client and make a request with the correct permissions to access the remote. In this manner, a user can be dynamically sent to the right federated identity provider for a remote without any up-front configuration or manual processes.
Following from the principle of keeping authentication knowledge in Git to a minimum, we modify Git to add all WWW-Authenticate values to the credential helper call.
Git sends over standard input:
protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite" wwwauth[]=Basic realm="login.example"
A credential helper that understands the extra wwwauth[n] property can decide on the "best" or correct authentication scheme, generate credentials for the request, and interact with the user.
The credential helper would then return over standard output:
protocol=https
host=example.com
path=foo.git
[email protected]
password=<BEARER-TOKEN>
Note that WWW-Authenticate supports multiple challenges, either in one header:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
or in multiple headers:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the implementation, Git will not merge or split up any of these WWW-Authenticate headers, and instead pass each header line as one credential helper property. The credential helper is responsible for splitting, merging, and otherwise parsing these header values.
An alternative option to sending the header fields individually would be to merge the header values in to one key=value property, for example:
...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
Future work
===========
In the future we can further expand the protocol to allow credential helpers decide the best authentication scheme. Today credential helpers are still only expected to return a username/password pair to Git, meaning the other authentication schemes that may be offered still need challenge responses sent via a Basic Authorization header. The changes outlined above still permit helpers to select and configure an available authentication mode, but require the remote for example to unpack a bearer token from a basic challenge.
More careful consideration is required in the handling of custom authentication schemes which may not have a username, or may require arbitrary additional request header values be set.
For example imagine a new "FooBar" authentication scheme that is surfaced in the following response:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"
With support for arbitrary authentication schemes, Git would call credential helpers with the following over standard input:
protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"
And then an enlightened credential helper could return over standard output:
protocol=https
host=example.com
authtype=FooBar
[email protected]
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF
Git would be expected to attach this authorization header to the next request:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF
Why not SSH?
============
There's nothing wrong with SSH. However, Git's Smart HTTP transport is widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport sometimes requires less client setup than SSH transport, and works in environments when SSH ports may be blocked. As long as Git supports HTTP transport, it should support common and popular HTTP authentication methods.
References
==========
* [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers
https://lore.kernel.org/git/[email protected]/
* [1] [PATCH 0/3] Correct credential helper discrepancies handling input
https://lore.kernel.org/git/[email protected]/
* [2] Git on the Server - The Protocols
https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols
* [3] HTTP Authentication: Basic and Digest Access Authentication
https://datatracker.ietf.org/doc/html/rfc2617
* [4] The Simple and Protected GSS-API Negotiation Mechanism
https://datatracker.ietf.org/doc/html/rfc2478
* [5] Git Credentials - Custom Helpers
https://git-scm.com/docs/gitcredentials#_custom_helpers
* [6] The OAuth 2.0 Authorization Framework
https://datatracker.ietf.org/doc/html/rfc6749
* [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
https://datatracker.ietf.org/doc/html/rfc7235
* [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
https://datatracker.ietf.org/doc/html/rfc6750
* [9] OpenID Connect Core 1.0
https://openid.net/specs/openid-connect-core-1_0.html
* [10] OpenID Connect Discovery 1.0
https://openid.net/specs/openid-connect-discovery-1_0.html
* [11] OpenID Connect Dynamic Client Registration 1.0
https://openid.net/specs/openid-connect-registration-1_0.html
* [12] Hypertext Transfer Protocol (HTTP/1.1)
https://datatracker.ietf.org/doc/html/rfc2616
Updates from RFC
================
* Submitted first three patches as separate submission:
https://lore.kernel.org/git/[email protected]/
* Various style fixes and updates to- and addition of comments.
* Drop the explicit integer index in new 'array' style credential helper attrbiutes ("key[n]=value" becomes just "key[]=value").
* Added test helper; a mini HTTP server, and several tests.
Updates in v3
=============
* Split final patch that added the test-http-server in to several, easier to review patches.
* Updated wording in git-credential.txt to clarify which side of the credential helper protocol is sending/receiving the new wwwauth and authtype attributes.
Updates in v4
=============
* Drop authentication scheme selection authtype attribute patches to greatly simplify the series; auth scheme selection is punted to a future series. This series still allows credential helpers to generate credentials and intelligently select correct identities for a given auth challenge.
Updates in v5
=============
* Libify parts of daemon.c and share implementation with test-http-server.
* Clarify test-http-server Git request regex pattern and auth logic comments.
* Use STD*_FILENO in place of 'magic' file descriptor numbers.
* Use strbuf_* functions in continuation header parsing.
* Use configuration file to configure auth for test-http-server rather than command-line arguments. Add ability to specify arbitrary extra headers that is useful for testing 'malformed' server responses.
* Use st_mult over unchecked multiplication in http.c curl callback functions.
* Fix some documentation line break issues.
* Reorder some commits to bring in the tests and test-http-server helper first and, then the WWW-Authentication changes, alongside tests to cover.
* Expose previously static strvec_push_nodup function.
* Merge the two timeout args for test-http-server (--timeout and --init-timeout) that were a hang-over from the original daemon.c but are no longer required here.
* Be more careful around continuation headers where they may be empty strings. Add more tests to cover these header types.
* Include standard trace2 tracing calls at start of test-http-server helper.
Updates in v6
=============
* Clarify the change to make logging optional in the check_dead_children() function during libification of daemon.c.
* Fix missing pointer dereference bugs identified in libification of child process handling functions for daemon.c.
* Add doc comments to child process handling function declarations in the daemon-utils.h header.
* Align function parameter names with variable names at callsites for libified daemon functions.
* Re-split out the test-http-server test helper commits in to smaller
patches: error response handling, request parsing, http-backend pass-through, simple authentication, arbitrary header support.
* Call out auth configuration file format for test-http-server test helper and supported options in commit messages, as well as a test to exercise and demonstrate these options.
* Permit auth.token and auth.challenge to appear in any order; create the struct auth_module just-in-time as options for that scheme are read. This simplifies the configuration authoring of the test-http-server test helper.
* Update tests to use auth.allowAnoymous in the patch that introduces the new test helper option.
* Drop the strvec_push_nodup() commit and update the implementation of HTTP request header line folding to use xstrdup and strvec_pop and _pushf.
* Use size_t instead of int in credential.c when iterating over the struct strvec credential members. Also drop the not required const and cast from the full_key definition and free.
* Replace in-tree test-credential-helper-reply.sh test cred helper script with the lib-credential-helper.sh reusable 'lib' test script and shell functions to configure the helper behaviour.
* Leverage sed over the while read $line loop in the test credential helper script.
Updates in v7
=============
* Address several whitespace and arg/param list alignment issues.
* Rethink the test-http-helper worker-mode error and result enum to be more simple and more informative to the nature of the error.
* Use uintmax_t to store the Content-Length of a request in the helper test-http-server. Maintain a bit flag to store if we received such a header.
* Return a "400 Bad Request" HTTP response if we fail to parse the request in the test-http-server.
* Add test case to cover request message parsing in test-http-server.
* Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in get_auth_module.
* Correctly free the split strbufs created in the header parsing loop in test-http-server.
* Avoid needless comparison > 0 for unsigned types.
* Always set optional outputs to NULL if not present in test helper config value handling.
* Remove an accidentally commented-out test cleanup line for one test case in t5556.
Matthew John Cheetham (12):
daemon: libify socket setup and option functions
daemon: libify child process handling functions
daemon: rename some esoteric/laboured terminology
test-http-server: add stub HTTP server test helper
test-http-server: add HTTP error response function
test-http-server: add HTTP request parsing
test-http-server: pass Git requests to http-backend
test-http-server: add simple authentication
test-http-server: add sending of arbitrary headers
http: replace unsafe size_t multiplication with st_mult
http: read HTTP WWW-Authenticate response headers
credential: add WWW-Authenticate header to cred requests
Documentation/git-credential.txt | 19 +- Makefile | 2 + contrib/buildsystems/CMakeLists.txt | 11 +- credential.c | 12 + credential.h | 15 + daemon-utils.c | 286 +++++++++ daemon-utils.h | 55 ++ daemon.c | 306 +-------- http.c | 98 ++- t/helper/.gitignore | 1 + t/helper/test-http-server.c | 943 ++++++++++++++++++++++++++++ t/lib-credential-helper.sh | 27 + t/t5556-http-auth.sh | 463 ++++++++++++++ 13 files changed, 1936 insertions(+), 302 deletions(-) create mode 100644 daemon-utils.c create mode 100644 daemon-utils.h create mode 100644 t/helper/test-http-server.c create mode 100644 t/lib-credential-helper.sh create mode 100755 t/t5556-http-auth.sh
base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v7
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v7
Pull-Request: https://github.com/gitgitgadget/git/pull/1352
Range-diff vs v6:
1: 74b0de14185 = 1: 74b0de14185 daemon: libify socket setup and option functions
2: b6ba344a671 = 2: b6ba344a671 daemon: libify child process handling functions
3: 9967401c972 = 3: 9967401c972 daemon: rename some esoteric/laboured terminology
4: d6e5e8825e8 ! 4: 17c890ee108 test-http-server: add stub HTTP server test helper
@@ t/helper/test-http-server.c (new)
+
+/*
+ * The code in this section is used by "worker" instances to service
-+ * a single connection from a client. The worker talks to the client
-+ * on 0 and 1.
++ * a single connection from a client. The worker talks to the client
++ * on stdin and stdout.
+ */
+
+enum worker_result {
@@ t/helper/test-http-server.c (new)
+ * Operation successful.
+ * Caller *might* keep the socket open and allow keep-alive.
+ */
-+ WR_OK = 0,
++ WR_OK = 0,
+
+ /*
-+ * Various errors while processing the request and/or the response.
++ * Fatal error that is not recoverable.
+ * Close the socket and clean up.
+ * Exit child-process with non-zero status.
+ */
-+ WR_IO_ERROR = 1<<0,
-+
-+ /*
-+ * Close the socket and clean up. Does not imply an error.
-+ */
-+ WR_HANGUP = 1<<1,
++ WR_FATAL_ERROR = 1,
+};
+
+static enum worker_result worker(void)
@@ t/helper/test-http-server.c (new)
+ while (1) {
+ if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
+ logerror("unable to write response");
-+ wr = WR_IO_ERROR;
++ wr = WR_FATAL_ERROR;
+ }
+
+ if (wr != WR_OK)
@@ t/helper/test-http-server.c (new)
+ close(STDIN_FILENO);
+ close(STDOUT_FILENO);
+
-+ return !!(wr & WR_IO_ERROR);
++ /* Only WR_OK should result in a non-zero exit code */
++ return wr != WR_OK;
+}
+
+static int max_connections = 32;
5: 79805f042b9 ! 5: 6e70e304cfe test-http-server: add HTTP error response function
@@ Commit message
## t/helper/test-http-server.c ##
@@ t/helper/test-http-server.c: enum worker_result {
- WR_HANGUP = 1<<1,
+ * Exit child-process with non-zero status.
+ */
+ WR_FATAL_ERROR = 1,
++
++ /*
++ * Close the socket and clean up. Does not imply an error.
++ */
++ WR_HANGUP = 2,
};
-+static enum worker_result send_http_error(
-+ int fd,
-+ int http_code, const char *http_code_name,
-+ int retry_after_seconds, struct string_list *response_headers,
-+ enum worker_result wr_in)
++static enum worker_result send_http_error(int fd, int http_code,
++ const char *http_code_name,
++ int retry_after_seconds,
++ struct string_list *response_headers,
++ enum worker_result wr_in)
+{
+ struct strbuf response_header = STRBUF_INIT;
+ struct strbuf response_content = STRBUF_INIT;
+ struct string_list_item *h;
+ enum worker_result wr;
+
-+ strbuf_addf(&response_content, "Error: %d %s\r\n",
-+ http_code, http_code_name);
++ strbuf_addf(&response_content, "Error: %d %s\r\n", http_code,
++ http_code_name);
++
+ if (retry_after_seconds > 0)
+ strbuf_addf(&response_content, "Retry-After: %d\r\n",
+ retry_after_seconds);
+
-+ strbuf_addf (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
++ strbuf_addf(&response_header, "HTTP/1.1 %d %s\r\n", http_code,
++ http_code_name);
+ strbuf_addstr(&response_header, "Cache-Control: private\r\n");
-+ strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
-+ strbuf_addf (&response_header, "Content-Length: %d\r\n", (int)response_content.len);
++ strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
++ strbuf_addf(&response_header, "Content-Length: %"PRIuMAX"\r\n",
++ (uintmax_t)response_content.len);
++
+ if (retry_after_seconds > 0)
-+ strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
-+ strbuf_addf( &response_header, "Server: test-http-server/%s\r\n", git_version_string);
-+ strbuf_addf( &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
++ strbuf_addf(&response_header, "Retry-After: %d\r\n",
++ retry_after_seconds);
++
++ strbuf_addf(&response_header, "Server: test-http-server/%s\r\n",
++ git_version_string);
++ strbuf_addf(&response_header, "Date: %s\r\n", show_date(time(NULL), 0,
++ DATE_MODE(RFC2822)));
++
+ if (response_headers)
+ for_each_string_list_item(h, response_headers)
+ strbuf_addf(&response_header, "%s\r\n", h->string);
@@ t/helper/test-http-server.c: enum worker_result {
+
+ if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
+ logerror("unable to write response header");
-+ wr = WR_IO_ERROR;
++ wr = WR_FATAL_ERROR;
+ goto done;
+ }
+
+ if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
+ logerror("unable to write response content body");
-+ wr = WR_IO_ERROR;
++ wr = WR_FATAL_ERROR;
+ goto done;
+ }
+
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
while (1) {
- if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
- logerror("unable to write response");
-- wr = WR_IO_ERROR;
+- wr = WR_FATAL_ERROR;
- }
+ wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
-+ NULL, WR_OK | WR_HANGUP);
++ NULL, WR_HANGUP);
if (wr != WR_OK)
break;
+@@ t/helper/test-http-server.c: static enum worker_result worker(void)
+ close(STDIN_FILENO);
+ close(STDOUT_FILENO);
+
+- /* Only WR_OK should result in a non-zero exit code */
+- return wr != WR_OK;
++ /* Only WR_OK and WR_HANGUP should result in a non-zero exit code */
++ return wr != WR_OK && wr != WR_HANGUP;
+ }
+
+ static int max_connections = 32;
6: 252098db219 ! 6: 43f1cdcbb82 test-http-server: add HTTP request parsing
@@ Commit message
test-http-server: add HTTP request parsing
Add ability to parse HTTP requests to the test-http-server test helper.
+ Introduce `struct req` to store request information including:
+
+ * HTTP method & version
+ * Request path and query parameters
+ * Headers
+ * Content type and length (from `Content-Type` and `-Length` headers)
+
+ Failure to parse the request results in a 400 Bad Request response to
+ the client. Note that we're not trying to support all possible requests
+ here, but just enough to exercise all code under test.
Signed-off-by: Matthew John Cheetham <[email protected]>
## t/helper/test-http-server.c ##
@@ t/helper/test-http-server.c: enum worker_result {
- WR_HANGUP = 1<<1,
- };
-
+ * Close the socket and clean up. Does not imply an error.
+ */
+ WR_HANGUP = 2,
++
++ /*
++ * Unexpected request message or error in request parsing.
++ * Respond with an 400 error. Close the socket and cleanup.
++ * Exit child-process with a non-zero status.
++ */
++ WR_CLIENT_ERROR = 3,
++};
++
+/*
+ * Fields from a parsed HTTP request.
+ */
@@ t/helper/test-http-server.c: enum worker_result {
+
+ struct string_list header_list;
+ const char *content_type;
-+ ssize_t content_length;
-+};
-+
++ uintmax_t content_length;
++ unsigned has_content_length:1;
+ };
+
+#define REQ__INIT { \
+ .start_line = STRBUF_INIT, \
+ .uri_path = STRBUF_INIT, \
+ .query_args = STRBUF_INIT, \
+ .header_list = STRING_LIST_INIT_NODUP, \
+ .content_type = NULL, \
-+ .content_length = -1 \
-+ }
++ .content_length = 0, \
++ .has_content_length = 0, \
++}
+
+static void req__release(struct req *req)
+{
@@ t/helper/test-http-server.c: enum worker_result {
+ string_list_clear(&req->header_list, 0);
+}
+
- static enum worker_result send_http_error(
- int fd,
- int http_code, const char *http_code_name,
+ static enum worker_result send_http_error(int fd, int http_code,
+ const char *http_code_name,
+ int retry_after_seconds,
@@ t/helper/test-http-server.c: done:
return wr;
}
@@ t/helper/test-http-server.c: done:
+ *
+ */
+ if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
-+ result = WR_OK | WR_HANGUP;
++ result = WR_HANGUP;
+ goto done;
+ }
+
@@ t/helper/test-http-server.c: done:
+ if (nr_start_line_fields != 3) {
+ logerror("could not parse request start-line '%s'",
+ req->start_line.buf);
-+ result = WR_IO_ERROR;
++ result = WR_CLIENT_ERROR;
+ goto done;
+ }
+
@@ t/helper/test-http-server.c: done:
+ if (strcmp(req->http_version, "HTTP/1.1")) {
+ logerror("unsupported version '%s' (expecting HTTP/1.1)",
+ req->http_version);
-+ result = WR_IO_ERROR;
++ result = WR_CLIENT_ERROR;
+ goto done;
+ }
+
@@ t/helper/test-http-server.c: done:
+ string_list_append(&req->header_list, hp);
+
+ /* also store common request headers as struct req members */
-+ if (skip_prefix(hp, "Content-Type: ", &hv)) {
++ if (skip_iprefix(hp, "Content-Type: ", &hv)) {
+ req->content_type = hv;
-+ } else if (skip_prefix(hp, "Content-Length: ", &hv)) {
-+ req->content_length = strtol(hv, &hp, 10);
++ } else if (skip_iprefix(hp, "Content-Length: ", &hv)) {
++ /*
++ * Content-Length is always non-negative, but has no
++ * upper bound according to RFC 7230 (§3.3.2).
++ */
++ intmax_t len = 0;
++ if (sscanf(hv, "%"PRIdMAX, &len) != 1 || len < 0 ||
++ len == INTMAX_MAX) {
++ logerror("invalid content-length: '%s'", hv);
++ result = WR_CLIENT_ERROR;
++ goto done;
++ }
++
++ req->content_length = (uintmax_t)len;
++ req->has_content_length = 1;
+ }
+ }
+
@@ t/helper/test-http-server.c: done:
+ trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
+ trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
+ trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
-+ if (req->content_length >= 0)
-+ trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
++ if (req->has_content_length)
++ trace2_printf("%s: clen: %"PRIuMAX, TR2_CAT,
++ req->content_length);
+ if (req->content_type)
+ trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
+ for_each_string_list_item(item, &req->header_list)
@@ t/helper/test-http-server.c: done:
+static enum worker_result dispatch(struct req *req)
+{
+ return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
-+ WR_OK | WR_HANGUP);
++ WR_HANGUP);
+}
+
static enum worker_result worker(void)
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
while (1) {
- wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
-- NULL, WR_OK | WR_HANGUP);
+- NULL, WR_HANGUP);
+ req__release(&req);
+
+ alarm(timeout);
+ wr = req__read(&req, 0);
+ alarm(0);
+
++ if (wr == WR_CLIENT_ERROR)
++ wr = send_http_error(STDOUT_FILENO, 400, "Bad Request",
++ -1, NULL, wr);
++
+ if (wr != WR_OK)
+ break;
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
if (wr != WR_OK)
break;
}
+
+ ## t/t5556-http-auth.sh (new) ##
+@@
++#!/bin/sh
++
++test_description='test http auth header and credential helper interop'
++
++TEST_NO_CREATE_REPO=1
++. ./test-lib.sh
++
++# Setup a repository
++#
++REPO_DIR="$TRASH_DIRECTORY"/repo
++
++SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
++
++PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
++
++test_expect_success 'setup repos' '
++ test_create_repo "$REPO_DIR" &&
++ git -C "$REPO_DIR" branch -M main
++'
++
++run_http_server_worker() {
++ (
++ cd "$REPO_DIR"
++ test-http-server --worker "$@" 2>"$SERVER_LOG" | tr -d "\r"
++ )
++}
++
++per_test_cleanup () {
++ rm -f OUT.* &&
++ rm -f IN.* &&
++}
++
++test_expect_success 'http auth server request parsing' '
++ test_when_finished "per_test_cleanup" &&
++
++ cat >auth.config <<-EOF &&
++ [auth]
++ allowAnonymous = true
++ EOF
++
++ echo "HTTP/1.1 400 Bad Request" >OUT.http400 &&
++ echo "HTTP/1.1 200 OK" >OUT.http200 &&
++
++ cat >IN.http.valid <<-EOF &&
++ GET /info/refs HTTP/1.1
++ Content-Length: 0
++ EOF
++
++ cat >IN.http.badfirstline <<-EOF &&
++ /info/refs GET HTTP
++ EOF
++
++ cat >IN.http.badhttpver <<-EOF &&
++ GET /info/refs HTTP/999.9
++ EOF
++
++ cat >IN.http.ltzlen <<-EOF &&
++ GET /info/refs HTTP/1.1
++ Content-Length: -1
++ EOF
++
++ cat >IN.http.badlen <<-EOF &&
++ GET /info/refs HTTP/1.1
++ Content-Length: not-a-number
++ EOF
++
++ cat >IN.http.overlen <<-EOF &&
++ GET /info/refs HTTP/1.1
++ Content-Length: 9223372036854775807
++ EOF
++
++ run_http_server_worker \
++ --auth-config="$TRASH_DIRECTORY/auth.config" <IN.http.valid \
++ | head -n1 >OUT.actual &&
++ test_cmp OUT.http200 OUT.actual &&
++
++ run_http_server_worker <IN.http.badfirstline | head -n1 >OUT.actual &&
++ test_cmp OUT.http400 OUT.actual &&
++
++ run_http_server_worker <IN.http.ltzlen | head -n1 >OUT.actual &&
++ test_cmp OUT.http400 OUT.actual &&
++
++ run_http_server_worker <IN.http.badlen | head -n1 >OUT.actual &&
++ test_cmp OUT.http400 OUT.actual &&
++
++ run_http_server_worker <IN.http.overlen | head -n1 >OUT.actual &&
++ test_cmp OUT.http400 OUT.actual
++'
++
++test_done
7: ab06ac9b965 ! 7: ca9c2787248 test-http-server: pass Git requests to http-backend
@@ t/helper/test-http-server.c: done:
+ return error(_("could not send '%s'"), ok);
+
+ strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
-+ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
-+ req->uri_path.buf);
++ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
+ strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
+ if (req->query_args.len)
-+ strvec_pushf(&cp.env, "QUERY_STRING=%s",
-+ req->query_args.buf);
++ strvec_pushf(&cp.env, "QUERY_STRING=%s", req->query_args.buf);
+ if (req->content_type)
-+ strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
-+ req->content_type);
-+ if (req->content_length >= 0)
-+ strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
-+ (intmax_t)req->content_length);
++ strvec_pushf(&cp.env, "CONTENT_TYPE=%s", req->content_type);
++ if (req->has_content_length)
++ strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIuMAX,
++ (uintmax_t)req->content_length);
+ cp.git_cmd = 1;
+ strvec_push(&cp.args, "http-backend");
+ res = run_command(&cp);
@@ t/helper/test-http-server.c: done:
+ return do__git(req);
+
return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
- WR_OK | WR_HANGUP);
+ WR_HANGUP);
}
- ## t/t5556-http-auth.sh (new) ##
-@@
-+#!/bin/sh
-+
-+test_description='test http auth header and credential helper interop'
-+
-+TEST_NO_CREATE_REPO=1
-+. ./test-lib.sh
-+
+ ## t/t5556-http-auth.sh ##
+@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'
+ TEST_NO_CREATE_REPO=1
+ . ./test-lib.sh
+
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
-+# Setup a repository
-+#
-+REPO_DIR="$TRASH_DIRECTORY"/repo
-+
+ # Setup a repository
+ #
+ REPO_DIR="$TRASH_DIRECTORY"/repo
+
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
+# any need to configure directory mappings etc - we only serve this
@@ t/t5556-http-auth.sh (new)
+# killing it by PID).
+#
+PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
-+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
-+
-+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
-+
-+test_expect_success 'setup repos' '
-+ test_create_repo "$REPO_DIR" &&
-+ git -C "$REPO_DIR" branch -M main
-+'
-+
+ SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
+
+ PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+@@ t/t5556-http-auth.sh: run_http_server_worker() {
+ )
+ }
+
+stop_http_server () {
+ if ! test -f "$PID_FILE"
+ then
@@ t/t5556-http-auth.sh (new)
+ return 1
+}
+
-+per_test_cleanup () {
+ per_test_cleanup () {
+ stop_http_server &&
-+ rm -f OUT.*
-+}
+ rm -f OUT.* &&
+ rm -f IN.* &&
+ }
+@@ t/t5556-http-auth.sh: test_expect_success 'http auth server request parsing' '
+ test_cmp OUT.http400 OUT.actual
+ '
+
+
+test_expect_success 'http auth anonymous no challenge' '
+ test_when_finished "per_test_cleanup" &&
@@ t/t5556-http-auth.sh (new)
+ git ls-remote $ORIGIN_URL
+'
+
-+test_done
+ test_done
8: a1ff55dd6e2 ! 8: b8d3e81b553 test-http-server: add simple authentication
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
+
strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
- strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
- req->uri_path.buf);
+ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
+ strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
return !!res;
}
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+
+static struct auth_module *get_auth_module(const char *scheme, int create)
+{
-+ int i;
+ struct auth_module *mod;
-+ for (i = 0; i < auth_modules_nr; i++) {
++ for (size_t i = 0; i < auth_modules_nr; i++) {
+ mod = auth_modules[i];
+ if (!strcasecmp(mod->scheme, scheme))
+ return mod;
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ struct auth_module *mod = xmalloc(sizeof(struct auth_module));
+ mod->scheme = xstrdup(scheme);
+ mod->challenge_params = NULL;
-+ CALLOC_ARRAY(mod->tokens, 1);
++ ALLOC_ARRAY(mod->tokens, 1);
+ string_list_init_dup(mod->tokens);
+
+ ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ for_each_string_list_item(hdr, &req->header_list) {
+ if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
+ split = strbuf_split_str(v, ' ', 2);
-+ if (!split[0] || !split[1]) continue;
-+
-+ /* trim trailing space ' ' */
-+ strbuf_setlen(split[0], split[0]->len - 1);
-+
-+ mod = get_auth_module(split[0]->buf, 0);
-+ if (mod) {
-+ result = AUTH_DENY;
-+
-+ for_each_string_list_item(token, mod->tokens) {
-+ if (!strcmp(split[1]->buf, token->string)) {
-+ result = AUTH_ALLOW;
-+ break;
++ if (split[0] && split[1]) {
++ /* trim trailing space ' ' */
++ strbuf_rtrim(split[0]);
++
++ mod = get_auth_module(split[0]->buf, 0);
++ if (mod) {
++ result = AUTH_DENY;
++
++ for_each_string_list_item(token, mod->tokens) {
++ if (!strcmp(split[1]->buf, token->string)) {
++ result = AUTH_ALLOW;
++ break;
++ }
+ }
-+ }
+
-+ goto done;
++ strbuf_list_free(split);
++ goto done;
++ }
+ }
++
++ strbuf_list_free(split);
+ }
+ }
+
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ &hdrs, *wr);
+ }
+
-+ strbuf_list_free(split);
+ string_list_clear(&hdrs, 0);
+
+ return result == AUTH_ALLOW ||
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ return -1;
+
+ /* trim trailing ':' */
-+ if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')
++ if (p[0]->len && p[0]->buf[p[0]->len - 1] == ':')
+ strbuf_setlen(p[0], p[0]->len - 1);
+
+ *scheme = strbuf_detach(p[0], NULL);
-+
-+ if (p[1])
-+ *val = strbuf_detach(p[1], NULL);
++ *val = p[1] ? strbuf_detach(p[1], NULL) : NULL;
+
+ strbuf_list_free(p);
+ return 0;
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ char *scheme = NULL;
+ char *token = NULL;
+ char *challenge = NULL;
-+ struct auth_module *mod = NULL;
++ struct auth_module *mod;
+
+ if (!strcmp(name, "auth.challenge")) {
+ if (split_auth_param(val, &scheme, &challenge)) {
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ return do__git(req, user);
return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
- WR_OK | WR_HANGUP);
+ WR_HANGUP);
@@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
pid_file = v;
continue;
@@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
## t/t5556-http-auth.sh ##
@@ t/t5556-http-auth.sh: per_test_cleanup () {
- rm -f OUT.*
+ stop_http_server &&
+ rm -f OUT.* &&
+ rm -f IN.* &&
++ rm -f auth.config
}
+ test_expect_success 'http auth server request parsing' '
+@@ t/t5556-http-auth.sh: test_expect_success 'http auth server request parsing' '
+ test_cmp OUT.http400 OUT.actual
+ '
+
+test_expect_success CURL 'http auth server auth config' '
-+ #test_when_finished "per_test_cleanup" &&
++ test_when_finished "per_test_cleanup" &&
+
+ cat >auth.config <<-EOF &&
+ [auth]
@@ t/t5556-http-auth.sh: per_test_cleanup () {
+
+ test_cmp OUT.expected OUT.actual
+'
-+
+
test_expect_success 'http auth anonymous no challenge' '
test_when_finished "per_test_cleanup" &&9: 76125cdf239 = 9: 2f97c94f679 test-http-server: add sending of arbitrary headers 10: cc9a220ed1f = 10: 4b1635b3f69 http: replace unsafe size_t multiplication with st_mult 11: bc1ac8d3eb3 = 11: 5f5e46038cf http: read HTTP WWW-Authenticate response headers 12: 7c8229f0b11 ! 12: 09164f77d56 credential: add WWW-Authenticate header to cred requests
@@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '
+setup_credential_helper
+
- stop_http_server () {
- if ! test -f "$PID_FILE"
- then
-@@ t/t5556-http-auth.sh: start_http_server () { -
- per_test_cleanup () {
+ run_http_server_worker() {
+ (
+ cd "$REPO_DIR"
+@@ t/t5556-http-auth.sh: per_test_cleanup () {
stop_http_server &&
-- rm -f OUT.*
-+ rm -f OUT.* &&
+ rm -f OUT.* &&
+ rm -f IN.* &&
+ rm -f *.cred &&
-+ rm -f auth.config
+ rm -f auth.config
}
- test_expect_success CURL 'http auth server auth config' '
@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
git ls-remote $ORIGIN_URL
'
Signature · 1 line
gitgitgadget
PATCH v7 1/12 daemon: libify socket setup and option functions
From: Matthew John Cheetham <[email protected]>
Extract functions for setting up listening sockets and keep-alive options from daemon.c to new daemon-utils.{c,h} files. Remove direct dependencies on global state by inlining the behaviour at the callsites for all libified functions.
--- Makefile | 1 + daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 23 ++++++ daemon.c | 214 +------------------------------------------------ 4 files changed, 237 insertions(+), 210 deletions(-) create mode 100644 daemon-utils.c create mode 100644 daemon-utils.h
Makefile +1 -0
@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o LIB_OBJS += csum-file.o LIB_OBJS += ctype.o LIB_OBJS += date.o +LIB_OBJS += daemon-utils.o LIB_OBJS += decorate.o LIB_OBJS += delta-islands.o LIB_OBJS += diagnose.o
daemon-utils.c new file +209 -0
209 lines not on this page · raw message
daemon-utils.h new file +23 -0
@@ -0,0 +1,23 @@ +#ifndef DAEMON_UTILS_H +#define DAEMON_UTILS_H + +#include "git-compat-util.h" +#include "string-list.h" + +typedef void (*log_fn )(const char *msg, ...); + +struct socketlist { +int *list; +size_t nr; +size_t alloc; +}; + +/* Enable sending of keep-alive messages on the socket. */ +void set_keep_alive (int sockfd,log_fn logerror); + +/* Setup a number of sockets to listen on the provided addresses. */ +void socksetup (struct string_list *listen_addr,int listen_port, +struct socketlist *socklist,int reuseaddr, +log_fn logerror); + +#endif
daemon.c +4 -210
246 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v7 2/12 daemon: libify child process handling functions
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Extract functions and structures for managing child processes started from the parent daemon-like process from daemon.c to the new shared daemon-utils.{c,h} files.
One minor functional change is introduced to check_dead_children() where the logging of a dead/disconnected child is now optional. With the 'libification' of these functions we extract the call to loginfo to a call to a function pointer, and guard the log message creation and logging behind a NULL check. Callers can now skip logging by passing NULL as the log_fn loginfo argument.
The behaviour of callers in daemon.c remains the same (save one extra NULL check) however as a pointer to loginfo is always passed.
--- daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++ daemon-utils.h | 32 ++++++++++++++++++ daemon.c | 92 +++----------------------------------------------- 3 files changed, 114 insertions(+), 87 deletions(-)
daemon-utils.c +77 -0
@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port, } } } + +static int addrcmp (const struct sockaddr_storage *s1, +const struct sockaddr_storage *s2) +{ +const struct sockaddr *sa1 = (const struct sockaddr *) s1; +const struct sockaddr *sa2 = (const struct sockaddr *) s2; + +if (sa1->sa_family != sa2->sa_family) +return sa1->sa_family - sa2->sa_family; +if (sa1->sa_family ==AF_INET ) +return memcmp (&((struct sockaddr_in *)s1)->sin_addr, + &((struct sockaddr_in *)s2)->sin_addr, +sizeof (struct in_addr )); +#ifndef NO_IPV6 +if (sa1->sa_family ==AF_INET6 ) +return memcmp (&((struct sockaddr_in6 *)s1)->sin6_addr, + &((struct sockaddr_in6 *)s2)->sin6_addr, +sizeof (struct in6_addr )); +#endif +return 0 ; +} + +void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, +struct child *firstborn ,unsigned int *live_children) +{ +struct child *newborn, **cradle; + +CALLOC_ARRAY (newborn,1 ); + (*live_children)++; +memcpy (&newborn->cld, cld,sizeof (*cld)); +memcpy (&newborn->address, addr, addrlen); +for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) +if (!addrcmp (&(*cradle)->address, &newborn->address)) +break ; + newborn->next = *cradle; + *cradle = newborn; +} + +void kill_some_child (struct child *firstborn) +{ +const struct child *blanket, *next; + +if (!(blanket = firstborn)) +return ; + +for (; (next = blanket->next); blanket = next) +if (!addrcmp (&blanket->address, &next->address)) { +kill (blanket->cld.pid,SIGTERM ); +break ; + } +} + +void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo) +{ +int status; +pid_t pid; + +struct child **cradle, *blanket; +for (cradle = &firstborn; (blanket = *cradle);) +if ((pid =waitpid (blanket->cld.pid, &status,WNOHANG )) >1 ) { +if (loginfo) { +const char *dead ="" ; +if (status) + dead =" (with error)" ; +loginfo ("[%" PRIuMAX"] Disconnected%s" , + (uintmax_t )pid, dead); + } + +/* remove the child */ + *cradle = blanket->next; + (*live_children)--; +child_process_clear (&blanket->cld); +free (blanket); + }else + cradle = &blanket->next; +}
daemon-utils.h +32 -0
@@ -2,6 +2,7 @@#define DAEMON_UTILS_H #include "git-compat-util.h" +#include "run-command.h" #include "string-list.h" typedef void (*log_fn )(const char *msg, ...);
@@ -20,4 +21,35 @@ void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist, int reuseaddr, log_fn logerror); +struct child { +struct child *next; +struct child_process cld; +struct sockaddr_storage address; +}; + +/* +* Add the child_process to the set of children and increment the number of +* live children. +*/ +void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, +struct child *firstborn,unsigned int *live_children); + +/* +* Kill the newest connection from a duplicate IP. +* +* This function should be called if the number of connections grows +* past the maximum number of allowed connections. +*/ +void kill_some_child (struct child *firstborn); + +/* +* Check for children that have disconnected and remove them from the +* active set, decrementing the number of live children. +* +* Optionally log the child PID that disconnected by passing a loginfo +* function. +*/ +void check_dead_children (struct child *firstborn,unsigned int *live_children, +log_fn loginfo); +#endif
daemon.c +5 -87
121 lines not on this page · raw message
Signature · 1 line
gitgitgadget
PATCH v7 3/12 daemon: rename some esoteric/laboured terminology
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Rename some of the variables and function arguments used to manage child processes. The existing names are esoteric; stretching an analogy too far to the point of being confusing to understand.
Rename "firstborn" to "first_child", "newborn" to "new_cld", "blanket" to "current" and "cradle" to "ptr".
--- daemon-utils.c | 46 +++++++++++++++++++++++----------------------- daemon-utils.h | 6 +++--- daemon.c | 10 +++++----- 3 files changed, 31 insertions(+), 31 deletions(-)
daemon-utils.c +23 -23
@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1, }void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, -struct child *firstborn ,unsigned int *live_children) +struct child *first_child,unsigned int *live_children) { -struct child *newborn, **cradle; +struct child *new_cld, **current; -CALLOC_ARRAY (newborn,1 ); +CALLOC_ARRAY (new_cld,1 ); (*live_children)++; -memcpy (&newborn->cld, cld,sizeof (*cld)); -memcpy (&newborn->address, addr, addrlen); -for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next) -if (!addrcmp (&(*cradle)->address, &newborn->address)) +memcpy (&new_cld->cld, cld,sizeof (*cld)); +memcpy (&new_cld->address, addr, addrlen); +for (current = &first_child; *current; current = &(*current)->next) +if (!addrcmp (&(*current)->address, &new_cld->address))break ; - newborn->next = *cradle; - *cradle = newborn; + new_cld->next = *current; + *current = new_cld; } -void kill_some_child (struct child *firstborn) +void kill_some_child (struct child *first_child) { -const struct child *blanket, *next; +const struct child *current, *next; -if (!(blanket = firstborn)) +if (!(current = first_child))return ; -for (; (next = blanket->next); blanket = next) -if (!addrcmp (&blanket->address, &next->address)) { -kill (blanket->cld.pid,SIGTERM ); +for (; (next = current->next); current = next) +if (!addrcmp (¤t->address, &next->address)) { +kill (current->cld.pid,SIGTERM );break ; } } -void check_dead_children (struct child *firstborn,unsigned int *live_children, +void check_dead_children (struct child *first_child,unsigned int *live_children,log_fn loginfo) {int status;pid_t pid; -struct child **cradle, *blanket; -for (cradle = &firstborn; (blanket = *cradle);) -if ((pid =waitpid (blanket->cld.pid, &status,WNOHANG )) >1 ) { +struct child **ptr, *current; +for (ptr = &first_child; (current = *ptr);) +if ((pid =waitpid (current->cld.pid, &status,WNOHANG )) >1 ) {if (loginfo) {const char *dead ="" ;if (status)
@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children, }/* remove the child */ - *cradle = blanket->next; + *ptr = current->next; (*live_children)--; -child_process_clear (&blanket->cld); -free (blanket); +child_process_clear (¤t->cld); +free (current); }else - cradle = &blanket->next; + ptr = ¤t->next; }
daemon-utils.h +3 -3
@@ -32,7 +32,7 @@ struct child {* live children. */ void add_child (struct child_process *cld,struct sockaddr *addr,socklen_t addrlen, -struct child *firstborn,unsigned int *live_children); +struct child *first_child,unsigned int *live_children);/* * Kill the newest connection from a duplicate IP.
@@ -40,7 +40,7 @@ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl* This function should be called if the number of connections grows * past the maximum number of allowed connections. */ -void kill_some_child (struct child *firstborn); +void kill_some_child (struct child *first_child);/* * Check for children that have disconnected and remove them from the
@@ -49,7 +49,7 @@ void kill_some_child(struct child *firstborn);* Optionally log the child PID that disconnected by passing a loginfo * function. */ -void check_dead_children (struct child *firstborn,unsigned int *live_children, +void check_dead_children (struct child *first_child,unsigned int *live_children,log_fn loginfo);#endif
daemon.c +5 -5
@@ -789,7 +789,7 @@ static int max_connections = 32;static unsigned int live_children; -static struct child *firstborn; +static struct child *first_child;static struct strvec cld_argv =STRVEC_INIT ;static void handle (int incoming,struct sockaddr *addr,socklen_t addrlen)
@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)struct child_process cld =CHILD_PROCESS_INIT ;if (max_connections && live_children >= max_connections) { -kill_some_child (firstborn); +kill_some_child (first_child);sleep (1 );/* give it some time to die */ -check_dead_children (firstborn, &live_children, loginfo); +check_dead_children (first_child, &live_children, loginfo);if (live_children >= max_connections) {close (incoming);logerror ("Too many children, dropping connection" );
@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)if (start_command (&cld))logerror ("unable to fork" );else -add_child (&cld, addr, addrlen, firstborn, &live_children); +add_child (&cld, addr, addrlen, first_child, &live_children); }static void child_handler (int signo)
@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)for (;;) {int i; -check_dead_children (firstborn, &live_children, loginfo); +check_dead_children (first_child, &live_children, loginfo);if (poll (pfd, socklist->nr,-1 ) <0 ) {if (errno !=EINTR ) {
Signature · 1 line
gitgitgadget
PATCH v7 4/12 test-http-server: add stub HTTP server test helper
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Introduce a mini HTTP server helper that in the future will be enhanced to provide a frontend for the git-http-backend, with support for arbitrary authentication schemes.
Right now, test-http-server is a pared-down copy of the git-daemon that always returns a 501 Not Implemented response to all callers.
--- Makefile | 1 + contrib/buildsystems/CMakeLists.txt | 11 +- t/helper/.gitignore | 1 + t/helper/test-http-server.c | 381 ++++++++++++++++++++++++++++ 4 files changed, 392 insertions(+), 2 deletions(-) create mode 100644 t/helper/test-http-server.c
Makefile +1 -0
@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o # Do not add more tests here unless they have extra dependencies. Add # them in TEST_BUILTINS_OBJS above. TEST_PROGRAMS_NEED_X += test-fake-ssh +TEST_PROGRAMS_NEED_X += test-http-server TEST_PROGRAMS_NEED_X += test-tool TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))
contrib/buildsystems/CMakeLists.txt +9 -2
@@ -961,6 +961,9 @@ if(BUILD_TESTING) add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c) target_link_libraries(test-fake-ssh common-main) +add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c) +target_link_libraries(test-http-server common-main) + #reftable-tests parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS") list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/")
@@ -980,6 +983,11 @@ if(MSVC) PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) set_target_properties(test-fake-ssh test-tool PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) + + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper) + set_target_properties(test-http-server + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper) endif() #wrapper scripts
@@ -987,8 +995,7 @@ set(wrapper_scripts git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar) set(wrapper_test_scripts - test-fake-ssh test-tool) - + test-http-server test-fake-ssh test-tool) foreach(script ${wrapper_scripts}) file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
t/helper/.gitignore +1 -0
@@ -1,2 +1,3 @@ /test-tool /test-fake-ssh +/test-http-server
t/helper/test-http-server.c new file +381 -0
381 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v7 04/12] test-http-server: add stub HTTP server test helper
Quoted 6 lines · Matthew John Cheetham via GitGitGadget
Introduce a mini HTTP server helper that in the future will be enhanced to provide a frontend for the git-http-backend, with support for arbitrary authentication schemes.
Right now, test-http-server is a pared-down copy of the git-daemon that always returns a 501 Not Implemented response to all callers.
This may be a dumb question, but I didn't see it raised or answered in the cover letter or earlier in the thread: what does this custom server give us that our current use of apache in the tests does not?
I'd imagine the answer is along the lines of: configuring apache to respond to auth in the way we'd like is hard and/or impossible. And if so, and if it's just "hard", I'd ask "how hard?".
Because I see a few downsides to introducing a custom server here:
1. It may or may not behave like real-world servers, which makes the test slightly less good. Not that apache can claim to cover all real-world behavior, but it's probably closer to reality (and that has flushed out interesting bugs and behaviors before).
2. It's a non-trivial amount of code, doing tricky things like daemonizing, socket setup and I/O, pidfiles, and so on. For example, it handles multiple listen addresses, and ipv6. Do we really need that? And we take shortcuts around things like CGI output buffering. I do see that you tried to reuse some existing code, but...
3. You're reusing parts of git-daemon, which I personally consider to be one of the absolute low-points of code quality inside git.git. I know that's a subjective statement, but my experience running it within GitHub was that there were a lot of rough edges, and we ended up rewriting several parts of it. In particular, the child-handling is inefficient (I seem to recall that it's quadratic in several places) and has odd behaviors (its kill_some_child() is basically nonsense, and can starve requests). Probably none of that matters for your use case in tests, which is likely doing one request at a time.
But then I'd wonder: do we really need those bits at all, then? In fact, would it be sufficient to write the server to handle one request at a time, without spawning a worker child at all?
I dunno. I know I am showing up to review quite late in the life of this patch series, and that probably makes me a bad person to start the review with "and could you re-do the whole test infrastructure". So if you want to tell me to get lost, I'd understand. But I had hoped that one day we could just delete all of daemon.c, and this moves in the opposite direction.
I think my order of preference (if you care ;) ) is:
1. Can we do it with apache?
2. If not, could we do it with a trivial application of some existing http server framework? I know that may mean extra dependencies, but there's a lot of perl in the test suite already, and it doesn't seem too terrible to me to require it for these tests.
3. If not, can we make the http-server code even more minimal?
Quoted 6 lines
Makefile | 1 + contrib/buildsystems/CMakeLists.txt | 11 +- t/helper/.gitignore | 1 + t/helper/test-http-server.c | 381 ++++++++++++++++++++++++++++ 4 files changed, 392 insertions(+), 2 deletions(-) create mode 100644 t/helper/test-http-server.c
If we do use this code, here are a few small bits I noticed:
t/helper/test-http-server.c line 170
Quoted 9 lines
t/helper/test-http-server.c new file +9
-0+static void child_handler (int signo) +{ +/* +* Otherwise empty handler because systemcalls will get interrupted +* upon signal receipt +* SysV needs the handler to be rearmed +*/ +signal (SIGCHLD , child_handler); +}
daemon.c has this, too. If we're going to share its child-handling code, should it maybe just handle this part, too?
t/helper/test-http-server.c line 177
Quoted 6 lines
t/helper/test-http-server.c new file +6
-0+static int service_loop (struct socketlist *socklist) +{ +struct pollfd *pfd; +int i; + +CALLOC_ARRAY (pfd, socklist->nr);
(Actually, Coverity noticed this, not me).
This pfd is never freed. I know this is copied from daemon.c, but in that file we never return from the function. Here you do break out of the loop and try to clean up; you'd want to free(pfd) there.
Quoted 16 lines
t/helper/test-http-server.c new file +16
-0+for (;;) { +int i; +int nr_ready; +int timeout = (pid_file ?100 :-1 ); + +check_dead_children (first_child, &live_children, loginfo); + + nr_ready =poll (pfd, socklist->nr, timeout); +if (nr_ready <0 ) { +if (errno !=EINTR ) { +logerror ("Poll failed, resuming: %s" , +strerror (errno)); +sleep (1 ); + } +continue ; + }
t/helper/test-http-server.c line 211
Quoted 10 lines
t/helper/test-http-server.c new file +10
-0+else if (nr_ready ==0 ) { +/* +* If we have a pid_file, then we watch it. +* If someone deletes it, we shutdown the service. +* The shell scripts in the test suite will use this. +*/ +if (!pid_file ||file_exists (pid_file)) +continue ; + goto shutdown; + }
I wondered how this would work, since removal of the pid file won't trigger poll(). But it looks like you set the timeout unconditionally in that case, so we're effectively polling for its removal every 100ms. It's not beautiful, but it should work reliably.
That also made me wonder about this timeout:
t/helper/test-http-server.c line 315
Quoted 4 lines
t/helper/test-http-server.c new file +4
-0+if (skip_prefix (arg,"--timeout=" , &v)) { + timeout =atoi (v); +continue ; + }
but it is not used. The "timeout" in service_loop shadows the global, and nobody ever looks at the global (however, it looks like a later patch adds an alarm() which uses it).
t/helper/test-http-server.c line 321
Quoted 6 lines
t/helper/test-http-server.c new file +6
-0+if (skip_prefix (arg,"--max-connections=" , &v)) { + max_connections =atoi (v); +if (max_connections <0 ) + max_connections =0 ;/* unlimited */ +continue ; + }
I don't think any caller ever uses --max-connections, though. This could be dropped, and that would simplify service_loop a bit.
t/helper/test-http-server.c line 339
Quoted 2 lines
t/helper/test-http-server.c new file +2
-0+if (listen_port ==0 ) + listen_port =DEFAULT_GIT_PORT ;
That's a funny default. Surely "80" or even "8080" would make more sense. But really, since our purpose is tests, isn't it a misconfiguration if the test does not tell us which port (which is generally dynamic based on the test number), and we should bail?
t/helper/test-http-server.c line 351
Quoted 11 lines
t/helper/test-http-server.c new file +11
-0+/* +* If no --listen=<addr> args are given, the setup_named_sock() +* code will use receive a NULL address and set INADDR_ANY. +* This exposes both internal and external interfaces on the +* port. +* +* Disallow that and default to the internal-use-only loopback +* address. +*/ +if (!listen_addr.nr) +string_list_append (&listen_addr,"127.0.0.1" );
Likewise, it seems like you could probably ditch --listen entirely, and just always listen on 127.0.0.1, for the purposes of the tests.
-Peff
PATCH v7 5/12 test-http-server: add HTTP error response function
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Introduce a function to the test-http-server test helper to write more full and valid HTTP error responses, including all the standard response headers like Server and Date.
--- t/helper/test-http-server.c | 76 +++++++++++++++++++++++++++++++++---- 1 file changed, 69 insertions(+), 7 deletions(-)
t/helper/test-http-server.c +69 -7
@@ -76,11 +76,75 @@ enum worker_result {* Exit child-process with non-zero status. */ WR_FATAL_ERROR =1 , + +/* +* Close the socket and clean up. Does not imply an error. +*/ + WR_HANGUP = 2, }; +static enum worker_result send_http_error (int fd,int http_code, +const char *http_code_name, +int retry_after_seconds, +struct string_list *response_headers, +enum worker_result wr_in) +{ +struct strbuf response_header =STRBUF_INIT ; +struct strbuf response_content =STRBUF_INIT ; +struct string_list_item *h; +enum worker_result wr; + +strbuf_addf (&response_content,"Error: %d %s\r\n" , http_code, + http_code_name); + +if (retry_after_seconds >0 ) +strbuf_addf (&response_content,"Retry-After: %d\r\n" , + retry_after_seconds); + +strbuf_addf (&response_header,"HTTP/1.1 %d %s\r\n" , http_code, + http_code_name); +strbuf_addstr (&response_header,"Cache-Control: private\r\n" ); +strbuf_addstr (&response_header,"Content-Type: text/plain\r\n" ); +strbuf_addf (&response_header,"Content-Length: %" PRIuMAX"\r\n" , + (uintmax_t )response_content.len); + +if (retry_after_seconds >0 ) +strbuf_addf (&response_header,"Retry-After: %d\r\n" , + retry_after_seconds); + +strbuf_addf (&response_header,"Server: test-http-server/%s\r\n" , + git_version_string); +strbuf_addf (&response_header,"Date: %s\r\n" ,show_date (time (NULL ),0 , +DATE_MODE (RFC2822 ))); + +if (response_headers) +for_each_string_list_item (h, response_headers) +strbuf_addf (&response_header,"%s\r\n" , h->string); +strbuf_addstr (&response_header,"\r\n" ); + +if (write_in_full (fd, response_header.buf, response_header.len) <0 ) { +logerror ("unable to write response header" ); + wr =WR_FATAL_ERROR ; + goto done; + } + +if (write_in_full (fd, response_content.buf, response_content.len) <0 ) { +logerror ("unable to write response content body" ); + wr =WR_FATAL_ERROR ; + goto done; + } + + wr = wr_in; + +done: +strbuf_release (&response_header); +strbuf_release (&response_content); + +return wr; +} +static enum worker_result worker (void ) { -const char *response ="HTTP/1.1 501 Not Implemented\r\n" ;char *client_addr =getenv ("REMOTE_ADDR" );char *client_port =getenv ("REMOTE_PORT" );enum worker_result wr =WR_OK ;
@@ -91,10 +155,8 @@ static enum worker_result worker(void)set_keep_alive (0 , logerror);while (1 ) { -if (write_in_full (STDOUT_FILENO , response,strlen (response)) <0 ) { -logerror ("unable to write response" ); - wr =WR_FATAL_ERROR ; - } + wr =send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 , +NULL ,WR_HANGUP );if (wr !=WR_OK )break ;
@@ -103,8 +165,8 @@ static enum worker_result worker(void)close (STDIN_FILENO );close (STDOUT_FILENO ); -/* Only WR_OK should result in a non-zero exit code */ -return wr !=WR_OK ; +/* Only WR_OK and WR_HANGUP should result in a non-zero exit code */ +return wr !=WR_OK && wr !=WR_HANGUP ; }static int max_connections =32 ;
Signature · 1 line
gitgitgadget
PATCH v7 6/12 test-http-server: add HTTP request parsing
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add ability to parse HTTP requests to the test-http-server test helper. Introduce struct req to store request information including:
* HTTP method & version
* Request path and query parameters
* Headers
* Content type and length (from Content-Type and -Length headers)
Failure to parse the request results in a 400 Bad Request response to the client. Note that we're not trying to support all possible requests here, but just enough to exercise all code under test.
--- t/helper/test-http-server.c | 202 +++++++++++++++++++++++++++++++++++- t/t5556-http-auth.sh | 90 ++++++++++++++++ 2 files changed, 290 insertions(+), 2 deletions(-) create mode 100755 t/t5556-http-auth.sh
t/helper/test-http-server.c +200 -2
@@ -81,8 +81,53 @@ enum worker_result {* Close the socket and clean up. Does not imply an error. */ WR_HANGUP =2 , + +/* +* Unexpected request message or error in request parsing. +* Respond with an 400 error. Close the socket and cleanup. +* Exit child-process with a non-zero status. +*/ + WR_CLIENT_ERROR = 3, +}; + +/* +* Fields from a parsed HTTP request. +*/ +struct req { +struct strbuf start_line; + +const char *method; +const char *http_version; + +struct strbuf uri_path; +struct strbuf query_args; + +struct string_list header_list; +const char *content_type; +uintmax_t content_length; +unsigned has_content_length:1 ; }; +#define REQ__INIT { \ + .start_line = STRBUF_INIT, \ + .uri_path = STRBUF_INIT, \ + .query_args = STRBUF_INIT, \ + .header_list = STRING_LIST_INIT_NODUP, \ + .content_type = NULL, \ + .content_length = 0, \ + .has_content_length = 0, \ +} + +static void req__release (struct req *req) +{ +strbuf_release (&req->start_line); + +strbuf_release (&req->uri_path); +strbuf_release (&req->query_args); + +string_list_clear (&req->header_list,0 ); +} + static enum worker_result send_http_error(int fd, int http_code, const char *http_code_name, int retry_after_seconds,
@@ -143,8 +188,150 @@ done:return wr; } +/* +* Read the HTTP request up to the start of the optional message-body. +* We do this byte-by-byte because we have keep-alive turned on and +* cannot rely on an EOF. +* +* https://tools.ietf.org/html/rfc7230 +* +* We cannot call die() here because our caller needs to properly +* respond to the client and/or close the socket before this +* child exits so that the client doesn't get a connection reset +* by peer error. +*/ +static enum worker_result req__read (struct req *req,int fd) +{ +struct strbuf h =STRBUF_INIT ; +struct string_list start_line_fields =STRING_LIST_INIT_DUP ; +int nr_start_line_fields; +const char *uri_target; +const char *query; +char *hp; +const char *hv; + +enum worker_result result =WR_OK ; + +/* +* Read line 0 of the request and split it into component parts: +* +* <method> SP <uri-target> SP <HTTP-version> CRLF +* +*/ +if (strbuf_getwholeline_fd (&req->start_line, fd,'\n' ) ==EOF ) { + result =WR_HANGUP ; + goto done; + } + +strbuf_trim_trailing_newline (&req->start_line); + + nr_start_line_fields =string_list_split (&start_line_fields, + req->start_line.buf, +' ' ,-1 ); +if (nr_start_line_fields !=3 ) { +logerror ("could not parse request start-line '%s'" , + req->start_line.buf); + result =WR_CLIENT_ERROR ; + goto done; + } + + req->method =xstrdup (start_line_fields.items[0 ].string); + req->http_version =xstrdup (start_line_fields.items[2 ].string); + + uri_target = start_line_fields.items[1 ].string; + +if (strcmp (req->http_version,"HTTP/1.1" )) { +logerror ("unsupported version '%s' (expecting HTTP/1.1)" , + req->http_version); + result =WR_CLIENT_ERROR ; + goto done; + } + + query =strchr (uri_target,'?' ); + +if (query) { +strbuf_add (&req->uri_path, uri_target, (query - uri_target)); +strbuf_trim_trailing_dir_sep (&req->uri_path); +strbuf_addstr (&req->query_args, query +1 ); + }else { +strbuf_addstr (&req->uri_path, uri_target); +strbuf_trim_trailing_dir_sep (&req->uri_path); + } + +/* +* Read the set of HTTP headers into a string-list. +*/ +while (1 ) { +if (strbuf_getwholeline_fd (&h, fd,'\n' ) ==EOF ) + goto done; +strbuf_trim_trailing_newline (&h); + +if (!h.len) + goto done;/* a blank line ends the header */ + + hp =strbuf_detach (&h,NULL ); +string_list_append (&req->header_list, hp); + +/* also store common request headers as struct req members */ +if (skip_iprefix (hp,"Content-Type: " , &hv)) { + req->content_type = hv; + }else if (skip_iprefix (hp,"Content-Length: " , &hv)) { +/* +* Content-Length is always non-negative, but has no +* upper bound according to RFC 7230 (§3.3.2). +*/ +intmax_t len =0 ; +if (sscanf (hv,"%" PRIdMAX, &len) !=1 || len <0 || + len ==INTMAX_MAX ) { +logerror ("invalid content-length: '%s'" , hv); + result =WR_CLIENT_ERROR ; + goto done; + } + + req->content_length = (uintmax_t )len; + req->has_content_length =1 ; + } + } + +/* +* We do not attempt to read the <message-body>, if it exists. +* We let our caller read/chunk it in as appropriate. +*/ + +done: +string_list_clear (&start_line_fields,0 ); + +/* +* This is useful for debugging the request, but very noisy. +*/ +if (trace2_is_enabled ()) { +struct string_list_item *item; +trace2_printf ("%s: %s" ,TR2_CAT , req->start_line.buf); +trace2_printf ("%s: hver: %s" ,TR2_CAT , req->http_version); +trace2_printf ("%s: hmth: %s" ,TR2_CAT , req->method); +trace2_printf ("%s: path: %s" ,TR2_CAT , req->uri_path.buf); +trace2_printf ("%s: qury: %s" ,TR2_CAT , req->query_args.buf); +if (req->has_content_length) +trace2_printf ("%s: clen: %" PRIuMAX,TR2_CAT , + req->content_length); +if (req->content_type) +trace2_printf ("%s: ctyp: %s" ,TR2_CAT , req->content_type); +for_each_string_list_item (item, &req->header_list) +trace2_printf ("%s: hdrs: %s" ,TR2_CAT , item->string); + } + +return result; +} + +static enum worker_result dispatch (struct req *req) +{ +return send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 ,NULL , +WR_HANGUP ); +} +static enum worker_result worker (void ) { +struct req req =REQ__INIT ;char *client_addr =getenv ("REMOTE_ADDR" );char *client_port =getenv ("REMOTE_PORT" );enum worker_result wr =WR_OK ;
@@ -155,9 +342,20 @@ static enum worker_result worker(void)set_keep_alive (0 , logerror);while (1 ) { - wr =send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 , -NULL ,WR_HANGUP ); +req__release (&req); + +alarm (timeout); + wr =req__read (&req,0 ); +alarm (0 ); + +if (wr ==WR_CLIENT_ERROR ) + wr =send_http_error (STDOUT_FILENO ,400 ,"Bad Request" , +-1 ,NULL , wr); + +if (wr !=WR_OK ) +break ; + wr =dispatch (&req);if (wr !=WR_OK )break ; }
t/t5556-http-auth.sh new file +90 -0
90 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v7 06/12] test-http-server: add HTTP request parsing
t/helper/test-http-server.c enum worker_result { · line 119
Quoted 9 lines · Matthew John Cheetham via GitGitGadget
t/helper/test-http-server.c +9
-0+#define REQ__INIT { \ + .start_line = STRBUF_INIT, \ + .uri_path = STRBUF_INIT, \ + .query_args = STRBUF_INIT, \ + .header_list = STRING_LIST_INIT_NODUP, \ + .content_type = NULL, \ + .content_length = 0, \ + .has_content_length = 0, \ +}
We declare header_list as nodup, but later we put actual duplicated strings in it:
t/helper/test-http-server.c done: · line 273
Quoted 2 lines
t/helper/test-http-server.c +2
-0+ hp =strbuf_detach (&h,NULL ); +string_list_append (&req->header_list, hp);
So later when we free it:
t/helper/test-http-server.c enum worker_result { · line 129
Quoted 9 lines
t/helper/test-http-server.c +9
-0+static void req__release (struct req *req) +{ +strbuf_release (&req->start_line); + +strbuf_release (&req->uri_path); +strbuf_release (&req->query_args); + +string_list_clear (&req->header_list,0 ); +}
the strings will be leaked. There are a lot of solutions here, including setting strdup_strings right before freeing. But it's probably reasonable to just use INIT_DUP, and then when storing, just do:
string_list_append(&req->header_list, h.buf);
Since "h" is filled by strbuf_getwholeline(), there's no need to erase the contents. It should reset the buffer itself (and so you end up re-using the same buffer, rather than freeing it for each loop). You will have to remember to strbuf_release() after the loop, though.
The leak isn't very big, and we hold onto it until the process ends anyway, but it will probably cause the leak-detector to complain.
(Yet another solution would just be to dump the trace as we parse the headers, rather than holding them, since that appears to be the only use of header_list).
Quoted 7 lines
t/helper/test-http-server.c +7
-0+/* +* Read the set of HTTP headers into a string-list. +*/ +while (1 ) { +if (strbuf_getwholeline_fd (&h, fd,'\n' ) ==EOF ) + goto done; +strbuf_trim_trailing_newline (&h);
t/helper/test-http-server.c done: · line 277
Quoted 10 lines
t/helper/test-http-server.c +10
-0+ +if (!h.len) + goto done;/* a blank line ends the header */ + + hp =strbuf_detach (&h,NULL ); +string_list_append (&req->header_list, hp); + +/* also store common request headers as struct req members */ +if (skip_iprefix (hp,"Content-Type: " , &hv)) { + req->content_type = hv;
I think this is stricter than necessary. The whitespace after the colon is optional, but can also be longer than just one space (or could be a tab). It's probably OK to be picky here since this is just for tests, but we'd want to make sure we're not this picky on the client side.
t/helper/test-http-server.c done: · line 289
Quoted 12 lines
t/helper/test-http-server.c +12
-0+ }else if (skip_iprefix (hp,"Content-Length: " , &hv)) { +/* +* Content-Length is always non-negative, but has no +* upper bound according to RFC 7230 (§3.3.2). +*/ +intmax_t len =0 ; +if (sscanf (hv,"%" PRIdMAX, &len) !=1 || len <0 || + len ==INTMAX_MAX ) { +logerror ("invalid content-length: '%s'" , hv); + result =WR_CLIENT_ERROR ; + goto done; + }
We usually avoid sscanf because it's error-checking sucks. For example, this will accept "123.garbage", but you can't tell because you have no clue how far it got. Something like strtoimax() is better. It probably doesn't matter much since this is test code, though I do think in the long run it would be nice to add scanf(), etc, to our list of banned functions (there are one or two other uses currently, though, so that isn't imminent).
-Peff
PATCH v7 7/12 test-http-server: pass Git requests to http-backend
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Teach the test-http-sever test helper to forward Git requests to the git-http-backend.
Introduce a new test script t5556-http-auth.sh that spins up the test HTTP server and attempts an ls-remote on the served repository, without any authentication.
--- t/helper/test-http-server.c | 68 ++++++++++++++++++++++++++++++ t/t5556-http-auth.sh | 83 +++++++++++++++++++++++++++++++++++++ 2 files changed, 151 insertions(+)
t/helper/test-http-server.c +68 -0
@@ -323,8 +323,76 @@ done:return result; } +static int is_git_request (struct req *req) +{ +static regex_t *smart_http_regex; +static int initialized; + +if (!initialized) { + smart_http_regex =xmalloc (sizeof (*smart_http_regex)); +/* +* This regular expression matches all dumb and smart HTTP +* requests that are currently in use, and defined in +* Documentation/gitprotocol-http.txt. +* +*/ +if (regcomp (smart_http_regex,"^/(HEAD|info/refs|" +"objects/info/[^/]+|git-(upload|receive)-pack)$" , +REG_EXTENDED )) { +warning ("could not compile smart HTTP regex" ); + smart_http_regex =NULL ; + } + initialized =1 ; + } + +return smart_http_regex && + !regexec (smart_http_regex, req->uri_path.buf,0 ,NULL ,0 ); +} + +static enum worker_result do__git (struct req *req) +{ +const char *ok ="HTTP/1.1 200 OK\r\n" ; +struct child_process cp =CHILD_PROCESS_INIT ; +int res; + +/* +* Note that we always respond with a 200 OK response even if the +* http-backend process exits with an error. This helper is intended +* only to be used to exercise the HTTP auth handling in the Git client, +* and specifically around authentication (not handled by http-backend). +* +* If we wanted to respond with a more 'valid' HTTP response status then +* we'd need to buffer the output of http-backend, wait for and grok the +* exit status of the process, then write the HTTP status line followed +* by the http-backend output. This is outside of the scope of this test +* helper's use at time of writing. +*/ +if (write (STDOUT_FILENO , ok,strlen (ok)) <0 ) +return error (_ ("could not send '%s'" ), ok); + +strvec_pushf (&cp.env,"REQUEST_METHOD=%s" , req->method); +strvec_pushf (&cp.env,"PATH_TRANSLATED=%s" , req->uri_path.buf); +strvec_push (&cp.env,"SERVER_PROTOCOL=HTTP/1.1" ); +if (req->query_args.len) +strvec_pushf (&cp.env,"QUERY_STRING=%s" , req->query_args.buf); +if (req->content_type) +strvec_pushf (&cp.env,"CONTENT_TYPE=%s" , req->content_type); +if (req->has_content_length) +strvec_pushf (&cp.env,"CONTENT_LENGTH=%" PRIuMAX, + (uintmax_t )req->content_length); + cp.git_cmd =1 ; +strvec_push (&cp.args,"http-backend" ); + res =run_command (&cp); +close (STDOUT_FILENO ); +close (STDIN_FILENO ); +return !!res; +} +static enum worker_result dispatch (struct req *req) { +if (is_git_request (req)) +return do__git (req); +return send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 ,NULL ,WR_HANGUP ); }
t/t5556-http-auth.sh +83 -0
104 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v7 07/12] test-http-server: pass Git requests to http-backend
Quoted 15 lines · Matthew John Cheetham via GitGitGadget
(quoted diff) +15
-0+static int is_git_request(struct req *req) +{ + static regex_t *smart_http_regex; + static int initialized; + + if (!initialized) { + smart_http_regex = xmalloc(sizeof(*smart_http_regex)); + /* + * This regular expression matches all dumb and smart HTTP + * requests that are currently in use, and defined in + * Documentation/gitprotocol-http.txt. + * + */ + if (regcomp(smart_http_regex, "^/(HEAD|info/refs|" + "objects/info/[^/]+|git-(upload|receive)-pack)$",
Review
Quoted 10 lines
(quoted diff) +10
-0+ REG_EXTENDED)) { + warning("could not compile smart HTTP regex"); + smart_http_regex = NULL; + } + initialized = 1; + } + + return smart_http_regex && + !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0); +}
Assigning NULL to smart_http_regex leaks the earlier allocation. You could free it, but I have to wonder why it is on the heap in the first place. Yes, you check for NULL and return 0 if it failed to compile, but...why would it? It's hard-coded. And if it does fail, wouldn't you want to fail immediately and loudly, because it means all of the tests are broken?
I.e., something like this is a bit simpler:
t/helper/test-http-server.c +4 -7
@@ -327,28 +327,25 @@ static enum worker_result req__read(struct req *req, int fd)static int is_git_request (struct req *req) { -static regex_t *smart_http_regex; +static regex_t smart_http_regex;static int initialized;if (!initialized) { - smart_http_regex =xmalloc (sizeof (*smart_http_regex));/* * This regular expression matches all dumb and smart HTTP * requests that are currently in use, and defined in * Documentation/gitprotocol-http.txt. * */ -if (regcomp (smart_http_regex,"^/(HEAD|info/refs|" +if (regcomp (&smart_http_regex,"^/(HEAD|info/refs|" "objects/info/[^/]+|git-(upload|receive)-pack)$" ,REG_EXTENDED )) { -warning ("could not compile smart HTTP regex" ); - smart_http_regex =NULL ; +die ("could not compile smart HTTP regex" ); } initialized =1 ; } -return smart_http_regex && - !regexec (smart_http_regex, req->uri_path.buf,0 ,NULL ,0 ); +return !regexec (&smart_http_regex, req->uri_path.buf,0 ,NULL ,0 ); }static enum worker_result do__git (struct req *req,const char *user)
Quoted 15 lines
(quoted diff) +15
-0+start_http_server () { + # + # Launch our server into the background in repo_dir. + # + ( + cd "$REPO_DIR" + test-http-server --verbose \ + --listen=127.0.0.1 \ + --port=$GIT_TEST_HTTP_PROTOCOL_PORT \ + --reuseaddr \ + --pid-file="$PID_FILE" \ + "$@" \ + 2>"$SERVER_LOG" & + ) + #
Review
Quoted 10 lines
(quoted diff) +10
-0+ # Give it a few seconds to get started. + # + for k in 0 1 2 3 4 + do + if test -f "$PID_FILE" + then + return 0 + fi + sleep 1 + done
Yuck. This makes the test take a long time to run, since it will almost always "sleep 1" each time (and it looks like you bring the server up and down in several tests). Worse, it's at risk of failing racily if it ever takes more than 5 seconds to start up. That should be uncommon, I'd think, but could happen on a heavily loaded system.
There's a race-less solution using fifos in lib-git-daemon.sh, where we wait for the "ready to rumble" line. It's kind of horrific, but it does work and is battle-tested.
-Peff
PATCH v7 8/12 test-http-server: add simple authentication
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add simple authentication to the test-http-server test helper. Authentication schemes and sets of valid tokens can be specified via a configuration file (in the normal gitconfig file format). Incoming requests are compared against the set of valid schemes and tokens and only approved if a matching token is found, or if no auth was provided and anonymous auth is enabled.
Configuration for auth includes a simple set of three options:
[auth]
challenge = <scheme>[:<challenge_params>]
token = <scheme>:[<token>]*
allowAnonymous = <bool>
auth.challenge allows you define what authentication schemes, and optional challenge parameters the server should use. Scheme names are unique and subsequently specified challenge parameters in the config file will replace previously specified ones.
auth.token allows you to define the set of value token values for an authentication scheme. This is a multi-var and each entry in the config file will append to the set of valid tokens for that scheme. Specifying an empty token value will clear the list of tokens so far for that scheme, i.e. token = <scheme>:.
auth.allowAnonymous controls whether or not unauthenticated requests (those without any Authorization headers) should succeed or not, and trigger a 401 Unauthorized response.
--- t/helper/test-http-server.c | 232 +++++++++++++++++++++++++++++++++++- t/t5556-http-auth.sh | 43 ++++++- 2 files changed, 272 insertions(+), 3 deletions(-)
t/helper/test-http-server.c +230 -2
@@ -7,6 +7,7 @@#include "version.h" #include "dir.h" #include "date.h" +#include "config.h" #define TR2_CAT "test-http-server"
@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =" [--timeout=<n>] [--max-connections=<n>]\n" " [--reuseaddr] [--pid-file=<file>]\n" " [--listen=<host_or_ipaddr>]* [--port=<n>]\n" +" [--auth-config=<file>]\n" ;static unsigned int timeout;
@@ -349,7 +351,7 @@ static int is_git_request(struct req *req) !regexec (smart_http_regex, req->uri_path.buf,0 ,NULL ,0 ); } -static enum worker_result do__git (struct req *req) +static enum worker_result do__git (struct req *req,const char *user) {const char *ok ="HTTP/1.1 200 OK\r\n" ;struct child_process cp =CHILD_PROCESS_INIT ;
@@ -366,10 +368,16 @@ static enum worker_result do__git(struct req *req)* exit status of the process, then write the HTTP status line followed * by the http-backend output. This is outside of the scope of this test * helper's use at time of writing. +* +* The important auth responses (401) we are handling prior to getting +* to this point. */ if (write (STDOUT_FILENO , ok,strlen (ok)) <0 )return error (_ ("could not send '%s'" ), ok); +if (user) +strvec_pushf (&cp.env,"REMOTE_USER=%s" , user); +strvec_pushf (&cp.env,"REQUEST_METHOD=%s" , req->method);strvec_pushf (&cp.env,"PATH_TRANSLATED=%s" , req->uri_path.buf);strvec_push (&cp.env,"SERVER_PROTOCOL=HTTP/1.1" );
@@ -388,10 +396,217 @@ static enum worker_result do__git(struct req *req)return !!res; } +enum auth_result { +/* No auth module matches the request. */ +AUTH_UNKNOWN =0 , + +/* Auth module denied the request. */ +AUTH_DENY =1 , + +/* Auth module successfully validated the request. */ +AUTH_ALLOW =2 , +}; + +struct auth_module { +char *scheme; +char *challenge_params; +struct string_list *tokens; +}; + +static int allow_anonymous; +static struct auth_module **auth_modules =NULL ; +static size_t auth_modules_nr =0 ; +static size_t auth_modules_alloc =0 ; + +static struct auth_module *get_auth_module (const char *scheme,int create) +{ +struct auth_module *mod; +for (size_t i =0 ; i < auth_modules_nr; i++) { + mod = auth_modules[i]; +if (!strcasecmp (mod->scheme, scheme)) +return mod; + } + +if (create) { +struct auth_module *mod =xmalloc (sizeof (struct auth_module )); + mod->scheme =xstrdup (scheme); + mod->challenge_params =NULL ; +ALLOC_ARRAY (mod->tokens,1 ); +string_list_init_dup (mod->tokens); + +ALLOC_GROW (auth_modules, auth_modules_nr +1 , auth_modules_alloc); + auth_modules[auth_modules_nr++] = mod; + +return mod; + } + +return NULL ; +} + +static int is_authed (struct req *req,const char **user,enum worker_result *wr) +{ +enum auth_result result =AUTH_UNKNOWN ; +struct string_list hdrs =STRING_LIST_INIT_NODUP ; +struct auth_module *mod; + +struct string_list_item *hdr; +struct string_list_item *token; +const char *v; +struct strbuf **split =NULL ; +int i; +char *challenge; + +/* +* Check all auth modules and try to validate the request. +* The first Authorization header that matches a known auth module +* scheme will be consulted to either approve or deny the request. +* If no module is found, or if there is no valid token, then 401 error. +* Otherwise, only permit the request if anonymous auth is enabled. +* It's atypical for user agents/clients to send multiple Authorization +* headers, but not explicitly forbidden or defined. +*/ +for_each_string_list_item (hdr, &req->header_list) { +if (skip_iprefix (hdr->string,"Authorization: " , &v)) { + split =strbuf_split_str (v,' ' ,2 ); +if (split[0 ] && split[1 ]) { +/* trim trailing space ' ' */ +strbuf_rtrim (split[0 ]); + + mod =get_auth_module (split[0 ]->buf,0 ); +if (mod) { + result =AUTH_DENY ; + +for_each_string_list_item (token, mod->tokens) { +if (!strcmp (split[1 ]->buf, token->string)) { + result =AUTH_ALLOW ; +break ; + } + } + +strbuf_list_free (split); + goto done; + } + } + +strbuf_list_free (split); + } + } + +done: +switch (result) { +case AUTH_ALLOW : +trace2_printf ("%s: auth '%s' ALLOW" ,TR2_CAT , mod->scheme); + *user ="VALID_TEST_USER" ; + *wr =WR_OK ; +break ; + +case AUTH_DENY : +trace2_printf ("%s: auth '%s' DENY" ,TR2_CAT , mod->scheme); +/* fall-through */ + +case AUTH_UNKNOWN : +if (result !=AUTH_DENY && allow_anonymous) +break ; + +for (i =0 ; i < auth_modules_nr; i++) { + mod = auth_modules[i]; +if (mod->challenge_params) + challenge =xstrfmt ("WWW-Authenticate: %s %s" , + mod->scheme, + mod->challenge_params); +else + challenge =xstrfmt ("WWW-Authenticate: %s" , + mod->scheme); +string_list_append (&hdrs, challenge); + } + + *wr =send_http_error (STDOUT_FILENO ,401 ,"Unauthorized" ,-1 , + &hdrs, *wr); + } + +string_list_clear (&hdrs,0 ); + +return result ==AUTH_ALLOW || + (result ==AUTH_UNKNOWN && allow_anonymous); +} + +static int split_auth_param (const char *str,char **scheme,char **val) +{ +struct strbuf **p =strbuf_split_str (str,':' ,2 ); + +if (!p[0 ]) +return -1 ; + +/* trim trailing ':' */ +if (p[0 ]->len && p[0 ]->buf[p[0 ]->len -1 ] ==':' ) +strbuf_setlen (p[0 ], p[0 ]->len -1 ); + + *scheme =strbuf_detach (p[0 ],NULL ); + *val = p[1 ] ?strbuf_detach (p[1 ],NULL ) :NULL ; + +strbuf_list_free (p); +return 0 ; +} + +static int read_auth_config (const char *name,const char *val,void *data) +{ +int ret =0 ; +char *scheme =NULL ; +char *token =NULL ; +char *challenge =NULL ; +struct auth_module *mod; + +if (!strcmp (name,"auth.challenge" )) { +if (split_auth_param (val, &scheme, &challenge)) { + ret =error ("invalid auth challenge '%s'" , val); + goto cleanup; + } + + mod =get_auth_module (scheme,1 ); + +/* Replace any existing challenge parameters */ +free (mod->challenge_params); + mod->challenge_params = challenge ?xstrdup (challenge) :NULL ; + }else if (!strcmp (name,"auth.token" )) { +if (split_auth_param (val, &scheme, &token)) { + ret =error ("invalid auth token '%s'" , val); + goto cleanup; + } + + mod =get_auth_module (scheme,1 ); + +/* +* Append to set of valid tokens unless an empty token value +* is provided, then clear the existing list. +*/ +if (token) +string_list_append (mod->tokens, token); +else +string_list_clear (mod->tokens,1 ); + }else if (!strcmp (name,"auth.allowanonymous" )) { + allow_anonymous =git_config_bool (name, val); + }else { +warning ("unknown auth config '%s'" , name); + } + +cleanup: +free (scheme); +free (token); +free (challenge); + +return ret; +} +static enum worker_result dispatch (struct req *req) { +enum worker_result wr =WR_OK ; +const char *user =NULL ; + +if (!is_authed (req, &user, &wr)) +return wr; +if (is_git_request (req)) -return do__git (req); +return do__git (req, user);return send_http_error (STDOUT_FILENO ,501 ,"Not Implemented" ,-1 ,NULL ,WR_HANGUP );
@@ -655,6 +870,19 @@ int cmd_main(int argc, const char **argv) pid_file = v;continue ; } +if (skip_prefix (arg,"--auth-config=" , &v)) { +if (!strlen (v)) { +error ("invalid argument - missing file path" ); +usage (test_http_auth_usage); + } + +if (git_config_from_file (read_auth_config, v,NULL )) { +error ("failed to read auth config file '%s'" , v); +usage (test_http_auth_usage); + } + +continue ; + }fprintf (stderr,"error: unknown argument '%s'\n" , arg);usage (test_http_auth_usage);
t/t5556-http-auth.sh +42 -1
59 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v7 08/12] test-http-server: add simple authentication
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 414
Quoted 5 lines · Matthew John Cheetham via GitGitGadget
t/helper/test-http-server.c +5
-0+struct auth_module { +char *scheme; +char *challenge_params; +struct string_list *tokens; +};
This is a really minor nit, but: why is "tokens" a pointer? It's always initialized, so you never need or want to test it for NULL.
That would make this:
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 435
Quoted 6 lines
t/helper/test-http-server.c +6
-0+if (create) { +struct auth_module *mod =xmalloc (sizeof (struct auth_module )); + mod->scheme =xstrdup (scheme); + mod->challenge_params =NULL ; +ALLOC_ARRAY (mod->tokens,1 ); +string_list_init_dup (mod->tokens);
simplify to:
string_list_init_dup(&mod->tokens);
and one does not have to wonder why we use ALLOC_ARRAY() there, but not when allocating the module itself. :)
Likewise you could skip freeing it, but since the memory is held until program end anyway, that doesn't happen either way.
Certainly what you have won't behave wrong; I'd consider this more like a coding style thing.
Quoted 14 lines
t/t5556-http-auth.sh +14
-0+ cat >auth.config <<-EOF && + [auth] + challenge = no-params + challenge = with-params:foo=\"bar\" p=1 + challenge = with-params:foo=\"replaced\" q=1 + + token = no-explicit-challenge:valid-token + token = no-explicit-challenge:also-valid + token = reset-tokens:these-tokens + token = reset-tokens:will-be-reset + token = reset-tokens: + token = reset-tokens:the-only-valid-one + + allowAnonymous = false
t/t5556-http-auth.sh test_expect_success 'http auth server request parsing' ' · line 190
Quoted 10 lines
t/t5556-http-auth.sh +10
-0+ EOF + + cat >OUT.expected <<-EOF && + WWW-Authenticate: no-params + WWW-Authenticate: with-params foo="replaced" q=1 + WWW-Authenticate: no-explicit-challenge + WWW-Authenticate: reset-tokens + + Error: 401 Unauthorized + EOF
OK, so I think now we are getting to the interesting part of what your custom http-server does compared to something like apache. And the answer so far is: custom WWW-Authenticate lines.
I think we could do that with mod_headers pretty easily. But presumably we also want to check that we are getting the correct tokens, generate a 401, etc.
I suspect this could all be done as a CGI wrapping git-http-backend. You can influence the HTTP response code by sending:
Status: 401 Authorization Required
WWW-Authenticate: whatever you want
And likewise you can see what the client sends by putting something like this in apache.conf:
SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
and then reading $HTTP_AUTHORIZATION as you like. At that point, it feels like a simple shell or perl script could then decide whether to return a 401 or not (and if not, then just exec git-http-backend to do the rest). And the scripts would be simple enough that you could have individual scripts to implement various schemes, rather than implementing this configuration scheme. You can control which script is run based on the URL; see the way we match /broken_smart/, etc, in t/lib-httpd/apache.conf.
-Peff
Re: [PATCH v7 08/12] test-http-server: add simple authentication
Quoted 11 lines · Jeff King
I suspect this could all be done as a CGI wrapping git-http-backend. You can influence the HTTP response code by sending:
Status: 401 Authorization Required
WWW-Authenticate: whatever you wantAnd likewise you can see what the client sends by putting something like this in apache.conf:
SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
Quoted 8 lines
and then reading $HTTP_AUTHORIZATION as you like. At that point, it feels like a simple shell or perl script could then decide whether to return a 401 or not (and if not, then just exec git-http-backend to do the rest). And the scripts would be simple enough that you could have individual scripts to implement various schemes, rather than implementing this configuration scheme. You can control which script is run based on the URL; see the way we match /broken_smart/, etc, in
t/lib-httpd/apache.conf.
And here's a minimally worked-out example of that approach. It's on top of your patches so I could use your credential-helper infrastructure in the test, but the intent is that it would replace all of the test-tool server patches and be rolled into t5556 as appropriate.
--- t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 6 ++++++ t/lib-httpd/custom-auth.sh | 18 ++++++++++++++++ t/t5563-simple-http-auth.sh | 42 +++++++++++++++++++++++++++++++++++++ 4 files changed, 67 insertions(+) create mode 100644 t/lib-httpd/custom-auth.sh create mode 100755 t/t5563-simple-http-auth.sh
t/lib-httpd.sh +1 -0
@@ -137,6 +137,7 @@ prepare_httpd() { install_script error-smart-http.sh install_script error.sh install_script apply-one-time-perl.sh + install_script custom-auth.sh ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
t/lib-httpd/apache.conf +6 -0
@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/ SetEnv GIT_HTTP_EXPORT_ALL SetEnv GIT_PROTOCOL </LocationMatch> +<LocationMatch /custom_auth/> + SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH} + SetEnv GIT_HTTP_EXPORT_ALL + CGIPassAuth on +</LocationMatch> ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/ ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/ ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/ ScriptAlias /error_smart/ error-smart-http.sh/ ScriptAlias /error/ error.sh/ ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1 +ScriptAliasMatch /custom_auth/(.*) custom-auth.sh/$1 <Directory ${GIT_EXEC_PATH}> Options FollowSymlinks </Directory>
t/lib-httpd/custom-auth.sh new file +18 -0
@@ -0,0 +1,18 @@ +#!/bin/sh + +# Our acceptable auth here is hard-coded, but we could +# read it from a file provided by individual tests, etc. +# +# base64("alice:secret-passwd") +USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== + +case "$HTTP_AUTHORIZATION" in +"Basic $USERPASS64") + exec "$GIT_EXEC_PATH"/git-http-backend + ;; +*) + echo 'Status: 401 Auth Required' + echo 'WWW-Authenticate: Basic realm="whatever"' + echo + ;; +esac
t/t5563-simple-http-auth.sh new file +42 -0
42 lines not on this page · raw message
Signature · 1 line
2.39.1.738.g5e5f8a2714
Re: [PATCH v7 08/12] test-http-server: add simple authentication
Quoted 7 lines · Jeff King
Quoted 2 lines
I suspect this could all be done as a CGI wrapping git-http-backend. You can influence the HTTP response code by sending:
...
And here's a minimally worked-out example of that approach. It's on top of your patches so I could use your credential-helper infrastructure in the test, but the intent is that it would replace all of the test-tool server patches and be rolled into t5556 as appropriate.
Thanks for helping Matthew's topic move forward. I very much like seeing apache used for tests in this sample approach, like all the other http tests we do with apache, instead of a custom server that we need to ensure that it mimics the real-world use cases and we have to maintain.
Re: [PATCH v7 08/12] test-http-server: add simple authentication
In reply to Matthew John Cheetham via GitGitGadgetQuoted 7 lines · Matthew John Cheetham via GitGitGadget
t/helper/test-http-server.c +7
-0+static int split_auth_param (const char *str,char **scheme,char **val) +{ +struct strbuf **p =strbuf_split_str (str,':' ,2 ); + +if (!p[0 ]) +return -1 ; +
t/helper/test-http-server.c static enum worker_result do__git(struct req *req) · line 549
Quoted 10 lines
t/helper/test-http-server.c +10
-0+/* trim trailing ':' */ +if (p[0 ]->len && p[0 ]->buf[p[0 ]->len -1 ] ==':' ) +strbuf_setlen (p[0 ], p[0 ]->len -1 ); + + *scheme =strbuf_detach (p[0 ],NULL ); + *val = p[1 ] ?strbuf_detach (p[1 ],NULL ) :NULL ; + +strbuf_list_free (p); +return 0 ; +}
Oh, I forgot one more Coverity-detected problem here when reviewing last night. The early "return -1" here leaks "p" (there are no strbufs in the resulting array, but strbuf_split_str() will still have allocated the array). It needs a call to strbuf_list_free(p) there.
-Peff
PATCH v7 9/12 test-http-server: add sending of arbitrary headers
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add the ability to send arbitrary headers in HTTP responses from the test-http-server. This is useful when we want to test 'malformed' response message handling.
Add the following option to the server auth config file:
[auth]
extraHeader = [<value>]*
Each auth.extraHeader value will be appended to the response headers verbatim.
--- t/helper/test-http-server.c | 6 ++++++ t/t5556-http-auth.sh | 7 +++++++ 2 files changed, 13 insertions(+)
t/helper/test-http-server.c +6 -0
@@ -417,6 +417,7 @@ static int allow_anonymous;static struct auth_module **auth_modules =NULL ;static size_t auth_modules_nr =0 ;static size_t auth_modules_alloc =0 ; +static struct strvec extra_headers =STRVEC_INIT ;static struct auth_module *get_auth_module (const char *scheme,int create) {
@@ -520,6 +521,9 @@ done:string_list_append (&hdrs, challenge); } +for (i =0 ; i < extra_headers.nr; i++) +string_list_append (&hdrs, extra_headers.v[i]); + *wr =send_http_error (STDOUT_FILENO ,401 ,"Unauthorized" ,-1 , &hdrs, *wr); }
@@ -585,6 +589,8 @@ static int read_auth_config(const char *name, const char *val, void *data)string_list_clear (mod->tokens,1 ); }else if (!strcmp (name,"auth.allowanonymous" )) { allow_anonymous =git_config_bool (name, val); + }else if (!strcmp (name,"auth.extraheader" )) { +strvec_push (&extra_headers, val); }else {warning ("unknown auth config '%s'" , name); }
t/t5556-http-auth.sh +7 -0
@@ -178,6 +178,10 @@ test_expect_success CURL 'http auth server auth config' ' token = reset-tokens:the-only-valid-one allowAnonymous = false + + extraHeader = X-Extra-Header: abc + extraHeader = X-Extra-Header: 123 + extraHeader = X-Another: header\twith\twhitespace! EOF cat >OUT.expected <<-EOF &&
@@ -185,6 +189,9 @@ test_expect_success CURL 'http auth server auth config' ' WWW-Authenticate: with-params foo="replaced" q=1 WWW-Authenticate: no-explicit-challenge WWW-Authenticate: reset-tokens + X-Extra-Header: abc + X-Extra-Header: 123 + X-Another: header with whitespace! Error: 401 Unauthorized EOF
Signature · 1 line
gitgitgadget
PATCH v7 10/12 http: replace unsafe size_t multiplication with st_mult
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Replace direct multiplication of two size_t parameters in curl response stream handling callback functions with st_mult to guard against overflows.
--- http.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
http.c +2 -2
@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;size_t fread_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct buffer *buffer = buffer_;if (size > buffer->buf.len - buffer->posn)
@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)size_t fwrite_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct strbuf *buffer = buffer_;strbuf_add (buffer, ptr, size);
Signature · 1 line
gitgitgadget
Re: [PATCH v7 10/12] http: replace unsafe size_t multiplication with st_mult
Quoted 3 lines · Matthew John Cheetham via GitGitGadget
Replace direct multiplication of two
size_tparameters in curl response stream handling callback functions withst_multto guard against overflows.
Hmm. So part of me says that more overflow detection is better than less, but...I really doubt this is doing anything, and it feels odd to me to do overflow checks when there is no allocation.
There are tons of integer multiplications in Git. Our usual strategy is to try to handle overflow like this when we're about to allocate a buffer, with the idea that we'll avoid a truncated size (that we may later fill with too many bytes).
In these cases, we could possibly avoid a weird or wrong result due to truncation, but I don't see how that is different than most of the rest of Git. What makes these worth touching?
Moreover...
http.c curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp) · line 182
Quoted 9 lines
http.c +1
-1@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)size_t fwrite_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct strbuf *buffer = buffer_;strbuf_add (buffer, ptr, size);
The caller is already claiming to have eltsize*nmemb bytes accessible via "ptr". How did it get such a buffer if that overflows size_t?
http.c static int http_schannel_use_ssl_cainfo; · line 152
Quoted 13 lines
http.c +1
-1@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;size_t fread_buffer (char *ptr,size_t eltsize,size_t nmemb,void *buffer_) { -size_t size = eltsize * nmemb; +size_t size =st_mult (eltsize, nmemb);struct buffer *buffer = buffer_;if (size > buffer->buf.len - buffer->posn)
Likewise the caller is asking us to fill a buffer that is eltsize*nmemb. So they must have allocated it already. How can it be bigger than a size_t?
In practice, of course, these are both coming from curl, and I strongly suspect that curl always sets "1" for eltsize anyway, since it's working with bytes. The two fields only exist to conform to the weird fread() interface for historical reasons.
So I don't think this patch is really hurting much. It just feels like a weird one-off that makes the code inconsistent. If somebody who was wanting to write similar code later asks "why is this one st_mult(), but not other multiplications", I wouldn't have an answer for them.
-Peff
PATCH v7 11/12 http: read HTTP WWW-Authenticate response headers
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Read and store the HTTP WWW-Authenticate response headers made for a particular request.
This will allow us to pass important authentication challenge information to credential helpers or others that would otherwise have been lost.
According to RFC2616 Section 4.2 [1], header field names are not case-sensitive meaning when collecting multiple values for the same field name, we can just use the case of the first observed instance of each field name and no normalisation is required.
libcurl only provides us with the ability to read all headers recieved for a particular request, including any intermediate redirect requests or proxies. The lines returned by libcurl include HTTP status lines delinating any intermediate requests such as "HTTP/1.1 200". We use these lines to reset the strvec of WWW-Authenticate header values as we encounter them in order to only capture the final response headers.
The collection of all header values matching the WWW-Authenticate header is complicated by the fact that it is legal for header fields to be continued over multiple lines, but libcurl only gives us one line at a time.
In the future [2] we may be able to leverage functions to read headers from libcurl itself, but as of today we must do this ourselves.
[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/
--- credential.c | 1 + credential.h | 15 +++++++++ http.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+)
credential.c +1 -0
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)free (c->username);free (c->password);string_list_clear (&c->helpers,0 ); +strvec_clear (&c->wwwauth_headers);credential_init (c); }
credential.h +15 -0
@@ -2,6 +2,7 @@#define CREDENTIAL_H #include "string-list.h" +#include "strvec.h" /** * The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {*/ struct string_list helpers; +/** +* A `strvec` of WWW-Authenticate header values. Each string +* is the value of a WWW-Authenticate header in an HTTP response, +* in the order they were received in the response. +*/ +struct strvec wwwauth_headers; + +/** +* Internal use only. Used to keep track of split header fields +* in order to fold multiple lines into one value. +*/ + unsigned header_is_last_match:1; + unsigned approved:1, configured:1, quit:1,
@@ -130,6 +144,7 @@ struct credential {#define CREDENTIAL_INIT { \ .helpers = STRING_LIST_INIT_DUP, \ + .wwwauth_headers = STRVEC_INIT, \ }/* Initialize a credential structure, setting all fields to empty. */
http.c +94 -0
106 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers
Quoted 8 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Read and store the HTTP WWW-Authenticate response headers made for a particular request.
This will allow us to pass important authentication challenge information to credential helpers or others that would otherwise have been lost.
Makes sense, and the code looks pretty reasonable overall.
A few observations:
Quoted 7 lines
credential.h +3
-0@@ -115,6 +116,19 @@ struct credential {*/ struct string_list helpers; +/** +* A `strvec` of WWW-Authenticate header values. Each string +* is the value of a WWW-Authenticate header in an HTTP response,
credential.h struct credential { · line 131
Quoted 10 lines
credential.h +10
-0+* in the order they were received in the response. +*/ +struct strvec wwwauth_headers; + +/** +* Internal use only. Used to keep track of split header fields +* in order to fold multiple lines into one value. +*/ + unsigned header_is_last_match:1; +
Stuffing this into a "struct credential" feels a little weird, just because it's specific to http parsing (especially this internal flag). And the credential code is seeing full header lines, not broken down at all.
I guess I would have expected some level of abstraction here between the credential subsystem and the http subsystem, where the latter is parsing and then sticking opaque data into the credential to ferry to the helpers.
But it probably isn't that big a deal either way. Even though there are non-http credentials, it's not too unreasonable for the credential system to be aware of http specifically.
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 188
Quoted 3 lines
http.c +3
-0+static size_t fwrite_wwwauth (char *ptr,size_t eltsize,size_t nmemb,void *p) +{ +size_t size =st_mult (eltsize, nmemb);
Here's that st_mult() again. Same comment as the previous patch. :)
Quoted 11 lines
http.c +11
-0+/* +* Header lines may not come NULL-terminated from libcurl so we must +* limit all scans to the maximum length of the header line, or leverage +* strbufs for all operations. +* +* In addition, it is possible that header values can be split over +* multiple lines as per RFC 2616 (even though this has since been +* deprecated in RFC 7230). A continuation header field value is +* identified as starting with a space or horizontal tab. +* +* The formal definition of a header field as given in RFC 2616 is:
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 213
Quoted 10 lines
http.c +10
-0+* +* message-header = field-name ":" [ field-value ] +* field-name = token +* field-value = *( field-content | LWS ) +* field-content = <the OCTETs making up the field-value +* and consisting of either *TEXT or combinations +* of token, separators, and quoted-string> +*/ + +strbuf_add (&buf, ptr, size);
OK, so we just copy the buffer. I don't think it would be too hard to handle the buffer as-is, but this does make things a bit easier. Given that we're going to immediately throw away the copy for anything except www-authenticate, we could perhaps wait until we've matched it. That does mean trimming the CRLF ourselves and using skip_prefix_mem() to match the start (you'd want skip_iprefix_mem(), of course, but it doesn't yet exist; I'll leave that as an exercise).
Maybe not worth it to save a few allocations, as an http request is already pretty heavyweight. Mostly I flagged it because this is going to run for every header of every request, even though most requests won't trigger it at all.
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 226
Quoted 12 lines
http.c +12
-0+/* Strip the CRLF that should be present at the end of each field */ +strbuf_trim_trailing_newline (&buf); + +/* Start of a new WWW-Authenticate header */ +if (skip_iprefix (buf.buf,"www-authenticate:" , &val)) { +while (isspace (*val)) + val++; + +strvec_push (values, val); + http_auth.header_is_last_match =1 ; + goto exit; + }
OK, this looks correct from my knowledge of the RFCs. I saw something about isspace() matching newlines, etc, in an earlier thread, but I think we'd never see a newline here, as we're expecting curl to be splitting on our behalf.
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 237
Quoted 10 lines
http.c +10
-0+/* +* This line could be a continuation of the previously matched header +* field. If this is the case then we should append this value to the +* end of the previously consumed value. +* Continuation lines start with at least one whitespace, maybe more, +* so we should collapse these down to a single SP (valid per the spec). +*/ +if (http_auth.header_is_last_match &&isspace (*buf.buf)) { +/* Trim leading whitespace from this continuation hdr line. */ +strbuf_ltrim (&buf);
OK, makes sense. This will memmove(), which is needlessly inefficient (we could just advance a pointer), but probably not a big deal in practice. Using the strbuf functions is a nice simplification.
Quoted 7 lines
http.c +7
-0+/* +* At this point we should always have at least one existing +* value, even if it is empty. Do not bother appending the new +* value if this continuation header is itself empty. +*/ +if (!values->nr) { +BUG ("should have at least one existing header value" );
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 255
Quoted 10 lines
http.c +10
-0+ }else if (buf.len) { +char *prev =xstrdup (values->v[values->nr -1 ]); + +/* Join two non-empty values with a single space. */ +const char *const sp = *prev ?" " :"" ; + +strvec_pop (values); +strvec_pushf (values,"%s%s%s" , prev, sp, buf.buf); +free (prev); + }
Likewise here we end up with an extra allocation of "prev", just because we can't pop/push in the right order. But that's probably OK in practice, as this is triggering only for the header we care about.
The concatenation itself makes the whole thing quadratic, but unless we are worried about a malicious server DoS-ing us with a billion www-authenticate continuations, I think we can disregard that.
-Peff
Re: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers
Quoted 38 lines · Jeff King
Quoted 8 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Read and store the HTTP WWW-Authenticate response headers made for a particular request.
This will allow us to pass important authentication challenge information to credential helpers or others that would otherwise have been lost.
Makes sense, and the code looks pretty reasonable overall.
A few observations:
Quoted 7 lines
credential.h +3
-0@@ -115,6 +116,19 @@ struct credential {*/ struct string_list helpers; +/** +* A `strvec` of WWW-Authenticate header values. Each string +* is the value of a WWW-Authenticate header in an HTTP response, credential.h struct credential { · line 131
Quoted 10 lines
credential.h +10
-0+* in the order they were received in the response. +*/ +struct strvec wwwauth_headers; + +/** +* Internal use only. Used to keep track of split header fields +* in order to fold multiple lines into one value. +*/ + unsigned header_is_last_match:1; +Stuffing this into a "struct credential" feels a little weird, just because it's specific to http parsing (especially this internal flag). And the credential code is seeing full header lines, not broken down at all.
Quoted 8 lines
I guess I would have expected some level of abstraction here between the credential subsystem and the http subsystem, where the latter is parsing and then sticking opaque data into the credential to ferry to the helpers.
But it probably isn't that big a deal either way. Even though there are non-http credentials, it's not too unreasonable for the credential system to be aware of http specifically.
I had considered possibly introducing an opaque property-bag style of 'protocol-specific properties' that, for example, http.c would add the WWW-Authenticate headers to as something like http.wwwauth[]. Other protocols (like smtp:// or cert://) could add their own properties if they needed or wanted to also.
Thoughts?
Quoted 5 lines
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 188
Quoted 3 lines
http.c +3
-0+static size_t fwrite_wwwauth (char *ptr,size_t eltsize,size_t nmemb,void *p) +{ +size_t size =st_mult (eltsize, nmemb);Here's that
st_mult()again. Same comment as the previous patch. :)
Yeah I'm gonna drop this. Your arguments make sense; it's not going to be a problem in reality :-)
Quoted 22 lines
Quoted 11 lines
http.c +11
-0+/* +* Header lines may not come NULL-terminated from libcurl so we must +* limit all scans to the maximum length of the header line, or leverage +* strbufs for all operations. +* +* In addition, it is possible that header values can be split over +* multiple lines as per RFC 2616 (even though this has since been +* deprecated in RFC 7230). A continuation header field value is +* identified as starting with a space or horizontal tab. +* +* The formal definition of a header field as given in RFC 2616 is: http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 213
Quoted 10 lines
http.c +10
-0+* +* message-header = field-name ":" [ field-value ] +* field-name = token +* field-value = *( field-content | LWS ) +* field-content = <the OCTETs making up the field-value +* and consisting of either *TEXT or combinations +* of token, separators, and quoted-string> +*/ + +strbuf_add (&buf, ptr, size);
Quoted 7 lines
OK, so we just copy the buffer. I don't think it would be too hard to handle the buffer as-is, but this does make things a bit easier. Given that we're going to immediately throw away the copy for anything except www-authenticate, we could perhaps wait until we've matched it. That does mean trimming the CRLF ourselves and using
skip_prefix_mem()to match the start (you'd wantskip_iprefix_mem(), of course, but it doesn't yet exist; I'll leave that as an exercise).
Fair point! I can replace most of these with operations over the curl ptr.
Quoted 65 lines
Maybe not worth it to save a few allocations, as an http request is already pretty heavyweight. Mostly I flagged it because this is going to run for every header of every request, even though most requests won't trigger it at all.
http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 226
Quoted 12 lines
http.c +12
-0+/* Strip the CRLF that should be present at the end of each field */ +strbuf_trim_trailing_newline (&buf); + +/* Start of a new WWW-Authenticate header */ +if (skip_iprefix (buf.buf,"www-authenticate:" , &val)) { +while (isspace (*val)) + val++; + +strvec_push (values, val); + http_auth.header_is_last_match =1 ; + goto exit; + }OK, this looks correct from my knowledge of the RFCs. I saw something about
isspace()matching newlines, etc, in an earlier thread, but I think we'd never see a newline here, as we're expecting curl to be splitting on our behalf.http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 237
Quoted 10 lines
http.c +10
-0+/* +* This line could be a continuation of the previously matched header +* field. If this is the case then we should append this value to the +* end of the previously consumed value. +* Continuation lines start with at least one whitespace, maybe more, +* so we should collapse these down to a single SP (valid per the spec). +*/ +if (http_auth.header_is_last_match &&isspace (*buf.buf)) { +/* Trim leading whitespace from this continuation hdr line. */ +strbuf_ltrim (&buf);OK, makes sense. This will
memmove(), which is needlessly inefficient (we could just advance a pointer), but probably not a big deal in practice. Using the strbuf functions is a nice simplification.Quoted 7 lines
http.c +7
-0+/* +* At this point we should always have at least one existing +* value, even if it is empty. Do not bother appending the new +* value if this continuation header is itself empty. +*/ +if (!values->nr) { +BUG ("should have at least one existing header value" );http.c size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_) · line 255
Quoted 10 lines
http.c +10
-0+ }else if (buf.len) { +char *prev =xstrdup (values->v[values->nr -1 ]); + +/* Join two non-empty values with a single space. */ +const char *const sp = *prev ?" " :"" ; + +strvec_pop (values); +strvec_pushf (values,"%s%s%s" , prev, sp, buf.buf); +free (prev); + }Likewise here we end up with an extra allocation of "prev", just because we can't pop/push in the right order. But that's probably OK in practice, as this is triggering only for the header we care about.
The concatenation itself makes the whole thing quadratic, but unless we are worried about a malicious server DoS-ing us with a billion www-authenticate continuations, I think we can disregard that.
-Peff
Re: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers
Quoted 9 lines · Matthew John Cheetham
Quoted 8 lines
I guess I would have expected some level of abstraction here between the credential subsystem and the http subsystem, where the latter is parsing and then sticking opaque data into the credential to ferry to the helpers.
But it probably isn't that big a deal either way. Even though there are non-http credentials, it's not too unreasonable for the credential system to be aware of http specifically.
Quoted 7 lines
I had considered possibly introducing an opaque property-bag style of 'protocol-specific properties' that, for example,
http.cwould add the WWW-Authenticate headers to as something likehttp.wwwauth[]. Other protocols (like smtp:// or cert://) could add their own properties if they needed or wanted to also.Thoughts?
At the protocol level, I don't see much point. wwwauth sufficiently implies "http", and any helper is free to ignore or respect keys as appropriate to what it can handle. A flat namespace is fine.
Here I was more talking about the internal implementation. Mostly it was just funky that this internal http state flag was stuck into the credential struct. I think it could be removed with some minor pain, but it's probably not too big a deal (the pain at all is only because we are having to bring this state across multiple curl callbacks).
So let's go with it for now.
-Peff
PATCH v7 12/12 credential: add WWW-Authenticate header to cred requests
In reply to Matthew John Cheetham via GitGitGadgetFrom: Matthew John Cheetham <[email protected]>
Add the value of the WWW-Authenticate response header to credential requests. Credential helpers that understand and support HTTP authentication and authorization can use this standard header (RFC 2616 Section 14.47 [1]) to generate valid credentials.
WWW-Authenticate headers can contain information pertaining to the authority, authentication mechanism, or extra parameters/scopes that are required.
The current I/O format for credential helpers only allows for unique names for properties/attributes, so in order to transmit multiple header values (with a specific order) we introduce a new convention whereby a C-style array syntax is used in the property name to denote multiple ordered values for the same property.
In this case we send multiple wwwauth[] properties where the order that the repeated attributes appear in the conversation reflects the order that the WWW-Authenticate headers appeared in the HTTP response.
Add a set of tests to exercise the HTTP authentication header parsing and the interop with credential helpers. Credential helpers will receive WWW-Authenticate information in credential requests.
[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47
--- Documentation/git-credential.txt | 19 ++- credential.c | 11 ++ t/lib-credential-helper.sh | 27 ++++ t/t5556-http-auth.sh | 242 +++++++++++++++++++++++++++++++ 4 files changed, 298 insertions(+), 1 deletion(-) create mode 100644 t/lib-credential-helper.sh
Documentation/git-credential.txt +18 -1
@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline. The key may contain any bytes except `=`, newline, or NUL. The value may contain any bytes except newline or NUL. -In both cases, all bytes are treated as-is (i.e., there is no quoting, +Attributes with keys that end with C-style array brackets `[]` can have +multiple values. Each instance of a multi-valued attribute forms an +ordered list of values - the order of the repeated attributes defines +the order of the values. An empty multi-valued attribute (`key[]=\n`) +acts to clear any previous entries and reset the list. + +In all cases, all bytes are treated as-is (i.e., there is no quoting, and one cannot transmit a value with newline or NUL in it). The list of attributes is terminated by a blank line or end-of-file.
@@ -160,6 +166,17 @@ empty string. Components which are missing from the URL (e.g., there is no username in the example above) will be left unset. +`wwwauth[]`:: + + When an HTTP response is received by Git that includes one or more + 'WWW-Authenticate' authentication headers, these will be passed by Git + to credential helpers. ++ +Each 'WWW-Authenticate' header value is passed as a multi-valued +attribute 'wwwauth[]', where the order of the attributes is the same as +they appear in the HTTP response. This attribute is 'one-way' from Git +to pass additional information to credential helpers. + Unrecognised attributes are silently discarded. GIT
credential.c +11 -0
@@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,fprintf (fp,"%s=%s\n" , key, value); } +static void credential_write_strvec (FILE *fp,const char *key, +const struct strvec *vec) +{ +char *full_key =xstrfmt ("%s[]" , key); +for (size_t i =0 ; i < vec->nr; i++) { +credential_write_item (fp, full_key, vec->v[i],0 ); + } +free (full_key); +} +void credential_write (const struct credential *c,FILE *fp) {credential_write_item (fp,"protocol" , c->protocol,1 );
@@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp)credential_write_item (fp,"path" , c->path,0 );credential_write_item (fp,"username" , c->username,0 );credential_write_item (fp,"password" , c->password,0 ); +credential_write_strvec (fp,"wwwauth" , &c->wwwauth_headers); } static int run_credential_helper(struct credential *c,
t/lib-credential-helper.sh new file +27 -0
@@ -0,0 +1,27 @@ +setup_credential_helper() { + test_expect_success 'setup credential helper' ' + CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" && + export CREDENTIAL_HELPER && + echo $CREDENTIAL_HELPER && + + write_script "$CREDENTIAL_HELPER" <<-\EOF + cmd=$1 + teefile=$cmd-query.cred + catfile=$cmd-reply.cred + sed -n -e "/^$/q" -e "p" >> $teefile + if test "$cmd" = "get"; then + cat $catfile + fi + EOF + ' +} + +set_credential_reply() { + cat >"$TRASH_DIRECTORY/$1-reply.cred" +} + +expect_credential_query() { + cat >"$TRASH_DIRECTORY/$1-expect.cred" && + test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \ + "$TRASH_DIRECTORY/$1-query.cred" +}
t/t5556-http-auth.sh +242 -0
264 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests
Quoted 10 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Add the value of the WWW-Authenticate response header to credential requests. Credential helpers that understand and support HTTP authentication and authorization can use this standard header (RFC 2616 Section 14.47 [1]) to generate valid credentials.
WWW-Authenticate headers can contain information pertaining to the authority, authentication mechanism, or extra parameters/scopes that are required.
I'm definitely on board with sending these to the helpers. It does feel a bit weird that we don't parse them at all, and just foist that on the helpers.
If I understand the RFC correctly, you can have multiple challenges per header, but also multiple headers. So:
WWW-Authenticate: Basic realm="foo", OtherAuth realm="bar"
WWW-Authenticate: YetAnotherScheme some-token
could be normalized as:
www-auth-challenge=Basic realm="foo"
www-auth-challenge=OtherAuth realm="bar"
www-auth-challenge=YetAnotherScheme some-token
which saves each helper from having to do the same work. Likewise, we can do a _little_ more parsing to get:
www-auth-basic=realm="foo"
www-auth-otherauth=realm="bar"
www-auth-yetanotherscheme=some-token
I don't think we can go beyond there, though, without understanding the syntax of individual schemes. Which is a shame, as one of the goals of the credential format was to let the helpers do as little as possible (so they can't get it wrong!). But helpers are stuck doing things like handling backslashed double-quotes, soaking up extra whitespace, etc.
I'm not really sure what we expect to see in the real world. I guess for your purposes, you are working on an already-big helper that is happy to just get the raw values and process them according to the rfc. I'm just wondering if there are use cases where somebody might want to do something with this header, but in a quick shell script kind of way. For example, my credential config is still:
[credential "https://github.com"] username = peff helper = "!f() { test $1 = get && echo password=$(pass ...); }; f"
That's an extreme example, but I'm wondering if there's _anything_ useful somebody would want to do in a similar quick-and-dirty kind of way. For example, deciding which cred to use based on basic realm, like:
realm=foo
while read line; do
case "$line" in
www-auth-basic=)
value=${line#*=}
# oops, we're just assuming it's realm= here, and we're # not handling quotes at all. I think it could technically be # realm=foo or realm="foo"
realm=${value#realm=}
;;
esac
done
echo password=$(pass "pats-by-realm/$realm")
which could be made a lot easier if we did more parsing (e.g., www-auth-basic-realm or something). I dunno. Maybe that is just opening up a can of worms, as we're stuffing structured data into a linearized key-value list. The nice thing about your proposal is that Git does not even have to know anything about these schemes; it's all the problem of the helper. My biggest fear is just that we'll want to shift that later, and we'll be stuck with this microformat forever.
Quoted 5 lines
The current I/O format for credential helpers only allows for unique names for properties/attributes, so in order to transmit multiple header values (with a specific order) we introduce a new convention whereby a C-style array syntax is used in the property name to denote multiple ordered values for the same property.
I don't know if this is strictly necessary. The semantics of duplicate keys are not really defined anywhere, and just because the implementations of current readers happen to replace duplicates for the current set of keys doesn't mean everything has to. So you could just define "wwwauth" to behave differently. But I don't mind having a syntactic marker to indicate this new type.
If you're at all convinced by what I said above, then we also might be able to get away with having unique keys anyway.
Quoted 6 lines
Documentation/git-credential.txt | 19 ++- credential.c | 11 ++ t/lib-credential-helper.sh | 27 ++++ t/t5556-http-auth.sh | 242 +++++++++++++++++++++++++++++++ 4 files changed, 298 insertions(+), 1 deletion(-) create mode 100644 t/lib-credential-helper.sh
The patch itself looks pretty reasonable to me.
One small thing I noticed:
Review
Quoted 1 line
(quoted diff) +1
-0+ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
As you undoubtedly figured out, the helper path is fed to the shell, so spaces in the trash directory are a problem. You've solved it here by adding a layer of double quotes, which handles spaces. But you'd run into problems if the absolute path that somebody is using for the test suite has a backslash or a double quote in it.
I don't know how careful we want to be here (or how careful we already are[1]), but one simple-ish solution is:
export CREDENTIAL_HELPER
git -c "credential.helper=!\"\$CREDENTIAL_HELPER\"" ...
I.e., letting the inner shell expand the variable itself. Another option is to put the helper into $TRASH_DIRECTORY/bin and add that to the $PATH.
I also wondered if it was worth having setup_credential_helper() just stick it in $TRASH_DIRECTORY/.gitconfig so that individual tests don't have to keep doing that ugly "-c" invocation. Or if you really want to have each test enable it, perhaps have set_credential_reply() turn it on via test_config (which will auto-remove it at the end of the test).
-Peff
[1] Curious, I tried cloning git into this directory:
mkdir '/tmp/foo/"horrible \"path\"'
and we do indeed already fail. The first breakage I saw was recent, but going further back, it looks like bin-wrappers don't correctly handle this case anyway. So maybe that's evidence that nobody would do something so ridiculous in practice.
Re: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests
Quoted 17 lines · Jeff King
Quoted 10 lines · Matthew John Cheetham via GitGitGadget
From: Matthew John Cheetham <[email protected]>
Add the value of the WWW-Authenticate response header to credential requests. Credential helpers that understand and support HTTP authentication and authorization can use this standard header (RFC 2616 Section 14.47 [1]) to generate valid credentials.
WWW-Authenticate headers can contain information pertaining to the authority, authentication mechanism, or extra parameters/scopes that are required.
I'm definitely on board with sending these to the helpers. It does feel a bit weird that we don't parse them at all, and just foist that on the helpers.
Quoted 5 lines
If I understand the RFC correctly, you can have multiple challenges per header, but also multiple headers. So:
WWW-Authenticate: Basic realm="foo", OtherAuth realm="bar"
WWW-Authenticate: YetAnotherScheme some-token
That is correct. It would be strange that server would respond with a mix of styles, but I guess it's not forbidden.
Quoted 13 lines
could be normalized as:
www-auth-challenge=Basic realm="foo"
www-auth-challenge=OtherAuth realm="bar"
www-auth-challenge=YetAnotherScheme some-tokenwhich saves each helper from having to do the same work. Likewise, we can do a _little_ more parsing to get:
www-auth-basic=realm="foo"
www-auth-otherauth=realm="bar"
www-auth-yetanotherscheme=some-token
Quoted 5 lines
I don't think we can go beyond there, though, without understanding the syntax of individual schemes. Which is a shame, as one of the goals of the credential format was to let the helpers do as little as possible (so they can't get it wrong!). But helpers are stuck doing things like handling backslashed double-quotes, soaking up extra whitespace, etc.
This key format wouldn't make it obviously easier for simple helpers to understand. Now they no longer have well-known keys but a key prefix.
My overall goal here is to have Git know less about auth, so it treats all values as totally opaque. The only logic added is around reconstructing folded headers, which is just HTTP and not auth specific.
Quoted 29 lines
I'm not really sure what we expect to see in the real world. I guess for your purposes, you are working on an already-big helper that is happy to just get the raw values and process them according to the rfc. I'm just wondering if there are use cases where somebody might want to do something with this header, but in a quick shell script kind of way. For example, my credential config is still:
[credential "https://github.com"] username = peff helper = "!f() { test $1 = get && echo password=$(pass ...); }; f"That's an extreme example, but I'm wondering if there's _anything_ useful somebody would want to do in a similar quick-and-dirty kind of way. For example, deciding which cred to use based on basic realm, like:
realm=foo
while read line; do
case "$line" in
www-auth-basic=)
value=${line#*=}
# oops, we're just assuming it's realm= here, and we're # not handling quotes at all. I think it could technically be # realm=foo or realm="foo"
realm=${value#realm=}
;;
esac
done
echo password=$(pass "pats-by-realm/$realm")
Quoted 7 lines
which could be made a lot easier if we did more parsing (e.g., www-auth-basic-realm or something). I dunno. Maybe that is just opening up a can of worms, as we're stuffing structured data into a linearized key-value list. The nice thing about your proposal is that Git does not even have to know anything about these schemes; it's all the problem of the helper. My biggest fear is just that we'll want to shift that later, and we'll be stuck with this microformat forever.
I'm not sure there's such a continuous scale between simple and 'complex' helpers that would mean there'd be a simple shell script generating OAuth or DPoP credentials instead of a helper written in a higher-level language where parsing the headers is one of the simpler challenges faced.
Quoted 6 lines
Quoted 5 lines
The current I/O format for credential helpers only allows for unique names for properties/attributes, so in order to transmit multiple header values (with a specific order) we introduce a new convention whereby a C-style array syntax is used in the property name to denote multiple ordered values for the same property.
Quoted 6 lines
I don't know if this is strictly necessary. The semantics of duplicate keys are not really defined anywhere, and just because the implementations of current readers happen to replace duplicates for the current set of keys doesn't mean everything has to. So you could just define "wwwauth" to behave differently. But I don't mind having a syntactic marker to indicate this new type.
I had considered another model whereby we forgo the key=value line model, and hide another format behind the 'final' terminating new-line. However I thought this would be even more distuptive.
Quoted 32 lines
If you're at all convinced by what I said above, then we also might be able to get away with having unique keys anyway.
Quoted 6 lines
Documentation/git-credential.txt | 19 ++- credential.c | 11 ++ t/lib-credential-helper.sh | 27 ++++ t/t5556-http-auth.sh | 242 +++++++++++++++++++++++++++++++ 4 files changed, 298 insertions(+), 1 deletion(-) create mode 100644 t/lib-credential-helper.shThe patch itself looks pretty reasonable to me.
One small thing I noticed:
Review
Quoted 1 line
(quoted diff) +1
-0+ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&As you undoubtedly figured out, the helper path is fed to the shell, so spaces in the trash directory are a problem. You've solved it here by adding a layer of double quotes, which handles spaces. But you'd run into problems if the absolute path that somebody is using for the test suite has a backslash or a double quote in it.
I don't know how careful we want to be here (or how careful we already are[1]), but one simple-ish solution is:
export
CREDENTIAL_HELPER
git -c "credential.helper=!\"\$CREDENTIAL_HELPER\"" ...I.e., letting the inner shell expand the variable itself. Another option is to put the helper into $TRASH_DIRECTORY/bin and add that to the $PATH.
Quoted 5 lines
I also wondered if it was worth having
setup_credential_helper()just stick it in $TRASH_DIRECTORY/.gitconfig so that individual tests don't have to keep doing that ugly "-c" invocation. Or if you really want to have each test enable it, perhaps haveset_credential_reply()turn it on viatest_config(which will auto-remove it at the end of the test).
Good ideas! I shall try those.
Quoted 10 lines
-Peff
[1] Curious, I tried cloning git into this directory:
mkdir '/tmp/foo/"horrible \"path\"'
and we do indeed already fail. The first breakage I saw was recent, but going further back, it looks like bin-wrappers don't correctly handle this case anyway. So maybe that's evidence that nobody would do something so ridiculous in practice.
Re: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests
Quoted 13 lines · Matthew John Cheetham
Quoted 13 lines
could be normalized as:
www-auth-challenge=Basic realm="foo"
www-auth-challenge=OtherAuth realm="bar"
www-auth-challenge=YetAnotherScheme some-tokenwhich saves each helper from having to do the same work. Likewise, we can do a _little_ more parsing to get:
www-auth-basic=realm="foo"
www-auth-otherauth=realm="bar"
www-auth-yetanotherscheme=some-token
Quoted 8 lines
Quoted 5 lines
I don't think we can go beyond there, though, without understanding the syntax of individual schemes. Which is a shame, as one of the goals of the credential format was to let the helpers do as little as possible (so they can't get it wrong!). But helpers are stuck doing things like handling backslashed double-quotes, soaking up extra whitespace, etc.
This key format wouldn't make it obviously easier for simple helpers to understand. Now they no longer have well-known keys but a key prefix.
Yes, though I don't think that's particularly complicated to parse. Either way we're just flattening a tuple of (a, b, c) from "a=b c" to "a-b=c". The value is in normalizing the syntax, so that helpers don't have to deal with both "a=b c d e" and ("a=b c", "a=d e") themselves.
Another way to do that normalization would be to have Git convert:
WWW-Authenticate: Basic realm="foo" OtherAuth realm="bar"
into:
WWW-Authenticate: Basic realm="foo"
WWW-Authenticate: OtherAuth realm="bar"
which then becomes (at the credential level):
www-auth[]=Basic realm="foo"
www-auth[]=OtherAuth realm="bar"
And likewise to normalize whitespace, etc, so each individual helper doesn't have to (or risk getting confused/exploited). That said...
Quoted 3 lines
My overall goal here is to have Git know less about auth, so it treats all values as totally opaque. The only logic added is around reconstructing folded headers, which is just HTTP and not auth specific.
Yeah, in general I agree with the notion that Git is mostly just passing around opaque tokens. We do have to understand some syntax (like folding!) at the HTTP level, so I think some syntactic normalization / simplification is reasonable.
BUT. I think you are right that embedding it into the schema of the helper protocol is probably bad. If the point is that the two forms of my Basic / OtherAuth example are semantically equivalent, then we can always decide later to convert between one and the other as a favor to helpers. Whereas baking it into the schema is a promise for Git to always parse and understand the headers.
So let me retract my suggestion, and we can leave "maybe normalize headers to save helpers some work" as a possible topic for later (if indeed it ever even becomes a problem in practice).
Quoted 22 lines
Quoted 14 lines
realm=foo
while read line; do
case "$line" in
www-auth-basic=)
value=${line#*=}
# oops, we're just assuming it's realm= here, and we're # not handling quotes at all. I think it could technically be # realm=foo or realm="foo"
realm=${value#realm=}
;;
esac
done
echo password=$(pass "pats-by-realm/$realm")Quoted 7 lines
which could be made a lot easier if we did more parsing (e.g., www-auth-basic-realm or something). I dunno. Maybe that is just opening up a can of worms, as we're stuffing structured data into a linearized key-value list. The nice thing about your proposal is that Git does not even have to know anything about these schemes; it's all the problem of the helper. My biggest fear is just that we'll want to shift that later, and we'll be stuck with this microformat forever.
Quoted 4 lines
I'm not sure there's such a continuous scale between simple and 'complex' helpers that would mean there'd be a simple shell script generating OAuth or DPoP credentials instead of a helper written in a higher-level language where parsing the headers is one of the simpler challenges faced.
For the most part, yeah. I tried to form the above example as something that was really just relying on "basic", but taking in more information / context than we currently provide (and that your patch would provide). I admit it's a stretch, though. Are there any servers which actually use a Basic realm to distinguish between two credential's you'd want to provide? I don't think I've seen one.
(Not to mention that people scripting helpers like this is probably pretty rare; I do, but you can probably consider me a special case. And if things got more complicated I'd just turn to Perl anyway. ;) ).
Quoted 3 lines
I had considered another model whereby we forgo the key=value line model, and hide another format behind the 'final' terminating new-line. However I thought this would be even more distuptive.
Yeah, if we can shoe-horn this into the existing key/value model, that's much better. The original intent with the final newline is that you could read multiple credentials in a list, though in the end I don't recall that we ever used that feature anyway.
-Peff
Quoted 69 lines · Matthew John Cheetham via GitGitGadget
Updates in v6
=============* Clarify the change to make logging optional in the
check_dead_children()function during libification ofdaemon.c.* Fix missing pointer dereference bugs identified in libification of child process handling functions for
daemon.c.* Add doc comments to child process handling function declarations in the
daemon-utils.hheader.* Align function parameter names with variable names at callsites for libified daemon functions.
* Re-split out the test-http-server test helper commits in to smaller
patches: error response handling, request parsing, http-backend pass-through, simple authentication, arbitrary header support.* Call out auth configuration file format for test-http-server test helper and supported options in commit messages, as well as a test to exercise and demonstrate these options.
* Permit auth.token and auth.challenge to appear in any order; create the struct
auth_modulejust-in-time as options for that scheme are read. This simplifies the configuration authoring of the test-http-server test helper.* Update tests to use auth.allowAnoymous in the patch that introduces the new test helper option.
* Drop the
strvec_push_nodup()commit and update the implementation of HTTP request header line folding to use xstrdup andstrvec_popand_pushf.* Use
size_tinstead of int incredential.cwhen iterating over the struct strvec credential members. Also drop the not required const and cast from thefull_keydefinition and free.* Replace in-tree
test-credential-helper-reply.shtest cred helper script with thelib-credential-helper.shreusable 'lib' test script and shell functions to configure the helper behaviour.* Leverage sed over the while read $line loop in the test credential helper script.
Updates in v7
=============* Address several whitespace and arg/param list alignment issues.
* Rethink the test-http-helper worker-mode error and result enum to be more simple and more informative to the nature of the error.
* Use
uintmax_tto store the Content-Length of a request in the helper test-http-server. Maintain a bit flag to store if we received such a header.* Return a "400 Bad Request" HTTP response if we fail to parse the request in the test-http-server.
* Add test case to cover request message parsing in test-http-server.
* Use
size_tandALLOC_ARRAYover int andCALLOC_ARRAYrespectively inget_auth_module.* Correctly free the split strbufs created in the header parsing loop in test-http-server.
Quoted 7 lines
* Avoid needless comparison > 0 for unsigned types.
* Always set optional outputs to NULL if not present in test helper config value handling.
* Remove an accidentally commented-out test cleanup line for one test case in t5556.
I've re-read the patches in this version; all of my comments from v5 have been addressed, and the additional updates w.r.t. other reviewer feedback all look good as well. At this point, I think the series is ready for 'next'.
Thanks!
Quoted 10 lines · Victoria Dye
Quoted 3 lines · Matthew John Cheetham via GitGitGadget
Updates in v6
=============
...I've re-read the patches in this version; all of my comments from v5 have been addressed, and the additional updates w.r.t. other reviewer feedback all look good as well. At this point, I think the series is ready for 'next'.
Thanks!
Thanks, both. Let's merge it down.
Quoted 6 lines · Junio C Hamano
Quoted 4 lines · Victoria Dye
Quoted 3 lines · Matthew John Cheetham via GitGitGadget
Updates in v6
=============
...
Quoted 8 lines
Quoted 6 lines
I've re-read the patches in this version; all of my comments from v5 have been addressed, and the additional updates w.r.t. other reviewer feedback all look good as well. At this point, I think the series is ready for 'next'.
Thanks!
Thanks, both. Let's merge it down.
Sorry, I'm a bit late to the party, but I left some comments just now (this topic had been on my review backlog for ages, but I never quite got to it).
Many of my comments were small bits that could be fixed on top (tiny leaks, etc). But some of my comments were of the form "no, do it totally differently". It may simply be too late for those ones, but let's see if Matthew finds anything compelling in them.
-Peff
Quoted 10 lines · Jeff King
Quoted 1 line
Thanks, both. Let's merge it down.
Sorry, I'm a bit late to the party, but I left some comments just now (this topic had been on my review backlog for ages, but I never quite got to it).
Many of my comments were small bits that could be fixed on top (tiny leaks, etc). But some of my comments were of the form "no, do it totally differently". It may simply be too late for those ones, but let's see if Matthew finds anything compelling in them.
I do not mind reverting the merge to 'next' to have an improved version. Your "do we really want to add a custom server based on questionable codebase whose quality as a test-bed for real world usage is dubious?" is a valid concern.
Hi Junio & Peff,
Quoted 13 lines · Junio C Hamano
Quoted 10 lines · Jeff King
Quoted 1 line
Thanks, both. Let's merge it down.
Sorry, I'm a bit late to the party, but I left some comments just now (this topic had been on my review backlog for ages, but I never quite got to it).
Many of my comments were small bits that could be fixed on top (tiny leaks, etc). But some of my comments were of the form "no, do it totally differently". It may simply be too late for those ones, but let's see if Matthew finds anything compelling in them.
Quoted 4 lines
I do not mind reverting the merge to 'next' to have an improved version. Your "do we really want to add a custom server based on questionable codebase whose quality as a test-bed for real world usage is dubious?" is a valid concern.
Except.
Except that this code base would have made for a fine base to potentially implement an HTTPS-based replacement for the aging and insecure git-daemon.
That code base (which is hardly as questionable codebase as you make it sound because it has been in use for years in a slightly different form) would have had the opportunity to mature in a relatively safe environment:
our test suite. And eventually, once robust enough, it could have been extended to allow for easy and painless yet secure ad-hoc serving of Git repositories, addressing the security concerns around git-daemon.
And now that we're throwing out that code we don't have that opportunity, making the goal to deprecate the git-daemon and replace it by something that is as easy to set up but talks HTTPS instead much, much harder to reach.
In addition, it causes a loss of test coverage because Apache is not available in all the setups where the "questionable" code would have had no problem being built and validating the credential code.
Windows, for example, will now go completely uncovered in CI regarding the new code.
Ciao,
Johannes
Quoted 35 lines · Johannes Schindelin
Hi Junio & Peff,
Quoted 13 lines · Junio C Hamano
Quoted 10 lines · Jeff King
Quoted 1 line
Thanks, both. Let's merge it down.
Sorry, I'm a bit late to the party, but I left some comments just now (this topic had been on my review backlog for ages, but I never quite got to it).
Many of my comments were small bits that could be fixed on top (tiny leaks, etc). But some of my comments were of the form "no, do it totally differently". It may simply be too late for those ones, but let's see if Matthew finds anything compelling in them.
Quoted 4 lines
I do not mind reverting the merge to 'next' to have an improved version. Your "do we really want to add a custom server based on questionable codebase whose quality as a test-bed for real world usage is dubious?" is a valid concern.
Except.
Except that this code base would have made for a fine base to potentially implement an HTTPS-based replacement for the aging and insecure git-daemon.
That code base (which is hardly as questionable codebase as you make it sound because it has been in use for years in a slightly different form) would have had the opportunity to mature in a relatively safe environment:
our test suite. And eventually, once robust enough, it could have been extended to allow for easy and painless yet secure ad-hoc serving of Git repositories, addressing the security concerns around git-daemon.
Quoted 4 lines
And now that we're throwing out that code we don't have that opportunity, making the goal to deprecate the git-daemon and replace it by something that is as easy to set up but talks HTTPS instead much, much harder to reach.
There's many reasons for why you almost never see a git:// URL in the wild anymore.
But if "easy and painless" was synonymous with "built with git" or "ships with git" as you seem to be using it, surely it would be more common than doing the same with http or https, which requires an external server?
So, easy for whom? Just us and our own test suite?
Having read both your reply & Jeff's[1] I don't think you're addressing the thrust of his argument.
You can share all those goals without the method of getting there requiring us to start maintaining our own webserver.
Quoted 6 lines
In addition, it causes a loss of test coverage because Apache is not available in all the setups where the "questionable" code would have had no problem being built and validating the credential code.
Windows, for example, will now go completely uncovered in CI regarding the new code.
I have not set up Apache on Windows, but binaries seem to be available for it[2]. We don't use those now, but is downloading, setting up & running them in CI really harder than emarking on a project of maintaining our own webserver, especially we've got an eye towards non-test suite use?
Even if we think that we'd like to have a webserver built when you "make git" I don't see why we'd go the NIH route of writing our own. Unlike the git:// protocol there's a *lot* of implementations of http(s):// servers.
If we think apache is too heavyweight for whatever reason, can't we add one of the many light http servers out there to contrib/ use it it from there?
1. https://lore.kernel.org/git/[email protected]/
2. https://httpd.apache.org/docs/2.4/platform/windows.html
Hi Ævar,
Quoted 35 lines · Ævar Arnfjörð Bjarmason
Quoted 33 lines · Johannes Schindelin
Quoted 13 lines · Junio C Hamano
Quoted 10 lines · Jeff King
Quoted 1 line
Thanks, both. Let's merge it down.
Sorry, I'm a bit late to the party, but I left some comments just now (this topic had been on my review backlog for ages, but I never quite got to it).
Many of my comments were small bits that could be fixed on top (tiny leaks, etc). But some of my comments were of the form "no, do it totally differently". It may simply be too late for those ones, but let's see if Matthew finds anything compelling in them.
Quoted 4 lines
I do not mind reverting the merge to 'next' to have an improved version. Your "do we really want to add a custom server based on questionable codebase whose quality as a test-bed for real world usage is dubious?" is a valid concern.
Except.
Except that this code base would have made for a fine base to potentially implement an HTTPS-based replacement for the aging and insecure git-daemon.
That code base (which is hardly as questionable codebase as you make it sound because it has been in use for years in a slightly different form) would have had the opportunity to mature in a relatively safe environment:
our test suite. And eventually, once robust enough, it could have been extended to allow for easy and painless yet secure ad-hoc serving of Git repositories, addressing the security concerns around git-daemon.
Quoted 7 lines
Quoted 4 lines
And now that we're throwing out that code we don't have that opportunity, making the goal to deprecate the git-daemon and replace it by something that is as easy to set up but talks HTTPS instead much, much harder to reach.
There's many reasons for why you almost never see a
git://URL in the wild anymore.
I am unwilling to accept that statement without any source to back it up. Thin air is no substitute for reliable evidence.
Quoted 4 lines
But if "easy and painless" was synonymous with "built with git" or "ships with git" as you seem to be using it, surely it would be more common than doing the same with http or https, which requires an external server?
Oh whoa... "requires an external server"?
My entire point was to suggest a way forward for an _internal_ server that speaks https:// instead of git://.
So I am not suggesting what you seem to have understood me to suggest.
Ciao,
Johannes
Quoted 44 lines · Johannes Schindelin
Hi Ævar,
Quoted 40 lines · Ævar Arnfjörð Bjarmason
Quoted 37 lines · Johannes Schindelin
Quoted 13 lines · Junio C Hamano
Quoted 10 lines · Jeff King
Quoted 1 line
Thanks, both. Let's merge it down.
Sorry, I'm a bit late to the party, but I left some comments just now (this topic had been on my review backlog for ages, but I never quite got to it).
Many of my comments were small bits that could be fixed on top (tiny leaks, etc). But some of my comments were of the form "no, do it totally differently". It may simply be too late for those ones, but let's see if Matthew finds anything compelling in them.
Quoted 4 lines
I do not mind reverting the merge to 'next' to have an improved version. Your "do we really want to add a custom server based on questionable codebase whose quality as a test-bed for real world usage is dubious?" is a valid concern.
Except.
Except that this code base would have made for a fine base to potentially implement an HTTPS-based replacement for the aging and insecure git-daemon.
That code base (which is hardly as questionable codebase as you make it sound because it has been in use for years in a slightly different form) would have had the opportunity to mature in a relatively safe environment:
our test suite. And eventually, once robust enough, it could have been extended to allow for easy and painless yet secure ad-hoc serving of Git repositories, addressing the security concerns around git-daemon.And now that we're throwing out that code we don't have that opportunity, making the goal to deprecate the git-daemon and replace it by something that is as easy to set up but talks HTTPS instead much, much harder to reach.
Quoted 5 lines
Quoted 2 lines
There's many reasons for why you almost never see a
git://URL in the wild anymore.I am unwilling to accept that statement without any source to back it up. Thin air is no substitute for reliable evidence.
Most people exposing git over the Internet use the ssh or http transport, and our own "git" protocol is relatively obscure.
If you need data I think major hosting sites not offering it, or deprecating it, is a pretty strong signal, e.g. Microsoft with:
https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git
But if you'll grant me that it's 50/50 git/other protocols (I think it's a *lot* more lopsided), then clearly combining git with 3rd party server components isn't the limiting factor on deploying it.
Which is the point I was going for.
Quoted 9 lines
Quoted 4 lines
But if "easy and painless" was synonymous with "built with git" or "ships with git" as you seem to be using it, surely it would be more common than doing the same with http or https, which requires an external server?
Oh whoa... "requires an external server"?
My entire point was to suggest a way forward for an _internal_ server that speaks https:// instead of
git://.
I understand that.
Quoted 1 line
So I am not suggesting what you seem to have understood me to suggest.
I wasn't suggesting that, and you seem to have not read my reply to the end, which should have addressed that.
Briefly, we'd like to be guaranteed to have regcomp() and regexec(), but did the Git project write its own regex engine?
No, we imported (with some minor tweaks) one from glibc/gawk (whatever current issues have cropped up with it lately...).
So can't we do the same for a httpd? If it really comes to "we must have it in-tree"?
It seems to me that there's a continuum here, which is at the very least:
1) We require an external package (e.g. ssh, or apache/httpd)
2) We require an external package *or* built-in (e.g. our SHA-1 implementations)
3) We use an external package as-is (sha1dc)
4) We adapt an external codebase, and perma-fork it (git-imap-send, although that example also kind of sucks)
5) We write it "in-house" from scratch.
It seems to me from reading the upthread that we're jumping straight from #1 to #5, and it's not clear to me why that is.
Not even that, we currently have CI tests running Apache on *nix boxes, but you're suggesting a loss of coverage on Windows
Is it really harder to just install (or even ship our own package of) Apache for Windows than it is to embark on PID file handling, logging, timeout management and the long tail of "80% is easy, the rest is really hard" of writing our own production-class httpd (as the suggestion is to have it eventually mature beyond the test suite)?
Maybe, all I'm saying (in trying to mediate the discussion between you and Jeff) is that it's not obvious to me why that is...
Hi Ævar,
Quoted 8 lines · Ævar Arnfjörð Bjarmason
we currently have CI tests running Apache on *nix boxes, but you're suggesting a loss of coverage on Windows
Is it really harder to just install (or even ship our own package of) Apache for Windows than it is to embark on PID file handling, logging, timeout management and the long tail of "80% is easy, the rest is really hard" of writing our own production-class httpd (as the suggestion is to have it eventually mature beyond the test suite)?
Yes, it _is_ that much harder, and it would result in yet more painful increases of the build times which have really gotten out of hand in the past year.
Ciao,
Johannes
Quoted 10 lines · Johannes Schindelin
Quoted 4 lines
I do not mind reverting the merge to 'next' to have an improved version. Your "do we really want to add a custom server based on questionable codebase whose quality as a test-bed for real world usage is dubious?" is a valid concern.
Except.
Except that this code base would have made for a fine base to potentially implement an HTTPS-based replacement for the aging and insecure git-daemon.
I'm skeptical that it is a good idea for Git to implement a custom http server from scratch. There are a lot of extended features people would want in a production-ready server. TLS, HTTP/2, and so on. The code under discussion is pretty stripped-down. A network service is also a pretty big attack surface for buffer overflows, etc.
It feels to me like the resources required to make it good enough for normal users to run would be substantial. And we'd be better off trying to integrate with an existing project that provides a web server (whether it's a lightweight server that supports us as a CGI, or a library that does most of the heavy lifting).
Quoted 2 lines
That code base (which is hardly as questionable codebase as you make it sound because it has been in use for years in a slightly different form)
Perhaps I'm being too hard on git-daemon. But my operational experience with it is that it has several flaws, mostly around the child-management code. We rewrote that code totally to make it usable at GitHub.
As a concrete example, the parent daemon process will do a linear walk over all children, calling waitpid() on each one. This makes handling N children quadratic, and the daemon grinds to a halt when there are many clients.
Quoted 6 lines
In addition, it causes a loss of test coverage because Apache is not available in all the setups where the "questionable" code would have had no problem being built and validating the credential code.
Windows, for example, will now go completely uncovered in CI regarding the new code.
I'm sympathetic there, though it's a problem for all of the existing http code, too. Is there some server option that would be easier to run everywhere, but that doesn't involve us writing a server from scratch? Certainly I have no particular love for apache or its configuration language.
-Peff
Hi Jeff,
Quoted 7 lines · Jeff King
Quoted 5 lines · Johannes Schindelin
Quoted 4 lines
I do not mind reverting the merge to 'next' to have an improved version. Your "do we really want to add a custom server based on questionable codebase whose quality as a test-bed for real world usage is dubious?" is a valid concern.
Quoted 8 lines
Quoted 5 lines
Except.
Except that this code base would have made for a fine base to potentially implement an HTTPS-based replacement for the aging and insecure git-daemon.
I'm skeptical that it is a good idea for Git to implement a custom http server from scratch.
To be clear: I never suggested to implement a generic HTTP server.
All I wanted was to have a replacement for git daemon that speaks
https:// instead of git://. It does not have to speak to every browser out there, it only needs to respond well when speaking to Git clients. That is a much, much smaller surface than "production-ready server, HTTP/2 and so on".
And while the proposed test helper was not quite complete in that way, and while it may have had much of the git daemon code that you would love to lose, it would have offered an incremental way forward.
I am afraid that this way forward is now blocked, and we're further away from dropping that git daemon code you wanted to drop than ever.
Ciao,
Johannes
Quoted 10 lines · Johannes Schindelin
Quoted 2 lines
I'm skeptical that it is a good idea for Git to implement a custom http server from scratch.
To be clear: I never suggested to implement a generic HTTP server.
All I wanted was to have a replacement for
git daemonthat speaks
https:// instead ofgit://. It does not have to speak to every browser out there, it only needs to respond well when speaking to Git clients. That is a much, much smaller surface than "production-ready server, HTTP/2 and so on".
I guess I don't see the point of having this in our test suite, though. We do want to test things like HTTP/2, SSL, and so on in our test suite. So either we have a split in our tests (some use apache, some don't, which presumably means many tests are still not run on Windows), or this custom HTTP server eventually grows to do all of those other things.
I can see the utility outside the tests of a quick "let me stand up an HTTP server to access Git" tool. But even there, I'd be considered with feature creep as regular users ignore any warnings about its lack of encryption/robustness, and so on. And it feels like something that could utilize work already done by others in making a web server. Yes, that's a new dependency for the tool, but there are a lot of options out there. Surely one of them is worth building on?
Quoted 6 lines
And while the proposed test helper was not quite complete in that way, and while it may have had much of the
git daemoncode that you would love to lose, it would have offered an incremental way forward.I am afraid that this way forward is now blocked, and we're further away from dropping that
git daemoncode you wanted to drop than ever.
I don't see how pushing the same code into an http server helps. If we could have incrementally improved it there, we could incrementally improve it in git-daemon, too.
-Peff
v7 cover Enhance credential helper protocol to include auth headers
In reply to Matthew John Cheetham via GitGitGadgetQuoted 45 lines
Future work
===========In the future we can further expand the protocol to allow credential helpers decide the best authentication scheme. Today credential helpers are still only expected to return a username/password pair to Git, meaning the other authentication schemes that may be offered still need challenge responses sent via a Basic Authorization header. The changes outlined above still permit helpers to select and configure an available authentication mode, but require the remote for example to unpack a bearer token from a basic challenge.
More careful consideration is required in the handling of custom authentication schemes which may not have a username, or may require arbitrary additional request header values be set.
For example imagine a new "FooBar" authentication scheme that is surfaced in the following response:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"With support for arbitrary authentication schemes, Git would call credential helpers with the following over standard input:
protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"And then an enlightened credential helper could return over standard output:
protocol=https
host=example.com
authtype=FooBar
[email protected]
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEFGit would be expected to attach this authorization header to the next request:
Quoted 6 lines
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF
Interesting! Can you tell us more about how you hope to use this at GitHub? Could this be used for OAuth 2.0 Demonstrating Proof-of-Possession at the Application Layer (DPoP)? https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop (some of the fields in your example look familiar).
Challenge responses are typically short lived [1]. What happens if a storage helper is configured before a challenge-response helper? We want to maintain composability of helpers.
[credential]
helper = storage # eg. cache or osxkeychain
helper = challenge-response # eg. oauth-dpop
Storage may return an expired challenge response stored earlier. This could be avoided by introducing an expiry attribute to the credential protocol. https://lore.kernel.org/git/[email protected]/T/#u
A monolithic helper configured alone doesn't have this problem -- it knows which parts of its output to store or discard.
Declaration of interest: I maintain a credential-generating OAuth helper composable with any storage helper. https://github.com/hickford/git-credential-oauth
[1] https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop#section-8
Quoted 45 lines · M Hickford
Quoted 45 lines
Future work
===========In the future we can further expand the protocol to allow credential helpers decide the best authentication scheme. Today credential helpers are still only expected to return a username/password pair to Git, meaning the other authentication schemes that may be offered still need challenge responses sent via a Basic Authorization header. The changes outlined above still permit helpers to select and configure an available authentication mode, but require the remote for example to unpack a bearer token from a basic challenge.
More careful consideration is required in the handling of custom authentication schemes which may not have a username, or may require arbitrary additional request header values be set.
For example imagine a new "FooBar" authentication scheme that is surfaced in the following response:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"With support for arbitrary authentication schemes, Git would call credential helpers with the following over standard input:
protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"And then an enlightened credential helper could return over standard output:
protocol=https
host=example.com
authtype=FooBar
[email protected]
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEFGit would be expected to attach this authorization header to the next request:
Quoted 8 lines
Quoted 6 lines
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEFInteresting! Can you tell us more about how you hope to use this at GitHub? Could this be used for OAuth 2.0 Demonstrating Proof-of-Possession at the Application Layer (DPoP)? https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop (some of the fields in your example look familiar).
This would be exactly the sort of thing that this would enable. DPoP is one example where the correct auth response requires more than just a username/ password pair in the Authorization header. We should also be returning standard headers like 'Authenticate-Info' on 200 OK responses along side the 'store' calls to helpers; they could contain nonces or other important auth information.
My end goal here is to extend the credential helper protocol such that that helpers can see more of the initial request challenge, and then modify the subsequent request directly, or configure and let curl handle it (the latter part not present in this iteration).
One thing not called out in this doc is really the need for some capability advertisement between Git and helpers - for example if the curl version in use supports CURLOPT_XOAUTH2_BEARER for bearer tokens.
Quoted 5 lines
Challenge responses are typically short lived [1]. What happens if a storage helper is configured before a challenge-response helper? We want to maintain composability of helpers.
[credential]
helper = storage # eg. cache or osxkeychain
helper = challenge-response # eg. oauth-dpop
I think really this sort of thing is where the credential helper protocol isn't designed for credential-generating helpers in mind, but only simple storage-only helpers. There is no affinity between get/erase/store commands meaning one helper may return a credential for another helper to store it. Not sure if this was ever the intention, over just the need to consult a list of helpers for a stored credential.
Quoted 7 lines
Storage may return an expired challenge response stored earlier. This could be avoided by introducing an expiry attribute to the credential protocol. https://lore.kernel.org/git/[email protected]/T/#u
A monolithic helper configured alone doesn't have this problem -- it knows which parts of its output to store or discard.
Declaration of interest: I maintain a credential-generating OAuth helper composable with any storage helper. https://github.com/hickford/git-credential-oauth
[1] https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop#section-8
Quoted 6 lines · Matthew John Cheetham
Quoted 5 lines
Challenge responses are typically short lived [1]. What happens if a storage helper is configured before a challenge-response helper? We want to maintain composability of helpers.
[credential]
helper = storage # eg. cache or osxkeychain
helper = challenge-response # eg. oauth-dpop
Quoted 6 lines
I think really this sort of thing is where the credential helper protocol isn't designed for credential-generating helpers in mind, but only simple storage-only helpers. There is no affinity between get/erase/store commands meaning one helper may return a credential for another helper to store it. Not sure if this was ever the intention, over just the need to consult a list of helpers for a stored credential.
I actually had envisioned helpers generating credentials. In fact, in the first iteration of the series, Git did not prompt for passwords at all! It would depend on git-credential-prompt to do so. But I ended up folding that in for simplicity.
I could well believe that there is not enough context passed around for helpers to make good decisions, though. It's both a feature and a bug that credentials from one helper get passed to another. It's good if you want to cache a generated credential. It's bad if you don't want credentials from one helper to leak to another, less-secure one.
So I'm open to improvements that help define and communicate that context.
-Peff
v8 cover Enhance credential helper protocol to include auth headers
In reply to Matthew John Cheetham via GitGitGadgetFollowing from my original RFC submission [0], this submission is considered ready for full review. This patch series is now based on top of current master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now separately submitted patches [1] to fix up the other credential helpers' behaviour.
In this patch series I update the existing credential helper design in order to allow for some new scenarios, and future evolution of auth methods that Git hosts may wish to provide. I outline the background, summary of changes and some challenges below.
Testing these new additions, I use a small CGI shell script that acts as a frontend to git-http-backend; simple authentication is configurable by files.
Background
==========
Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the authentication capabilities of this protocol to address limitations (see below).
The Smart HTTP protocol in Git supports a few different types of HTTP authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478) [4]. Git uses a extensible model where credential helpers can provide credentials for protocols [5]. Several helpers support alternatives such as OAuth authentication (RFC 6749) [6], but this is typically done as an extension. For example, a helper might use basic auth and set the password to an OAuth Bearer access token. Git uses standard input and output to communicate with credential helpers.
After a HTTP 401 response, Git would call a credential helper with the following over standard input:
protocol=https
host=example.com
And then a credential helper would return over standard output:
protocol=https
host=example.com
[email protected]
password=<BEARER-TOKEN>
Git then the following request to the remote, including the standard HTTP Authorization header (RFC 7235 Section 4.2) [7]:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64([email protected]:<BEARER-TOKEN>)
Credential helpers are encouraged (see gitcredentials.txt) to return the minimum information necessary.
Limitations
===========
Because this credential model was built mostly for password based authentication systems, it's somewhat limited. In particular:
1. To generate valid credentials, additional information about the request (or indeed the requestee and their device) may be required. For example, OAuth is based around scopes. A scope, like "git.read", might be required to read data from the remote. However, the remote cannot tell the credential helper what scope is required for this request.
2. This system is not fully extensible. Each time a new type of authentication (like OAuth Bearer) is invented, Git needs updates before credential helpers can take advantage of it (or leverage a new capability in libcurl).
Goals
=====
* As a user with multiple federated cloud identities:
* Reach out to a remote and have my credential helper automatically prompt me for the correct identity.
* Allow credential helpers to differentiate between different authorities or authentication/authorization challenge types, even from the same DNS hostname (and without needing to use credential.useHttpPath).
* Leverage existing authentication systems built-in to many operating systems and devices to boost security and reduce reliance on passwords.
* As a Git host and/or cloud identity provider:
* Enforce security policies (like requiring two-factor authentication) dynamically.
* Allow integration with third party standard based identity providers in enterprises allowing customers to have a single plane of control for critical identities with access to source code.
Design Principles
=================
* Use the existing infrastructure. Git credential helpers are an already-working model.
* Follow widely-adopted time-proven open standards, avoid net new ideas in the authentication space.
* Minimize knowledge of authentication in Git; maintain modularity and extensibility.
Proposed Changes
================
1. Teach Git to read HTTP response headers, specifically the standard WWW-Authenticate (RFC 7235 Section 4.1) headers.
2. Teach Git to include extra information about HTTP responses that require authentication when calling credential helpers. Specifically the WWW-Authenticate header information.
Because the extra information forms an ordered list, and the existing credential helper I/O format only provides for simple key=value pairs, we introduce a new convention for transmitting an ordered list of values. Key names that are suffixed with a C-style array syntax should have values considered to form an order list, i.e. key[]=value, where the order of the key=value pairs in the stream specifies the order.
For the WWW-Authenticate header values we opt to use the key wwwauth[].
Handling the WWW-Authenticate header in detail
==============================================
RFC 6750 [8] envisions that OAuth Bearer resource servers would give responses that include WWW-Authenticate headers, for example:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
Specifically, a WWW-Authenticate header consists of a scheme and arbitrary attributes, depending on the scheme. This pattern enables generic OAuth or OpenID Connect [9] authorities. Note that it is possible to have several WWW-Authenticate challenges in a response.
First Git attempts to make a request, unauthenticated, which fails with a 401 response and includes WWW-Authenticate header(s).
Next, Git invokes a credential helper which may prompt the user. If the user approves, a credential helper can generate a token (or any auth challenge response) to be used for that request.
For example: with a remote that supports bearer tokens from an OpenID Connect [9] authority, a credential helper can use OpenID Connect's Discovery [10] and Dynamic Client Registration [11] to register a client and make a request with the correct permissions to access the remote. In this manner, a user can be dynamically sent to the right federated identity provider for a remote without any up-front configuration or manual processes.
Following from the principle of keeping authentication knowledge in Git to a minimum, we modify Git to add all WWW-Authenticate values to the credential helper call.
Git sends over standard input:
protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite" wwwauth[]=Basic realm="login.example"
A credential helper that understands the extra wwwauth[n] property can decide on the "best" or correct authentication scheme, generate credentials for the request, and interact with the user.
The credential helper would then return over standard output:
protocol=https
host=example.com
path=foo.git
[email protected]
password=<BEARER-TOKEN>
Note that WWW-Authenticate supports multiple challenges, either in one header:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
or in multiple headers:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the implementation, Git will not merge or split up any of these WWW-Authenticate headers, and instead pass each header line as one credential helper property. The credential helper is responsible for splitting, merging, and otherwise parsing these header values.
An alternative option to sending the header fields individually would be to merge the header values in to one key=value property, for example:
...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
Future work
===========
In the future we can further expand the protocol to allow credential helpers decide the best authentication scheme. Today credential helpers are still only expected to return a username/password pair to Git, meaning the other authentication schemes that may be offered still need challenge responses sent via a Basic Authorization header. The changes outlined above still permit helpers to select and configure an available authentication mode, but require the remote for example to unpack a bearer token from a basic challenge.
More careful consideration is required in the handling of custom authentication schemes which may not have a username, or may require arbitrary additional request header values be set.
For example imagine a new "FooBar" authentication scheme that is surfaced in the following response:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"
With support for arbitrary authentication schemes, Git would call credential helpers with the following over standard input:
protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"
And then an enlightened credential helper could return over standard output:
protocol=https
host=example.com
authtype=FooBar
[email protected]
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF
Git would be expected to attach this authorization header to the next request:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF
Why not SSH?
============
There's nothing wrong with SSH. However, Git's Smart HTTP transport is widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport sometimes requires less client setup than SSH transport, and works in environments when SSH ports may be blocked. As long as Git supports HTTP transport, it should support common and popular HTTP authentication methods.
References
==========
* [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers
https://lore.kernel.org/git/[email protected]/
* [1] [PATCH 0/3] Correct credential helper discrepancies handling input
https://lore.kernel.org/git/[email protected]/
* [2] Git on the Server - The Protocols
https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols
* [3] HTTP Authentication: Basic and Digest Access Authentication
https://datatracker.ietf.org/doc/html/rfc2617
* [4] The Simple and Protected GSS-API Negotiation Mechanism
https://datatracker.ietf.org/doc/html/rfc2478
* [5] Git Credentials - Custom Helpers
https://git-scm.com/docs/gitcredentials#_custom_helpers
* [6] The OAuth 2.0 Authorization Framework
https://datatracker.ietf.org/doc/html/rfc6749
* [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
https://datatracker.ietf.org/doc/html/rfc7235
* [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
https://datatracker.ietf.org/doc/html/rfc6750
* [9] OpenID Connect Core 1.0
https://openid.net/specs/openid-connect-core-1_0.html
* [10] OpenID Connect Discovery 1.0
https://openid.net/specs/openid-connect-discovery-1_0.html
* [11] OpenID Connect Dynamic Client Registration 1.0
https://openid.net/specs/openid-connect-registration-1_0.html
* [12] Hypertext Transfer Protocol (HTTP/1.1)
https://datatracker.ietf.org/doc/html/rfc2616
Updates from RFC
================
* Submitted first three patches as separate submission:
https://lore.kernel.org/git/[email protected]/
* Various style fixes and updates to- and addition of comments.
* Drop the explicit integer index in new 'array' style credential helper attrbiutes ("key[n]=value" becomes just "key[]=value").
* Added test helper; a mini HTTP server, and several tests.
Updates in v3
=============
* Split final patch that added the test-http-server in to several, easier to review patches.
* Updated wording in git-credential.txt to clarify which side of the credential helper protocol is sending/receiving the new wwwauth and authtype attributes.
Updates in v4
=============
* Drop authentication scheme selection authtype attribute patches to greatly simplify the series; auth scheme selection is punted to a future series. This series still allows credential helpers to generate credentials and intelligently select correct identities for a given auth challenge.
Updates in v5
=============
* Libify parts of daemon.c and share implementation with test-http-server.
* Clarify test-http-server Git request regex pattern and auth logic comments.
* Use STD*_FILENO in place of 'magic' file descriptor numbers.
* Use strbuf_* functions in continuation header parsing.
* Use configuration file to configure auth for test-http-server rather than command-line arguments. Add ability to specify arbitrary extra headers that is useful for testing 'malformed' server responses.
* Use st_mult over unchecked multiplication in http.c curl callback functions.
* Fix some documentation line break issues.
* Reorder some commits to bring in the tests and test-http-server helper first and, then the WWW-Authentication changes, alongside tests to cover.
* Expose previously static strvec_push_nodup function.
* Merge the two timeout args for test-http-server (--timeout and --init-timeout) that were a hang-over from the original daemon.c but are no longer required here.
* Be more careful around continuation headers where they may be empty strings. Add more tests to cover these header types.
* Include standard trace2 tracing calls at start of test-http-server helper.
Updates in v6
=============
* Clarify the change to make logging optional in the check_dead_children() function during libification of daemon.c.
* Fix missing pointer dereference bugs identified in libification of child process handling functions for daemon.c.
* Add doc comments to child process handling function declarations in the daemon-utils.h header.
* Align function parameter names with variable names at callsites for libified daemon functions.
* Re-split out the test-http-server test helper commits in to smaller
patches: error response handling, request parsing, http-backend pass-through, simple authentication, arbitrary header support.
* Call out auth configuration file format for test-http-server test helper and supported options in commit messages, as well as a test to exercise and demonstrate these options.
* Permit auth.token and auth.challenge to appear in any order; create the struct auth_module just-in-time as options for that scheme are read. This simplifies the configuration authoring of the test-http-server test helper.
* Update tests to use auth.allowAnoymous in the patch that introduces the new test helper option.
* Drop the strvec_push_nodup() commit and update the implementation of HTTP request header line folding to use xstrdup and strvec_pop and _pushf.
* Use size_t instead of int in credential.c when iterating over the struct strvec credential members. Also drop the not required const and cast from the full_key definition and free.
* Replace in-tree test-credential-helper-reply.sh test cred helper script with the lib-credential-helper.sh reusable 'lib' test script and shell functions to configure the helper behaviour.
* Leverage sed over the while read $line loop in the test credential helper script.
Updates in v7
=============
* Address several whitespace and arg/param list alignment issues.
* Rethink the test-http-helper worker-mode error and result enum to be more simple and more informative to the nature of the error.
* Use uintmax_t to store the Content-Length of a request in the helper test-http-server. Maintain a bit flag to store if we received such a header.
* Return a "400 Bad Request" HTTP response if we fail to parse the request in the test-http-server.
* Add test case to cover request message parsing in test-http-server.
* Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in get_auth_module.
* Correctly free the split strbufs created in the header parsing loop in test-http-server.
* Avoid needless comparison > 0 for unsigned types.
* Always set optional outputs to NULL if not present in test helper config value handling.
* Remove an accidentally commented-out test cleanup line for one test case in t5556.
Updates in v8
=============
* Drop custom HTTP test helper tool in favour of using a CGI shell script and Apache; avoiding the need to implement an HTTP server.
* Avoid allocations in header reading callback unless we have a header we care about; act on the char* from libcurl directly rather than create a strbuf for each header.
* Drop st_mult overflow guarding function in curl callback functions; we're not allocating memory based on the resulting value and just adds to potential confusion in the future.
Matthew John Cheetham (3):
t5563: add tests for basic and anoymous HTTP access
http: read HTTP WWW-Authenticate response headers
credential: add WWW-Authenticate header to cred requests
Documentation/git-credential.txt | 19 +- credential.c | 12 ++ credential.h | 15 ++ git-compat-util.h | 22 +++ http.c | 120 ++++++++++++ t/lib-httpd.sh | 1 + t/lib-httpd/apache.conf | 6 + t/lib-httpd/nph-custom-auth.sh | 42 ++++ t/t5563-simple-http-auth.sh | 322 +++++++++++++++++++++++++++++++ 9 files changed, 558 insertions(+), 1 deletion(-) create mode 100755 t/lib-httpd/nph-custom-auth.sh create mode 100755 t/t5563-simple-http-auth.sh
base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v8
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v8
Pull-Request: https://github.com/gitgitgadget/git/pull/1352
Range-diff vs v7:
1: 74b0de14185 < -: ----------- daemon: libify socket setup and option functions
2: b6ba344a671 < -: ----------- daemon: libify child process handling functions
3: 9967401c972 < -: ----------- daemon: rename some esoteric/laboured terminology
4: 17c890ee108 < -: ----------- test-http-server: add stub HTTP server test helper
5: 6e70e304cfe < -: ----------- test-http-server: add HTTP error response function
6: 43f1cdcbb82 ! 1: d362f7016d3 test-http-server: add HTTP request parsing
@@ Metadata
Author: Matthew John Cheetham <[email protected]>
## Commit message ##
- test-http-server: add HTTP request parsing
+ t5563: add tests for basic and anoymous HTTP access
- Add ability to parse HTTP requests to the test-http-server test helper.
- Introduce `struct req` to store request information including:
+ Add a test showing simple anoymous HTTP access to an unprotected
+ repository, that results in no credential helper invocations.
+ Also add a test demonstrating simple basic authentication with
+ simple credential helper support.
- * HTTP method & version
- * Request path and query parameters
- * Headers
- * Content type and length (from `Content-Type` and `-Length` headers)
-
- Failure to parse the request results in a 400 Bad Request response to
- the client. Note that we're not trying to support all possible requests
- here, but just enough to exercise all code under test.
+ Leverage a no-parsed headers (NPH) CGI script so that we can directly
+ control the HTTP responses to simulate a multitude of good, bad and ugly
+ remote server implementations around auth.
Signed-off-by: Matthew John Cheetham <[email protected]>
- ## t/helper/test-http-server.c ##
-@@ t/helper/test-http-server.c: enum worker_result {
- * Close the socket and clean up. Does not imply an error.
- */
- WR_HANGUP = 2,
-+
-+ /*
-+ * Unexpected request message or error in request parsing.
-+ * Respond with an 400 error. Close the socket and cleanup.
-+ * Exit child-process with a non-zero status.
-+ */
-+ WR_CLIENT_ERROR = 3,
-+};
-+
-+/*
-+ * Fields from a parsed HTTP request.
-+ */
-+struct req {
-+ struct strbuf start_line;
-+
-+ const char *method;
-+ const char *http_version;
-+
-+ struct strbuf uri_path;
-+ struct strbuf query_args;
-+
-+ struct string_list header_list;
-+ const char *content_type;
-+ uintmax_t content_length;
-+ unsigned has_content_length:1;
- };
+ ## t/lib-httpd.sh ##
+@@ t/lib-httpd.sh: prepare_httpd() {
+ install_script error-smart-http.sh
+ install_script error.sh
+ install_script apply-one-time-perl.sh
++ install_script nph-custom-auth.sh
-+#define REQ__INIT { \
-+ .start_line = STRBUF_INIT, \
-+ .uri_path = STRBUF_INIT, \
-+ .query_args = STRBUF_INIT, \
-+ .header_list = STRING_LIST_INIT_NODUP, \
-+ .content_type = NULL, \
-+ .content_length = 0, \
-+ .has_content_length = 0, \
-+}
-+
-+static void req__release(struct req *req)
-+{
-+ strbuf_release(&req->start_line);
-+
-+ strbuf_release(&req->uri_path);
-+ strbuf_release(&req->query_args);
-+
-+ string_list_clear(&req->header_list, 0);
-+}
-+
- static enum worker_result send_http_error(int fd, int http_code,
- const char *http_code_name,
- int retry_after_seconds,
-@@ t/helper/test-http-server.c: done:
- return wr;
- }
-
-+/*
-+ * Read the HTTP request up to the start of the optional message-body.
-+ * We do this byte-by-byte because we have keep-alive turned on and
-+ * cannot rely on an EOF.
-+ *
-+ * https://tools.ietf.org/html/rfc7230
-+ *
-+ * We cannot call die() here because our caller needs to properly
-+ * respond to the client and/or close the socket before this
-+ * child exits so that the client doesn't get a connection reset
-+ * by peer error.
-+ */
-+static enum worker_result req__read(struct req *req, int fd)
-+{
-+ struct strbuf h = STRBUF_INIT;
-+ struct string_list start_line_fields = STRING_LIST_INIT_DUP;
-+ int nr_start_line_fields;
-+ const char *uri_target;
-+ const char *query;
-+ char *hp;
-+ const char *hv;
-+
-+ enum worker_result result = WR_OK;
-+
-+ /*
-+ * Read line 0 of the request and split it into component parts:
-+ *
-+ * <method> SP <uri-target> SP <HTTP-version> CRLF
-+ *
-+ */
-+ if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
-+ result = WR_HANGUP;
-+ goto done;
-+ }
-+
-+ strbuf_trim_trailing_newline(&req->start_line);
-+
-+ nr_start_line_fields = string_list_split(&start_line_fields,
-+ req->start_line.buf,
-+ ' ', -1);
-+ if (nr_start_line_fields != 3) {
-+ logerror("could not parse request start-line '%s'",
-+ req->start_line.buf);
-+ result = WR_CLIENT_ERROR;
-+ goto done;
-+ }
-+
-+ req->method = xstrdup(start_line_fields.items[0].string);
-+ req->http_version = xstrdup(start_line_fields.items[2].string);
-+
-+ uri_target = start_line_fields.items[1].string;
-+
-+ if (strcmp(req->http_version, "HTTP/1.1")) {
-+ logerror("unsupported version '%s' (expecting HTTP/1.1)",
-+ req->http_version);
-+ result = WR_CLIENT_ERROR;
-+ goto done;
-+ }
-+
-+ query = strchr(uri_target, '?');
-+
-+ if (query) {
-+ strbuf_add(&req->uri_path, uri_target, (query - uri_target));
-+ strbuf_trim_trailing_dir_sep(&req->uri_path);
-+ strbuf_addstr(&req->query_args, query + 1);
-+ } else {
-+ strbuf_addstr(&req->uri_path, uri_target);
-+ strbuf_trim_trailing_dir_sep(&req->uri_path);
-+ }
-+
-+ /*
-+ * Read the set of HTTP headers into a string-list.
-+ */
-+ while (1) {
-+ if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
-+ goto done;
-+ strbuf_trim_trailing_newline(&h);
-+
-+ if (!h.len)
-+ goto done; /* a blank line ends the header */
-+
-+ hp = strbuf_detach(&h, NULL);
-+ string_list_append(&req->header_list, hp);
-+
-+ /* also store common request headers as struct req members */
-+ if (skip_iprefix(hp, "Content-Type: ", &hv)) {
-+ req->content_type = hv;
-+ } else if (skip_iprefix(hp, "Content-Length: ", &hv)) {
-+ /*
-+ * Content-Length is always non-negative, but has no
-+ * upper bound according to RFC 7230 (§3.3.2).
-+ */
-+ intmax_t len = 0;
-+ if (sscanf(hv, "%"PRIdMAX, &len) != 1 || len < 0 ||
-+ len == INTMAX_MAX) {
-+ logerror("invalid content-length: '%s'", hv);
-+ result = WR_CLIENT_ERROR;
-+ goto done;
-+ }
-+
-+ req->content_length = (uintmax_t)len;
-+ req->has_content_length = 1;
-+ }
-+ }
-+
-+ /*
-+ * We do not attempt to read the <message-body>, if it exists.
-+ * We let our caller read/chunk it in as appropriate.
-+ */
-+
-+done:
-+ string_list_clear(&start_line_fields, 0);
-+
-+ /*
-+ * This is useful for debugging the request, but very noisy.
-+ */
-+ if (trace2_is_enabled()) {
-+ struct string_list_item *item;
-+ trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
-+ trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
-+ trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
-+ trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
-+ trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
-+ if (req->has_content_length)
-+ trace2_printf("%s: clen: %"PRIuMAX, TR2_CAT,
-+ req->content_length);
-+ if (req->content_type)
-+ trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
-+ for_each_string_list_item(item, &req->header_list)
-+ trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
-+ }
-+
-+ return result;
-+}
-+
-+static enum worker_result dispatch(struct req *req)
-+{
-+ return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
-+ WR_HANGUP);
-+}
-+
- static enum worker_result worker(void)
- {
-+ struct req req = REQ__INIT;
- char *client_addr = getenv("REMOTE_ADDR");
- char *client_port = getenv("REMOTE_PORT");
- enum worker_result wr = WR_OK;
-@@ t/helper/test-http-server.c: static enum worker_result worker(void)
- set_keep_alive(0, logerror);
+ ln -s "$LIB_HTTPD_MODULE_PATH" "$HTTPD_ROOT_PATH/modules"
- while (1) {
-- wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
-- NULL, WR_HANGUP);
-+ req__release(&req);
+
+ ## t/lib-httpd/apache.conf ##
+@@ t/lib-httpd/apache.conf: Alias /auth/dumb/ www/auth/dumb/
+ SetEnv GIT_HTTP_EXPORT_ALL
+ SetEnv GIT_PROTOCOL
+ </LocationMatch>
++<LocationMatch /custom_auth/>
++ SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
++ SetEnv GIT_HTTP_EXPORT_ALL
++ CGIPassAuth on
++</LocationMatch>
+ ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/
+ ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/
+ ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/
+@@ t/lib-httpd/apache.conf: ScriptAlias /broken_smart/ broken-smart-http.sh/
+ ScriptAlias /error_smart/ error-smart-http.sh/
+ ScriptAlias /error/ error.sh/
+ ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1
++ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1
+ <Directory ${GIT_EXEC_PATH}>
+ Options FollowSymlinks
+ </Directory>
+
+ ## t/lib-httpd/nph-custom-auth.sh (new) ##
+@@
++#!/bin/sh
+
-+ alarm(timeout);
-+ wr = req__read(&req, 0);
-+ alarm(0);
++VALID_CREDS_FILE=custom-auth.valid
++CHALLENGE_FILE=custom-auth.challenge
++ANONYMOUS_FILE=custom-auth.anonymous
+
-+ if (wr == WR_CLIENT_ERROR)
-+ wr = send_http_error(STDOUT_FILENO, 400, "Bad Request",
-+ -1, NULL, wr);
++#
++# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.
++#
++# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid
++# credential for the current request. Each line in the file is considered a
++# valid HTTP Authorization header value. For example:
++#
++# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
++#
++# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers
++# in a 401 response if no valid authentication credentials were included in the
++# request. For example:
++#
++# WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
++# WWW-Authenticate: Basic realm="example.com"
++#
+
-+ if (wr != WR_OK)
-+ break;
-
-+ wr = dispatch(&req);
- if (wr != WR_OK)
- break;
- }
++if test -f "$ANONYMOUS_FILE" || (test -f "$VALID_CREDS_FILE" && \
++ grep -qi "^${HTTP_AUTHORIZATION:-nopenopnope}$" "$VALID_CREDS_FILE")
++then
++ # Note that although git-http-backend returns a status line, it
++ # does so using a CGI 'Status' header. Because this script is an
++ # No Parsed Headers (NPH) script, we must return a real HTTP
++ # status line.
++ # This is only a test script, so we don't bother to check for
++ # the actual status from git-http-backend and always return 200.
++ echo 'HTTP/1.1 200 OK'
++ exec "$GIT_EXEC_PATH"/git-http-backend
++fi
++
++echo 'HTTP/1.1 401 Authorization Required'
++if test -f "$CHALLENGE_FILE"
++then
++ cat "$CHALLENGE_FILE"
++fi
++echo
- ## t/t5556-http-auth.sh (new) ##
+ ## t/t5563-simple-http-auth.sh (new) ##
@@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
-+TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
-+
-+# Setup a repository
-+#
-+REPO_DIR="$TRASH_DIRECTORY"/repo
-+
-+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
-+
-+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
-+
-+test_expect_success 'setup repos' '
-+ test_create_repo "$REPO_DIR" &&
-+ git -C "$REPO_DIR" branch -M main
++. "$TEST_DIRECTORY"/lib-httpd.sh
++
++start_httpd
++
++test_expect_success 'setup_credential_helper' '
++ mkdir -p "$TRASH_DIRECTORY/bin" &&
++ PATH=$PATH:"$TRASH_DIRECTORY/bin" &&
++ export PATH &&
++
++ CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
++ write_script "$CREDENTIAL_HELPER" <<-\EOF
++ cmd=$1
++ teefile=$cmd-query.cred
++ catfile=$cmd-reply.cred
++ sed -n -e "/^$/q" -e "p" >> $teefile
++ if test "$cmd" = "get"; then
++ cat $catfile
++ fi
++ EOF
+'
+
-+run_http_server_worker() {
-+ (
-+ cd "$REPO_DIR"
-+ test-http-server --worker "$@" 2>"$SERVER_LOG" | tr -d "\r"
-+ )
++set_credential_reply() {
++ cat >"$TRASH_DIRECTORY/$1-reply.cred"
++}
++
++expect_credential_query() {
++ cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
++ test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
++ "$TRASH_DIRECTORY/$1-query.cred"
+}
+
+per_test_cleanup () {
-+ rm -f OUT.* &&
-+ rm -f IN.* &&
++ rm -f *.cred &&
++ rm -f "$HTTPD_ROOT_PATH"/custom-auth.*
+}
+
-+test_expect_success 'http auth server request parsing' '
-+ test_when_finished "per_test_cleanup" &&
++test_expect_success 'setup repository' '
++ test_commit foo &&
++ git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
++ git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
++'
+
-+ cat >auth.config <<-EOF &&
-+ [auth]
-+ allowAnonymous = true
-+ EOF
++test_expect_success 'access anonymous no challenge' '
++ test_when_finished "per_test_cleanup" &&
++ touch "$HTTPD_ROOT_PATH/custom-auth.anonymous" &&
++ git ls-remote "$HTTPD_URL/custom_auth/repo.git"
++'
+
-+ echo "HTTP/1.1 400 Bad Request" >OUT.http400 &&
-+ echo "HTTP/1.1 200 OK" >OUT.http200 &&
++test_expect_success 'access using basic auth' '
++ test_when_finished "per_test_cleanup" &&
+
-+ cat >IN.http.valid <<-EOF &&
-+ GET /info/refs HTTP/1.1
-+ Content-Length: 0
++ set_credential_reply get <<-EOF &&
++ username=alice
++ password=secret-passwd
+ EOF
+
-+ cat >IN.http.badfirstline <<-EOF &&
-+ /info/refs GET HTTP
++ cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
++ Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+ EOF
+
-+ cat >IN.http.badhttpver <<-EOF &&
-+ GET /info/refs HTTP/999.9
++ cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
++ WWW-Authenticate: Basic realm="example.com"
+ EOF
+
-+ cat >IN.http.ltzlen <<-EOF &&
-+ GET /info/refs HTTP/1.1
-+ Content-Length: -1
-+ EOF
++ test_config_global credential.helper test-helper &&
++ git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
-+ cat >IN.http.badlen <<-EOF &&
-+ GET /info/refs HTTP/1.1
-+ Content-Length: not-a-number
++ expect_credential_query get <<-EOF &&
++ protocol=http
++ host=$HTTPD_DEST
+ EOF
+
-+ cat >IN.http.overlen <<-EOF &&
-+ GET /info/refs HTTP/1.1
-+ Content-Length: 9223372036854775807
++ expect_credential_query store <<-EOF
++ protocol=http
++ host=$HTTPD_DEST
++ username=alice
++ password=secret-passwd
+ EOF
-+
-+ run_http_server_worker \
-+ --auth-config="$TRASH_DIRECTORY/auth.config" <IN.http.valid \
-+ | head -n1 >OUT.actual &&
-+ test_cmp OUT.http200 OUT.actual &&
-+
-+ run_http_server_worker <IN.http.badfirstline | head -n1 >OUT.actual &&
-+ test_cmp OUT.http400 OUT.actual &&
-+
-+ run_http_server_worker <IN.http.ltzlen | head -n1 >OUT.actual &&
-+ test_cmp OUT.http400 OUT.actual &&
-+
-+ run_http_server_worker <IN.http.badlen | head -n1 >OUT.actual &&
-+ test_cmp OUT.http400 OUT.actual &&
-+
-+ run_http_server_worker <IN.http.overlen | head -n1 >OUT.actual &&
-+ test_cmp OUT.http400 OUT.actual
+'
+
+test_done
7: ca9c2787248 < -: ----------- test-http-server: pass Git requests to http-backend
8: b8d3e81b553 < -: ----------- test-http-server: add simple authentication
9: 2f97c94f679 < -: ----------- test-http-server: add sending of arbitrary headers
10: 4b1635b3f69 < -: ----------- http: replace unsafe size_t multiplication with st_mult
11: 5f5e46038cf ! 2: cd9a02ba94e http: read HTTP WWW-Authenticate response headers
@@ credential.h: struct credential {
/* Initialize a credential structure, setting all fields to empty. */
+ ## git-compat-util.h ##
+@@ git-compat-util.h: static inline int skip_iprefix(const char *str, const char *prefix,
+ return 0;
+ }
+
++/*
++ * Like skip_prefix_mem, but compare case-insensitively. Note that the
++ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte
++ * characters or locale-specific conversions).
++ */
++static inline int skip_iprefix_mem(const char *buf, size_t len,
++ const char *prefix,
++ const char **out, size_t *outlen)
++{
++ size_t prefix_len = strlen(prefix);
++ if (len < prefix_len)
++ return 0;
++
++ if (!strncasecmp(buf, prefix, prefix_len)){
++ *out = buf + prefix_len;
++ *outlen = len - prefix_len;
++ return 1;
++ }
++
++ return 0;
++}
++
+ static inline int strtoul_ui(char const *s, int base, unsigned int *result)
+ {
+ unsigned long ul;
+
## http.c ##
@@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
return nmemb;
}
++/*
++ * A folded header continuation line starts with at least one single whitespace
++ * character. It is not a continuation line if the line is *just* a newline.
++ * The RFC for HTTP states that CRLF is the header field line ending, but some
++ * servers may use LF only; we accept both.
++ */
++static inline int is_hdr_continuation(const char *ptr, const size_t size)
++{
++ /* totally empty line or normal header */
++ if (!size || !isspace(*ptr))
++ return 0;
++
++ /* empty line with LF line ending */
++ if (size == 1 && ptr[0] == '\n')
++ return 0;
++
++ /* empty line with CRLF line ending */
++ if (size == 2 && ptr[0] == '\r' && ptr[1] == '\n')
++ return 0;
++
++ return 1;
++}
++
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
-+ size_t size = st_mult(eltsize, nmemb);
++ size_t size = eltsize * nmemb;
+ struct strvec *values = &http_auth.wwwauth_headers;
+ struct strbuf buf = STRBUF_INIT;
+ const char *val;
++ size_t val_len;
+
+ /*
+ * Header lines may not come NULL-terminated from libcurl so we must
@@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
+ * of token, separators, and quoted-string>
+ */
+
-+ strbuf_add(&buf, ptr, size);
-+
-+ /* Strip the CRLF that should be present at the end of each field */
-+ strbuf_trim_trailing_newline(&buf);
-+
+ /* Start of a new WWW-Authenticate header */
-+ if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
-+ while (isspace(*val))
-+ val++;
++ if (skip_iprefix_mem(ptr, size, "www-authenticate:", &val, &val_len)) {
++ strbuf_add(&buf, val, val_len);
+
-+ strvec_push(values, val);
++ /*
++ * Strip the CRLF that should be present at the end of each
++ * field as well as any trailing or leading whitespace from the
++ * value.
++ */
++ strbuf_trim(&buf);
++
++ strvec_push(values, buf.buf);
+ http_auth.header_is_last_match = 1;
+ goto exit;
+ }
@@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
+ * This line could be a continuation of the previously matched header
+ * field. If this is the case then we should append this value to the
+ * end of the previously consumed value.
-+ * Continuation lines start with at least one whitespace, maybe more,
-+ * so we should collapse these down to a single SP (valid per the spec).
+ */
-+ if (http_auth.header_is_last_match && isspace(*buf.buf)) {
-+ /* Trim leading whitespace from this continuation hdr line. */
-+ strbuf_ltrim(&buf);
++ if (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {
++ /*
++ * Trim the CRLF and any leading or trailing from this line.
++ */
++ strbuf_add(&buf, ptr, size);
++ strbuf_trim(&buf);
+
+ /*
+ * At this point we should always have at least one existing
@@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff
+ * We only care about the last HTTP request response's headers so clear
+ * the existing array.
+ */
-+ if (istarts_with(buf.buf, "http/"))
++ if (!strncasecmp(ptr, "http/", 5))
+ strvec_clear(values);
+
+exit:
12: 09164f77d56 ! 3: 149aedf5501 credential: add WWW-Authenticate header to cred requests
@@ credential.c: void credential_write(const struct credential *c, FILE *fp)
static int run_credential_helper(struct credential *c,
- ## t/lib-credential-helper.sh (new) ##
-@@
-+setup_credential_helper() {
-+ test_expect_success 'setup credential helper' '
-+ CREDENTIAL_HELPER="$TRASH_DIRECTORY/credential-helper.sh" &&
-+ export CREDENTIAL_HELPER &&
-+ echo $CREDENTIAL_HELPER &&
-+
-+ write_script "$CREDENTIAL_HELPER" <<-\EOF
-+ cmd=$1
-+ teefile=$cmd-query.cred
-+ catfile=$cmd-reply.cred
-+ sed -n -e "/^$/q" -e "p" >> $teefile
-+ if test "$cmd" = "get"; then
-+ cat $catfile
-+ fi
-+ EOF
-+ '
-+}
+ ## t/t5563-simple-http-auth.sh ##
+@@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '
+ expect_credential_query get <<-EOF &&
+ protocol=http
+ host=$HTTPD_DEST
++ wwwauth[]=Basic realm="example.com"
++ EOF
+
-+set_credential_reply() {
-+ cat >"$TRASH_DIRECTORY/$1-reply.cred"
-+}
++ expect_credential_query store <<-EOF
++ protocol=http
++ host=$HTTPD_DEST
++ username=alice
++ password=secret-passwd
++ EOF
++'
+
-+expect_credential_query() {
-+ cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
-+ test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
-+ "$TRASH_DIRECTORY/$1-query.cred"
-+}
-
- ## t/t5556-http-auth.sh ##
-@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'
-
- TEST_NO_CREATE_REPO=1
- . ./test-lib.sh
-+. "$TEST_DIRECTORY"/lib-credential-helper.sh
-
- test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
-
-@@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '
- git -C "$REPO_DIR" branch -M main
- '
-
-+setup_credential_helper
-+
- run_http_server_worker() {
- (
- cd "$REPO_DIR"
-@@ t/t5556-http-auth.sh: per_test_cleanup () {
- stop_http_server &&
- rm -f OUT.* &&
- rm -f IN.* &&
-+ rm -f *.cred &&
- rm -f auth.config
- }
-
-@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
- git ls-remote $ORIGIN_URL
- '
-
-+test_expect_success 'http auth www-auth headers to credential helper basic valid' '
++test_expect_success 'access using basic auth invalid credentials' '
+ test_when_finished "per_test_cleanup" &&
-+ # base64("alice:secret-passwd")
-+ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
-+ export USERPASS64 &&
-+
-+ cat >auth.config <<-EOF &&
-+ [auth]
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
++
++ set_credential_reply get <<-EOF &&
++ username=baduser
++ password=wrong-passwd
+ EOF
+
-+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
++ cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
++ Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
++ EOF
+
-+ set_credential_reply get <<-EOF &&
-+ protocol=http
-+ host=$HOST_PORT
-+ username=alice
-+ password=secret-passwd
++ cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
++ WWW-Authenticate: Basic realm="example.com"
+ EOF
+
-+ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++ test_config_global credential.helper test-helper &&
++ test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+ expect_credential_query get <<-EOF &&
+ protocol=http
-+ host=$HOST_PORT
-+ wwwauth[]=basic realm="example.com"
++ host=$HTTPD_DEST
++ wwwauth[]=Basic realm="example.com"
+ EOF
+
-+ expect_credential_query store <<-EOF
++ expect_credential_query erase <<-EOF
+ protocol=http
-+ host=$HOST_PORT
-+ username=alice
-+ password=secret-passwd
++ host=$HTTPD_DEST
++ username=baduser
++ password=wrong-passwd
++ wwwauth[]=Basic realm="example.com"
+ EOF
+'
+
-+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '
++test_expect_success 'access using basic auth with extra challenges' '
+ test_when_finished "per_test_cleanup" &&
-+ # base64("alice:secret-passwd")
-+ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
-+ export USERPASS64 &&
-+
-+ cat >auth.config <<-EOF &&
-+ [auth]
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
-+ extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\"id.example.com\"
-+ EOF
-+
-+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ set_credential_reply get <<-EOF &&
-+ protocol=http
-+ host=$HOST_PORT
+ username=alice
+ password=secret-passwd
+ EOF
+
-+ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++ cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
++ Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
++ EOF
++
++ cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
++ WWW-Authenticate: FooBar param1="value1" param2="value2"
++ WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
++ WWW-Authenticate: Basic realm="example.com"
++ EOF
++
++ test_config_global credential.helper test-helper &&
++ git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+ expect_credential_query get <<-EOF &&
+ protocol=http
-+ host=$HOST_PORT
-+ wwwauth[]=basic realm="example.com"
-+ wwwauth[]=bEaRer auThoRiTy="id.example.com"
++ host=$HTTPD_DEST
++ wwwauth[]=FooBar param1="value1" param2="value2"
++ wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
++ wwwauth[]=Basic realm="example.com"
+ EOF
+
+ expect_credential_query store <<-EOF
+ protocol=http
-+ host=$HOST_PORT
++ host=$HTTPD_DEST
+ username=alice
+ password=secret-passwd
+ EOF
+'
+
-+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '
++test_expect_success 'access using basic auth mixed-case wwwauth header name' '
+ test_when_finished "per_test_cleanup" &&
-+ # base64("alice:secret-passwd")
-+ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
-+ export USERPASS64 &&
-+
-+ cat >auth.config <<-EOF &&
-+ [auth]
-+ challenge = "bearer:authority=\"id.example.com\"\\n q=1\\n \\t p=0"
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
-+ EOF
-+
-+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ set_credential_reply get <<-EOF &&
-+ protocol=http
-+ host=$HOST_PORT
+ username=alice
+ password=secret-passwd
+ EOF
+
-+ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++ cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
++ Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
++ EOF
++
++ cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
++ www-authenticate: foobar param1="value1" param2="value2"
++ WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
++ WwW-aUtHeNtIcAtE: baSiC realm="example.com"
++ EOF
++
++ test_config_global credential.helper test-helper &&
++ git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+ expect_credential_query get <<-EOF &&
+ protocol=http
-+ host=$HOST_PORT
-+ wwwauth[]=bearer authority="id.example.com" q=1 p=0
-+ wwwauth[]=basic realm="example.com"
++ host=$HTTPD_DEST
++ wwwauth[]=foobar param1="value1" param2="value2"
++ wwwauth[]=BEARER authorize_uri="id.example.com" p=1 q=0
++ wwwauth[]=baSiC realm="example.com"
+ EOF
+
+ expect_credential_query store <<-EOF
+ protocol=http
-+ host=$HOST_PORT
++ host=$HTTPD_DEST
+ username=alice
+ password=secret-passwd
+ EOF
+'
+
-+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '
++test_expect_success 'access using basic auth with wwwauth header continuations' '
+ test_when_finished "per_test_cleanup" &&
-+ # base64("alice:secret-passwd")
-+ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
-+ export USERPASS64 &&
-+
-+ cat >auth.config <<-EOF &&
-+ [auth]
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
-+ extraheader = "WWW-Authenticate:"
-+ extraheader = " "
-+ extraheader = " bearer authority=\"id.example.com\""
-+ EOF
-+
-+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ set_credential_reply get <<-EOF &&
-+ protocol=http
-+ host=$HOST_PORT
+ username=alice
+ password=secret-passwd
+ EOF
+
-+ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++ cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
++ Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
++ EOF
++
++ # Note that leading and trailing whitespace is important to correctly
++ # simulate a continuation/folded header.
++ cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
++ WWW-Authenticate: FooBar param1="value1"
++ param2="value2"
++ WWW-Authenticate: Bearer authorize_uri="id.example.com"
++ p=1
++ q=0
++ WWW-Authenticate: Basic realm="example.com"
++ EOF
++
++ test_config_global credential.helper test-helper &&
++ git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+ expect_credential_query get <<-EOF &&
+ protocol=http
-+ host=$HOST_PORT
-+ wwwauth[]=basic realm="example.com"
-+ wwwauth[]=bearer authority="id.example.com"
++ host=$HTTPD_DEST
++ wwwauth[]=FooBar param1="value1" param2="value2"
++ wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
++ wwwauth[]=Basic realm="example.com"
+ EOF
+
+ expect_credential_query store <<-EOF
+ protocol=http
-+ host=$HOST_PORT
++ host=$HTTPD_DEST
+ username=alice
+ password=secret-passwd
+ EOF
+'
+
-+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '
++test_expect_success 'access using basic auth with wwwauth header empty continuations' '
+ test_when_finished "per_test_cleanup" &&
-+ # base64("alice:secret-passwd")
-+ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
-+ export USERPASS64 &&
-+
-+ cat >auth.config <<-EOF &&
-+ [auth]
-+ challenge = "foobar:alg=test widget=1"
-+ challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
-+ EOF
-+
-+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ set_credential_reply get <<-EOF &&
-+ protocol=http
-+ host=$HOST_PORT
+ username=alice
+ password=secret-passwd
+ EOF
+
-+ git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
++ cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
++ Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
++ EOF
++
++ CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
++
++ # Note that leading and trailing whitespace is important to correctly
++ # simulate a continuation/folded header.
++ printf "">$CHALLENGE &&
++ printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
++ printf " \r\n" >>$CHALLENGE &&
++ printf " param2=\"value2\"\r\n" >>$CHALLENGE &&
++ printf "WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>$CHALLENGE &&
++ printf " p=1\r\n" >>$CHALLENGE &&
++ printf " \r\n" >>$CHALLENGE &&
++ printf " q=0\r\n" >>$CHALLENGE &&
++ printf "WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>$CHALLENGE &&
++
++ test_config_global credential.helper test-helper &&
++ git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+ expect_credential_query get <<-EOF &&
+ protocol=http
-+ host=$HOST_PORT
-+ wwwauth[]=foobar alg=test widget=1
-+ wwwauth[]=bearer authority="id.example.com" q=1 p=0
-+ wwwauth[]=basic realm="example.com"
++ host=$HTTPD_DEST
++ wwwauth[]=FooBar param1="value1" param2="value2"
++ wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
++ wwwauth[]=Basic realm="example.com"
+ EOF
+
+ expect_credential_query store <<-EOF
+ protocol=http
-+ host=$HOST_PORT
++ host=$HTTPD_DEST
+ username=alice
+ password=secret-passwd
+ EOF
+'
+
-+test_expect_success 'http auth www-auth headers to credential helper invalid' '
++test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '
+ test_when_finished "per_test_cleanup" &&
-+ # base64("alice:secret-passwd")
-+ USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&
-+ export USERPASS64 &&
-+
-+ cat >auth.config <<-EOF &&
-+ [auth]
-+ challenge = "bearer:authority=\"id.example.com\" q=1 p=0"
-+ challenge = basic:realm=\"example.com\"
-+ token = basic:$USERPASS64
-+ EOF
-+
-+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ set_credential_reply get <<-EOF &&
-+ protocol=http
-+ host=$HOST_PORT
+ username=alice
-+ password=invalid-passwd
++ password=secret-passwd
+ EOF
+
-+ test_must_fail git -c "credential.helper=!\"$CREDENTIAL_HELPER\"" ls-remote $ORIGIN_URL &&
-+
-+ expect_credential_query get <<-EOF &&
-+ protocol=http
-+ host=$HOST_PORT
-+ wwwauth[]=bearer authority="id.example.com" q=1 p=0
-+ wwwauth[]=basic realm="example.com"
++ cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
++ Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+ EOF
+
-+ expect_credential_query erase <<-EOF
-+ protocol=http
-+ host=$HOST_PORT
-+ username=alice
-+ password=invalid-passwd
-+ wwwauth[]=bearer authority="id.example.com" q=1 p=0
-+ wwwauth[]=basic realm="example.com"
-+ EOF
-+'
++ CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
++
++ # Note that leading and trailing whitespace is important to correctly
++ # simulate a continuation/folded header.
++ printf "">$CHALLENGE &&
++ printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >$CHALLENGE &&
++ printf " \r\n" >>$CHALLENGE &&
++ printf "\tparam2=\"value2\"\r\n" >>$CHALLENGE &&
++ printf "WWW-Authenticate: Basic realm=\"example.com\"" >>$CHALLENGE &&
+
- test_done
++ test_config_global credential.helper test-helper &&
++ git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
++
++ expect_credential_query get <<-EOF &&
++ protocol=http
++ host=$HTTPD_DEST
++ wwwauth[]=FooBar param1="value1" param2="value2"
++ wwwauth[]=Basic realm="example.com"
+ EOF
+
+ expect_credential_query store <<-EOFSignature · 1 line
gitgitgadget