Skip to contentSkip to conversation

gitweb: safely output binary files for 'blob_plain' action

PATCH In review Jakub Narebski 3 participants 6 messages Jun 15, 2016

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 gitweb: safely output binary files for 'blob_plain' action

gitweb tries now to output correct Content-Type header for 'blob_plain' action; for now text/plain for text files, appropriate image MIME type for *.png, *.gif and *.jpg/*.jpeg files, and application/octet-stream for other binary files.

Introduced new configuration variables: $default_blob_plain_mimetype and $default_text_plain_charset (only 'utf-8' is guaranteed to work for the latter).

binmode changed to ':raw' in git_blob_plain for output of non-text files.

---
 gitweb/gitweb.cgi |   43 +++++++++++++++++++++++++++++++++++++++----
 1 files changed, 39 insertions(+), 4 deletions(-)
gitweb/gitweb.cgi +39 -4
index 9d902b7..b37ec50 100755
@@ -39,12 +39,16 @@ # html text to include at home page
 my $home_text =		"indextext.html";
 
 # URI of default stylesheet
-my $stylesheet = 	"gitweb.css";
+my $stylesheet =	"gitweb.css";
 
 # source of projects list
 #my $projects_list =	$projectroot;
 my $projects_list =	"index/index.aux";
 
+# default blob_plain mimetype and default charset for text/plain blob
+my $default_blob_plain_mimetype = 'text/plain';
+my $default_text_plain_charset  = 'utf-8';	# can be undefined
+
 # input validation and dispatch
 my $action = $cgi->param('a');
 if (defined $action) {
@@ -1354,15 +1358,46 @@ sub git_blob {
 	git_footer_html();
 }
 
+sub git_blob_plain_mimetype {
+	my $fd = shift;
+	my $filename = shift;
+
+	# just in case
+	return $default_blob_plain_mimetype unless $fd;
+
+	if (-T $fd) {
+		return 'text/plain' .
+		       ($default_text_plain_charset ? '; charset='.$default_text_plain_charset : '');
+	} elsif (! $filename) {
+		return 'application/octet-stream';
+	} elsif ($filename =~ m/\.png$/i) {
+		return 'image/png';
+	} elsif ($filename =~ m/\.gif$/i) {
+		return 'image/gif';
+	} elsif ($filename =~ m/\.jpe?g$/i) {
+		return 'image/jpeg';
+	} else {
+		return 'application/octet-stream';
+	}
+}
+
 sub git_blob_plain {
-	my $save_as = "$hash.txt";
+	open my $fd, "-|", "$gitbin/git-cat-file blob $hash" or return;
+	my $type = git_blob_plain_mimetype($fd, $file_name);
+	
+	# save as filename, even when no $file_name is given
+	my $save_as = "$hash";
 	if (defined $file_name) {
 		$save_as = $file_name;
+	} elsif ($type =~ m/^text\//) {
+		$save_as .= '.txt';
 	}
-	print $cgi->header(-type => "text/plain", -charset => 'utf-8', '-content-disposition' => "inline; filename=\"$save_as\"");
-	open my $fd, "-|", "$gitbin/git-cat-file blob $hash" or return;
+	
+	print $cgi->header(-type => "$type", '-content-disposition' => "inline; filename=\"$save_as\"");
 	undef $/;
+	binmode STDOUT, ':raw' unless $type =~ m/^text\//;
 	print <$fd>;
+	binmode STDOUT, ':utf8' unless $type =~ m/^text\//;
 	$/ = "\n";
 	close $fd;
 }
Signature · 1 line
1.3.0
#1 lore raw

Dear diary, on Sat, Jun 17, 2006 at 01:32:15PM CEST, I got a letter where Jakub Narebski <[email protected]> said that...

Quoted 3 lines

Introduced new configuration variables: $default_blob_plain_mimetype and $default_text_plain_charset (only 'utf-8' is guaranteed to work for the latter).

Nah, defaulting to 'utf-8' is horrible - usually, you just don't have a clue and should refrain from sending any charset information at all, so I think undef is a much saner default.

Signature · 3 lines
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
A person is just about as big as the things that make them angry.
#2 lore raw
Quoted 9 lines · Petr Baudis

Dear diary, on Sat, Jun 17, 2006 at 01:32:15PM CEST, I got a letter where Jakub Narebski <[email protected]> said that...

Quoted 3 lines

Introduced new configuration variables: $default_blob_plain_mimetype and $default_text_plain_charset (only 'utf-8' is guaranteed to work for the latter).

Nah, defaulting to 'utf-8' is horrible - usually, you just don't have a clue and should refrain from sending any charset information at all, so I think undef is a much saner default.

Concurred. I see Jakub's second patch to make this configurable, but I wonder about a few things:

- we might want to have a configuration mechanism in place before enhancing gitweb. My gut feeling is that we can use [gitweb] section in project.git/config (and probably duplicate first and deprecate later existing "description" as well).

- the blob charset should be per path -- otherwise the feature would be not useful for projects that maintains bunch of po files.

In other words, something like this:

(in torvalds/linux-2.6.git/config)

[gitweb]
description = "Linus's kernel tree"
; defaultblobcharset = "latin1"
blobmimemapfile = "mime-map"

(in torvalds/linux-2.6.git/mime-map, first match decides)

fs/nls/nls_euc-jp.c text/plain; charset=euc_jp
*.c text/plain; charset=utf-8
*.h text/plain; charset=utf-8

I do not think defaultblobcharset above is a good idea though. You could just have the last entry in mime-map file to be:

* text/plain; charset=latin1

#3 lore raw

Dear diary, on Sat, Jun 17, 2006 at 11:13:30PM CEST, I got a letter where Junio C Hamano <[email protected]> said that...

Quoted 5 lines

- we might want to have a configuration mechanism in place before enhancing gitweb. My gut feeling is that we can use [gitweb] section in project.git/config (and probably duplicate first and deprecate later existing "description" as well).

Agreed. (I planned to back this up with a patch, then looked at the clock.)

Hmm, after I'm over my exam period, since there's now another .pl thing in the git tree I might start working on some kind of universal Git.pm interface. I'm gonna need it for Cogito in the longer term anyway. ;-)

Quoted 13 lines

- the blob charset should be per path -- otherwise the feature would be not useful for projects that maintains bunch of po files.

In other words, something like this:

(in torvalds/linux-2.6.git/config)

[gitweb]
description = "Linus's kernel tree"
; defaultblobcharset = "latin1"
blobmimemapfile = "mime-map"

Quoted 5 lines

(in torvalds/linux-2.6.git/mime-map, first match decides)

fs/nls/nls_euc-jp.c text/plain; charset=euc_jp
*.c text/plain; charset=utf-8
*.h text/plain; charset=utf-8

You could as well just support the mime.types format and load /etc/mime.types for this kind of mapping (see below for a patch). The advantage is that this pretty much covers all the MIME types you will need, the disadvantage is that it's less flexible and the charset part wouldn't probably fit in nicely.

We could obviously do both. :-)

---
[PATCH] Support for the standard mime.types map in gitweb

gitweb will try to look up the filename mimetype in /etc/mime.types and optionally a user-configured mime.types map as well.

---

Depends on Jakub's mime patches.

 gitweb/gitweb.cgi |   44 ++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 44 insertions(+), 0 deletions(-)
gitweb/gitweb.cgi +44 -0
index 9250548..0116531 100755
@@ -46,6 +46,11 @@ # default blob_plain mimetype and defaul
 my $default_blob_plain_mimetype = 'text/plain';
 my $default_text_plain_charset  = undef;       # was: 'utf-8'
 
+# file to use for guessing MIME types before trying /etc/mime.types
+# (relative to the current git repository)
+my $mimetypes_file              = undef;
+
+
 # input validation and dispatch
 my $action = $cgi->param('a');
 if (defined $action) {
@@ -1414,6 +1419,40 @@ sub git_blob {
 	git_footer_html();
 }
 
+sub mimetype_guess_file {
+	my $filename = shift;
+	my $mimemap = shift;
+	-r $mimemap or return undef;
+
+	my %mimemap;
+	open(MIME, $mimemap) or return undef;
+	while (<MIME>) {
+		my ($mime, $exts) = split(/\t+/);
+		my @exts = split(/\s+/, $exts);
+		foreach my $ext (@exts) {
+			$mimemap{$ext} = $mime;
+		}
+	}
+	close(MIME);
+
+	$filename =~ /\.(.*?)$/;
+	return $mimemap{$1};
+}
+
+sub mimetype_guess {
+	my $filename = shift;
+	my $mime;
+	$filename =~ /\./ or return undef;
+
+	if ($mimetypes_file) {
+		my $file = $mimetypes_file;
+		$file =~ m#^/# or $file = "$projectroot/$path/$file";
+		$mime = mimetype_guess_file($filename, $file);
+	}
+	$mime ||= mimetype_guess_file($filename, '/etc/mime.types');
+	return $mime;
+}
+
 sub git_blob_plain_mimetype {
 	my $fd = shift;
 	my $filename = shift;
@@ -1421,6 +1460,11 @@ sub git_blob_plain_mimetype {
 	# just in case
 	return $default_blob_plain_mimetype unless $fd;
 
+	if ($filename) {
+		my $mime = mimetype_guess($filename);
+		$mime and return $mime;
+	}
+
 	if (-T $fd) {
 		return 'text/plain' .
 		       ($default_text_plain_charset ? '; charset='.$default_text_plain_charset : '');
Signature · 3 lines
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
A person is just about as big as the things that make them angry.
#4 lore raw
Quoted 15 lines · Petr Baudis
Quoted 9 lines

In other words, something like this:

(in torvalds/linux-2.6.git/config)

[gitweb]
description = "Linus's kernel tree"
; defaultblobcharset = "latin1"
blobmimemapfile = "mime-map"

Quoted 5 lines

(in torvalds/linux-2.6.git/mime-map, first match decides)

fs/nls/nls_euc-jp.c text/plain; charset=euc_jp
*.c text/plain; charset=utf-8
*.h text/plain; charset=utf-8

Quoted 5 lines

You could as well just support the mime.types format and load /etc/mime.types for this kind of mapping (see below for a patch). The advantage is that this pretty much covers all the MIME types you will need, the disadvantage is that it's less flexible and the charset part wouldn't probably fit in nicely.

Ah, I thought Jakub's patch was already taking care of mime.types but apparently that was not the case. As you say, using /etc/mime.types for this is obviously a good point to start.

Quoted 1 line

We could obviously do both. :-)

The point of my example was about charset part; comparing the suffix part only is not good enough for .po files, so we should obviously do both.

#5 lore raw

PATCH 1/1 gitweb: text files for 'blob_plain' action without charset by default

In reply to Petr Baudis

$default_text_plain_charset is undefined (no specified charset) by default. Additionally ':raw' layer for binmode is used for outputting file content.

---
This patch depends on the previous patch in the thread:
"gitweb: safely output binary files for 'blob_plain' action"
Contrary to the previous patch it is not based on other unrelated gitweb.cgi patches (this changes only line numbers in patch).

 gitweb/gitweb.cgi |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

53209981db06a5dde7c59caada279bf63d329da8

gitweb/gitweb.cgi +3 -3
index acac1f4..f082e5d 100755
@@ -44,7 +44,7 @@ my $projects_list =   "index/index.aux";
 
 # default blob_plain mimetype and default charset for text/plain blob
 my $default_blob_plain_mimetype = 'text/plain';
-my $default_text_plain_charset  = 'utf-8';     # can be undefined
+my $default_text_plain_charset  = undef;       # was: 'utf-8'
 
 # input validation and dispatch
 my $action = $cgi->param('a');
@@ -1451,9 +1451,9 @@ sub git_blob_plain {
 
        print $cgi->header(-type => "$type", '-content-disposition' => "inline; filename=\"$save_as\"");
        undef $/;
-       binmode STDOUT, ':raw' unless $type =~ m/^text\//;
+       binmode STDOUT, ':raw';
        print <$fd>;
-       binmode STDOUT, ':utf8' unless $type =~ m/^text\//;
+       binmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi
        $/ = "\n";
        close $fd;
 }
Signature · 1 line
1.3.0