Skip to contentSkip to conversation

apt 0.7.14: runtime error fix when tmpdir is very long

PATCH Applied Robert Yang 2 participants 6 messages Jul 4, 2012 – Jul 9, 2012

lore
Share & tools
lore
#0 lore raw

cover apt 0.7.14: runtime error fix when tmpdir is very long

The following changes since commit a6b6df1350149c116050cb93c3c7b4802c709d31:

task-core-tools-debug: Added openssh-sftp-server. (2012-07-03 14:52:38 +0100)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib robert/apt
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/apt

Robert Yang (2):
apt 0.7.14: runtime error: filename too long (tmpdir length) apt 0.7.14: runtime error: Method file has died unexpectedly

 .../apt/apt-0.7.14/allocate-larger-memory.patch    |   75 ++++++++++++++++++++
 .../apt/apt-0.7.14/truncate-filename.patch         |   35 +++++++++
 meta/recipes-devtools/apt/apt-native_0.7.14.bb     |    2 +-
 meta/recipes-devtools/apt/apt.inc                  |    2 +
 meta/recipes-devtools/apt/apt_0.7.14.bb            |    2 +-
 5 files changed, 114 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-devtools/apt/apt-0.7.14/allocate-larger-memory.patch
 create mode 100644 meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch
#1 lore raw

PATCH 1/2 apt 0.7.14: runtime error: filename too long (tmpdir length)

when the tmpdir dir is longer than 220, there is no files saved in tmp/sysroots/x86_64-linux/var/lib/apt/lists/ after run apt-get update, this is because apt-get uses the path as the file name, but the file name can't be longer than 255 according to /usr/include/linux/limits.h.

[YOCTO #2688]

---
 .../apt/apt-0.7.14/truncate-filename.patch         |   35 ++++++++++++++++++++
 meta/recipes-devtools/apt/apt-native_0.7.14.bb     |    2 +-
 meta/recipes-devtools/apt/apt.inc                  |    1 +
 meta/recipes-devtools/apt/apt_0.7.14.bb            |    2 +-
 4 files changed, 38 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch
meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch new file +35 -0
new file mode 100644
index 0000000..db1c42b
@@ -0,0 +1,35 @@
+strutl.cc: the filename can't be longer than 255
+
+The URItoFileName translates the path into the filename, but the
+filename can't be longer than 255 according to
+/usr/include/linux/limits.h.
+
+Truncate it when it is longer than 240 (leave some spaces for
+".Packages" and "._Release" suffix)
+
+Upstream-Status: Pending
+Signed-off-by: Robert Yang <[email protected]>
+---
+ apt-pkg/contrib/strutl.cc |    7 ++++++-
+ 1 file changed, 6 insertions(+), 1 deletion(-)
+
+diff --git a/apt-pkg/contrib/strutl.cc b/apt-pkg/contrib/strutl.cc
+--- a/apt-pkg/contrib/strutl.cc
++++ b/apt-pkg/contrib/strutl.cc
+@@ -399,7 +399,12 @@ string URItoFileName(const string &URI)
+    // "\x00-\x20{}|\\\\^\\[\\]<>\"\x7F-\xFF";
+    string NewURI = QuoteString(U,"\\|{}[]<>\"^~_=!@#$%^&*");
+    replace(NewURI.begin(),NewURI.end(),'/','_');
+-   return NewURI;
++
++   // Truncate from the head when it is longer than 240
++   if(NewURI.length() > 240)
++       return NewURI.substr(NewURI.length() - 240, NewURI.length() - 1);
++   else
++       return NewURI;
+ }
+ 									/*}}}*/
+ // Base64Encode - Base64 Encoding routine for short strings		/*{{{*/
+-- 
+1.7.10.4
+
meta/recipes-devtools/apt/apt-native_0.7.14.bb +1 -1
index ca5476b..38e1fe6 100644
@@ -1,6 +1,6 @@
 require apt-native.inc
 
-PR = "r8"
+PR = "r9"
 
 SRC_URI += "file://nodoc.patch \
             file://noconfigure.patch \
meta/recipes-devtools/apt/apt.inc +1 -0
index 1f6343a..563eda1 100644
@@ -8,6 +8,7 @@ SRC_URI = "${DEBIAN_MIRROR}/main/a/apt/apt_${PV}.tar.gz \
            file://localefixes.patch \
            file://makerace.patch \
            file://remove-redeclaration.patch \
+           file://truncate-filename.patch \
            "
 
 inherit autotools gettext
meta/recipes-devtools/apt/apt_0.7.14.bb +1 -1
index a627728..12dd1f2 100644
@@ -3,7 +3,7 @@ RDEPENDS_${PN} = "dpkg"
 LIC_FILES_CHKSUM = "file://COPYING.GPL;md5=0636e73ff0215e8d672dc4c32c317bb3"
 require apt.inc
 
-PR = "r13"
+PR = "r14"
 
 SRC_URI += "file://nodoc.patch \
             file://includes-fix.patch "
Signature · 1 line
1.7.1
#2 lore raw

Re: [PATCH 1/2] apt 0.7.14: runtime error: filename too long (tmpdir length)

Quoted 41 lines · Robert Yang

when the tmpdir dir is longer than 220, there is no files saved in tmp/sysroots/x86_64-linux/var/lib/apt/lists/ after run apt-get update, this is because apt-get uses the path as the file name, but the file name can't be longer than 255 according to /usr/include/linux/limits.h.

[YOCTO #2688]

---
  .../apt/apt-0.7.14/truncate-filename.patch         |   35 ++++++++++++++++++++
  meta/recipes-devtools/apt/apt-native_0.7.14.bb     |    2 +-
  meta/recipes-devtools/apt/apt.inc                  |    1 +
  meta/recipes-devtools/apt/apt_0.7.14.bb            |    2 +-
  4 files changed, 38 insertions(+), 2 deletions(-)

create mode 100644 meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch

meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch new file +19 -0
new file mode 100644
index 0000000..db1c42b
@@ -0,0 +1,35 @@
+strutl.cc: the filename can't be longer than 255
+
+The URItoFileName translates the path into the filename, but the
+filename can't be longer than 255 according to
+/usr/include/linux/limits.h.
+
+Truncate it when it is longer than 240 (leave some spaces for
+".Packages" and "._Release" suffix)
+
+Upstream-Status: Pending
+Signed-off-by: Robert Yang <[email protected]>
+---
+ apt-pkg/contrib/strutl.cc |    7 ++++++-
+ 1 file changed, 6 insertions(+), 1 deletion(-)
+
+diff --git a/apt-pkg/contrib/strutl.cc b/apt-pkg/contrib/strutl.cc
+--- a/apt-pkg/contrib/strutl.cc
++++ b/apt-pkg/contrib/strutl.cc
+@@ -399,7 +399,12 @@ string URItoFileName(const string &URI)

meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch line 29

Quoted 10 lines
meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch new file +10 -0
+    // "\x00-\x20{}|\\\\^\\[\\]<>\"\x7F-\xFF";
+    string NewURI = QuoteString(U,"\\|{}[]<>\"^~_=!@#$%^&*");
+    replace(NewURI.begin(),NewURI.end(),'/','_');
+-   return NewURI;
++
++   // Truncate from the head when it is longer than 240
++   if(NewURI.length() > 240)
++       return NewURI.substr(NewURI.length() - 240, NewURI.length() - 1);
++   else
++       return NewURI;

Robert, I am not super familiar with this part of the DPKG code, what uses the URItoFileName() and what effect does the truncation have?

Sau!

Quoted 44 lines
meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch new file +6 -0
+ }
+ 									/*}}}*/
+ // Base64Encode - Base64 Encoding routine for short strings		/*{{{*/
+--
+1.7.10.4
+
meta/recipes-devtools/apt/apt-native_0.7.14.bb +1 -1
index ca5476b..38e1fe6 100644
@@ -1,6 +1,6 @@
  require apt-native.inc
 
-PR = "r8"
+PR = "r9"
 
  SRC_URI += "file://nodoc.patch \
              file://noconfigure.patch \
meta/recipes-devtools/apt/apt.inc +1 -0
index 1f6343a..563eda1 100644
@@ -8,6 +8,7 @@ SRC_URI = "${DEBIAN_MIRROR}/main/a/apt/apt_${PV}.tar.gz \
             file://localefixes.patch \
             file://makerace.patch \
             file://remove-redeclaration.patch \
+           file://truncate-filename.patch \
             "
 
  inherit autotools gettext
meta/recipes-devtools/apt/apt_0.7.14.bb +1 -1
index a627728..12dd1f2 100644
@@ -3,7 +3,7 @@ RDEPENDS_${PN} = "dpkg"
  LIC_FILES_CHKSUM = "file://COPYING.GPL;md5=0636e73ff0215e8d672dc4c32c317bb3"
  require apt.inc
 
-PR = "r13"
+PR = "r14"
 
  SRC_URI += "file://nodoc.patch \
              file://includes-fix.patch "
#3 lore raw

Re: [PATCH 1/2] apt 0.7.14: runtime error: filename too long (tmpdir length)

Quoted 48 lines · Saul Wold
Quoted 47 lines · Robert Yang

when the tmpdir dir is longer than 220, there is no files saved in tmp/sysroots/x86_64-linux/var/lib/apt/lists/ after run apt-get update, this is because apt-get uses the path as the file name, but the file name can't be longer than 255 according to /usr/include/linux/limits.h.

[YOCTO #2688]

---
  .../apt/apt-0.7.14/truncate-filename.patch         |   35 ++++++++++++++++++++
  meta/recipes-devtools/apt/apt-native_0.7.14.bb     |    2 +-
  meta/recipes-devtools/apt/apt.inc                  |    1 +
  meta/recipes-devtools/apt/apt_0.7.14.bb            |    2 +-
  4 files changed, 38 insertions(+), 2 deletions(-)

create mode 100644 meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch

meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch new file +24 -0
new file mode 100644
index 0000000..db1c42b
@@ -0,0 +1,35 @@
+strutl.cc: the filename can't be longer than 255
+
+The URItoFileName translates the path into the filename, but the
+filename can't be longer than 255 according to
+/usr/include/linux/limits.h.
+
+Truncate it when it is longer than 240 (leave some spaces for
+".Packages" and "._Release" suffix)
+
+Upstream-Status: Pending
+Signed-off-by: Robert Yang <[email protected]>
+---
+ apt-pkg/contrib/strutl.cc |    7 ++++++-
+ 1 file changed, 6 insertions(+), 1 deletion(-)
+
+diff --git a/apt-pkg/contrib/strutl.cc b/apt-pkg/contrib/strutl.cc
+--- a/apt-pkg/contrib/strutl.cc
++++ b/apt-pkg/contrib/strutl.cc
+@@ -399,7 +399,12 @@ string URItoFileName(const string &URI)
+    // "\x00-\x20{}|\\\\^\\[\\]<>\"\x7F-\xFF";
+    string NewURI = QuoteString(U,"\\|{}[]<>\"^~_=!@#$%^&*");
+    replace(NewURI.begin(),NewURI.end(),'/','_');
+-   return NewURI;
++
Quoted 9 lines

meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch line 29

Quoted 5 lines
meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch new file +5 -0
++   // Truncate from the head when it is longer than 240
++   if(NewURI.length() > 240)
++       return NewURI.substr(NewURI.length() - 240, NewURI.length() - 1);
++   else
++       return NewURI;

Robert, I am not super familiar with this part of the DPKG code, what uses the URItoFileName() and what effect does the truncation have?

Hi Saul,

The filename which is used /var/lib/apt/lists/ is from URItoFileName(), for example, there is a line in my /etc/apt/sources.list:

deb http://mirrors.163.com/ubuntu/ maverick main restricted universe multiverse

Then the URItoFileName will change this line to a filename:

[snip] mirrors.163.com_ubuntu_dists_maverick_main_binary-i386_Packages mirrors.163.com_ubuntu_dists_maverick_restricted_binary-i386_Packages [snip]

When the filename is longer than 240, truncate the head of them, e.g.:

ubuntu_dists_maverick_main_binary-i386_Packages ubuntu_dists_maverick_restricted_binary-i386_Packages

I think this is fine for us since the end part of the path is uniq. Another solution is that use the md5sum for the filename no matter how long it is, or use the "translated_path+part_of_the_md5sum", this is easy to do, but it is a little big change to apt, and we don't need.

// Robert

Quoted 52 lines

Sau!

Quoted 47 lines
meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch new file +6 -0
+ }
+                                     /*}}}*/
+ // Base64Encode - Base64 Encoding routine for short strings        /*{{{*/
+--
+1.7.10.4
+
meta/recipes-devtools/apt/apt-native_0.7.14.bb +1 -1
index ca5476b..38e1fe6 100644
@@ -1,6 +1,6 @@
  require apt-native.inc
 
-PR = "r8"
+PR = "r9"
 
  SRC_URI += "file://nodoc.patch \
              file://noconfigure.patch \
meta/recipes-devtools/apt/apt.inc +1 -0
index 1f6343a..563eda1 100644
@@ -8,6 +8,7 @@ SRC_URI = "${DEBIAN_MIRROR}/main/a/apt/apt_${PV}.tar.gz \
             file://localefixes.patch \
             file://makerace.patch \
             file://remove-redeclaration.patch \
+           file://truncate-filename.patch \
             "
 
  inherit autotools gettext
meta/recipes-devtools/apt/apt_0.7.14.bb +1 -1
index a627728..12dd1f2 100644
@@ -3,7 +3,7 @@ RDEPENDS_${PN} = "dpkg"
  LIC_FILES_CHKSUM = "file://COPYING.GPL;md5=0636e73ff0215e8d672dc4c32c317bb3"
  require apt.inc
 
-PR = "r13"
+PR = "r14"
 
  SRC_URI += "file://nodoc.patch \
              file://includes-fix.patch "
#4 lore raw

PATCH 2/2 apt 0.7.14: runtime error: Method file has died unexpectedly

In reply to Robert Yang

When the length of the tmpdir is longer than 400, there is an error when run "apt-get update":

Method file has died unexpectedly!

This is because the "char S[1024]" is not enough for long URI, S[2048] would be enough.

[YOCTO #2689]

---
 .../apt/apt-0.7.14/allocate-larger-memory.patch    |   75 ++++++++++++++++++++
 meta/recipes-devtools/apt/apt-native_0.7.14.bb     |    2 +-
 meta/recipes-devtools/apt/apt.inc                  |    1 +
 meta/recipes-devtools/apt/apt_0.7.14.bb            |    2 +-
 4 files changed, 78 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-devtools/apt/apt-0.7.14/allocate-larger-memory.patch
meta/recipes-devtools/apt/apt-0.7.14/allocate-larger-memory.patch new file +75 -0
new file mode 100644
index 0000000..36e1499
@@ -0,0 +1,75 @@
+Method file has died unexpectedly
+
+"Method file has died unexpectedly!", this is because the "char S[1024]"
+is not enough for the long the URI, "char S[2048]" would be enough.
+
+It would be boring if we use malloc here since we can't know how much
+memory is needed except strelen() every component of it. So similarly
+use "char S[2048]" as it did before.
+
+Upstream-Status: Pending
+Signed-off-by: Robert Yang <[email protected]>
+---
+ apt-pkg/acquire-method.cc |   12 ++++++------
+ 1 file changed, 6 insertions(+), 6 deletions(-)
+
+diff --git a/apt-pkg/acquire-method.cc b/apt-pkg/acquire-method.cc
+--- a/apt-pkg/acquire-method.cc
++++ b/apt-pkg/acquire-method.cc
+@@ -95,7 +95,7 @@ void pkgAcqMethod::Fail(string Err,bool Transient)
+ 	 *I = ' ';
+    }
+    
+-   char S[1024];
++   char S[2048];
+    if (Queue != 0)
+    {
+       snprintf(S,sizeof(S)-50,"400 URI Failure\nURI: %s\n"
+@@ -132,7 +132,7 @@ void pkgAcqMethod::URIStart(FetchResult &Res)
+    if (Queue == 0)
+       abort();
+    
+-   char S[1024] = "";
++   char S[2048] = "";
+    char *End = S;
+    
+    End += snprintf(S,sizeof(S),"200 URI Start\nURI: %s\n",Queue->Uri.c_str());
+@@ -160,7 +160,7 @@ void pkgAcqMethod::URIDone(FetchResult &Res, FetchResult *Alt)
+    if (Queue == 0)
+       abort();
+    
+-   char S[1024] = "";
++   char S[2048] = "";
+    char *End = S;
+    
+    End += snprintf(S,sizeof(S),"201 URI Done\nURI: %s\n",Queue->Uri.c_str());
+@@ -242,7 +242,7 @@ void pkgAcqMethod::URIDone(FetchResult &Res, FetchResult *Alt)
+    to be ackd */
+ bool pkgAcqMethod::MediaFail(string Required,string Drive)
+ {
+-   char S[1024];
++   char S[2048];
+    snprintf(S,sizeof(S),"403 Media Failure\nMedia: %s\nDrive: %s\n\n",
+ 	    Required.c_str(),Drive.c_str());
+ 
+@@ -411,7 +411,7 @@ void pkgAcqMethod::Log(const char *Format,...)
+    va_start(args,Format);
+ 
+    // sprintf the description
+-   char S[1024];
++   char S[2048];
+    unsigned int Len = snprintf(S,sizeof(S)-4,"101 Log\nURI: %s\n"
+ 			       "Message: ",CurrentURI.c_str());
+ 
+@@ -435,7 +435,7 @@ void pkgAcqMethod::Status(const char *Format,...)
+    va_start(args,Format);
+ 
+    // sprintf the description
+-   char S[1024];
++   char S[2048];
+    unsigned int Len = snprintf(S,sizeof(S)-4,"102 Status\nURI: %s\n"
+ 			       "Message: ",CurrentURI.c_str());
+ 
+-- 
+1.7.10.4
+
meta/recipes-devtools/apt/apt-native_0.7.14.bb +1 -1
index 38e1fe6..46f39f5 100644
@@ -1,6 +1,6 @@
 require apt-native.inc
 
-PR = "r9"
+PR = "r10"
 
 SRC_URI += "file://nodoc.patch \
             file://noconfigure.patch \
meta/recipes-devtools/apt/apt.inc +1 -0
index 563eda1..973ccb5 100644
@@ -9,6 +9,7 @@ SRC_URI = "${DEBIAN_MIRROR}/main/a/apt/apt_${PV}.tar.gz \
            file://makerace.patch \
            file://remove-redeclaration.patch \
            file://truncate-filename.patch \
+           file://allocate-larger-memory.patch \
            "
 
 inherit autotools gettext
meta/recipes-devtools/apt/apt_0.7.14.bb +1 -1
index 12dd1f2..667b700 100644
@@ -3,7 +3,7 @@ RDEPENDS_${PN} = "dpkg"
 LIC_FILES_CHKSUM = "file://COPYING.GPL;md5=0636e73ff0215e8d672dc4c32c317bb3"
 require apt.inc
 
-PR = "r14"
+PR = "r15"
 
 SRC_URI += "file://nodoc.patch \
             file://includes-fix.patch "
Signature · 1 line
1.7.1
Applied
#5 lore raw
In reply to Robert Yang
Quoted 18 lines · Robert Yang

The following changes since commit a6b6df1350149c116050cb93c3c7b4802c709d31:

   task-core-tools-debug: Added openssh-sftp-server. (2012-07-03 14:52:38 +0100)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib robert/apt
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/apt

Robert Yang (2):
apt 0.7.14: runtime error: filename too long (tmpdir length) apt 0.7.14: runtime error: Method file has died unexpectedly

  .../apt/apt-0.7.14/allocate-larger-memory.patch    |   75 ++++++++++++++++++++
  .../apt/apt-0.7.14/truncate-filename.patch         |   35 +++++++++
  meta/recipes-devtools/apt/apt-native_0.7.14.bb     |    2 +-
  meta/recipes-devtools/apt/apt.inc                  |    2 +
  meta/recipes-devtools/apt/apt_0.7.14.bb            |    2 +-
  5 files changed, 114 insertions(+), 2 deletions(-)
Quoted 9 lines

create mode 100644 meta/recipes-devtools/apt/apt-0.7.14/allocate-larger-memory.patch create mode 100644 meta/recipes-devtools/apt/apt-0.7.14/truncate-filename.patch

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Merged into OE-Core

Thanks
Sau!