Skip to contentSkip to conversation

bitbake: prserv: fix ResourceWarning due to unclosed socket

PATCH In review [email protected] 4 participants 4 messages Nov 11, 2019 – Nov 23, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bitbake: prserv: fix ResourceWarning due to unclosed socket

From: Gavin Li <[email protected]>

With PRSERV_HOST = "localhost:0", this message would occasionally pop up during the initial cache read:

WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)>
value = pickled.load()

The file location stated is irrelevant; it just happens to be wherever CPython decides to run the garbage collector. The issue is that after we fork off a PRServer, self.socket is also duplicated. The parent side of it also needs to be closed.

---
 bitbake/lib/prserv/serv.py | 1 +
 1 file changed, 1 insertion(+)
bitbake/lib/prserv/serv.py +1 -0
index be3acec36a..1d9148b817 100644
@@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
         try:
             pid = os.fork()
             if pid > 0:
+                self.socket.close() # avoid ResourceWarning in parent
                 return pid
         except OSError as e:
             raise Exception("%s [%d]" % (e.strerror, e.errno))
Signature · 1 line
2.23.0
#1 lore raw

✗ patchtest: failure for bitbake: prserv: fix ResourceWarning due to unclosed socket

== Series Details ==

Series: bitbake: prserv: fix ResourceWarning due to unclosed socket
Revision: 1
URL : https://patchwork.openembedded.org/series/21085/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue Series sent to the wrong mailing list or some patches from the series correspond to different mailing lists [test_target_mailing_list]

  Suggested fix    Send the series again to the correct mailing list (ML)
  Suggested ML     [email protected] [http://git.openembedded.org/bitbake/]
  Patch's path:    bitbake/lib/prserv/serv.py

* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at 4dc804b2de)

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
#2 lore raw
In reply to [email protected]

Thanks, but it still starts and terminates PRServer immediately, right?

With your fix, all my builds now start with:

NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 34335, PID: 9429
NOTE: Terminating PRServer...
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 45753, PID: 9433

before this I was using:
https://github.com/shr-distribution/bitbake/commit/728d9ec3b3448bba91a3cc0d505b804e702aec4b
which doesn't start it just to terminate it, but RP said that my fix doesn't look correct, so I haven't sent it to ML. BTW: bitbake patches should go to bitbake-devel ML not oe-core ML.

Quoted 42 lines · On Mon, Nov 11, 2019 at 11:15 PM <[email protected]> wrote:

From: Gavin Li <[email protected]>

With PRSERV_HOST = "localhost:0", this message would occasionally pop up during the initial cache read:

WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446:
ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 45655)>
value = pickled.load()

The file location stated is irrelevant; it just happens to be wherever CPython decides to run the garbage collector. The issue is that after we fork off a PRServer, self.socket is also duplicated. The parent side of it also needs to be closed.

---
 bitbake/lib/prserv/serv.py | 1 +
 1 file changed, 1 insertion(+)
bitbake/lib/prserv/serv.py +1 -1
index be3acec36a..1d9148b817 100644
@@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
         try:
             pid = os.fork()
             if pid > 0:
+                self.socket.close() # avoid ResourceWarning in parent
                 return pid
         except OSError as e:
             raise Exception("%s [%d]" % (e.strerror, e.errno))
--

2.23.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#3 lore raw
Quoted 11 lines · Martin Jansa

Thanks, but it still starts and terminates PRServer immediately, right?

With your fix, all my builds now start with:

NOTE: Started PRServer with DBfile: /OE/build/oe- core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 34335, PID: 9429
NOTE: Terminating PRServer...
NOTE: Started PRServer with DBfile: /OE/build/oe- core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 45753, PID: 9433

Quoted 5 lines

before this I was using:
https://github.com/shr-distribution/bitbake/commit/728d9ec3b3448bba91a3cc0d505b804e702aec4b
which doesn't start it just to terminate it, but RP said that my fix doesn't look correct, so I haven't sent it to ML. BTW: bitbake patches should go to bitbake-devel ML not oe-core ML.

FWIW I finally got around to fixing this. The resource issue was half the problem, the unnecessary restarts are now also addressed. This was particularly important to get right for the context of memory resident bitbake.

Cheers,

Richard