Skip to contentSkip to conversation

trace-cmd: Update python plugin for Python 3

PATCH In review Steven Rostedt 2 participants 10 messages Jul 19, 2019 – Jul 26, 2019

lore
Share & tools
lore
#0 lore raw

cover trace-cmd: Update python plugin for Python 3

As reported here:

https://bugzilla.kernel.org/show_bug.cgi?id=204231

The trace-cmd python plugin doesn't build properly for Python 3. As Python 2 is going to be EOL soon, these should fix that. Note, I'm not strong in using python, and these changes basically came from googling how to handle these deprecations. Please have a look to see if I didn't break anything.

Steven Rostedt (VMware) (3):
trace-cmd: Replace PySting_FromString() with PyUnicode_FromString()
trace-cmd: Use PyMemoryView_FromMemory() for Python 3
trace-cmd: Use PyLong_AsLong() for Python 3

----

 python/ctracecmd.i | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)
#1 lore raw

PATCH 1/3 trace-cmd: Replace PySting_FromString() with PyUnicode_FromString()

From: "Steven Rostedt (VMware)" <[email protected]>

Python 3 has deprecated PyString_FromString(), but both Python 2 and Python 3 have PyUnicode_FromString() which should be equivalent (at least according to google). As Python 2 is going to be EOL soon, we need to support Python 3.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=204231

---
 python/ctracecmd.i | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
python/ctracecmd.i +4 -4
index 65a3d5144b88..09d1d6414fc1 100644
@@ -108,7 +108,7 @@ static PyObject *py_field_get_stack(struct tep_handle *pevent,
 		    ((int)addr == -1))
 			break;
 		func = tep_find_function(event->tep, addr);
-		if (PyList_Append(list, PyString_FromString(func))) {
+		if (PyList_Append(list, PyUnicode_FromString(func))) {
 			Py_DECREF(list);
 			return NULL;
 		}
@@ -162,10 +162,10 @@ static PyObject *py_field_get_str(struct tep_format_field *f, struct tep_record
 		 */
 		offset = val & 0xffff;
 
-		return PyString_FromString((char *)r->data + offset);
+		return PyUnicode_FromString((char *)r->data + offset);
 	}
 
-	return PyString_FromStringAndSize((char *)r->data + f->offset,
+	return PyUnicode_FromStringAndSize((char *)r->data + f->offset,
 				strnlen((char *)r->data + f->offset, f->size));
 }
 
@@ -177,7 +177,7 @@ static PyObject *py_format_get_keys(struct tep_event *ef)
 	list = PyList_New(0);
 
 	for (f = ef->format.fields; f; f = f->next) {
-		if (PyList_Append(list, PyString_FromString(f->name))) {
+		if (PyList_Append(list, PyUnicode_FromString(f->name))) {
 			Py_DECREF(list);
 			return NULL;
 		}
Signature · 1 line
2.20.1
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] trace-cmd: Replace PySting_FromString() with PyUnicode_FromString()

python/ctracecmd.i static PyObject *py_field_get_stack(struct tep_handle *pevent, · line 112

Quoted 9 lines · Steven Rostedt
python/ctracecmd.i +1 -1
@@ -108,7 +108,7 @@ static PyObject *py_field_get_stack(struct tep_handle *pevent,
 		    ((int)addr == -1))
 			break;
 		func = tep_find_function(event->tep, addr);
-		if (PyList_Append(list, PyString_FromString(func))) {
+		if (PyList_Append(list, PyUnicode_FromString(func))) {
 			Py_DECREF(list);

This assumes that the source code is not using extended identifiers (per C99, gcc -fextended-identifiers). I think that's probably a reasonable assumption :-)

In theory, a bit safer would be to use PyBytes_FromString() and let the python plugin worry about the encoding.

That really applies to all of the instances here, I think, since the strings eventually come from (macros in the) code.

johannes

#3 lore raw

PATCH 2/3 trace-cmd: Use PyMemoryView_FromMemory() for Python 3

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Python 3 has deprecated PyBuffer_FromMemory() and instead has PyMemoryView_FromMemory(). Add a helper function that uses the latter if Python 3 is detected. As Python 2 is going to be EOL soon, we need to support Python 3.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=204231

---
 python/ctracecmd.i | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
python/ctracecmd.i +11 -2
index 09d1d6414fc1..63e5dcb813f1 100644
@@ -117,6 +117,15 @@ static PyObject *py_field_get_stack(struct tep_handle *pevent,
 	return list;
 }
 
+static PyObject *fromMemory(void *buf, size_t len)
+{
+#if PY_MAJOR_VERSION >= 3
+		return PyMemoryView_FromMemory(buf, len, PyBUF_READ);
+#else
+		return PyBuffer_FromMemory(buf, len);
+#endif
+}
+
 static PyObject *py_field_get_data(struct tep_format_field *f, struct tep_record *r)
 {
 	if (!strncmp(f->type, "__data_loc ", 11)) {
@@ -137,10 +146,10 @@ static PyObject *py_field_get_data(struct tep_format_field *f, struct tep_record
 		offset = val & 0xffff;
 		len = val >> 16;
 
-		return PyBuffer_FromMemory((char *)r->data + offset, len);
+		return fromMemory(r->data + offset, len);
 	}
 
-	return PyBuffer_FromMemory((char *)r->data + f->offset, f->size);
+	return fromMemory(r->data + f->offset, f->size);
 }
 
 static PyObject *py_field_get_str(struct tep_format_field *f, struct tep_record *r)
Signature · 1 line
2.20.1
Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] trace-cmd: Use PyMemoryView_FromMemory() for Python 3

Quoted 8 lines · Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Python 3 has deprecated PyBuffer_FromMemory() and instead has PyMemoryView_FromMemory(). Add a helper function that uses the latter if Python 3 is detected. As Python 2 is going to be EOL soon, we need to support Python 3.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=204231

johannes

#5 lore raw

PATCH 3/3 trace-cmd: Use PyLong_AsLong() for Python 3

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Python 3 has deprecated PyInt_AS_LONG. Add code to use PyLong_AsLong() if Python 3 is detected. As Python 2 is going to be EOL soon, we need to support Python 3.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=204231

---
 python/ctracecmd.i | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
python/ctracecmd.i +10 -3
index 63e5dcb813f1..2601d39a76be 100644
@@ -117,14 +117,21 @@ static PyObject *py_field_get_stack(struct tep_handle *pevent,
 	return list;
 }
 
+#if PY_MAJOR_VERSION >= 3
 static PyObject *fromMemory(void *buf, size_t len)
 {
-#if PY_MAJOR_VERSION >= 3
 		return PyMemoryView_FromMemory(buf, len, PyBUF_READ);
+}
+#define PY_INT_AS_LONG PyLong_AsLong
 #else
+static PyObject *fromMemory(void *buf, size_t len)
+{
 		return PyBuffer_FromMemory(buf, len);
-#endif
 }
+#define PY_INT_AS_LONG PyInt_AS_LONG
+#endif
+
+
 
 static PyObject *py_field_get_data(struct tep_format_field *f, struct tep_record *r)
 {
@@ -226,7 +233,7 @@ static int python_callback(struct trace_seq *s,
 			Py_XDECREF(result);
 			return 0;
 		}
-		r = PyInt_AS_LONG(result);
+		r = PY_INT_AS_LONG(result);
 	} else if (result == Py_None)
 		r = 0;
 	else
Signature · 1 line
2.20.1
Reviewed-by on v1 3/3
#6 lore raw

Re: [PATCH 3/3] trace-cmd: Use PyLong_AsLong() for Python 3

Quoted 1 line · Steven Rostedt

+#if PY_MAJOR_VERSION >= 3

[...]

Quoted 2 lines

+#define PY_INT_AS_LONG PyLong_AsLong
#else

[...]

python/ctracecmd.i static PyObject *py_field_get_stack(struct tep_handle *pevent, · line 132

Quoted 2 lines
python/ctracecmd.i +2 -0
+#define PY_INT_AS_LONG PyInt_AS_LONG
+#endif

I probably would've gone for only having a single

#define PyLong_AsLong PyInt_AS_LONG

in the python2 case, but doesn't really matter.

johannes

#7 lore raw

Re: [PATCH 3/3] trace-cmd: Use PyLong_AsLong() for Python 3

Quoted 12 lines · On Fri, 26 Jul 2019 09:31:57 +0200 Johannes Berg <[email protected]>…
Quoted 1 line · Steven Rostedt

+#if PY_MAJOR_VERSION >= 3

[...]

Quoted 2 lines

+#define PY_INT_AS_LONG PyLong_AsLong
#else

[...]

python/ctracecmd.i static PyObject *py_field_get_stack(struct tep_handle *pevent, · line 132

Quoted 2 lines
python/ctracecmd.i +2 -0
+#define PY_INT_AS_LONG PyInt_AS_LONG
+#endif  

I probably would've gone for only having a single

Quoted 7 lines

#define PyLong_AsLong PyInt_AS_LONG

in the python2 case, but doesn't really matter.

Hi Johannes,

Thanks for the review (unfortunately, due to the rush to get KernelShark 1.0 out, I already applied them, I should have Cc'd you on them too).

Should we make any of the above changes you mentioned to any of these patches? I'm looking at releasing trace-cmd 2.8.4 with all these changes and will hold off if you think it would be better to update them.

Thanks!

-- Steve

#8 lore raw

Re: [PATCH 3/3] trace-cmd: Use PyLong_AsLong() for Python 3

Hi,

Quoted 9 lines

Thanks for the review (unfortunately, due to the rush to get KernelShark 1.0 out, I already applied them, I should have Cc'd you on them too).

Should we make any of the above changes you mentioned to any of these patches? I'm looking at releasing trace-cmd 2.8.4 with all these changes and will hold off if you think it would be better to update them.

I don't think it matters.

The PyLong thing was just really cosmetic, could've saved some lines of code.

The bytes vs. unicode - well, python assumes that you give it a utf-8 string, so even if we start using non-ASCII identifiers (which I think is highly unlikely!), we'll likely use utf-8 in the kernel, and it would either way not be an issue.

johannes

#9 lore raw

Re: [PATCH 3/3] trace-cmd: Use PyLong_AsLong() for Python 3

Quoted 4 lines · On Fri, 26 Jul 2019 17:22:58 +0200 Johannes Berg <[email protected]>…

The bytes vs. unicode - well, python assumes that you give it a utf-8 string, so even if we start using non-ASCII identifiers (which I think is highly unlikely!), we'll likely use utf-8 in the kernel, and it would either way not be an issue.

Thanks Johannes for explaining.

-- Steve