Skip to contentSkip to conversation

Crash in cgroup_procs_show

PATCH In review Laura Abbott 4 participants 9 messages Dec 13, 2017 – Jan 8, 2018 cgroup

lore
Share & tools
lore
#0 lore raw

Hi,

Fedora got a bug report on 4.14.4 of a crash on reboot https://bugzilla.redhat.com/show_bug.cgi?id=1525279

 BUG: unable to handle kernel NULL pointer dereference at 0000000000000254
 IP: __task_pid_nr_ns+0xc7/0xf0
 PGD 0 P4D 0
 Oops: 0000 [#1] SMP
 Modules linked in: fuse vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_nat xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle bridge iptable_raw stp llc iptable_security ebtable_filter ebtables ip6table_filter ip6_tables sunrpc vfat fat edac_mce_amd eeepc_wmi kvm_amd asus_wmi wmi_bmof sparse_keymap rfkill video kvm snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi irqbypass crct10dif_pclmul snd_hda_intel crc32_pclmul snd_hda_codec ghash_clmulni_intel
  snd_hda_core snd_hwdep snd_seq joydev snd_seq_device snd_pcm snd_timer sp5100_tco i2c_piix4 ccp snd soundcore tpm_tis shpchp tpm_tis_core tpm wmi acpi_cpufreq binfmt_misc btrfs xor zstd_decompress zstd_compress xxhash raid6_pq hid_logitech_hidpp amdkfd amd_iommu_v2 amdgpu i2c_algo_bit drm_kms_helper ttm drm crc32c_intel r8169 hid_logitech_dj mii nvme nvme_core i2c_dev
 CPU: 2 PID: 1 Comm: systemd Not tainted 4.14.4-200.fc26.x86_64 #1
 Hardware name: System manufacturer System Product Name/PRIME B350M-A, BIOS 3203 11/09/2017
 task: ffff88c4baee8000 task.stack: ffff96d5c3158000
 RIP: 0010:__task_pid_nr_ns+0xc7/0xf0
 RSP: 0018:ffff96d5c315bd50 EFLAGS: 00010206
 RAX: 0000000000000000 RBX: ffff88c4b68c6000 RCX: 0000000000000250
 RDX: ffffffffa5e47960 RSI: 0000000000000000 RDI: ffff88c490f6ab00
 RBP: ffff96d5c315bd50 R08: 0000000000001000 R09: 0000000000000005
 R10: ffff88c4be006b80 R11: ffff88c42f1b8004 R12: ffff96d5c315bf18
 R13: ffff88c42d7dd200 R14: ffff88c490f6a510 R15: ffff88c4b68c6000
 FS:  00007f9446f8ea00(0000) GS:ffff88c4be680000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000254 CR3: 00000007f956f000 CR4: 00000000003406e0
 Call Trace:
  cgroup_procs_show+0x19/0x30
  cgroup_seqfile_show+0x4c/0xb0
  kernfs_seq_show+0x21/0x30
  seq_read+0x2ec/0x3f0
  kernfs_fop_read+0x134/0x180
  __vfs_read+0x37/0x160
  ? security_file_permission+0x9b/0xc0
  vfs_read+0x8e/0x130
  SyS_read+0x55/0xc0
  entry_SYSCALL_64_fastpath+0x1a/0xa5
 RIP: 0033:0x7f94455f942d
 RSP: 002b:00007ffe81ba2d00 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
 RAX: ffffffffffffffda RBX: 00005574e2233f00 RCX: 00007f94455f942d
 RDX: 0000000000001000 RSI: 00005574e2321a90 RDI: 000000000000002b
 RBP: 0000000000000000 R08: 00005574e2321a90 R09: 00005574e231de60
 R10: 00007f94458c8b38 R11: 0000000000000293 R12: 00007f94458c8ae0
 R13: 00007ffe81ba3800 R14: 0000000000000000 R15: 00005574e2116560
 Code: 04 74 0e 89 f6 48 8d 04 76 48 8d 04 c5 f0 05 00 00 48 8b bf b8 05 00 00 48 01 c7 31 c0 48 8b 0f 48 85 c9 74 18 8b b2 30 08 00 00 <3b> 71 04 77 0d 48 c1 e6 05 48 01 f1 48 3b 51 38 74 09 5d c3 8b
 RIP: __task_pid_nr_ns+0xc7/0xf0 RSP: ffff96d5c315bd50

It looks like a few others reported similar problems

https://marc.info/?l=linux-kernel&m=151210295120013
https://marc.info/?l=linux-kernel&m=151151651301420

I haven't yet asked the reporter to try and bisect. Is there a fix queued up for this yet?

Thanks,
Laura

maintainer
#1 lore raw

Hello, Laura.

Quoted 7 lines · Laura Abbott

Hi,

Fedora got a bug report on 4.14.4 of a crash on reboot https://bugzilla.redhat.com/show_bug.cgi?id=1525279

 BUG: unable to handle kernel NULL pointer dereference at 0000000000000254
 IP: __task_pid_nr_ns+0xc7/0xf0

Any chance you can map this back to the source line with addr2line?

Thanks.

Signature · 1 line
tejun
#2 lore raw
Quoted 8 lines · Tejun Heo

Hello, Laura.

Quoted 5 lines · Laura Abbott

Hi,

Fedora got a bug report on 4.14.4 of a crash on reboot https://bugzilla.redhat.com/show_bug.cgi?id=1525279

Quoted 7 lines
Quoted 2 lines
  BUG: unable to handle kernel NULL pointer dereference at 0000000000000254
  IP: __task_pid_nr_ns+0xc7/0xf0

Any chance you can map this back to the source line with addr2line?

Thanks.

annotated with decode_stacktrace.sh

IP: __task_pid_nr_ns (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/pid.c:506 /usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/pid.c:535)

cgroup_procs_show (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/cgroup/cgroup.c:4240)
cgroup_seqfile_show (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/cgroup/cgroup.c:3413)
kernfs_seq_show (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/kernfs/file.c:169)
seq_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/seq_file.c:269)
kernfs_fop_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/kernfs/file.c:252)
__vfs_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/read_write.c:416)
? security_file_permission (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/include/linux/fsnotify.h:56 /usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/security/security.c:867) vfs_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/read_write.c:448)
SyS_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/read_write.c:574 /usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/read_write.c:566) entry_SYSCALL_64_fastpath (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/arch/x86/entry/entry_64.S:206)

All code
========

   0:	04 74                	add    $0x74,%al
   2:	0e                   	(bad)
   3:	89 f6                	mov    %esi,%esi
   5:	48 8d 04 76          	lea    (%rsi,%rsi,2),%rax
   9:	48 8d 04 c5 f0 05 00 	lea    0x5f0(,%rax,8),%rax
  10:	00
  11:	48 8b bf b8 05 00 00 	mov    0x5b8(%rdi),%rdi
  18:	48 01 c7             	add    %rax,%rdi
  1b:	31 c0                	xor    %eax,%eax
  1d:	48 8b 0f             	mov    (%rdi),%rcx
  20:	48 85 c9             	test   %rcx,%rcx
  23:	74 18                	je     0x3d
  25:	8b b2 30 08 00 00    	mov    0x830(%rdx),%esi
  2b:*	3b 71 04             	cmp    0x4(%rcx),%esi		<-- trapping instruction
  2e:	77 0d                	ja     0x3d
  30:	48 c1 e6 05          	shl    $0x5,%rsi
  34:	48 01 f1             	add    %rsi,%rcx
  37:	48 3b 51 38          	cmp    0x38(%rcx),%rdx
  3b:	74 09                	je     0x46
  3d:	5d                   	pop    %rbp
  3e:	c3                   	retq
  3f:	8b                   	.byte 0x8b

Code starting with the faulting instruction
===========================================

   0:	3b 71 04             	cmp    0x4(%rcx),%esi
   3:	77 0d                	ja     0x12
   5:	48 c1 e6 05          	shl    $0x5,%rsi
   9:	48 01 f1             	add    %rsi,%rcx
   c:	48 3b 51 38          	cmp    0x38(%rcx),%rdx
  10:	74 09                	je     0x1b
  12:	5d                   	pop    %rbp
  13:	c3                   	retq
  14:	8b                   	.byte 0x8b
maintainer
#3 lore raw

Hello, Laura.

Quoted 14 lines · Laura Abbott
Quoted 13 lines · Tejun Heo

Hello, Laura.

Quoted 7 lines · Laura Abbott

Hi,

Fedora got a bug report on 4.14.4 of a crash on reboot https://bugzilla.redhat.com/show_bug.cgi?id=1525279

 BUG: unable to handle kernel NULL pointer dereference at 0000000000000254
 IP: __task_pid_nr_ns+0xc7/0xf0

Any chance you can map this back to the source line with addr2line?

Quoted 7 lines
Quoted 2 lines

Thanks.

annotated with decode_stacktrace.sh

IP: __task_pid_nr_ns (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/pid.c:506 /usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/pid.c:535)

Thank you very much for the trace. I *think* I found the place where the css task iterator points to a task without holding its ref allowing the task to go away before dereferenced. I'll update once I learn more.

Thanks.

Signature · 1 line
tejun
maintainer
#4 lore raw

PATCH 1/1 cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC

Hello,

Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. I could reproduce the problem reliably and am pretty sure this is the right fix but I'd greatly appreciate if you guys can confirm the fix too.

Thank you very much.

------ 8< ------
From 74d0833c659a8a54735e5efdd44f4b225af68586 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Wed, 20 Dec 2017 07:09:19 -0800

While teaching css_task_iter to handle skipping over tasks which aren't group leaders, bc2fb7ed089f ("cgroup: add @flags to css_task_iter_start() and implement CSS_TASK_ITER_PROCS") introduced a silly bug.

CSS_TASK_ITER_PROCS is implemented by repeating css_task_iter_advance() while the advanced cursor is pointing to a non-leader thread. However, the cursor variable, @l, wasn't updated when the iteration has to advance to the next css_set and the following repetition would operate on the terminal @l from the previous iteration which isn't pointing to a valid task leading to oopses like the following or infinite looping.

  BUG: unable to handle kernel NULL pointer dereference at 0000000000000254
  IP: __task_pid_nr_ns+0xc7/0xf0
  PGD 0 P4D 0
  Oops: 0000 [#1] SMP
  ...
  CPU: 2 PID: 1 Comm: systemd Not tainted 4.14.4-200.fc26.x86_64 #1
  Hardware name: System manufacturer System Product Name/PRIME B350M-A, BIOS 3203 11/09/2017
  task: ffff88c4baee8000 task.stack: ffff96d5c3158000
  RIP: 0010:__task_pid_nr_ns+0xc7/0xf0
  RSP: 0018:ffff96d5c315bd50 EFLAGS: 00010206
  RAX: 0000000000000000 RBX: ffff88c4b68c6000 RCX: 0000000000000250
  RDX: ffffffffa5e47960 RSI: 0000000000000000 RDI: ffff88c490f6ab00
  RBP: ffff96d5c315bd50 R08: 0000000000001000 R09: 0000000000000005
  R10: ffff88c4be006b80 R11: ffff88c42f1b8004 R12: ffff96d5c315bf18
  R13: ffff88c42d7dd200 R14: ffff88c490f6a510 R15: ffff88c4b68c6000
  FS:  00007f9446f8ea00(0000) GS:ffff88c4be680000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 0000000000000254 CR3: 00000007f956f000 CR4: 00000000003406e0
  Call Trace:
   cgroup_procs_show+0x19/0x30
   cgroup_seqfile_show+0x4c/0xb0
   kernfs_seq_show+0x21/0x30
   seq_read+0x2ec/0x3f0
   kernfs_fop_read+0x134/0x180
   __vfs_read+0x37/0x160
   ? security_file_permission+0x9b/0xc0
   vfs_read+0x8e/0x130
   SyS_read+0x55/0xc0
   entry_SYSCALL_64_fastpath+0x1a/0xa5
  RIP: 0033:0x7f94455f942d
  RSP: 002b:00007ffe81ba2d00 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
  RAX: ffffffffffffffda RBX: 00005574e2233f00 RCX: 00007f94455f942d
  RDX: 0000000000001000 RSI: 00005574e2321a90 RDI: 000000000000002b
  RBP: 0000000000000000 R08: 00005574e2321a90 R09: 00005574e231de60
  R10: 00007f94458c8b38 R11: 0000000000000293 R12: 00007f94458c8ae0
  R13: 00007ffe81ba3800 R14: 0000000000000000 R15: 00005574e2116560
  Code: 04 74 0e 89 f6 48 8d 04 76 48 8d 04 c5 f0 05 00 00 48 8b bf b8 05 00 00 48 01 c7 31 c0 48 8b 0f 48 85 c9 74 18 8b b2 30 08 00 00 <3b> 71 04 77 0d 48 c1 e6 05 48 01 f1 48 3b 51 38 74 09 5d c3 8b
  RIP: __task_pid_nr_ns+0xc7/0xf0 RSP: ffff96d5c315bd50

Fix it by moving the initialization of the cursor below the repeat label. While at it, rename it to @next for readability.

Fixes bc2fb7ed089f ("cgroup: add @flags to css_task_iter_start() and implement CSS_TASK_ITER_PROCS")
Cc [email protected] # v4.14+
---
 kernel/cgroup/cgroup.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
kernel/cgroup/cgroup.c +6 -8
index f4c2f8c..2cf06c2 100644
@@ -4125,26 +4125,24 @@ static void css_task_iter_advance_css_set(struct css_task_iter *it)
 
 static void css_task_iter_advance(struct css_task_iter *it)
 {
-	struct list_head *l = it->task_pos;
+	struct list_head *next;
 
 	lockdep_assert_held(&css_set_lock);
-	WARN_ON_ONCE(!l);
-
 repeat:
 	/*
 	 * Advance iterator to find next entry.  cset->tasks is consumed
 	 * first and then ->mg_tasks.  After ->mg_tasks, we move onto the
 	 * next cset.
 	 */
-	l = l->next;
+	next = it->task_pos->next;
 
-	if (l == it->tasks_head)
-		l = it->mg_tasks_head->next;
+	if (next == it->tasks_head)
+		next = it->mg_tasks_head->next;
 
-	if (l == it->mg_tasks_head)
+	if (next == it->mg_tasks_head)
 		css_task_iter_advance_css_set(it);
 	else
-		it->task_pos = l;
+		it->task_pos = next;
 
 	/* if PROCS, skip over tasks which aren't group leaders */
 	if ((it->flags & CSS_TASK_ITER_PROCS) && it->task_pos &&
Signature · 1 line
2.9.5
#5 lore raw

Re: [PATCH cgroup/for-4.15-fixes] cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC

This resolves the bug for me.

Thank you,
George

Quoted 120 lines · Tejun Heo

Hello,

Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. I could reproduce the problem reliably and am pretty sure this is the right fix but I'd greatly appreciate if you guys can confirm the fix too.

Thank you very much.

------ 8< ------
From 74d0833c659a8a54735e5efdd44f4b225af68586 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Wed, 20 Dec 2017 07:09:19 -0800

While teaching css_task_iter to handle skipping over tasks which aren't group leaders, bc2fb7ed089f ("cgroup: add @flags to css_task_iter_start() and implement CSS_TASK_ITER_PROCS") introduced a silly bug.

CSS_TASK_ITER_PROCS is implemented by repeating css_task_iter_advance() while the advanced cursor is pointing to a non-leader thread. However, the cursor variable, @l, wasn't updated when the iteration has to advance to the next css_set and the following repetition would operate on the terminal @l from the previous iteration which isn't pointing to a valid task leading to oopses like the following or infinite looping.

  BUG: unable to handle kernel NULL pointer dereference at 0000000000000254
  IP: __task_pid_nr_ns+0xc7/0xf0
  PGD 0 P4D 0
  Oops: 0000 [#1] SMP
  ...
  CPU: 2 PID: 1 Comm: systemd Not tainted 4.14.4-200.fc26.x86_64 #1
  Hardware name: System manufacturer System Product Name/PRIME B350M-A, BIOS 3203 11/09/2017
  task: ffff88c4baee8000 task.stack: ffff96d5c3158000
  RIP: 0010:__task_pid_nr_ns+0xc7/0xf0
  RSP: 0018:ffff96d5c315bd50 EFLAGS: 00010206
  RAX: 0000000000000000 RBX: ffff88c4b68c6000 RCX: 0000000000000250
  RDX: ffffffffa5e47960 RSI: 0000000000000000 RDI: ffff88c490f6ab00
  RBP: ffff96d5c315bd50 R08: 0000000000001000 R09: 0000000000000005
  R10: ffff88c4be006b80 R11: ffff88c42f1b8004 R12: ffff96d5c315bf18
  R13: ffff88c42d7dd200 R14: ffff88c490f6a510 R15: ffff88c4b68c6000
  FS:  00007f9446f8ea00(0000) GS:ffff88c4be680000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 0000000000000254 CR3: 00000007f956f000 CR4: 00000000003406e0
  Call Trace:
   cgroup_procs_show+0x19/0x30
   cgroup_seqfile_show+0x4c/0xb0
   kernfs_seq_show+0x21/0x30
   seq_read+0x2ec/0x3f0
   kernfs_fop_read+0x134/0x180
   __vfs_read+0x37/0x160
   ? security_file_permission+0x9b/0xc0
   vfs_read+0x8e/0x130
   SyS_read+0x55/0xc0
   entry_SYSCALL_64_fastpath+0x1a/0xa5
  RIP: 0033:0x7f94455f942d
  RSP: 002b:00007ffe81ba2d00 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
  RAX: ffffffffffffffda RBX: 00005574e2233f00 RCX: 00007f94455f942d
  RDX: 0000000000001000 RSI: 00005574e2321a90 RDI: 000000000000002b
  RBP: 0000000000000000 R08: 00005574e2321a90 R09: 00005574e231de60
  R10: 00007f94458c8b38 R11: 0000000000000293 R12: 00007f94458c8ae0
  R13: 00007ffe81ba3800 R14: 0000000000000000 R15: 00005574e2116560
  Code: 04 74 0e 89 f6 48 8d 04 76 48 8d 04 c5 f0 05 00 00 48 8b bf b8 05 00 00 48 01 c7 31 c0 48 8b 0f 48 85 c9 74 18 8b b2 30 08 00 00 <3b> 71 04 77 0d 48 c1 e6 05 48 01 f1 48 3b 51 38 74 09 5d c3 8b
  RIP: __task_pid_nr_ns+0xc7/0xf0 RSP: ffff96d5c315bd50

Fix it by moving the initialization of the cursor below the repeat label. While at it, rename it to @next for readability.

Fixes bc2fb7ed089f ("cgroup: add @flags to css_task_iter_start() and implement CSS_TASK_ITER_PROCS")
Cc [email protected] # v4.14+
---
 kernel/cgroup/cgroup.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
kernel/cgroup/cgroup.c +6 -9
index f4c2f8c..2cf06c2 100644
@@ -4125,26 +4125,24 @@ static void css_task_iter_advance_css_set(struct css_task_iter *it)
 
 static void css_task_iter_advance(struct css_task_iter *it)
 {
-       struct list_head *l = it->task_pos;
+       struct list_head *next;
 
        lockdep_assert_held(&css_set_lock);
-       WARN_ON_ONCE(!l);
-
 repeat:
        /*
         * Advance iterator to find next entry.  cset->tasks is consumed
         * first and then ->mg_tasks.  After ->mg_tasks, we move onto the
         * next cset.
         */
-       l = l->next;
+       next = it->task_pos->next;
 
-       if (l == it->tasks_head)
-               l = it->mg_tasks_head->next;
+       if (next == it->tasks_head)
+               next = it->mg_tasks_head->next;
 
-       if (l == it->mg_tasks_head)
+       if (next == it->mg_tasks_head)
                css_task_iter_advance_css_set(it);
        else
-               it->task_pos = l;
+               it->task_pos = next;
 
        /* if PROCS, skip over tasks which aren't group leaders */
        if ((it->flags & CSS_TASK_ITER_PROCS) && it->task_pos &&
--

2.9.5

#6 lore raw

Re: [PATCH cgroup/for-4.15-fixes] cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC

In reply to Tejun Heo
Quoted 6 lines · Tejun Heo

Hello,

Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. I could reproduce the problem reliably and am pretty sure this is the right fix but I'd greatly appreciate if you guys can confirm the fix too.

Fixed here as well, 4.14.8 stable with your fix on top.

Thanks,
Rouven

#7 lore raw

Re: [PATCH cgroup/for-4.15-fixes] cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC

In reply to Tejun Heo

Hello,

Quoted 2 lines · Tejun Heo

Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. [...]

I can't find your fixes in master yet, did you perhaps forget to push out?

Thanks and a happy new year,
Rouven

maintainer
#8 lore raw

Re: [PATCH cgroup/for-4.15-fixes] cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC

Quoted 9 lines · Rouven Czerwinski

Hello,

Quoted 2 lines · Tejun Heo

Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. [...]

I can't find your fixes in master yet, did you perhaps forget to push out?

Thanks for reminding me. Sent out the pull request.

Happy new year!

Signature · 1 line
tejun