linux 5.14.3: free_user_ns causes NULL pointer dereference

Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window

https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u

Logs below:

------------[ cut here ]------------
WARNING: CPU: 1 PID: 26546 at kernel/ucount.c:253 dec_ucount+0x43/0x50
Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_magn_3d hid_sensor_rotation hid_sensor_accel_3d hid_sensor_gyro_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader nft_counter intel_ishtp_hid snd_hda_codec_hdmi intel_rapl_msr xt_mark ipt_REJECT nf_reject_ipv4 snd_ctl_led xt_LOG snd_hda_codec_conexant nf_log_syslog snd_hda_codec_generic xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv4 mei_hdcp snd_hda_intel nft_compat wmi_bmof nf_tables intel_rapl_common libcrc32c think_lmi intel_tcc_cooling snd_intel_dspcfg firmware_attributes_class nfnetlink iwlmv

intel_wmi_thunderbolt mac80211 x86_pkg_temp_thermal snd_hda_codec intel_powerclamp coretemp libarc4 vfat fat kvm_intel rapl intel_cstate snd_hwdep intel_uncore iwlwifi snd_hda_core mousedev joydev snd_pcm psmouse cfg80211 snd_timer mei_me ucsi_acpi wacom intel_ish_ipc intel_xhci_usb_role_switch mei intel_pch_thermal typec_ucsi roles typec intel_ishtp wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore rfkill tpm_crb i2c_hid_acpi i2c_hid acpi_pad tpm_tis mac_hid tpm_tis_core pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd xhci_pci rtsx_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm kvm irqbypass

CPU: 1 PID: 26546 Comm: kworker/1:1 Not tainted 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
Workqueue: events free_user_ns
RIP: 0010:dec_ucount+0x43/0x50
Code: 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 48 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff <0f> 0b eb e7 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 f8 48
RSP: 0018:ffffa82cc2bd7e60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffffa2f53298ee50 RCX: ffffa2f3c0061000
RDX: ffffa2f3c0061020 RSI: ffffffffffffffff RDI: ffffa2f3c0061000
RBP: ffffa2f53298ebe0 R08: 0000000000000020 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffffa2f3c0061000
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffffa2f599680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000628f892be9f8 CR3: 000000002880e004 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 free_user_ns+0x73/0x110
 process_one_work+0x1e1/0x380
 worker_thread+0x50/0x3a0
 ? rescuer_thread+0x360/0x360
 kthread+0x127/0x150
 ? set_kthread_struct+0x40/0x40
 ret_from_fork+0x22/0x30
---[ end trace eb7a8d38b64b2d3a ]---
BUG: kernel NULL pointer dereference, address: 00000000000001e8
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: 0000 [#1] SMP PTI
CPU: 1 PID: 26546 Comm: kworker/1:1 Tainted: G        W         5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
Workqueue: events free_user_ns
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
RSP: 0018:ffffa82cc2bd7e60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffffa2f53298ee50 RCX: ffffa2f3c0061000
RDX: ffffa2f3c0061020 RSI: ffffffffffffffff RDI: ffffa2f3c0061000
RBP: ffffa2f53298ebe0 R08: 0000000000000020 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffffa2f3c0061000
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffffa2f599680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 000000002880e004 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 free_user_ns+0x73/0x110
 process_one_work+0x1e1/0x380
 worker_thread+0x50/0x3a0
 ? rescuer_thread+0x360/0x360
 kthread+0x127/0x150
 ? set_kthread_struct+0x40/0x40
 ret_from_fork+0x22/0x30

Quoted 3 lines · Jordan Glover [email protected] writes:

Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window

https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u

So that warning comes from:

void dec_ucount(struct ucounts *ucounts, enum ucount_type type)

{

struct ucounts *iter;

for (iter = ucounts; iter; iter = iter->ns->ucounts) {

    	long dec = atomic_long_dec_if_positive(&iter->ucount[type]);

    	WARN_ON_ONCE(dec < 0);
    }
    put_ucounts(ucounts);

}

Which certainly looks like a reference count bug. It could also be a

memory stomp somewhere close.

Do you have any idea what else was going on? This location is the

symptom but not the actual cause.

Eric

Quoted 31 lines · On Wednesday, September 15th, 2021 at 9:02 PM, <[email protected]> wrote:

Quoted 3 lines · Jordan Glover [email protected] writes:

Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window

https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u

So that warning comes from:

void dec_ucount(struct ucounts *ucounts, enum ucount_type type)

{

struct ucounts *iter;

for (iter = ucounts; iter; iter = iter->ns->ucounts) {

      long dec = atomic_long_dec_if_positive(&iter->ucount[type]);

      WARN_ON_ONCE(dec < 0);
    }
    put_ucounts(ucounts);

}

Which certainly looks like a reference count bug. It could also be a

memory stomp somewhere close.

Do you have any idea what else was going on? This location is the

Quoted 3 lines

symptom but not the actual cause.

Eric

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

Quoted 36 lines · Jordan Glover

Quoted 33 lines · On Wednesday, September 15th, 2021 at 9:02 PM, <[email protected]> wrote:

Quoted 3 lines · Jordan Glover [email protected] writes:

Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window

https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u

So that warning comes from:

void dec_ucount(struct ucounts *ucounts, enum ucount_type type)

{

struct ucounts *iter;

for (iter = ucounts; iter; iter = iter->ns->ucounts) {

      long dec = atomic_long_dec_if_positive(&iter->ucount[type]);

      WARN_ON_ONCE(dec < 0);
    }
    put_ucounts(ucounts);

}

Which certainly looks like a reference count bug. It could also be a

memory stomp somewhere close.

Do you have any idea what else was going on? This location is the

symptom but not the actual cause.

Quoted 5 lines

Quoted 1 line

Eric

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

I have been able to steadily reproduce this for a while. But I haven't had time to look into it. I'd appreciate any help.

Quoted 38 lines · Yu Zhao

Quoted 36 lines · Jordan Glover

Quoted 33 lines · On Wednesday, September 15th, 2021 at 9:02 PM, <[email protected]> wrote:

Quoted 3 lines · Jordan Glover [email protected] writes:

Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window

https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u

So that warning comes from:

void dec_ucount(struct ucounts *ucounts, enum ucount_type type)

{

struct ucounts *iter;

for (iter = ucounts; iter; iter = iter->ns->ucounts) {

      long dec = atomic_long_dec_if_positive(&iter->ucount[type]);

      WARN_ON_ONCE(dec < 0);
    }
    put_ucounts(ucounts);

}

Which certainly looks like a reference count bug. It could also be a

memory stomp somewhere close.

Do you have any idea what else was going on? This location is the

symptom but not the actual cause.

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

Eric

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

I have been able to steadily reproduce this for a while. But I haven't had time to look into it. I'd appreciate any help.

It would be very helpful if you could look farther back in your logs and see if you can also see:

WARNING: CPU: 1 PID: 351 at kernel/ucount.c:253 dec_ucount+0x43/0x5

Or anything else preceding the use-after-free.

I am inclined to think they are the same issue but without seeing the WARN_ON_ONCE I can't safely conclude that.

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

Quoted 4 lines · Jordan Glover

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

Heh, it jut happened again. This time closing firefox alone had such effect:

------------[ cut here ]------------
WARNING: CPU: 1 PID: 351 at kernel/ucount.c:253 dec_ucount+0x43/0x50
Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
 snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
 fb_sys_fops cec drm kvm irqbypass
CPU: 1 PID: 351 Comm: kworker/1:3 Not tainted 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
Workqueue: events free_user_ns
RIP: 0010:dec_ucount+0x43/0x50
Code: 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 48 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff <0f> 0b eb e7 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 f8 48
RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056ffceff6b10 CR3: 0000000147a0e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 free_user_ns+0x73/0x110
 process_one_work+0x1e1/0x380
 worker_thread+0x50/0x3a0
 ? rescuer_thread+0x360/0x360
 kthread+0x127/0x150
 ? set_kthread_struct+0x40/0x40
 ret_from_fork+0x22/0x30
---[ end trace ff45ac39689f43c1 ]---
BUG: kernel NULL pointer dereference, address: 00000000000001e8
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: 0000 [#1] SMP PTI
CPU: 1 PID: 351 Comm: kworker/1:3 Tainted: G        W         5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
Workqueue: events free_user_ns
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 free_user_ns+0x73/0x110
 process_one_work+0x1e1/0x380
 worker_thread+0x50/0x3a0
 ? rescuer_thread+0x360/0x360
 kthread+0x127/0x150
 ? set_kthread_struct+0x40/0x40
 ret_from_fork+0x22/0x30
Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
 snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
 fb_sys_fops cec drm kvm irqbypass
CR2: 00000000000001e8
---[ end trace ff45ac39689f43c2 ]---
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Quoted 8 lines · Jordan Glover

Quoted 4 lines · Jordan Glover

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

Heh, it jut happened again. This time closing firefox alone had such effect:

Ok. It looks like he have a couple of folks seeing issues here.

I thought we had all of the issues sorted out for the release of v5.14, but it looks like there is still some little bug left.

If Alex doesn't beat me to it I will see if I can come up with a debugging patch to make it easy to help track down where the reference
count is going wrong. It will be a little bit as my brain is mush at
the moment.

Eric

Quoted 74 lines

------------[ cut here ]------------
WARNING: CPU: 1 PID: 351 at kernel/ucount.c:253 dec_ucount+0x43/0x50
Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
 snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
 fb_sys_fops cec drm kvm irqbypass
CPU: 1 PID: 351 Comm: kworker/1:3 Not tainted 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
Workqueue: events free_user_ns
RIP: 0010:dec_ucount+0x43/0x50
Code: 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 48 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff <0f> 0b eb e7 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 f8 48
RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056ffceff6b10 CR3: 0000000147a0e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 free_user_ns+0x73/0x110
 process_one_work+0x1e1/0x380
 worker_thread+0x50/0x3a0
 ? rescuer_thread+0x360/0x360
 kthread+0x127/0x150
 ? set_kthread_struct+0x40/0x40
 ret_from_fork+0x22/0x30
---[ end trace ff45ac39689f43c1 ]---
BUG: kernel NULL pointer dereference, address: 00000000000001e8
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: 0000 [#1] SMP PTI
CPU: 1 PID: 351 Comm: kworker/1:3 Tainted: G        W         5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
Workqueue: events free_user_ns
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 free_user_ns+0x73/0x110
 process_one_work+0x1e1/0x380
 worker_thread+0x50/0x3a0
 ? rescuer_thread+0x360/0x360
 kthread+0x127/0x150
 ? set_kthread_struct+0x40/0x40
 ret_from_fork+0x22/0x30
Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
 snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
 fb_sys_fops cec drm kvm irqbypass
CR2: 00000000000001e8
---[ end trace ff45ac39689f43c2 ]---
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Quoted 8 lines · Jordan Glover [email protected] writes:

Quoted 3 lines · On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover…

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

Heh, it jut happened again. This time closing firefox alone had such effect:

Ok. It looks like he have a couple of folks seeing issues here. I thought we had all of the issues sorted out for the release of v5.14, but it looks like there is still some little bug left.

If Alex doesn't beat me to it I will see if I can come up with a debugging patch to make it easy to help track down where the reference count is going wrong. It will be a little bit as my brain is mush at the moment.

Eric

Quoted 20 lines · On Thursday, September 16th, 2021 at 5:30 PM, <[email protected]> wrote:

Quoted 8 lines · Jordan Glover [email protected] writes:

Quoted 3 lines · On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover…

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

Heh, it jut happened again. This time closing firefox alone had such effect:

Ok. It looks like he have a couple of folks seeing issues here. I thought we had all of the issues sorted out for the release of v5.14, but it looks like there is still some little bug left.

If Alex doesn't beat me to it I will see if I can come up with a debugging patch to make it easy to help track down where the reference count is going wrong. It will be a little bit as my brain is mush at the moment.

Quoted 1 line

Eric

As the issue persist in 5.14.7 I would be very interested in such patch.

For now the thing is mostly reproducible when I close several tabs in ff then close the browser in short period of time. When I close tabs then wait out a bit then close the browser it doesn't happen so I guess some interrupted cleanup triggers it.

Quoted 36 lines · Jordan Glover

Quoted 20 lines · On Thursday, September 16th, 2021 at 5:30 PM, [email protected] wrote:

Quoted 9 lines · Jordan Glover [email protected] writes:

Quoted 3 lines · On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover…

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

Heh, it jut happened again. This time closing firefox alone had such

effect:

Ok. It looks like he have a couple of folks seeing issues here.

I thought we had all of the issues sorted out for the release of v5.14,

but it looks like there is still some little bug left.

If Alex doesn't beat me to it I will see if I can come up with a

Quoted 7 lines

debugging patch to make it easy to help track down where the reference

count is going wrong. It will be a little bit as my brain is mush at

the moment.

Eric

As the issue persist in 5.14.7 I would be very interested in such patch.

For now the thing is mostly reproducible when I close several tabs in ff then

close the browser in short period of time. When I close tabs then wait out

Quoted 3 lines

a bit then close the browser it doesn't happen so I guess some interrupted

cleanup triggers it.

I'm still investigating, but I would like to rule out one option.

Could you check out the patch?

kernel/ucount.c +0 -0

index bb51849e6375..f23f906f4f62 100644

--- a/kernel/ucount.c

+++ b/kernel/ucount.c

kernel/ucount.c +0 -5

@@ -201,11 +201,14 @@ void put_ucounts(struct ucounts *ucounts)
 
 {
 
 unsigned long flags;
 
-         if (atomic_dec_and_lock_irqsave(&ucounts->count, &ucounts_lock, flags)) {
 
 
 
-         spin_lock_irqsave(&ucounts_lock, flags);
 
 
-         if (atomic_dec_and_test(&ucounts->count)) {
 
                  hlist_del_init(&ucounts->node);
 
                  spin_unlock_irqrestore(&ucounts_lock, flags);
                  kfree(ucounts);
 
 
-                 return;
          }
 
 
-         spin_unlock_irqrestore(&ucounts_lock, flags);

}

static inline bool atomic_long_inc_below(atomic_long_t *v, int u)

---------------------------------------------------------------------

Rgrds, legion

Quoted 3 lines

I'm still investigating, but I would like to rule out one option.

Could you check out the patch?

Thx, I added it to my kernel and will report in few days. Does this patch try to fix the issue or make it easier to track?

Quoted 2 lines · Jordan Glover

Quoted 2 lines

I'm still investigating, but I would like to rule out one option.

Quoted 5 lines

Quoted 1 line

Could you check out the patch?

Thx, I added it to my kernel and will report in few days. Does this patch try to fix the issue or make it easier to track?

I suspect the error is caused by a race between allow_ucounts() and put_ucounts(). I think this patch could solve the problem.

Adding Rune Kleveland to the discussion as he also seems to have reproduced the issue.

Alex and I have been starring at the code and the reports and this bug is hiding well. Here is what we have figured out so far.

Both the warning from free_user_ns calling dec_ucount that Jordan Glover reported and the KASAN error that Yu Zhao has reported appear to have the same cause. Using a ucounts structure after it has been freed and reallocated as something else.

I have just skimmed through the recent report from Rune Kleveland and it appears also to be a use after free. Especially since the second failure in the log is slub complaining about trying to free the ucounts data structure.

We looked through the users of put_ucounts and we don't see any obvious buggy users that would be freeing the data structure early.

Alex has tried to reproduce this so far is not having any luck. Folks can you tell what compiler versions you are using and share your kernel config with us? That might help.

The little debug diff below is my guess of what is happening. If the folks who can reproduce this issue can try the patch below and let me know if the warnings fire that would be appreciated. It is still not enough to track down the bug but at least it will confirm my current hypothesis about how things look before there is a use of memory after it is freed.

Thank you,
Eric

kernel/cred.c +6 -0

index f784e08c2fbd..e7ffaa3cf5a6 100644
@@ -120,6 +120,12 @@ static void put_cred_rcu(struct rcu_head *rcu)
 	if (cred->group_info)
 		put_group_info(cred->group_info);
 	free_uid(cred->user);
+#if 1
+	if ((cred->ucounts == cred->user_ns->ucounts) &&
+	    (atomic_read(&cred->ucounts->count) == 1)) {
+		WARN_ONCE(1, "put_cred_rcu: ucount count 1\n");
+	}
+#endif
 	if (cred->ucounts)
 		put_ucounts(cred->ucounts);
 	put_user_ns(cred->user_ns);

kernel/exit.c +7 -0

index 91a43e57a32e..60fd88b34c1a 100644
@@ -743,6 +743,13 @@ void __noreturn do_exit(long code)
 	if (unlikely(!tsk->pid))
 		panic("Attempted to kill the idle task!");
 
+#if 1
+	if ((tsk->cred->ucounts == tsk->cred->user_ns->ucounts) &&
+	    (atomic_read(tsk->cred->ucounts->count) == 1)) {
+		WARN_ONCE(1, "do_exit: ucount count 1\n");
+	}
+#endif
+
 	/*
 	 * If do_exit is called because this processes oopsed, it's possible
 	 * that get_fs() was left as KERNEL_DS, so reset it to USER_DS before

Quoted 16 lines · [email protected] (Eric W. Biederman) writes:

Adding Rune Kleveland to the discussion as he also seems to have reproduced the issue.

Alex and I have been starring at the code and the reports and this bug is hiding well. Here is what we have figured out so far.

Both the warning from free_user_ns calling dec_ucount that Jordan Glover reported and the KASAN error that Yu Zhao has reported appear to have the same cause. Using a ucounts structure after it has been freed and reallocated as something else.

I have just skimmed through the recent report from Rune Kleveland and it appears also to be a use after free. Especially since the second failure in the log is slub complaining about trying to free the ucounts data structure.

Quoted 6 lines

We looked through the users of put_ucounts and we don't see any obvious buggy users that would be freeing the data structure early.

Alex has tried to reproduce this so far is not having any luck. Folks can you tell what compiler versions you are using and share your kernel config with us? That might help.

Quoted 23 lines · [email protected] (Eric W. Biederman) writes:

Adding Rune Kleveland to the discussion as he also seems to have reproduced the issue.

Alex and I have been starring at the code and the reports and this bug is hiding well. Here is what we have figured out so far.

Both the warning from free_user_ns calling dec_ucount that Jordan Glover reported and the KASAN error that Yu Zhao has reported appear to have the same cause. Using a ucounts structure after it has been freed and reallocated as something else.

I have just skimmed through the recent report from Rune Kleveland and it appears also to be a use after free. Especially since the second failure in the log is slub complaining about trying to free the ucounts data structure.

We looked through the users of put_ucounts and we don't see any obvious buggy users that would be freeing the data structure early.

Alex has tried to reproduce this so far is not having any luck. Folks can you tell what compiler versions you are using and share your kernel config with us? That might help.

Quoted 6 lines

The little debug diff below is my guess of what is happening. If the folks who can reproduce this issue can try the patch below and let me know if the warnings fire that would be appreciated. It is still not enough to track down the bug but at least it will confirm my current hypothesis about how things look before there is a use of memory after it is freed.

Bah. Scratch that test patch. I just double checked myself and cred->ucounts and cred->user_ns->ucounts should never be equal, as the user namespace is counted in it's parent user namespace.

That observation now tells me I have a parent user namespace that went corrupt.

Back to the drawing board.

Quoted 38 lines

Thank you,
Eric

kernel/cred.c +6 -0

index f784e08c2fbd..e7ffaa3cf5a6 100644
@@ -120,6 +120,12 @@ static void put_cred_rcu(struct rcu_head *rcu)
  	if (cred->group_info)
  		put_group_info(cred->group_info);
  	free_uid(cred->user);
+#if 1
+	if ((cred->ucounts == cred->user_ns->ucounts) &&
+	    (atomic_read(&cred->ucounts->count) == 1)) {
+		WARN_ONCE(1, "put_cred_rcu: ucount count 1\n");
+	}
+#endif
  	if (cred->ucounts)
  		put_ucounts(cred->ucounts);
  	put_user_ns(cred->user_ns);

kernel/exit.c +7 -0

index 91a43e57a32e..60fd88b34c1a 100644
@@ -743,6 +743,13 @@ void __noreturn do_exit(long code)
  	if (unlikely(!tsk->pid))
  		panic("Attempted to kill the idle task!");
 
+#if 1
+	if ((tsk->cred->ucounts == tsk->cred->user_ns->ucounts) &&
+	    (atomic_read(tsk->cred->ucounts->count) == 1)) {
+		WARN_ONCE(1, "do_exit: ucount count 1\n");
+	}
+#endif
+
  	/*
  	 * If do_exit is called because this processes oopsed, it's possible
  	 * that get_fs() was left as KERNEL_DS, so reset it to USER_DS before

Just wanted to let you know that I still get these on stock Fedora kernel 5.14.10 on the IBM blades. But it took 10 hours before the first

server crashed. The other 4 still runs fine since 15 hours ago. So for me it seems more stable now, but that could just be a coincidence.

Best regards,

Rune

------------[ cut here ]------------
kernel BUG at mm/slub.c:321!

invalid opcode: 0000 [#1] SMP PTI

CPU: 22 PID: 1838853 Comm: python3 Not tainted 5.14.10-200.fc34.x86_64 #1
Hardware name: IBM BladeCenter HS22 -[7870TKN]-/68Y8161, BIOS 
-[P9E164CUS-1.28]- 04/17/2018
RIP: 0010:__slab_free+0x245/0x4a0
Code: 0f b6 5c 24 1b 44 8b 44 24 1c 48 89 44 24 08 48 8b 54 24 20 4c 8b 

4c 24 28 e9 8a fe ff ff 41 f7 45 08 00 0d 21 00 75 98 eb 8d <0f> 0b 49 3b 54 24 28 0f 85 53 ff ff ff 49 8b 44 24 08 4>
RSP: 0018:ffffb71dcfd6fda0 EFLAGS: 00010246
RAX: ffff9c5480d35860 RBX: ffff9c5480d35800 RCX: ffff9c5480d35800
RDX: 00000000802a0029 RSI: ffffeb41da034d00 RDI: ffff9c4f00042800
RBP: ffffb71dcfd6fe50 R08: 0000000000000001 R09: ffffffff9210b6a5
R10: ffff9c548102e000 R11: 0000000062667658 R12: ffffeb41da034d00
R13: ffff9c4f00042800 R14: ffff9c5480d35800 R15: ffff9c5480d35800 FS:  00007f7a89765740(0000) GS:ffff9c6c1fc80000(0000) knlGS:0000000000000000 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7a7ad7f4b0 CR3: 0000000564af4002 CR4: 00000000000206e0

Call Trace:
  ? filename_lookup+0x135/0x1b0
  ? put_ucounts+0x65/0x70
  kfree+0x369/0x3c0
  put_ucounts+0x65/0x70
  put_cred_rcu+0x70/0xd0
  do_faccessat+0x113/0x240
  do_syscall_64+0x3b/0x90
  entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f7a899cc44b
Code: 77 05 c3 0f 1f 40 00 48 8b 15 29 1a 0d 00 f7 d8 64 89 02 48 c7 c0 

ff ff ff ff c3 0f 1f 40 00 f3 0f 1e fa b8 15 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 f9 19 0>
RSP: 002b:00007ffd01fa9ce8 EFLAGS: 00000202 ORIG_RAX: 0000000000000015
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f7a899cc44b
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f79a96d6e10
RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f7a7c1fb930
R10: 00007f79a96d6000 R11: 0000000000000202 R12: 00007ffd01fa9d00
R13: 0000000000000001 R14: 0000556841045c90 R15: 00000000ffffff9c

Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs 
lockd grace sunrpc fscache netfs nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 
nft_fib nft_reject_inet nf_reject_ipv4 nf_rejec>
---[ end trace 0a81b150eacde1d5 ]---
RIP: 0010:__slab_free+0x245/0x4a0
Code: 0f b6 5c 24 1b 44 8b 44 24 1c 48 89 44 24 08 48 8b 54 24 20 4c 8b 

4c 24 28 e9 8a fe ff ff 41 f7 45 08 00 0d 21 00 75 98 eb 8d <0f> 0b 49 3b 54 24 28 0f 85 53 ff ff ff 49 8b 44 24 08 4>
RSP: 0018:ffffb71dcfd6fda0 EFLAGS: 00010246
RAX: ffff9c5480d35860 RBX: ffff9c5480d35800 RCX: ffff9c5480d35800
RDX: 00000000802a0029 RSI: ffffeb41da034d00 RDI: ffff9c4f00042800
RBP: ffffb71dcfd6fe50 R08: 0000000000000001 R09: ffffffff9210b6a5
R10: ffff9c548102e000 R11: 0000000062667658 R12: ffffeb41da034d00
R13: ffff9c4f00042800 R14: ffff9c5480d35800 R15: ffff9c5480d35800 FS:  00007f7a89765740(0000) GS:ffff9c6c1fc80000(0000) knlGS:0000000000000000 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7a7ad7f4b0 CR3: 0000000564af4002 CR4: 00000000000206e0

------------[ cut here ]------------

Hi!

Just wanted to let you know that I still get these on stock Fedora kernel 5.14.10 on the IBM blades. But it took 10 hours before the first server crashed. The other 4 still runs fine since 15 hours ago. So for me it seems more stable now, but that could just be a coincidence.

In commit fda31c50292a ("signal: avoid double atomic counter increments for user accounting") Linus made a clever optimization to how rlimits and the struct user_struct. Unfortunately that optimization does not work in the obvious way when moved to nested rlimits. The problem is that the last decrement of the per user namespace per user sigpending counter might also be the last decrement of the sigpending counter in the parent user namespace as well. Which means that simply freeing the leaf ucount in __free_sigqueue is not enough.

Maintain the optimization and handle the tricky cases by introducing inc_rlimit_get_ucounts and dec_rlimit_put_ucounts.

By moving the entire optimization into functions that perform all of the work it becomes possible to ensure that every level is handled properly.

I wish we had a single user across all of the threads whose rlimit could be charged so we did not need this complexity.

Cc [email protected]

Fixes d64696905554 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")

Signed-off-by "Eric W. Biederman" <[email protected]>

---

With a lot of help from Alex who found a way I could reproduce this I believe I have found the issue.

Could people who are seeing this issue test and verify this solves the problem for them?

 include/linux/user_namespace.h |  2 ++
 kernel/signal.c                | 25 +++++----------------
 kernel/ucount.c                | 41 ++++++++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 19 deletions(-)

include/linux/user_namespace.h +2 -0

index eb70cabe6e7f..33a4240e6a6f 100644
@@ -127,6 +127,8 @@ static inline long get_ucounts_value(struct ucounts *ucounts, enum ucount_type t
 
 long inc_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
 bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type);
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type);
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max);
 
 static inline void set_rlimit_ucount_max(struct user_namespace *ns,

kernel/signal.c +6 -19

index a3229add4455..762de58c6e76 100644
@@ -425,22 +425,10 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
 	 */
 	rcu_read_lock();
 	ucounts = task_ucounts(t);
-	sigpending = inc_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-	switch (sigpending) {
-	case 1:
-		if (likely(get_ucounts(ucounts)))
-			break;
-		fallthrough;
-	case LONG_MAX:
-		/*
-		 * we need to decrease the ucount in the userns tree on any
-		 * failure to avoid counts leaking.
-		 */
-		dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-		rcu_read_unlock();
-		return NULL;
-	}
+	sigpending = inc_rlimit_get_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
 	rcu_read_unlock();
+	if (sigpending == LONG_MAX)
+		return NULL;
 
 	if (override_rlimit || likely(sigpending <= task_rlimit(t, RLIMIT_SIGPENDING))) {
 		q = kmem_cache_alloc(sigqueue_cachep, gfp_flags);

@@ -449,8 +437,7 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
 	}
 
 	if (unlikely(q == NULL)) {
-		if (dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1))
-			put_ucounts(ucounts);
+		dec_rlimit_put_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
 	} else {
 		INIT_LIST_HEAD(&q->list);
 		q->flags = sigqueue_flags;

@@ -463,8 +450,8 @@ static void __sigqueue_free(struct sigqueue *q)
 {
 	if (q->flags & SIGQUEUE_PREALLOC)
 		return;
-	if (q->ucounts && dec_rlimit_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING, 1)) {
-		put_ucounts(q->ucounts);
+	if (q->ucounts) {
+		dec_rlimit_put_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING);
 		q->ucounts = NULL;
 	}
 	kmem_cache_free(sigqueue_cachep, q);

kernel/ucount.c +20 -0

index 3b7e176cf7a2..687d77aa66bb 100644
@@ -285,6 +285,47 @@ bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v)
 	return (new == 0);
 }
 
+static void do_dec_rlimit_put_ucounts(struct ucounts *ucounts,
+				struct ucounts *last, enum ucount_type type)
+{
+	struct ucounts *iter;
+	for (iter = ucounts; iter != last; iter = iter->ns->ucounts) {
+		long dec = atomic_long_add_return(-1, &iter->ucount[type]);
+		WARN_ON_ONCE(dec < 0);
+		if (dec == 0)
+			put_ucounts(iter);
+	}
+}
+
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+	do_dec_rlimit_put_ucounts(ucounts, NULL, type);
+}
+
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+	struct ucounts *iter;

kernel/ucount.c +6 -0

+	WARN_ON_ONCE(dec < 0);
+unwind:
+	do_dec_rlimit_put_ucounts(ucounts, iter, type);
+	return LONG_MAX;
+}
+
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max)
 {
 	struct ucounts *iter;

Signature · 1 line

2.20.1

Quoted 134 lines · Eric W. Biederman

In commit fda31c50292a ("signal: avoid double atomic counter increments for user accounting") Linus made a clever optimization to how rlimits and the struct user_struct. Unfortunately that optimization does not work in the obvious way when moved to nested rlimits. The problem is that the last decrement of the per user namespace per user sigpending counter might also be the last decrement of the sigpending counter in the parent user namespace as well. Which means that simply freeing the leaf ucount in __free_sigqueue is not enough.

Maintain the optimization and handle the tricky cases by introducing inc_rlimit_get_ucounts and dec_rlimit_put_ucounts.

By moving the entire optimization into functions that perform all of the work it becomes possible to ensure that every level is handled properly.

I wish we had a single user across all of the threads whose rlimit could be charged so we did not need this complexity.

Cc [email protected]

Fixes d64696905554 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")

Signed-off-by "Eric W. Biederman" <[email protected]>

---

With a lot of help from Alex who found a way I could reproduce this I believe I have found the issue.

Could people who are seeing this issue test and verify this solves the problem for them?

 include/linux/user_namespace.h |  2 ++
 kernel/signal.c                | 25 +++++----------------
 kernel/ucount.c                | 41 ++++++++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 19 deletions(-)

include/linux/user_namespace.h +2 -0

index eb70cabe6e7f..33a4240e6a6f 100644
@@ -127,6 +127,8 @@ static inline long get_ucounts_value(struct ucounts *ucounts, enum ucount_type t
 
 long inc_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
 bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type);
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type);
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max);
 
 static inline void set_rlimit_ucount_max(struct user_namespace *ns,

kernel/signal.c +6 -19

index a3229add4455..762de58c6e76 100644
@@ -425,22 +425,10 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
 	 */
 	rcu_read_lock();
 	ucounts = task_ucounts(t);
-	sigpending = inc_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-	switch (sigpending) {
-	case 1:
-		if (likely(get_ucounts(ucounts)))
-			break;
-		fallthrough;
-	case LONG_MAX:
-		/*
-		 * we need to decrease the ucount in the userns tree on any
-		 * failure to avoid counts leaking.
-		 */
-		dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-		rcu_read_unlock();
-		return NULL;
-	}
+	sigpending = inc_rlimit_get_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
 	rcu_read_unlock();
+	if (sigpending == LONG_MAX)
+		return NULL;
 
 	if (override_rlimit || likely(sigpending <= task_rlimit(t, RLIMIT_SIGPENDING))) {
 		q = kmem_cache_alloc(sigqueue_cachep, gfp_flags);

@@ -449,8 +437,7 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
 	}
 
 	if (unlikely(q == NULL)) {
-		if (dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1))
-			put_ucounts(ucounts);
+		dec_rlimit_put_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
 	} else {
 		INIT_LIST_HEAD(&q->list);
 		q->flags = sigqueue_flags;

@@ -463,8 +450,8 @@ static void __sigqueue_free(struct sigqueue *q)
 {
 	if (q->flags & SIGQUEUE_PREALLOC)
 		return;
-	if (q->ucounts && dec_rlimit_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING, 1)) {
-		put_ucounts(q->ucounts);
+	if (q->ucounts) {
+		dec_rlimit_put_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING);
 		q->ucounts = NULL;
 	}
 	kmem_cache_free(sigqueue_cachep, q);

kernel/ucount.c +25 -0

index 3b7e176cf7a2..687d77aa66bb 100644
@@ -285,6 +285,47 @@ bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v)
 	return (new == 0);
 }
 
+static void do_dec_rlimit_put_ucounts(struct ucounts *ucounts,
+				struct ucounts *last, enum ucount_type type)
+{
+	struct ucounts *iter;
+	for (iter = ucounts; iter != last; iter = iter->ns->ucounts) {
+		long dec = atomic_long_add_return(-1, &iter->ucount[type]);
+		WARN_ON_ONCE(dec < 0);
+		if (dec == 0)
+			put_ucounts(iter);
+	}
+}
+
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+	do_dec_rlimit_put_ucounts(ucounts, NULL, type);
+}
+
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+	struct ucounts *iter;
+	long dec, ret = 0;
+
+	for (iter = ucounts; iter; iter = iter->ns->ucounts) {
+		long max = READ_ONCE(iter->ns->ucount_max[type]);
+		long new = atomic_long_add_return(1, &iter->ucount[type]);

kernel/ucount.c bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v) · line 317

Quoted 5 lines

kernel/ucount.c +5 -0

+		if (new < 0 || new > max)
+			goto unwind;
+		else if (iter == ucounts)
+			ret = new;
+		if ((new == 1) && (get_ucounts(iter) != iter))

get_ucounts can do put_ucounts. Are you sure it's correct to use get_ucounts here?

kernel/ucount.c bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v) · line 322

Quoted 5 lines

kernel/ucount.c +5 -0

+			goto dec_unwind;
+	}
+	return ret;
+dec_unwind:
+	dec = atomic_long_add_return(1, &iter->ucount[type]);

Should be -1 ?

Quoted 4 lines

kernel/ucount.c +4 -0

+	WARN_ON_ONCE(dec < 0);
+unwind:
+	do_dec_rlimit_put_ucounts(ucounts, iter, type);
+	return LONG_MAX;

Quoted 8 lines

kernel/ucount.c +2 -0

+}
+
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max)
 {
 	struct ucounts *iter;

Signature · 1 line

2.20.1

Quoted 135 lines · Alexey Gladkov

Quoted 134 lines · Eric W. Biederman

In commit fda31c50292a ("signal: avoid double atomic counter increments for user accounting") Linus made a clever optimization to how rlimits and the struct user_struct. Unfortunately that optimization does not work in the obvious way when moved to nested rlimits. The problem is that the last decrement of the per user namespace per user sigpending counter might also be the last decrement of the sigpending counter in the parent user namespace as well. Which means that simply freeing the leaf ucount in __free_sigqueue is not enough.

Maintain the optimization and handle the tricky cases by introducing inc_rlimit_get_ucounts and dec_rlimit_put_ucounts.

By moving the entire optimization into functions that perform all of the work it becomes possible to ensure that every level is handled properly.

I wish we had a single user across all of the threads whose rlimit could be charged so we did not need this complexity.

Cc [email protected]

Fixes d64696905554 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")

Signed-off-by "Eric W. Biederman" <[email protected]>

---

With a lot of help from Alex who found a way I could reproduce this I believe I have found the issue.

Could people who are seeing this issue test and verify this solves the problem for them?

 include/linux/user_namespace.h |  2 ++
 kernel/signal.c                | 25 +++++----------------
 kernel/ucount.c                | 41 ++++++++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 19 deletions(-)

include/linux/user_namespace.h +2 -0

index eb70cabe6e7f..33a4240e6a6f 100644
@@ -127,6 +127,8 @@ static inline long get_ucounts_value(struct ucounts *ucounts, enum ucount_type t
 
 long inc_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
 bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type);
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type);
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max);
 
 static inline void set_rlimit_ucount_max(struct user_namespace *ns,

kernel/signal.c +6 -19

index a3229add4455..762de58c6e76 100644
@@ -425,22 +425,10 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
      */
     rcu_read_lock();
     ucounts = task_ucounts(t);
-    sigpending = inc_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-    switch (sigpending) {
-    case 1:
-            if (likely(get_ucounts(ucounts)))
-                    break;
-            fallthrough;
-    case LONG_MAX:
-            /*
-             * we need to decrease the ucount in the userns tree on any
-             * failure to avoid counts leaking.
-             */
-            dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-            rcu_read_unlock();
-            return NULL;
-    }
+    sigpending = inc_rlimit_get_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
     rcu_read_unlock();
+    if (sigpending == LONG_MAX)
+            return NULL;
 
     if (override_rlimit || likely(sigpending <= task_rlimit(t, RLIMIT_SIGPENDING))) {
             q = kmem_cache_alloc(sigqueue_cachep, gfp_flags);

@@ -449,8 +437,7 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
     }
 
     if (unlikely(q == NULL)) {
-            if (dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1))
-                    put_ucounts(ucounts);
+            dec_rlimit_put_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
     } else {
             INIT_LIST_HEAD(&q->list);
             q->flags = sigqueue_flags;

@@ -463,8 +450,8 @@ static void __sigqueue_free(struct sigqueue *q)
 {
     if (q->flags & SIGQUEUE_PREALLOC)
             return;
-    if (q->ucounts && dec_rlimit_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING, 1)) {
-            put_ucounts(q->ucounts);
+    if (q->ucounts) {
+            dec_rlimit_put_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING);
             q->ucounts = NULL;
     }
     kmem_cache_free(sigqueue_cachep, q);

kernel/ucount.c +25 -0

index 3b7e176cf7a2..687d77aa66bb 100644
@@ -285,6 +285,47 @@ bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v)
     return (new == 0);
 }
 
+static void do_dec_rlimit_put_ucounts(struct ucounts *ucounts,
+                            struct ucounts *last, enum ucount_type type)
+{
+    struct ucounts *iter;
+    for (iter = ucounts; iter != last; iter = iter->ns->ucounts) {
+            long dec = atomic_long_add_return(-1, &iter->ucount[type]);
+            WARN_ON_ONCE(dec < 0);
+            if (dec == 0)
+                    put_ucounts(iter);
+    }
+}
+
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+    do_dec_rlimit_put_ucounts(ucounts, NULL, type);
+}
+
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+    struct ucounts *iter;
+    long dec, ret = 0;
+
+    for (iter = ucounts; iter; iter = iter->ns->ucounts) {
+            long max = READ_ONCE(iter->ns->ucount_max[type]);
+            long new = atomic_long_add_return(1, &iter->ucount[type]);

Quoted 8 lines

kernel/ucount.c bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v) · line 317

Quoted 5 lines

kernel/ucount.c +5 -0

+            if (new < 0 || new > max)
+                    goto unwind;
+            else if (iter == ucounts)
+                    ret = new;
+            if ((new == 1) && (get_ucounts(iter) != iter))

get_ucounts can do put_ucounts. Are you sure it's correct to use get_ucounts here?

My only concern would be if we could not run inc_rlimit_get_ucounts would not be safe to call under rcu_read_lock(). I don't see anything in get_ucounts or put_ucounts that would not be safe under rcu_read_lock().

For get_ucounts we do need to test to see if it fails. Either by testing for NULL or testing to see if it does not return the expected ucount.

Does that make sense or do you have another concern?

Quoted 6 lines

Quoted 5 lines

kernel/ucount.c +5 -0

+                    goto dec_unwind;
+    }
+    return ret;
+dec_unwind:
+    dec = atomic_long_add_return(1, &iter->ucount[type]);

Quoted 1 line

Should be -1 ?

Yes it should. I will fix and resend.

Quoted 4 lines

Quoted 4 lines

kernel/ucount.c +4 -0

+    WARN_ON_ONCE(dec < 0);
+unwind:
+    do_dec_rlimit_put_ucounts(ucounts, iter, type);
+    return LONG_MAX;

Quoted 8 lines

Quoted 8 lines

kernel/ucount.c +2 -1

+}
+
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max)
 {
     struct ucounts *iter;
--

2.20.1

Eric

Quoted 10 lines · Eric W. Biederman

Quoted 7 lines · Alexey Gladkov

Quoted 5 lines · Eric W. Biederman

kernel/ucount.c +5 -0

+                    goto dec_unwind;
+    }
+    return ret;
+dec_unwind:
+    dec = atomic_long_add_return(1, &iter->ucount[type]);

Quoted 3 lines

Quoted 1 line

Should be -1 ?

Yes it should. I will fix and resend.

Or just atomic_long_dec_return().

In commit fda31c50292a ("signal: avoid double atomic counter increments for user accounting") Linus made a clever optimization to how rlimits and the struct user_struct. Unfortunately that optimization does not work in the obvious way when moved to nested rlimits. The problem is that the last decrement of the per user namespace per user sigpending counter might also be the last decrement of the sigpending counter in the parent user namespace as well. Which means that simply freeing the leaf ucount in __free_sigqueue is not enough.

Maintain the optimization and handle the tricky cases by introducing inc_rlimit_get_ucounts and dec_rlimit_put_ucounts.

By moving the entire optimization into functions that perform all of the work it becomes possible to ensure that every level is handled properly.

I wish we had a single user across all of the threads whose rlimit could be charged so we did not need this complexity.

Cc [email protected]

Fixes d64696905554 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")

Signed-off-by "Eric W. Biederman" <[email protected]>

---

With a lot of help from Alex who found a way I could reproduce this I believe I have found the issue.

Could people who are seeing this issue test and verify this solves the problem for them?

 include/linux/user_namespace.h |  2 ++
 kernel/signal.c                | 25 +++++----------------
 kernel/ucount.c                | 41 ++++++++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 19 deletions(-)

include/linux/user_namespace.h +2 -0

index eb70cabe6e7f..33a4240e6a6f 100644
@@ -127,6 +127,8 @@ static inline long get_ucounts_value(struct ucounts *ucounts, enum ucount_type t
 
 long inc_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
 bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type);
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type);
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max);
 
 static inline void set_rlimit_ucount_max(struct user_namespace *ns,

kernel/signal.c +6 -19

index a3229add4455..762de58c6e76 100644
@@ -425,22 +425,10 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
 	 */
 	rcu_read_lock();
 	ucounts = task_ucounts(t);
-	sigpending = inc_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-	switch (sigpending) {
-	case 1:
-		if (likely(get_ucounts(ucounts)))
-			break;
-		fallthrough;
-	case LONG_MAX:
-		/*
-		 * we need to decrease the ucount in the userns tree on any
-		 * failure to avoid counts leaking.
-		 */
-		dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-		rcu_read_unlock();
-		return NULL;
-	}
+	sigpending = inc_rlimit_get_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
 	rcu_read_unlock();
+	if (sigpending == LONG_MAX)
+		return NULL;
 
 	if (override_rlimit || likely(sigpending <= task_rlimit(t, RLIMIT_SIGPENDING))) {
 		q = kmem_cache_alloc(sigqueue_cachep, gfp_flags);

@@ -449,8 +437,7 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
 	}
 
 	if (unlikely(q == NULL)) {
-		if (dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1))
-			put_ucounts(ucounts);
+		dec_rlimit_put_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
 	} else {
 		INIT_LIST_HEAD(&q->list);
 		q->flags = sigqueue_flags;

@@ -463,8 +450,8 @@ static void __sigqueue_free(struct sigqueue *q)
 {
 	if (q->flags & SIGQUEUE_PREALLOC)
 		return;
-	if (q->ucounts && dec_rlimit_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING, 1)) {
-		put_ucounts(q->ucounts);
+	if (q->ucounts) {
+		dec_rlimit_put_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING);
 		q->ucounts = NULL;
 	}
 	kmem_cache_free(sigqueue_cachep, q);

kernel/ucount.c +0 -0

index 3b7e176cf7a2..687d77aa66bb 100644
@@ -285,6 +285,47 @@ bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v)
 	return (new == 0);
 }
 

kernel/ucount.c +11 -0

+}
+
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+	do_dec_rlimit_put_ucounts(ucounts, NULL, type);
+}
+
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+	struct ucounts *iter;
+	long dec, ret = 0;

kernel/ucount.c +10 -0

+	}
+	return ret;
+dec_unwind:
+	dec = atomic_long_add_return(1, &iter->ucount[type]);
+	WARN_ON_ONCE(dec < 0);
+unwind:
+	do_dec_rlimit_put_ucounts(ucounts, iter, type);
+	return LONG_MAX;
+}
+
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max)
 {
 	struct ucounts *iter;

Signature · 1 line

2.20.1

Quoted 109 lines · On Fri, 15 Oct 2021 17:10:58 -0500 Eric W. Biederman wrote:

In commit fda31c50292a ("signal: avoid double atomic counter increments for user accounting") Linus made a clever optimization to how rlimits and the struct user_struct. Unfortunately that optimization does not work in the obvious way when moved to nested rlimits. The problem is that the last decrement of the per user namespace per user sigpending counter might also be the last decrement of the sigpending counter in the parent user namespace as well. Which means that simply freeing the leaf ucount in __free_sigqueue is not enough.

Maintain the optimization and handle the tricky cases by introducing inc_rlimit_get_ucounts and dec_rlimit_put_ucounts.

By moving the entire optimization into functions that perform all of the work it becomes possible to ensure that every level is handled properly.

I wish we had a single user across all of the threads whose rlimit could be charged so we did not need this complexity.

Cc [email protected]

Fixes d64696905554 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")

Signed-off-by "Eric W. Biederman" <[email protected]>

---

With a lot of help from Alex who found a way I could reproduce this I believe I have found the issue.

Could people who are seeing this issue test and verify this solves the problem for them?

 include/linux/user_namespace.h |  2 ++
 kernel/signal.c                | 25 +++++----------------
 kernel/ucount.c                | 41 ++++++++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 19 deletions(-)

include/linux/user_namespace.h +2 -0

index eb70cabe6e7f..33a4240e6a6f 100644
@@ -127,6 +127,8 @@ static inline long get_ucounts_value(struct ucounts *ucounts, enum ucount_type t
 
 long inc_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
 bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type);
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type);
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max);
 
 static inline void set_rlimit_ucount_max(struct user_namespace *ns,

kernel/signal.c +6 -19

index a3229add4455..762de58c6e76 100644
@@ -425,22 +425,10 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
 	 */
 	rcu_read_lock();
 	ucounts = task_ucounts(t);
-	sigpending = inc_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-	switch (sigpending) {
-	case 1:
-		if (likely(get_ucounts(ucounts)))
-			break;
-		fallthrough;
-	case LONG_MAX:
-		/*
-		 * we need to decrease the ucount in the userns tree on any
-		 * failure to avoid counts leaking.
-		 */
-		dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-		rcu_read_unlock();
-		return NULL;
-	}
+	sigpending = inc_rlimit_get_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
 	rcu_read_unlock();
+	if (sigpending == LONG_MAX)
+		return NULL;
 
 	if (override_rlimit || likely(sigpending <= task_rlimit(t, RLIMIT_SIGPENDING))) {
 		q = kmem_cache_alloc(sigqueue_cachep, gfp_flags);

@@ -449,8 +437,7 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
 	}
 
 	if (unlikely(q == NULL)) {
-		if (dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1))
-			put_ucounts(ucounts);
+		dec_rlimit_put_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
 	} else {
 		INIT_LIST_HEAD(&q->list);
 		q->flags = sigqueue_flags;

@@ -463,8 +450,8 @@ static void __sigqueue_free(struct sigqueue *q)
 {
 	if (q->flags & SIGQUEUE_PREALLOC)
 		return;
-	if (q->ucounts && dec_rlimit_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING, 1)) {
-		put_ucounts(q->ucounts);
+	if (q->ucounts) {
+		dec_rlimit_put_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING);
 		q->ucounts = NULL;
 	}
 	kmem_cache_free(sigqueue_cachep, q);

kernel/ucount.c +0 -0

index 3b7e176cf7a2..687d77aa66bb 100644
@@ -285,6 +285,47 @@ bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v)
 	return (new == 0);
 }
 

kernel/ucount.c bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v) · line 297

Quoted 10 lines

kernel/ucount.c +10 -0

+static void do_dec_rlimit_put_ucounts(struct ucounts *ucounts,
+				struct ucounts *last, enum ucount_type type)
+{
+	struct ucounts *iter;
+	for (iter = ucounts; iter != last; iter = iter->ns->ucounts) {
+		long dec = atomic_long_add_return(-1, &iter->ucount[type]);
+		WARN_ON_ONCE(dec < 0);
+		if (dec == 0)
+			put_ucounts(iter);
+	}

Given kfree in put_ucounts(), this has difficulty surviving tests like kasan if the put pairs with the get in the below inc_rlimit_get_ucounts().

Quoted 15 lines

kernel/ucount.c +15 -0

+}
+
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+	do_dec_rlimit_put_ucounts(ucounts, NULL, type);
+}
+
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+	struct ucounts *iter;
+	long dec, ret = 0;
+
+	for (iter = ucounts; iter; iter = iter->ns->ucounts) {
+		long max = READ_ONCE(iter->ns->ucount_max[type]);
+		long new = atomic_long_add_return(1, &iter->ucount[type]);

kernel/ucount.c bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v) · line 318

Quoted 6 lines

kernel/ucount.c +6 -0

+		if (new < 0 || new > max)
+			goto unwind;
+		else if (iter == ucounts)
+			ret = new;
+		if ((new == 1) && (get_ucounts(iter) != iter))
+			goto dec_unwind;

Add a line of comment for get to ease readers.

Hillf

Quoted 15 lines

kernel/ucount.c +10 -0

+	}
+	return ret;
+dec_unwind:
+	dec = atomic_long_add_return(1, &iter->ucount[type]);
+	WARN_ON_ONCE(dec < 0);
+unwind:
+	do_dec_rlimit_put_ucounts(ucounts, iter, type);
+	return LONG_MAX;
+}
+
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max)
 {
 	struct ucounts *iter;

Signature · 1 line

2.20.1

In commit fda31c50292a ("signal: avoid double atomic counter increments for user accounting") Linus made a clever optimization to how rlimits and the struct user_struct. Unfortunately that optimization does not work in the obvious way when moved to nested rlimits. The problem is that the last decrement of the per user namespace per user sigpending counter might also be the last decrement of the sigpending counter in the parent user namespace as well. Which means that simply freeing the leaf ucount in __free_sigqueue is not enough.

Maintain the optimization and handle the tricky cases by introducing inc_rlimit_get_ucounts and dec_rlimit_put_ucounts.

By moving the entire optimization into functions that perform all of the work it becomes possible to ensure that every level is handled properly.

I wish we had a single user across all of the threads whose rlimit could be charged so we did not need this complexity.

Hi!

After applying the below patch, the 5 most problematic servers have run without any issues for 23 hours. That never happened before the patch on 5.14, so the patch seems to have fixed the issue for me.

On Monday there will be more load on the servers, which caused them to crash faster without the patch. I will let you know if it happens again.

Best regards,
Rune

Quoted 20 lines · Eric W. Biederman

In commit fda31c50292a ("signal: avoid double atomic counter increments for user accounting") Linus made a clever optimization to how rlimits and the struct user_struct. Unfortunately that optimization does not work in the obvious way when moved to nested rlimits. The problem is that the last decrement of the per user namespace per user sigpending counter might also be the last decrement of the sigpending counter in the parent user namespace as well. Which means that simply freeing the leaf ucount in __free_sigqueue is not enough.

Maintain the optimization and handle the tricky cases by introducing inc_rlimit_get_ucounts and dec_rlimit_put_ucounts.

By moving the entire optimization into functions that perform all of the work it becomes possible to ensure that every level is handled properly.

I wish we had a single user across all of the threads whose rlimit could be charged so we did not need this complexity.

On Sun, Oct 17, 2021 at 10:47 AM Rune Kleveland

Quoted 2 lines · [email protected] wrote:

Hi!

Quoted 5 lines

After applying the below patch, the 5 most problematic servers have run

without any issues for 23 hours. That never happened before the patch on

5.14, so the patch seems to have fixed the issue for me.

Confirm. I couldn't reproduce the problem on 5.14 either.

Quoted 41 lines

On Monday there will be more load on the servers, which caused them to

crash faster without the patch. I will let you know if it happens again.

Best regards,

Rune

Quoted 31 lines · Eric W. Biederman

In commit fda31c50292a ("signal: avoid double atomic counter

increments for user accounting") Linus made a clever optimization to

how rlimits and the struct user_struct. Unfortunately that

optimization does not work in the obvious way when moved to nested

rlimits. The problem is that the last decrement of the per user

namespace per user sigpending counter might also be the last decrement

of the sigpending counter in the parent user namespace as well. Which

means that simply freeing the leaf ucount in __free_sigqueue is not

enough.

Maintain the optimization and handle the tricky cases by introducing

inc_rlimit_get_ucounts and dec_rlimit_put_ucounts.

By moving the entire optimization into functions that perform all of

the work it becomes possible to ensure that every level is handled

properly.

I wish we had a single user across all of the threads whose rlimit

could be charged so we did not need this complexity.

Hi "Eric,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linux/master]
[also build test ERROR on v5.15-rc6]
[cannot apply to linus/master jmorris-security/next-testing next-20211022] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Eric-W-Biederman/ucounts-Pair-inc_rlimit_ucounts-with-dec_rlimit_ucoutns-in-commit_creds/20211019-022345
base:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ec681c53f8d2d0ee362ff67f5b98dd8263c15002
config: arc-allyesconfig (attached as .config)
compiler: arceb-elf-gcc (GCC) 11.2.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/78e52b4abddecd7a6666f768874ee737c292830d
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Eric-W-Biederman/ucounts-Pair-inc_rlimit_ucounts-with-dec_rlimit_ucoutns-in-commit_creds/20211019-022345 git checkout 78e52b4abddecd7a6666f768874ee737c292830d
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross ARCH=arc

If you fix the issue, kindly add following tag as appropriate

Reported-by kernel test robot <[email protected]>

All errors (new ones prefixed by >>):

In file included from arch/arc/include/asm/bug.h:30,
from include/linux/bug.h:5,
from include/linux/thread_info.h:13,
from include/asm-generic/preempt.h:5,
from ./arch/arc/include/generated/asm/preempt.h:1, from include/linux/preempt.h:78,
from include/linux/rcupdate.h:27,
from include/linux/rculist.h:11,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from security/keys/process_keys.c:9:
security/keys/process_keys.c: In function 'key_change_session_keyring':

Quoted 1 line

Quoted 1 line

security/keys/process_keys.c:923:30: error: format '%s' expects a matching 'char *' argument [-Werror=format=]

     923 |                 WARN_ONCE(1, "In %s get_ucounts failed\n");
         |                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
   include/asm-generic/bug.h:91:62: note: in definition of macro '__WARN_printf'
      91 |                 warn_slowpath_fmt(__FILE__, __LINE__, taint, arg);      \
         |                                                              ^~~
   include/linux/once_lite.h:19:25: note: in expansion of macro 'WARN'
      19 |                         func(__VA_ARGS__);                              \
         |                         ^~~~
   include/asm-generic/bug.h:150:9: note: in expansion of macro 'DO_ONCE_LITE_IF'
     150 |         DO_ONCE_LITE_IF(condition, WARN, 1, format)
         |         ^~~~~~~~~~~~~~~
   security/keys/process_keys.c:923:17: note: in expansion of macro 'WARN_ONCE'
     923 |                 WARN_ONCE(1, "In %s get_ucounts failed\n");
         |                 ^~~~~~~~~
   security/keys/process_keys.c:923:35: note: format string is defined here
     923 |                 WARN_ONCE(1, "In %s get_ucounts failed\n");
         |                                  ~^
         |                                   |
         |                                   char *

cc1: all warnings being treated as errors

vim +923 security/keys/process_keys.c

   906	
   907	/*
   908	 * Replace a process's session keyring on behalf of one of its children when
   909	 * the target  process is about to resume userspace execution.
   910	 */
   911	void key_change_session_keyring(struct callback_head *twork)
   912	{
   913		const struct cred *old = current_cred();
   914		struct cred *new = container_of(twork, struct cred, rcu);
   915	
   916		if (unlikely(current->flags & PF_EXITING)) {
   917			put_cred(new);
   918			return;
   919		}
   920	
   921		/* If get_ucounts fails more bits are needed in the refcount */
   922		if (unlikely(!get_ucounts(old->ucounts))) {

Quoted 1 line

923 WARN_ONCE(1, "In %s get_ucounts failed\n");

   924			put_cred(new);
   925			return;
   926		}
   927	
   928		new->  uid	= old->  uid;
   929		new-> euid	= old-> euid;
   930		new-> suid	= old-> suid;
   931		new->fsuid	= old->fsuid;
   932		new->  gid	= old->  gid;
   933		new-> egid	= old-> egid;
   934		new-> sgid	= old-> sgid;
   935		new->fsgid	= old->fsgid;
   936		new->user	= get_uid(old->user);
   937		new->ucounts	= old->ucounts;
   938		new->user_ns	= get_user_ns(old->user_ns);
   939		new->group_info	= get_group_info(old->group_info);
   940	
   941		new->securebits	= old->securebits;
   942		new->cap_inheritable	= old->cap_inheritable;
   943		new->cap_permitted	= old->cap_permitted;
   944		new->cap_effective	= old->cap_effective;
   945		new->cap_ambient	= old->cap_ambient;
   946		new->cap_bset		= old->cap_bset;
   947	
   948		new->jit_keyring	= old->jit_keyring;
   949		new->thread_keyring	= key_get(old->thread_keyring);
   950		new->process_keyring	= key_get(old->process_keyring);
   951	
   952		security_transfer_creds(new, old);
   953	
   954		commit_creds(new);
   955	}
   956	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

The following changes are a set of miscellaneous fixes that makes the ucount code a little bit easier to read. There are all things that I ran into while helping hunt the crazy reference count bug.

I am aiming these at the next merge window and 5.16 rather than bug fixes to get into the current 5.15.

Eric W. Biederman (4):
ucounts: In set_cred_ucounts assume new->ucounts is non-NULL
ucounts: Remove unnecessary test for NULL ucount in get_ucounts
ucounts: Add get_ucounts_or_wrap for clarity
ucounts: Use atomic_long_sub_return for clarity

 kernel/cred.c   |  5 ++---
 kernel/ucount.c | 20 +++++++++++++-------
 2 files changed, 15 insertions(+), 10 deletions(-)

While digging into the previous ucount kernel crashes I found some minor bugs in the ucount code. When hit these bugs all result in a ucount either being counted in the wrong location or leak of a struct ucounts.

Nothing particularly serious but certainly things that should be fixed.

Eric W. Biederman (3):
ucounts: Pair inc_rlimit_ucounts with dec_rlimit_ucoutns in commit_creds
ucounts: Proper error handling in set_cred_ucounts
ucounts: Move get_ucounts from cred_alloc_blank to key_change_session_keyring

 kernel/cred.c                | 9 ++++-----
 security/keys/process_keys.c | 8 ++++++++
 2 files changed, 12 insertions(+), 5 deletions(-)

In commit fda31c50292a ("signal: avoid double atomic counter increments for user accounting") Linus made a clever optimization to how rlimits and the struct user_struct. Unfortunately that optimization does not work in the obvious way when moved to nested rlimits. The problem is that the last decrement of the per user namespace per user sigpending counter might also be the last decrement of the sigpending counter in the parent user namespace as well. Which means that simply freeing the leaf ucount in __free_sigqueue is not enough.

Maintain the optimization and handle the tricky cases by introducing inc_rlimit_get_ucounts and dec_rlimit_put_ucounts.

By moving the entire optimization into functions that perform all of the work it becomes possible to ensure that every level is handled properly.

The new function inc_rlimit_get_ucounts returns 0 on failure to increment the ucount. This is different than inc_rlimit_ucounts which increments the ucounts and returns LONG_MAX if the ucount counter has exceeded it's maximum or it wrapped (to indicate the counter needs to decremented).

I wish we had a single user to account all pending signals to across all of the threads of a process so this complexity was not necessary

Cc [email protected]

Fixes d64696905554 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")

v1: https://lkml.kernel.org/r/87mtnavszx.fsf_-_@disp2133

Tested-by Rune Kleveland <[email protected]>

Tested-by Yu Zhao <[email protected]>

Tested-by Jordan Glover <[email protected]>

Signed-off-by "Eric W. Biederman" <[email protected]>

---

This is my version of the fix with all of the feedback rolled in. I have tested it and believe this is ready to send out.

If folks code take a once over and see if I have spotted things.

For the people who are testing or have tested this I have added your tested-by's please let me know if you mind.

Eric

 include/linux/user_namespace.h |  2 ++
 kernel/signal.c                | 25 +++++------------
 kernel/ucount.c                | 49 ++++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+), 19 deletions(-)

include/linux/user_namespace.h +2 -0

index eb70cabe6e7f..33a4240e6a6f 100644
@@ -127,6 +127,8 @@ static inline long get_ucounts_value(struct ucounts *ucounts, enum ucount_type t
 
 long inc_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
 bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type);
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type);
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max);
 
 static inline void set_rlimit_ucount_max(struct user_namespace *ns,

kernel/signal.c +6 -19

index a3229add4455..13d2505a14a0 100644
@@ -425,22 +425,10 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
         */
        rcu_read_lock();
        ucounts = task_ucounts(t);
-       sigpending = inc_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-       switch (sigpending) {
-       case 1:
-               if (likely(get_ucounts(ucounts)))
-                       break;
-               fallthrough;
-       case LONG_MAX:
-               /*
-                * we need to decrease the ucount in the userns tree on any
-                * failure to avoid counts leaking.
-                */
-               dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
-               rcu_read_unlock();
-               return NULL;
-       }
+       sigpending = inc_rlimit_get_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
        rcu_read_unlock();
+       if (!sigpending)
+               return NULL;
 
        if (override_rlimit || likely(sigpending <= task_rlimit(t, RLIMIT_SIGPENDING))) {
                q = kmem_cache_alloc(sigqueue_cachep, gfp_flags);

@@ -449,8 +437,7 @@ __sigqueue_alloc(int sig, struct task_struct *t, gfp_t gfp_flags,
        }
 
        if (unlikely(q == NULL)) {
-               if (dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING, 1))
-                       put_ucounts(ucounts);
+               dec_rlimit_put_ucounts(ucounts, UCOUNT_RLIMIT_SIGPENDING);
        } else {
                INIT_LIST_HEAD(&q->list);
                q->flags = sigqueue_flags;

@@ -463,8 +450,8 @@ static void __sigqueue_free(struct sigqueue *q)
 {
        if (q->flags & SIGQUEUE_PREALLOC)
                return;
-       if (q->ucounts && dec_rlimit_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING, 1)) {
-               put_ucounts(q->ucounts);
+       if (q->ucounts) {
+               dec_rlimit_put_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING);
                q->ucounts = NULL;
        }
        kmem_cache_free(sigqueue_cachep, q);

kernel/ucount.c +49 -1

index bb51849e6375..eb03f3c68375 100644
@@ -284,6 +284,55 @@ bool dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v)
        return (new == 0);
 }
 
+static void do_dec_rlimit_put_ucounts(struct ucounts *ucounts,
+                               struct ucounts *last, enum ucount_type type)
+{
+       struct ucounts *iter, *next;
+       for (iter = ucounts; iter != last; iter = next) {
+               long dec = atomic_long_add_return(-1, &iter->ucount[type]);
+               WARN_ON_ONCE(dec < 0);
+               next = iter->ns->ucounts;
+               if (dec == 0)
+                       put_ucounts(iter);
+       }
+}
+
+void dec_rlimit_put_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+       do_dec_rlimit_put_ucounts(ucounts, NULL, type);
+}
+
+long inc_rlimit_get_ucounts(struct ucounts *ucounts, enum ucount_type type)
+{
+       /* Caller must hold a reference to ucounts */
+       struct ucounts *iter;
+       long dec, ret = 0;
+
+       for (iter = ucounts; iter; iter = iter->ns->ucounts) {
+               long max = READ_ONCE(iter->ns->ucount_max[type]);
+               long new = atomic_long_add_return(1, &iter->ucount[type]);
+               if (new < 0 || new > max)
+                       goto unwind;
+               if (iter == ucounts)
+                       ret = new;
+               /*
+                * Grab an extra ucount reference for the caller when
+                * the rlimit count was previously 0.
+                */
+               if (new != 1)
+                       continue;
+               if (!get_ucounts(iter))
+                       goto dec_unwind;
+       }
+       return ret;
+dec_unwind:
+       dec = atomic_long_add_return(-1, &iter->ucount[type]);
+       WARN_ON_ONCE(dec < 0);
+unwind:
+       do_dec_rlimit_put_ucounts(ucounts, iter, type);
+       return 0;
+}
+
 bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, unsigned long max)
 {
        struct ucounts *iter;
--

2.20.1

Quoted 23 lines · [email protected] (Eric W. Biederman) writes:

Adding Rune Kleveland to the discussion as he also seems to have reproduced the issue.

Alex and I have been starring at the code and the reports and this bug is hiding well. Here is what we have figured out so far.

Both the warning from free_user_ns calling dec_ucount that Jordan Glover reported and the KASAN error that Yu Zhao has reported appear to have the same cause. Using a ucounts structure after it has been freed and reallocated as something else.

I have just skimmed through the recent report from Rune Kleveland and it appears also to be a use after free. Especially since the second failure in the log is slub complaining about trying to free the ucounts data structure.

We looked through the users of put_ucounts and we don't see any obvious buggy users that would be freeing the data structure early.

Alex has tried to reproduce this so far is not having any luck. Folks can you tell what compiler versions you are using and share your kernel config with us? That might help.

Quoted 6 lines

The little debug diff below is my guess of what is happening. If the folks who can reproduce this issue can try the patch below and let me know if the warnings fire that would be appreciated. It is still not enough to track down the bug but at least it will confirm my current hypothesis about how things look before there is a use of memory after it is freed.

Bah. Scratch that test patch. I just double checked myself and cred->ucounts and cred->user_ns->ucounts should never be equal, as the user namespace is counted in it's parent user namespace.

That observation now tells me I have a parent user namespace that went corrupt.

Back to the drawing board.

Quoted 28 lines

Thank you,
Eric

kernel/cred.c +6 -0

index f784e08c2fbd..e7ffaa3cf5a6 100644
@@ -120,6 +120,12 @@ static void put_cred_rcu(struct rcu_head *rcu)
 	if (cred->group_info)
 		put_group_info(cred->group_info);
 	free_uid(cred->user);
+#if 1
+	if ((cred->ucounts == cred->user_ns->ucounts) &&
+	    (atomic_read(&cred->ucounts->count) == 1)) {
+		WARN_ONCE(1, "put_cred_rcu: ucount count 1\n");
+	}
+#endif
 	if (cred->ucounts)
 		put_ucounts(cred->ucounts);
 	put_user_ns(cred->user_ns);

kernel/exit.c +0 -0

index 91a43e57a32e..60fd88b34c1a 100644
@@ -743,6 +743,13 @@ void __noreturn do_exit(long code)
 	if (unlikely(!tsk->pid))
 		panic("Attempted to kill the idle task!");
 

kernel/exit.c void __noreturn do_exit(long code) · line 755

Quoted 10 lines

kernel/exit.c +7 -0

+#if 1
+	if ((tsk->cred->ucounts == tsk->cred->user_ns->ucounts) &&
+	    (atomic_read(tsk->cred->ucounts->count) == 1)) {
+		WARN_ONCE(1, "do_exit: ucount count 1\n");
+	}
+#endif
+
 	/*
 	 * If do_exit is called because this processes oopsed, it's possible
 	 * that get_fs() was left as KERNEL_DS, so reset it to USER_DS before

Quoted 4 lines · Alexey Gladkov

Quoted 2 lines · Jordan Glover

Quoted 2 lines

I'm still investigating, but I would like to rule out one option.

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

Could you check out the patch?

Thx, I added it to my kernel and will report in few days. Does this patch try to fix the issue or make it easier to track?

I suspect the error is caused by a race between allow_ucounts() and put_ucounts(). I think this patch could solve the problem.

Thanks for your help. Still can reproduce the problem with the change suggested.

[ 7761.885966] ==================================================================
[ 7761.893462] BUG: KASAN: use-after-free in dec_ucount+0x50/0xd8
[ 7761.899491] Write of size 8 at addr ffffff80c537b140 by task
kworker/u16:3/10303
[ 7761.907110]
[ 7761.908668] CPU: 0 PID: 10303 Comm: kworker/u16:3 Not tainted
5.14.0-lockdep+ #1
[ 7761.916289] Hardware name: Google Lazor (rev3+) with KB Backlight (DT)
[ 7761.923021] Workqueue: netns cleanup_net
[ 7761.927106] Call trace:
[ 7761.929648]  dump_backtrace+0x0/0x42c
[ 7761.933442]  show_stack+0x24/0x30
[ 7761.936878]  dump_stack_lvl+0xd0/0x100
[ 7761.940763]  print_address_description+0x30/0x304
[ 7761.945628]  kasan_report+0x190/0x1d8
[ 7761.949418]  kasan_check_range+0x1ac/0x1bc
[ 7761.953655]  __kasan_check_write+0x44/0x54
[ 7761.957891]  dec_ucount+0x50/0xd8
[ 7761.961334]  cleanup_net+0x630/0x718
[ 7761.965036]  process_one_work+0x7b4/0x10ec
[ 7761.969274]  worker_thread+0x800/0xcf4
[ 7761.973152]  kthread+0x2a8/0x358
[ 7761.976496]  ret_from_fork+0x10/0x18
[ 7761.980201]
[ 7761.981761] Allocated by task 4840:
[ 7761.985366]  kasan_save_stack+0x38/0x68
[ 7761.989342]  __kasan_kmalloc+0x9c/0xb8
[ 7761.993222]  kmem_cache_alloc_trace+0x2a4/0x370
[ 7761.997905]  alloc_ucounts+0x150/0x374
[ 7762.001787]  set_cred_ucounts+0x198/0x248
[ 7762.005935]  __sys_setresuid+0x31c/0x4f8
[ 7762.009993]  __arm64_sys_setresuid+0x84/0x98
[ 7762.014410]  invoke_syscall+0xd4/0x2c8
[ 7762.018292]  el0_svc_common+0x124/0x200
[ 7762.022265]  do_el0_svc_compat+0x54/0x64
[ 7762.026325]  el0_svc_compat+0x24/0x34
[ 7762.030124]  el0t_32_sync_handler+0xc0/0xf0
[ 7762.034451]  el0t_32_sync+0x19c/0x1a0
[ 7762.038241]
[ 7762.039799] Freed by task 0:
[ 7762.042778]  kasan_save_stack+0x38/0x68
[ 7762.046747]  kasan_set_track+0x28/0x3c
[ 7762.050625]  kasan_set_free_info+0x28/0x4c
[ 7762.054857]  ____kasan_slab_free+0x118/0x164
[ 7762.059277]  __kasan_slab_free+0x18/0x28
[ 7762.063339]  kfree+0x2f8/0x500
[ 7762.066505]  put_ucounts+0x11c/0x134
[ 7762.070209]  put_cred_rcu+0x1bc/0x35c
[ 7762.074006]  rcu_core+0xa68/0x1b20
[ 7762.077538]  rcu_core_si+0x1c/0x28
[ 7762.081061]  __do_softirq+0x4bc/0xedc
[ 7762.084851]
[ 7762.086401] The buggy address belongs to the object at ffffff80c537b100
[ 7762.086401]  which belongs to the cache kmalloc-256 of size 256
[ 7762.099267] The buggy address is located 64 bytes inside of
[ 7762.099267]  256-byte region [ffffff80c537b100, ffffff80c537b200)
[ 7762.111248] The buggy address belongs to the page:
[ 7762.116185] page:fffffffe0314de00 refcount:1 mapcount:0
mapping:0000000000000000 index:0xffffff80c537ad00 pfn:0x145378
[ 7762.127180] head:fffffffe0314de00 order:3 compound_mapcount:0
compound_pincount:0
[ 7762.134881] flags: 0x8000000000010200(slab|head|zone=2)
[ 7762.140286] raw: 8000000000010200 fffffffe02799408 fffffffe02020808
ffffff808000c980
[ 7762.148263] raw: ffffff80c537ad00 0000000000200004 00000001ffffffff
0000000000000000
[ 7762.156228] page dumped because: kasan: bad access detected
[ 7762.161974]
[ 7762.163532] Memory state around the buggy address:
[ 7762.168475]  ffffff80c537b000: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7762.175915]  ffffff80c537b080: fc fc fc fc fc fc fc fc fc fc fc fc

fc fc fc fc

[ 7762.183346] >ffffff80c537b100: fa fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7762.190774]                                            ^
[ 7762.196258]  ffffff80c537b180: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7762.203689]  ffffff80c537b200: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7762.211125] ==================================================================

Quoted 89 lines · On Thu, 30 Sep 2021 16:27:34 -0600 Yu Zhao wrote:

Quoted 4 lines · Alexey Gladkov

Quoted 2 lines · Jordan Glover

Quoted 2 lines

I'm still investigating, but I would like to rule out one option.

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

Could you check out the patch?

Thx, I added it to my kernel and will report in few days. Does this patch try to fix the issue or make it easier to track?

I suspect the error is caused by a race between allow_ucounts() and put_ucounts(). I think this patch could solve the problem.

Thanks for your help. Still can reproduce the problem with the change suggested.

[ 7761.885966] ==================================================================
[ 7761.893462] BUG: KASAN: use-after-free in dec_ucount+0x50/0xd8
[ 7761.899491] Write of size 8 at addr ffffff80c537b140 by task
kworker/u16:3/10303
[ 7761.907110]
[ 7761.908668] CPU: 0 PID: 10303 Comm: kworker/u16:3 Not tainted
5.14.0-lockdep+ #1
[ 7761.916289] Hardware name: Google Lazor (rev3+) with KB Backlight (DT)
[ 7761.923021] Workqueue: netns cleanup_net
[ 7761.927106] Call trace:
[ 7761.929648]  dump_backtrace+0x0/0x42c
[ 7761.933442]  show_stack+0x24/0x30
[ 7761.936878]  dump_stack_lvl+0xd0/0x100
[ 7761.940763]  print_address_description+0x30/0x304
[ 7761.945628]  kasan_report+0x190/0x1d8
[ 7761.949418]  kasan_check_range+0x1ac/0x1bc
[ 7761.953655]  __kasan_check_write+0x44/0x54
[ 7761.957891]  dec_ucount+0x50/0xd8
[ 7761.961334]  cleanup_net+0x630/0x718
[ 7761.965036]  process_one_work+0x7b4/0x10ec
[ 7761.969274]  worker_thread+0x800/0xcf4
[ 7761.973152]  kthread+0x2a8/0x358
[ 7761.976496]  ret_from_fork+0x10/0x18
[ 7761.980201]
[ 7761.981761] Allocated by task 4840:
[ 7761.985366]  kasan_save_stack+0x38/0x68
[ 7761.989342]  __kasan_kmalloc+0x9c/0xb8
[ 7761.993222]  kmem_cache_alloc_trace+0x2a4/0x370
[ 7761.997905]  alloc_ucounts+0x150/0x374
[ 7762.001787]  set_cred_ucounts+0x198/0x248
[ 7762.005935]  __sys_setresuid+0x31c/0x4f8
[ 7762.009993]  __arm64_sys_setresuid+0x84/0x98
[ 7762.014410]  invoke_syscall+0xd4/0x2c8
[ 7762.018292]  el0_svc_common+0x124/0x200
[ 7762.022265]  do_el0_svc_compat+0x54/0x64
[ 7762.026325]  el0_svc_compat+0x24/0x34
[ 7762.030124]  el0t_32_sync_handler+0xc0/0xf0
[ 7762.034451]  el0t_32_sync+0x19c/0x1a0
[ 7762.038241]
[ 7762.039799] Freed by task 0:
[ 7762.042778]  kasan_save_stack+0x38/0x68
[ 7762.046747]  kasan_set_track+0x28/0x3c
[ 7762.050625]  kasan_set_free_info+0x28/0x4c
[ 7762.054857]  ____kasan_slab_free+0x118/0x164
[ 7762.059277]  __kasan_slab_free+0x18/0x28
[ 7762.063339]  kfree+0x2f8/0x500
[ 7762.066505]  put_ucounts+0x11c/0x134
[ 7762.070209]  put_cred_rcu+0x1bc/0x35c
[ 7762.074006]  rcu_core+0xa68/0x1b20
[ 7762.077538]  rcu_core_si+0x1c/0x28
[ 7762.081061]  __do_softirq+0x4bc/0xedc
[ 7762.084851]
[ 7762.086401] The buggy address belongs to the object at ffffff80c537b100
[ 7762.086401]  which belongs to the cache kmalloc-256 of size 256
[ 7762.099267] The buggy address is located 64 bytes inside of
[ 7762.099267]  256-byte region [ffffff80c537b100, ffffff80c537b200)
[ 7762.111248] The buggy address belongs to the page:
[ 7762.116185] page:fffffffe0314de00 refcount:1 mapcount:0
mapping:0000000000000000 index:0xffffff80c537ad00 pfn:0x145378
[ 7762.127180] head:fffffffe0314de00 order:3 compound_mapcount:0
compound_pincount:0
[ 7762.134881] flags: 0x8000000000010200(slab|head|zone=2)
[ 7762.140286] raw: 8000000000010200 fffffffe02799408 fffffffe02020808
ffffff808000c980
[ 7762.148263] raw: ffffff80c537ad00 0000000000200004 00000001ffffffff
0000000000000000
[ 7762.156228] page dumped because: kasan: bad access detected
[ 7762.161974]
[ 7762.163532] Memory state around the buggy address:
[ 7762.168475]  ffffff80c537b000: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7762.175915]  ffffff80c537b080: fc fc fc fc fc fc fc fc fc fc fc fc

fc fc fc fc

Quoted 8 lines

[ 7762.183346] >ffffff80c537b100: fa fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7762.190774]                                            ^
[ 7762.196258]  ffffff80c537b180: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7762.203689]  ffffff80c537b200: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7762.211125] ==================================================================

Could you please check if it is due to count underflow? Given nothing wrong on the other side based on the efforts
"We looked through the users of put_ucounts and we don't see any obvious buggy users that would be freeing the data structure early."

Could you please check if it is due to count underflow? Given nothing wrong

on the other side based on the efforts

"We looked through the users of put_ucounts and we don't see any obvious buggy

users that would be freeing the data structure early."

Thanks

Hillf

--- linux-5.14.4/kernel/ucount.c

+++ b/kernel/ucount.c

(quoted diff) +0 -4

@@ -152,7 +152,10 @@ static void hlist_add_ucounts(struct uco
 
 struct ucounts *get_ucounts(struct ucounts *ucounts)
 
 {
 
-   if (ucounts && atomic_add_negative(1, &ucounts->count)) {
 
-   if (!ucounts)
 
-         return NULL;
 
 
-   WARN_ON(!atomic_read(&ucounts->count));

For me above patch changed slightly the printed output. Now the warning comes from 'cleanup_net' instead of 'free_user_ns'. My system was also still responsive after the bug occurred which didn't happen previously. I can't say if this means anything or if this is result of above patch or instability of my reproducer.

------------[ cut here ]------------
WARNING: CPU: 2 PID: 27643 at kernel/ucount.c:256 dec_ucount+0x43/0x50
Modules linked in: <cut>
CPU: 2 PID: 27643 Comm: kworker/u8:3 Not tainted 5.14.9 #1 0274f3d0712a6dad=
c9a2cf8341ae333de732a31a
Workqueue: netns cleanup_net
RIP: 0010:dec_ucount+0x43/0x50
Code: 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b =
41 10 48 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 fd fc ff ff <0f> 0b eb e7 66 0=
f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 f8 48
RSP: 0018:ffffb34fc34cfe30 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffffa448eec5f3b0 RCX: ffffa447cfe1f540
RDX: ffffa447cfe1f580 RSI: ffffffffffffffff RDI: ffffa447c445c780
RBP: ffffa448eec5f380 R08: 0000000000000040 R09: ffffa44a196ac040
R10: 00000000001436be R11: 0000000000000259 R12: ffffb34fc34cfe10
R13: ffffb34fc34cfe40 R14: 00000000ffffffff R15: ffffa448eec5d414
FS:  0000000000000000(0000) GS:ffffa44a19700000(0000) knlGS:000000000000000=
0
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000072a95d359030 CR3: 000000000b20e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
cleanup_net+0x2e2/0x370
process_one_work+0x1e1/0x380
worker_thread+0x50/0x3a0
? rescuer_thread+0x360/0x360
kthread+0x127/0x150
? set_kthread_struct+0x40/0x40
ret_from_fork+0x22/0x30
---[ end trace e5fdc3317f00d0e8 ]---
BUG: kernel NULL pointer dereference, address: 00000000000001e8
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] SMP PTI
CPU: 2 PID: 27643 Comm: kworker/u8:3 Tainted: G        W         5.14.9 #1 =
0274f3d0712a6dadc9a2cf8341ae333de732a31a
Workqueue: netns cleanup_net
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 =
89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 0=
0 00 48 85 c9 75 d9 e9 fd fc ff ff 0f 0b eb e7 66
RSP: 0018:ffffb34fc34cfe30 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffffa448eec5f3b0 RCX: ffffa447cfe1f540
RDX: ffffa447cfe1f580 RSI: ffffffffffffffff RDI: ffffa447c445c780
RBP: ffffa448eec5f380 R08: 0000000000000040 R09: ffffa44a196ac040
R10: 00000000001436be R11: 0000000000000259 R12: ffffb34fc34cfe10
R13: ffffb34fc34cfe40 R14: 00000000ffffffff R15: ffffa448eec5d414
FS:  0000000000000000(0000) GS:ffffa44a19700000(0000) knlGS:000000000000000=
0
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 000000000b20e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
cleanup_net+0x2e2/0x370
process_one_work+0x1e1/0x380
worker_thread+0x50/0x3a0
? rescuer_thread+0x360/0x360
kthread+0x127/0x150
? set_kthread_struct+0x40/0x40
ret_from_fork+0x22/0x30
Modules linked in: <cut>
CR2: 00000000000001e8
---[ end trace e5fdc3317f00d0e9 ]---
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 =
89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 0=
0 00 48 85 c9 75 d9 e9 fd fc ff ff 0f 0b eb e7 66
RSP: 0018:ffffb34fc34cfe30 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffffa448eec5f3b0 RCX: ffffa447cfe1f540
RDX: ffffa447cfe1f580 RSI: ffffffffffffffff RDI: ffffa447c445c780
RBP: ffffa448eec5f380 R08: 0000000000000040 R09: ffffa44a196ac040
R10: 00000000001436be R11: 0000000000000259 R12: ffffb34fc34cfe10
R13: ffffb34fc34cfe40 R14: 00000000ffffffff R15: ffffa448eec5d414
FS:  0000000000000000(0000) GS:ffffa44a19700000(0000) knlGS:000000000000000=
0
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 000000000b20e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Quoted 32 lines · Jordan Glover

Quoted 20 lines · On Thursday, September 16th, 2021 at 5:30 PM, [email protected] wrote:

Quoted 9 lines · Jordan Glover [email protected] writes:

Quoted 3 lines · On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover…

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

Heh, it jut happened again. This time closing firefox alone had such

effect:

Ok. It looks like he have a couple of folks seeing issues here.

I thought we had all of the issues sorted out for the release of v5.14,

but it looks like there is still some little bug left.

If Alex doesn't beat me to it I will see if I can come up with a

Quoted 7 lines

debugging patch to make it easy to help track down where the reference

count is going wrong. It will be a little bit as my brain is mush at

the moment.

Eric

As the issue persist in 5.14.7 I would be very interested in such patch.

Quoted 7 lines

For now the thing is mostly reproducible when I close several tabs in ff then

close the browser in short period of time. When I close tabs then wait out

a bit then close the browser it doesn't happen so I guess some interrupted

cleanup triggers it.

I'm still investigating, but I would like to rule out one option.

Could you check out the patch?

kernel/ucount.c +0 -0

index bb51849e6375..f23f906f4f62 100644

--- a/kernel/ucount.c

+++ b/kernel/ucount.c

kernel/ucount.c +0 -5

@@ -201,11 +201,14 @@ void put_ucounts(struct ucounts *ucounts)
 
 {
 
 unsigned long flags;
 
-         if (atomic_dec_and_lock_irqsave(&ucounts->count, &ucounts_lock, flags)) {
 
 
 
-         spin_lock_irqsave(&ucounts_lock, flags);
 
 
-         if (atomic_dec_and_test(&ucounts->count)) {
 
                  hlist_del_init(&ucounts->node);
 
                  spin_unlock_irqrestore(&ucounts_lock, flags);
                  kfree(ucounts);
 
 
-                 return;
          }
 
 
-         spin_unlock_irqrestore(&ucounts_lock, flags);

}

static inline bool atomic_long_inc_below(atomic_long_t *v, int u)

---------------------------------------------------------------------

Rgrds, legion