Panic#1 Part16 <0>[ 661.333520] gem_exec-2454 20.l... 238223377us : __intel_context_do_pin_ww: 0000:00:02.0 bcs0: context:1d6 active <0>[ 661.458292] gem_exec-2454 20.l... 238223378us : intel_timeline_pin: 0000:00:02.0 timeline:1d6 using HWSP offset:fedc2000 <0>[ 661.591302] gem_exec-2454 20.l... 238223384us : __intel_context_do_pin_ww: 0000:00:02.0 bcs0: context:1d6 pin ring:{start:fedba000, head:0b00, tail:0b00} <0>[ 661.616876] watchdog: BUG: soft lockup - CPU#1 stuck for 71s! [prometheus-node:772] <4>[ 661.616878] Modules linked in: vgem snd_hda_codec_intelhdmi snd_hda_codec_hdmi i915 prime_numbers drm_buddy ttm drm_display_helper cec rc_core i2c_algo_bit kunit overlay binfmt_misc hid_sensor_custom intel_uncore_frequency intel_uncore_frequency_common x86_pkg_temp_thermal intel_powerclamp coretemp hid_sensor_hub kvm_intel snd_hda_intel hid_generic snd_intel_dspcfg cmdlinepart kvm snd_hda_codec intel_ishtp_hid snd_hda_core hid spi_nor snd_hwdep mei_gsc_proxy irqbypass mtd ghash_clmulni_intel snd_pcm aesni_intel intel_rapl_msr wmi_bmof rapl intel_cstate nls_iso8859_1 i2c_i801 ucsi_acpi snd_timer spi_intel_pci i2c_mux mei_me typec_ucsi snd spi_intel video i2c_smbus soundcore mei typec idma64 intel_ish_ipc e1000e intel_ishtp intel_vpu intel_pmc_core processor_thermal_device_pci processor_thermal_device intel_skl_int3472_tps68470 processor_thermal_wt_hint tps68470_regulator platform_temperature_control clk_tps68470 processor_thermal_soc_slider platform_profile processor_thermal_rfim int3403_thermal processor_thermal_rapl Panic#1 Part15 <4>[ 661.616896] pmt_telemetry pmt_discovery intel_rapl_common pmt_class processor_thermal_wt_req intel_skl_int3472_discrete acpi_tad processor_thermal_power_floor thunderbolt intel_skl_int3472_common processor_thermal_mbox igen6_edac int340x_thermal_zone intel_pmc_ssram_telemetry intel_hid int3400_thermal acpi_pad wmi sparse_keymap intel_vsec acpi_thermal_rel pinctrl_meteorlake dm_multipath msr nvme_fabrics fuse efi_pstore nfnetlink autofs4 <4>[ 661.616904] irq event stamp: 1752176 <4>[ 661.616904] hardirqs last enabled at (1752175): [] irqentry_exit+0x6a/0x780 <4>[ 661.616906] hardirqs last disabled at (1752176): [] sysvec_apic_timer_interrupt+0x11/0xc0 <4>[ 661.616908] softirqs last enabled at (1752164): [] __irq_exit_rcu+0x13f/0x160 <4>[ 661.616910] softirqs last disabled at (1752153): [] __irq_exit_rcu+0x13f/0x160 <4>[ 661.616912] CPU: 1 UID: 110 PID: 772 Comm: prometheus-node Tainted: G S U W L 7.0.0-Patchwork_164993v2-gda91cb4e6137+ #1 PREEMPT(lazy) <4>[ 661.616914] Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER, [W]=WARN, [L]=SOFTLOCKUP <4>[ 661.616914] Hardware name: Intel Corporation Meteor Lake Client Platform/MTL-P LP5x T3 RVP, BIOS MTLPFWI1.R00.3471.D91.2401310918 01/31/2024 <4>[ 661.616915] RIP: 0010:smp_call_function_many_cond+0x1a9/0x870 <4>[ 661.616916] Code: 73 35 4c 63 e0 49 8b 1f 49 81 fc 00 20 00 00 0f 83 8d 06 00 00 4a 03 1c e5 60 31 26 83 8b 53 08 48 89 de 83 e2 01 74 0a f3 90 <8b> 4e 08 83 e1 01 75 f6 83 c0 01 eb b1 48 83 c4 60 5b 41 5c 41 5d Panic#1 Part14 <4>[ 661.616917] RSP: 0000:ffffc90009643b70 EFLAGS: 00000202 <4>[ 661.616917] RAX: 0000000000000003 RBX: ffff88885f5c2240 RCX: 0000000000000001 <4>[ 661.616918] RDX: 0000000000000001 RSI: ffff88885f5c2240 RDI: 0000000000000000 <4>[ 661.616918] RBP: ffffc90009643bf8 R08: 0000000000000000 R09: 0000000000000000 <4>[ 661.616919] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003 <4>[ 661.616919] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88885f4b6f40 <4>[ 661.616920] FS: 00007177898b6740(0000) GS:ffff8888db517000(0000) knlGS:0000000000000000 <4>[ 661.616920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 661.616921] CR2: 00007177402630d8 CR3: 0000000120a1f001 CR4: 0000000000f72ef0 <4>[ 661.616921] PKRU: 55555554 <4>[ 661.616922] Call Trace: <4>[ 661.616922] <4>[ 661.616924] ? __pfx_flush_tlb_func+0x10/0x10 <4>[ 661.616927] on_each_cpu_cond_mask+0x24/0x60 <4>[ 661.616928] native_flush_tlb_multi+0x172/0x220 <4>[ 661.616931] flush_tlb_mm_range+0x2d1/0x830 <4>[ 661.616933] ? _raw_spin_lock+0x37/0x60 <4>[ 661.616934] ? pte_offset_map_lock+0x97/0x180 <4>[ 661.616938] ptep_clear_flush+0x69/0x90 <4>[ 661.616940] do_wp_page+0x85a/0xef0 <4>[ 661.616943] __handle_mm_fault+0xaec/0x1030 <4>[ 661.616946] ? lock_vma_under_rcu+0xda/0x2f0 <4>[ 661.616951] handle_mm_fault+0x12c/0x300 <4>[ 661.616953] do_user_addr_fault+0x1d6/0x8e0 <4>[ 661.616957] exc_page_fault+0x96/0x2c0 <4>[ 661.616960] asm_exc_page_fault+0x27/0x30 <4>[ 661.616960] RIP: 0033:0x43695c Panic#1 Part13 <4>[ 661.616962] Code: f9 03 75 09 48 8d 05 73 3b d2 00 eb 07 48 8d 05 62 3b d2 00 48 8b 4c 24 40 48 8b 19 48 89 1a 48 8b 18 48 89 5a 08 90 48 89 d3 <48> 87 11 90 48 87 18 90 90 48 8d 05 b4 76 d8 00 e8 cf bc fd ff 48 <4>[ 661.616962] RSP: 002b:000000c0002b34f0 EFLAGS: 00010246 <4>[ 661.616963] RAX: 000000000115a4a0 RBX: 000071774004e620 RCX: 00007177402630d8 <4>[ 661.616964] RDX: 000071774004e620 RSI: 000000c0002b3580 RDI: 000071774004e650 <4>[ 661.616964] RBP: 000000c0002b3538 R08: 0000000000003620 R09: 0000000000000003 <4>[ 661.616965] R10: 0000000000000002 R11: f6519b57865a1c24 R12: 0000000000fcba68 <4>[ 661.616965] R13: 000000c0002b3580 R14: 000000c000007340 R15: 0000000000007fff <4>[ 661.616970] <6>[ 661.616970] Sending NMI from CPU 1 to CPUs 0,2-21: <0>[ 661.758306] gem_exec-2454 20..... 238225046us : __i915_request_commit: 0000:00:02.0 bcs0: fence 1d6:406, current 404 <4>[ 661.848908] NMI backtrace for cpu 12 <4>[ 661.848910] CPU: 12 UID: 0 PID: 2962 Comm: kworker/u89:313 Tainted: G S U W L 7.0.0-Patchwork_164993v2-gda91cb4e6137+ #1 PREEMPT(lazy) <4>[ 661.848912] Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER, [W]=WARN, [L]=SOFTLOCKUP <4>[ 661.848912] Hardware name: Intel Corporation Meteor Lake Client Platform/MTL-P LP5x T3 RVP, BIOS MTLPFWI1.R00.3471.D91.2401310918 01/31/2024 <4>[ 661.848914] Workqueue: writeback wb_workfn (flush-259:0) <4>[ 661.848917] RIP: 0010:smp_call_function_many_cond+0x1a9/0x870 <4>[ 661.848919] Code: 73 35 4c 63 e0 49 8b 1f 49 81 fc 00 20 00 00 0f 83 8d 06 00 00 4a 03 1c e5 60 31 26 83 8b 53 08 48 89 de 83 e2 01 74 0a f3 90 <8b> 4e 08 83 e1 01 75 f6 83 c0 01 eb b1 48 83 c4 60 5b 41 5c 41 5d Panic#1 Part12 <4>[ 661.848920] RSP: 0018:ffffc90049da3300 EFLAGS: 00000202 <4>[ 661.848921] RAX: 0000000000000003 RBX: ffff88885f5c22e0 RCX: 0000000000000001 <4>[ 661.848921] RDX: 0000000000000001 RSI: ffff88885f5c22e0 RDI: 0000000000000000 <4>[ 661.848922] RBP: ffffc90049da3388 R08: 0000000000000000 R09: 0000000000000000 <4>[ 661.848922] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003 <4>[ 661.848923] R13: 0000000000000001 R14: 000000000000000c R15: ffff88885fa36f40 <4>[ 661.848924] FS: 0000000000000000(0000) GS:ffff8888dba97000(0000) knlGS:0000000000000000 <4>[ 661.848924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 661.848925] CR2: 00007fb673cb5150 CR3: 000000000344c006 CR4: 0000000000f72ef0 <4>[ 661.848926] PKRU: 55555554 <4>[ 661.848926] Call Trace: <4>[ 661.848926] <4>[ 661.848928] ? __pfx_flush_tlb_func+0x10/0x10 <4>[ 661.848931] on_each_cpu_cond_mask+0x24/0x60 <4>[ 661.848932] native_flush_tlb_multi+0x172/0x220 <4>[ 661.848935] flush_tlb_mm_range+0x2d1/0x830 <4>[ 661.848937] ? page_vma_mapped_walk+0x661/0xe50 <4>[ 661.848940] ptep_clear_flush+0x69/0x90 <4>[ 661.848942] page_vma_mkclean_one.constprop.0+0x15e/0x2e0 <4>[ 661.848945] page_mkclean_one+0x88/0xc0 <4>[ 661.848948] __rmap_walk_file+0xdf/0x200 <4>[ 661.848950] rmap_walk+0x43/0xa0 <4>[ 661.848952] folio_mkclean+0xb1/0xc0 <4>[ 661.848954] ? __pfx_page_mkclean_one+0x10/0x10 <4>[ 661.848955] ? __pfx_invalid_mkclean_vma+0x10/0x10 <4>[ 661.848956] folio_clear_dirty_for_io+0x6e/0x2d0 <4>[ 661.848959] mpage_submit_folio+0x30/0xc0 <4>[ 661.848961] mpage_process_page_bufs+0xcb/0x190 Panic#1 Part11 <4>[ 661.848963] mpage_prepare_extent_to_map+0x1e2/0x590 <4>[ 661.848965] ? mark_held_locks+0x46/0x90 <4>[ 661.848968] ? srso_alias_untrain_ret+0x3/0x10 <4>[ 661.848970] ? _raw_spin_unlock_irqrestore+0x51/0x80 <4>[ 661.848972] ? __create_object+0x68/0xc0 <4>[ 661.848976] ? kmemleak_alloc+0x4a/0xa0 <4>[ 661.848978] ? kmem_cache_alloc_noprof+0x49c/0x690 <4>[ 661.848981] ? ext4_init_io_end+0x1f/0x60 <4>[ 661.848984] ext4_do_writepages+0x41b/0x11f0 <4>[ 661.848987] ? do_writepages+0xb7/0x170 <4>[ 661.848991] ext4_writepages+0x109/0x2e0 <4>[ 661.848992] ? ext4_writepages+0x109/0x2e0 <4>[ 661.848997] do_writepages+0xb7/0x170 <4>[ 661.848999] __writeback_single_inode+0x5a/0x600 <4>[ 661.849002] writeback_sb_inodes+0x2ac/0x8e0 <4>[ 661.849012] __writeback_inodes_wb+0x54/0xf0 <4>[ 661.849015] wb_writeback+0x201/0x480 <4>[ 661.849017] ? mark_held_locks+0x46/0x90 <4>[ 661.849020] ? get_nr_dirty_inodes+0x60/0xc0 <4>[ 661.849024] wb_workfn+0x421/0x530 <4>[ 661.849028] process_one_work+0x239/0x760 <4>[ 661.849031] worker_thread+0x200/0x3f0 <4>[ 661.849033] ? __pfx_worker_thread+0x10/0x10 <4>[ 661.849034] kthread+0x10d/0x150 <4>[ 661.849036] ? __pfx_kthread+0x10/0x10 <4>[ 661.849039] ret_from_fork+0x3d4/0x480 <4>[ 661.849040] ? __pfx_kthread+0x10/0x10 <4>[ 661.849043] ret_from_fork_asm+0x1a/0x30 <4>[ 661.849048] <4>[ 661.849049] NMI backtrace for cpu 20 <4>[ 661.849051] CPU: 20 UID: 1000 PID: 2991 Comm: pool-1-thread-1 Tainted: G S U W L 7.0.0-Patchwork_164993v2-gda91cb4e6137+ #1 PREEMPT(lazy) <4>[ 661.849054] Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER, [W]=WARN, [L]=SOFTLOCKUP Panic#1 Part10 <4>[ 661.849054] Hardware name: Intel Corporation Meteor Lake Client Platform/MTL-P LP5x T3 RVP, BIOS MTLPFWI1.R00.3471.D91.2401310918 01/31/2024 <4>[ 661.849055] RIP: 0010:smp_call_function_many_cond+0x1ac/0x870 <4>[ 661.849057] Code: 63 e0 49 8b 1f 49 81 fc 00 20 00 00 0f 83 8d 06 00 00 4a 03 1c e5 60 31 26 83 8b 53 08 48 89 de 83 e2 01 74 0a f3 90 8b 4e 08 <83> e1 01 75 f6 83 c0 01 eb b1 48 83 c4 60 5b 41 5c 41 5d 41 5e 41 <4>[ 661.849059] RSP: 0018:ffffc90049e5bb58 EFLAGS: 00000202 <4>[ 661.849060] RAX: 0000000000000003 RBX: ffff88885f5c23e0 RCX: 0000000000000011 <4>[ 661.849061] RDX: 0000000000000001 RSI: ffff88885f5c23e0 RDI: 0000000000000000 <4>[ 661.849062] RBP: ffffc90049e5bbe0 R08: 0000000000000000 R09: 0000000000000000 <4>[ 661.849063] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003 <4>[ 661.849063] R13: 0000000000000001 R14: 0000000000000014 R15: ffff88885fe36f40 <4>[ 661.849064] FS: 000076382cb2e6c0(0000) GS:ffff8888dbe97000(0000) knlGS:0000000000000000 <4>[ 661.849066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 661.849067] CR2: 00007638657f02b0 CR3: 000000012f635003 CR4: 0000000000f72ef0 <4>[ 661.849068] PKRU: 55555554 <4>[ 661.849068] Call Trace: <4>[ 661.849069] <4>[ 661.849071] ? __pfx_flush_tlb_func+0x10/0x10 <4>[ 661.849075] on_each_cpu_cond_mask+0x24/0x60 <4>[ 661.849077] native_flush_tlb_multi+0x172/0x220 <4>[ 661.849080] flush_tlb_mm_range+0x2d1/0x830 <4>[ 661.849082] ? madvise_do_behavior.part.0+0x9a/0x230 <4>[ 661.849084] ? trace_hardirqs_on+0x22/0x100 <4>[ 661.849090] tlb_finish_mmu+0x135/0x1b0 <4>[ 661.849093] do_madvise+0x12e/0x190 Panic#1 Part9 <4>[ 661.849100] __x64_sys_madvise+0x2b/0x40 <4>[ 661.849102] x64_sys_call+0x2149/0x26e0 <4>[ 661.849105] do_syscall_64+0xdd/0x1440 <4>[ 661.849109] ? do_futex+0x18e/0x260 <4>[ 661.849111] ? __x64_sys_futex+0x111/0x1f0 <4>[ 661.849114] ? handle_mm_fault+0x1e7/0x300 <4>[ 661.849118] ? trace_hardirqs_on_prepare+0xe1/0x100 <4>[ 661.849121] ? do_syscall_64+0x22e/0x1440 <4>[ 661.849124] ? exc_page_fault+0xbd/0x2c0 <4>[ 661.849128] entry_SYSCALL_64_after_hwframe+0x76/0x7e <4>[ 661.849130] RIP: 0033:0x7638659250eb <4>[ 661.849131] Code: 8b 55 f8 64 48 2b 14 25 28 00 00 00 75 02 c9 c3 e8 ea 2d 01 00 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 1c 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d fd dc 0d 00 f7 d8 64 89 01 48 <4>[ 661.849132] RSP: 002b:000076382cb2dec8 EFLAGS: 00000206 ORIG_RAX: 000000000000001c <4>[ 661.849133] RAX: ffffffffffffffda RBX: 000076382cb2ecdc RCX: 00007638659250eb <4>[ 661.849134] RDX: 0000000000000004 RSI: 00000000000fa000 RDI: 000076382ca2f000 <4>[ 661.849135] RBP: 000076382cb2df70 R08: 000076382cb2e6c0 R09: 0000000000000007 <4>[ 661.849136] R10: 0000000000000008 R11: 0000000000000206 R12: 000076382ca2f000 <4>[ 661.849137] R13: 0000000000100000 R14: 000000000000000b R15: 000076382ce2cfe0 <4>[ 661.849142] <4>[ 661.849143] NMI backtrace for cpu 13 skipped: idling at intel_idle+0x65/0xc0 <4>[ 661.849148] NMI backtrace for cpu 2 skipped: idling at intel_idle+0x65/0xc0 <4>[ 661.849151] NMI backtrace for cpu 21 <4>[ 661.849153] CPU: 21 UID: 1000 PID: 2990 Comm: pool-1-thread-1 Tainted: G S U W L 7.0.0-Patchwork_164993v2-gda91cb4e6137+ #1 PREEMPT(lazy) Panic#1 Part8 <4>[ 661.849155] Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER, [W]=WARN, [L]=SOFTLOCKUP <4>[ 661.849156] Hardware name: Intel Corporation Meteor Lake Client Platform/MTL-P LP5x T3 RVP, BIOS MTLPFWI1.R00.3471.D91.2401310918 01/31/2024 <4>[ 661.849157] RIP: 0010:smp_call_function_many_cond+0x1a9/0x870 <4>[ 661.849159] Code: 73 35 4c 63 e0 49 8b 1f 49 81 fc 00 20 00 00 0f 83 8d 06 00 00 4a 03 1c e5 60 31 26 83 8b 53 08 48 89 de 83 e2 01 74 0a f3 90 <8b> 4e 08 83 e1 01 75 f6 83 c0 01 eb b1 48 83 c4 60 5b 41 5c 41 5d <4>[ 661.849160] RSP: 0018:ffffc9000055ba88 EFLAGS: 00000202 <4>[ 661.849161] RAX: 0000000000000003 RBX: ffff88885f5c2400 RCX: 0000000000000001 <4>[ 661.849162] RDX: 0000000000000001 RSI: ffff88885f5c2400 RDI: 0000000000000000 <4>[ 661.849163] RBP: ffffc9000055bb10 R08: 0000000000000000 R09: 0000000000000000 <4>[ 661.849164] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003 <4>[ 661.849164] R13: 0000000000000001 R14: 0000000000000015 R15: ffff88885feb6f40 <4>[ 661.849165] FS: 000076382d12e6c0(0000) GS:ffff8888dbf17000(0000) knlGS:0000000000000000 <4>[ 661.849166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 661.849167] CR2: 00007638657f0130 CR3: 000000012f635002 CR4: 0000000000f72ef0 <4>[ 661.849169] PKRU: 55555554 <4>[ 661.849169] Call Trace: <4>[ 661.849170] <4>[ 661.849172] ? __pfx_flush_tlb_func+0x10/0x10 <4>[ 661.849176] on_each_cpu_cond_mask+0x24/0x60 <4>[ 661.849177] native_flush_tlb_multi+0x8d/0x220 <4>[ 661.849180] flush_tlb_mm_range+0x2d1/0x830 <4>[ 661.849183] ? madvise_do_behavior.part.0+0x9a/0x230 <4>[ 661.849187] tlb_finish_mmu+0x135/0x1b0