Ubuntu LXC einfach aus

UnBreakable · 11. Januar 2024

Halo zusammen,

Ich habe ein Ubuntu unter Proxmox als LXC Container am laufen, welches per Cron regelmäßig ein Script ausführt. Es kommt nun sehr oft vor, dass dieser Container plötzlich einfach aus ist.

Ich vermute nicht, dass es an Proxmox liegt, weil ich einige Container und VM am laufen habe, welche dieses Problem nicht haben. Um das Problem herauszufinden habe ich mal ins Syslog geschaut, da ist der letzte Eintrag folgender:

kernel: [8483670.204018] dump_header+0x50/0x290

Ich vermute (vor allem, da scheinbar ein dump file angelegt wurde), dass aus irgendeinem Grund der Kernel abgestürzt ist. Wie finde ich nun heraus, warum?

aluis · 11. Januar 2024

Ich würde an deiner Stelle das Debugloglevel erhöhen, in der Hoffnung, dass dann etwas plausibleres oder etwas ergooglebares raus kommt. Wichtig ist ja, was vor dem Absturz passierte.

Falls du ein Linux-Geek bist, einfach einen Kerneldebugger anschmeißen oder den dump analysieren, aber dann hättest du wohl hier nicht die Frage gestellt.

riversource · 11. Januar 2024

UnBreakable schrieb:
kernel: [8483670.204018] dump_header+0x50/0x290

War das ein Kernel Oops? Dann kann man mit dieser einen Zeile wenig anfangen. Der Rest ist der interessante Teil und lässt auch Rückschlüsse auf die Ursache zu.

Ansonsten haben die Container keinen eigenen Kernel, sondern arbeiten auf dem Kernel des Hosts. Der Kernel an sich kann also nicht abgestürzt sein, sonst wäre das gesamte System inkl. Proxmox gecrasht. Da wird also eher ein User Space Prozess aus dem Ruder laufen und vom Kernel abgeschossen. Eventuell kann folgendes Kommando noch helfen:

Code:

lxc warning list

UnBreakable · 4. Februar 2024

So, nach längerer Zeit kann ich mich mal wieder dem Thema annehmen.

Ich würde an deiner Stelle das Debugloglevel erhöhen, in der Hoffnung, dass dann etwas plausibleres oder etwas ergooglebares raus kommt. Wichtig ist ja, was vor dem Absturz passierte.

Wo und wie mache ich das? (Also vom Container oder von Proxmox?

Falls du ein Linux-Geek bist, einfach einen Kerneldebugger anschmeißen oder den dump analysieren, aber dann hättest du wohl hier nicht die Frage gestellt.

Nein, wie man bestimmt merkt, bin ich das nicht ;-)

War das ein Kernel Oops?

Keine Ahnung. Woran erkenne ich das?

Der Rest ist der interessante Teil und lässt auch Rückschlüsse auf die Ursache zu.

Welcher Rest?

Ansonsten haben die Container keinen eigenen Kernel, sondern arbeiten auf dem Kernel des Hosts.

Stimmt, daran habe ich nicht gedacht.

Eventuell kann folgendes Kommando noch helfen:

Wo gebe ich diesen Befehl ein?
Wenn ich den auf meinem Proxmoxhost eingebe, wird der Befehl lxc nicht gefunden. Auch im Container nicht.

riversource · 4. Februar 2024

Generell als Hinweis: Ein Forum ist Hilfe zur Selbsthilfe. Man bekommt in einem Forum nicht alles auf dem Silbertablett serviert, sondern man bekommt Hinweise, was man tun muss. Wie man es tun muss, kann man dann selber herausfinden, z.B. mit Hilfe von Anleitungen oder Google.

UnBreakable schrieb:
Keine Ahnung. Woran erkenne ich das?

Ist jetzt klar, wie du das herausfindest, oder?

UnBreakable schrieb:
Welcher Rest?

Du wirst doch hoffentlich in der Lage sein, einen zusammenhängenden Block an Kernelmeldungen zu identifizieren? Und sei es nur anhand des Timestamps oder des Prefixes? Es ist auch keine Anforderung, in einem Forum den gesunden Menschenverstand komplett auszuschalten.

UnBreakable schrieb:
Wo gebe ich diesen Befehl ein?

Wo läuft denn LXC?

UnBreakable schrieb:
Wenn ich den auf meinem Proxmoxhost eingebe, wird der Befehl lxc nicht gefunden.

Das heißt ja nicht, dass er nicht da ist. Nutzerrechte, Pfadangaben, ... Linux Basiswissen halt. Ein weiterer hervorragender Anlass, dich weiterzubilden.

UnBreakable · 4. Februar 2024

Du wirst doch hoffentlich in der Lage sein, einen zusammenhängenden Block an Kernelmeldungen zu identifizieren? Und sei es nur anhand des Timestamps oder des Prefixes? Es ist auch keine Anforderung, in einem Forum den gesunden Menschenverstand komplett auszuschalten.

Ich hoffe du bekommst ähnlich arrogante Antworten, wenn du mal nach Hilfe zu einem Thema suchst, von dem du wenig Ahnung hast. Antworte dann doch einfach gar nicht wenn ich dir zu ungebildet und dumm bin.

oicfar · 4. Februar 2024

UnBreakable schrieb:
Halo zusammen,

Ich habe ein Ubuntu unter Proxmox als LXC Container am laufen, welches per Cron regelmäßig ein Script ausführt. Es kommt nun sehr oft vor, dass dieser Container plötzlich einfach aus ist.

Macht der Cron was spezielles?

Ist dein Proxmox auf dem aktuellen Stand?

Ergänzung (4. Februar 2024)

Seltsam, dass im Proxmox Forum (https://forum.proxmox.com/threads/ubuntu-lxc-einfach-aus.139556/) noch niemand reagiert hat.

UnBreakable · 4. Februar 2024

Ist dein Proxmox auf dem aktuellen Stand?

Ja.

Macht der Cron was spezielles?

Eigentlich nicht. Ich mounte zwei Shares (Samba Windows), prüfe ob es neue Dateien gibt und kopiere diese, falls dem so ist. Die Dateien sind bis zu 120 GB groß.

oicfar · 4. Februar 2024

Vielleicht hilft dir das

https://forum.proxmox.com/threads/enable-install-crash-dump.127256/

weiter.

Hast du für Proxmox Monitoring (z.B. Prometheus+Grafana) am Start? So was ist immer hilfreich um Probleme zu erkennen.

UnBreakable · 4. Februar 2024

Hast du für Proxmox Monitoring (z.B. Prometheus+Grafana) am Start? So was ist immer hilfreich um Probleme zu erkennen.

Ja, habe ich. Arbeitsspeicher/CPU/Festplattenplatz ist nichts über 80%.

Vielleicht hilft dir das

https://forum.proxmox.com/threads/enable-install-crash-dump.127256/

Danke, ich hoffe ich kann mir das morgen mal in Ruhe genauer anschauen.

oicfar · 4. Februar 2024

UnBreakable schrieb:
Ja, habe ich. Arbeitsspeicher/CPU/Festplattenplatz ist nichts über 80%.

Es geht eher um das Verhalten. Wenn z.B. was passiert, dann kann es an der einen oder anderen Stelle im Monitoring Ausschläge in den Charts geben, die dann auch Hinweise geben können.

UnBreakable schrieb:
Eigentlich nicht. Ich mounte zwei Shares (Samba Windows), prüfe ob es neue Dateien gibt und kopiere diese, falls dem so ist. Die Dateien sind bis zu 120 GB groß.

Wird nur kopiert oder noch was anderes mit den großen Dateien angestellt?

Und die Frage wäre, ob es zu dem Problem kommt, wenn die ganz große Dateien kopiert werden. Ich würde da noch Logging (Starte mit dem Kopiervorgang von Datei X mit der Größe Y ..., Kopiervorgang beendet, ... oder so was) in das Skript einbauen. Vielleicht hilft das bei der Problemanalyse. Und dann weißt du, ob das Skript eine Rolle spielt oder nicht. Und ob die Dateigröße eine Rolle spielt.

UnBreakable · 6. Februar 2024

Es geht eher um das Verhalten. Wenn z.B. was passiert, dann kann es an der einen oder anderen Stelle im Monitoring Ausschläge in den Charts geben, die dann auch Hinweise geben können.

Da konnte ich keine Auffälligkeiten feststellen.

Wird nur kopiert oder noch was anderes mit den großen Dateien angestellt?

Nein. Es wird nur geprüft, ob die Datei auf der Gegenseite bereits existiert usw.
Die Quelle ist dabei eine Windows-Share und das Ziel eine Linux Freigabe.
Also weder das Ziel, noch die Quelle ist die Maschine die das Problem verursacht.

Und dann weißt du, ob das Skript eine Rolle spielt oder nicht.

Also da bin ich mir eigentlich so gut wie sicher. Der Container macht nämlich nichts anderes, den habe ich extra dafür erstellt. Ich habe auch testweise das Script deaktiviert. Dann gibt es keine Probleme.

Logging (Starte mit dem Kopiervorgang von Datei X mit der Größe Y ..., Kopiervorgang beendet, ... oder so was) in das Skript einbauen. Vielleicht hilft das bei der Problemanalyse.

Das ist bereits enthalten. Das Problem tritt mitten im Kopiervorgang auf. Also zumindest ist die letzte Meldung, dass der Kopiervorgang gestartet wurde.

Ob es nur bei den großen Dateien der Fall ist, kann ich gerade noch nicht sagen. Ich habe nämlich nun den Arbeitsspeicher einfach mal testweise verdoppelt. Bisher ist nun das Problem nicht mehr aufgetreten. Ich werde es nun noch weiter beobachten.

Aber: Sollte das Problem nun wirklich behoben sein: Wie kann das sein, obwohl die Auslastung nie bei 100% war. Und vor allem: Wie hätte ich das herausfinden können und ist es normal, dass der Container einfach beendet wird?

redjack1000 · 6. Februar 2024

UnBreakable schrieb:
Wie kann das sein, obwohl die Auslastung nie bei 100% war. Und vor allem: Wie hätte ich das herausfinden können und ist es normal, dass der Container einfach beendet wird?

Mir den von dir gelieferten Daten

UnBreakable schrieb:
kernel: [8483670.204018] dump_header+0x50/0x290

kann man deine Fragen nicht beantworten, bestenfalls könnte man anfangen zu raten.

CU
redjack

oicfar · 6. Februar 2024

D.h. ich würde es auch mal richtig testen

UnBreakable schrieb:
Das ist bereits enthalten. Das Problem tritt mitten im Kopiervorgang auf. Also zumindest ist die letzte Meldung, dass der Kopiervorgang gestartet wurde.

Und wie wird die Datei kopiert?

UnBreakable schrieb:
Ob es nur bei den großen Dateien der Fall ist, kann ich gerade noch nicht sagen. Ich habe nämlich nun den Arbeitsspeicher einfach mal testweise verdoppelt. Bisher ist nun das Problem nicht mehr aufgetreten. Ich werde es nun noch weiter beobachten.

Wie viel RAM waren zuvor und nun zugewiesen?

Ergänzung (6. Februar 2024)

UnBreakable schrieb:
Das ist bereits enthalten. Das Problem tritt mitten im Kopiervorgang auf. Also zumindest ist die letzte Meldung, dass der Kopiervorgang gestartet wurde.

Man könnte dann wieder weniger RAM zuweisen und den Kopiervorgang manuell paar mal starten um zu sehen, ob der Fehler passiert oder nicht.

riversource · 6. Februar 2024

Bevor du jetzt anfängst, im Nebel zu stochern: Die Hinweise, wie du das Problem identifizieren kannst, hast du bekommen. Das wird erheblich schneller zum Ziel führen.

UnBreakable · 8. Februar 2024

Und wie wird die Datei kopiert?

Ich nutze shutil.copy

Wie viel RAM waren zuvor und nun zugewiesen?

Eigentlich sowieso überdimenisioniert.
Vorher 512 MB, dann 1024 MB
Auslastung ist auch keine 10%

Man könnte dann wieder weniger RAM zuweisen und den Kopiervorgang manuell paar mal starten um zu sehen, ob der Fehler passiert oder nicht.

Ja, das habe ich sowieso gemacht und wollte damit testen.
Ich habe den Arbeitsspeicher wieder reduziert auf den ursprünglich Wert.
Komischerweise stürzte vorher das System täglich ab. Jetzt seit mehreren Tage garnicht, obwohl ich nichts geändert habe.
Ich verstehe es nicht. Mal schauen, ob das Problem doch wieder auftritt.

UnBreakable · 13. Februar 2024

So, heute ist das Problem erneut aufgetreten.
Ich habe nun zumindest mehr Informationen:

Bash:

2024-02-13T13:36:31.807002+01:00 pve kernel: [12367942.963455] kworker/u16:8 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
2024-02-13T13:36:31.807016+01:00 pve kernel: [12367942.963460] CPU: 6 PID: 198323 Comm: kworker/u16:8 Tainted: P           O       6.2.16-12-pve #1
2024-02-13T13:36:31.807017+01:00 pve kernel: [12367942.963463] Hardware name: Dell Inc. OptiPlex 7050/0XHGV1, BIOS 1.20.0 03/23/2022
2024-02-13T13:36:31.807019+01:00 pve kernel: [12367942.963465] Workqueue: writeback wb_workfn (flush-cifs-130)
2024-02-13T13:36:31.807020+01:00 pve kernel: [12367942.963471]  <TASK>
2024-02-13T13:36:31.807021+01:00 pve kernel: [12367942.963474]  dump_stack_lvl+0x48/0x70
2024-02-13T13:36:31.807022+01:00 pve kernel: [12367942.963478]  dump_stack+0x10/0x20
2024-02-13T13:36:31.807023+01:00 pve kernel: [12367942.963480]  dump_header+0x50/0x290
2024-02-13T13:36:31.807036+01:00 pve kernel: [12367942.963483]  oom_kill_process+0x10d/0x1c0
2024-02-13T13:36:31.807038+01:00 pve kernel: [12367942.963485]  out_of_memory+0x23c/0x570
2024-02-13T13:36:31.807039+01:00 pve kernel: [12367942.963488]  mem_cgroup_out_of_memory+0x145/0x170
2024-02-13T13:36:31.807040+01:00 pve kernel: [12367942.963491]  try_charge_memcg+0x731/0x820
2024-02-13T13:36:31.807040+01:00 pve kernel: [12367942.963494]  ? __mod_memcg_state+0x47/0xd0
2024-02-13T13:36:31.807041+01:00 pve kernel: [12367942.963497]  mem_cgroup_charge_skmem+0x40/0xf0
2024-02-13T13:36:31.807042+01:00 pve kernel: [12367942.963499]  __sk_mem_raise_allocated+0xcc/0x4f0
2024-02-13T13:36:31.807043+01:00 pve kernel: [12367942.963502]  __sk_mem_schedule+0x30/0x50
2024-02-13T13:36:31.807044+01:00 pve kernel: [12367942.963503]  tcp_wmem_schedule+0x41/0x90
2024-02-13T13:36:31.807045+01:00 pve kernel: [12367942.963506]  tcp_sendmsg_locked+0x5ae/0xc90
2024-02-13T13:36:31.807046+01:00 pve kernel: [12367942.963509]  tcp_sendmsg+0x2c/0x50
2024-02-13T13:36:31.807047+01:00 pve kernel: [12367942.963511]  inet_sendmsg+0x42/0x80
2024-02-13T13:36:31.807048+01:00 pve kernel: [12367942.963513]  sock_sendmsg+0x66/0x70
2024-02-13T13:36:31.807049+01:00 pve kernel: [12367942.963517]  smb_send_kvec+0x84/0x190 [cifs]
2024-02-13T13:36:31.807050+01:00 pve kernel: [12367942.963561]  ? release_sock+0x8f/0xb0
2024-02-13T13:36:31.807051+01:00 pve kernel: [12367942.963563]  __smb_send_rqst+0x2ef/0x7b0 [cifs]
2024-02-13T13:36:31.807052+01:00 pve kernel: [12367942.963616]  smb_send_rqst+0x19b/0x200 [cifs]
2024-02-13T13:36:31.807052+01:00 pve kernel: [12367942.963654]  cifs_call_async+0x144/0x330 [cifs]
2024-02-13T13:36:31.807053+01:00 pve kernel: [12367942.963689]  ? __pfx_smb2_writev_callback+0x10/0x10 [cifs]
2024-02-13T13:36:31.807054+01:00 pve kernel: [12367942.963740]  smb2_async_writev+0x4bf/0x740 [cifs]
2024-02-13T13:36:31.807055+01:00 pve kernel: [12367942.963777]  ? __pfx_cifs_writedata_release+0x10/0x10 [cifs]
2024-02-13T13:36:31.807056+01:00 pve kernel: [12367942.963809]  cifs_writepages+0xa4c/0x1230 [cifs]
2024-02-13T13:36:31.807057+01:00 pve kernel: [12367942.963840]  ? cifs_writepages+0xa4c/0x1230 [cifs]
2024-02-13T13:36:31.807058+01:00 pve kernel: [12367942.963873]  do_writepages+0xcf/0x1e0
2024-02-13T13:36:31.807059+01:00 pve kernel: [12367942.963876]  __writeback_single_inode+0x44/0x370
2024-02-13T13:36:31.807060+01:00 pve kernel: [12367942.963878]  ? fprop_fraction_percpu+0x34/0x90
2024-02-13T13:36:31.807061+01:00 pve kernel: [12367942.963881]  writeback_sb_inodes+0x21b/0x500
2024-02-13T13:36:31.807061+01:00 pve kernel: [12367942.963885]  __writeback_inodes_wb+0x54/0x100
2024-02-13T13:36:31.807062+01:00 pve kernel: [12367942.963886]  ? queue_io+0x115/0x120
2024-02-13T13:36:31.807071+01:00 pve kernel: [12367942.963888]  wb_writeback+0x20e/0x300
2024-02-13T13:36:31.807072+01:00 pve kernel: [12367942.963890]  wb_workfn+0x334/0x4d0
2024-02-13T13:36:31.807073+01:00 pve kernel: [12367942.963892]  ? try_to_wake_up+0x6c/0x600
2024-02-13T13:36:31.807074+01:00 pve kernel: [12367942.963896]  process_one_work+0x222/0x430
2024-02-13T13:36:31.807075+01:00 pve kernel: [12367942.963899]  worker_thread+0x50/0x3e0
2024-02-13T13:36:31.807076+01:00 pve kernel: [12367942.963901]  ? __pfx_worker_thread+0x10/0x10
2024-02-13T13:36:31.807077+01:00 pve kernel: [12367942.963903]  kthread+0xe6/0x110
2024-02-13T13:36:31.807078+01:00 pve kernel: [12367942.963905]  ? __pfx_kthread+0x10/0x10
2024-02-13T13:36:31.807079+01:00 pve kernel: [12367942.963908]  ret_from_fork+0x29/0x50
2024-02-13T13:36:31.807080+01:00 pve kernel: [12367942.963911]  </TASK>
2024-02-13T13:36:31.807080+01:00 pve kernel: [12367942.963934] memory: usage 524288kB, limit 524288kB, failcnt 21619
2024-02-13T13:36:31.807081+01:00 pve kernel: [12367942.963936] swap: usage 40388kB, limit 524288kB, failcnt 0
2024-02-13T13:36:31.807082+01:00 pve kernel: [12367942.963937] Memory cgroup stats for /lxc/108:
2024-02-13T13:36:31.807083+01:00 pve kernel: [12367942.963965] anon 0
2024-02-13T13:36:31.807084+01:00 pve kernel: [12367942.963965] file 518873088
2024-02-13T13:36:31.807085+01:00 pve kernel: [12367942.963965] kernel 17805312
2024-02-13T13:36:31.807086+01:00 pve kernel: [12367942.963965] kernel_stack 491520
2024-02-13T13:36:31.807087+01:00 pve kernel: [12367942.963965] pagetables 1482752
2024-02-13T13:36:31.807088+01:00 pve kernel: [12367942.963965] sec_pagetables 0
2024-02-13T13:36:31.807089+01:00 pve kernel: [12367942.963965] percpu 701080
2024-02-13T13:36:31.807090+01:00 pve kernel: [12367942.963965] sock 135168
2024-02-13T13:36:31.807090+01:00 pve kernel: [12367942.963965] vmalloc 487424
2024-02-13T13:36:31.807091+01:00 pve kernel: [12367942.963965] shmem 0
2024-02-13T13:36:31.807092+01:00 pve kernel: [12367942.963965] zswap 0
2024-02-13T13:36:31.807093+01:00 pve kernel: [12367942.963965] zswapped 0
2024-02-13T13:36:31.807094+01:00 pve kernel: [12367942.963965] file_mapped 0
2024-02-13T13:36:31.807095+01:00 pve kernel: [12367942.963965] file_dirty 0
2024-02-13T13:36:31.807095+01:00 pve kernel: [12367942.963965] file_writeback 518848512
2024-02-13T13:36:31.807096+01:00 pve kernel: [12367942.963965] swapcached 0
2024-02-13T13:36:31.807097+01:00 pve kernel: [12367942.963965] anon_thp 0
2024-02-13T13:36:31.807098+01:00 pve kernel: [12367942.963965] file_thp 0
2024-02-13T13:36:31.807099+01:00 pve kernel: [12367942.963965] shmem_thp 0
2024-02-13T13:36:31.807100+01:00 pve kernel: [12367942.963965] inactive_anon 0
2024-02-13T13:36:31.807100+01:00 pve kernel: [12367942.963965] active_anon 0
2024-02-13T13:36:31.807101+01:00 pve kernel: [12367942.963965] inactive_file 507572224
2024-02-13T13:36:31.807102+01:00 pve kernel: [12367942.963965] active_file 11358208
2024-02-13T13:36:31.807103+01:00 pve kernel: [12367942.963965] unevictable 0
2024-02-13T13:36:31.807104+01:00 pve kernel: [12367942.963965] slab_reclaimable 11699240
2024-02-13T13:36:31.807105+01:00 pve kernel: [12367942.963965] slab_unreclaimable 2323000
2024-02-13T13:36:31.807106+01:00 pve kernel: [12367942.963965] slab 14022240
2024-02-13T13:36:31.807107+01:00 pve kernel: [12367942.963971] Tasks state (memory values in pages):
2024-02-13T13:36:31.807107+01:00 pve kernel: [12367942.963972] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-02-13T13:36:31.807147+01:00 pve kernel: [12367942.963974] [ 183713]     0 183713     4589       96    77824      704             0 systemd
2024-02-13T13:36:31.807148+01:00 pve kernel: [12367942.963979] [ 184092]     0 184092      705       32    45056        0             0 agetty
2024-02-13T13:36:31.807149+01:00 pve kernel: [12367942.963983] [ 184194]     0 184194    10308       36    73728      160             0 master
2024-02-13T13:36:31.807150+01:00 pve kernel: [12367942.963985] [ 184195]   102 184195    10389       64    69632       96             0 pickup
2024-02-13T13:36:31.807151+01:00 pve kernel: [12367942.963988] [ 184196]   102 184196    10400       64    73728      128             0 qmgr
2024-02-13T13:36:31.807152+01:00 pve kernel: [12367942.963990] [ 183828]     0 183828     7835       32    81920      256          -250 systemd-journal
2024-02-13T13:36:31.807153+01:00 pve kernel: [12367942.963994] [ 183866]   105 183866     4031       32    73728      256             0 systemd-network
2024-02-13T13:36:31.807154+01:00 pve kernel: [12367942.963997] [ 183879]   110 183879     2026       96    53248       64             0 rpcbind
2024-02-13T13:36:31.807155+01:00 pve kernel: [12367942.964000] [ 183882]   106 183882     6384       87    94208     1056             0 systemd-resolve
2024-02-13T13:36:31.807155+01:00 pve kernel: [12367942.964003] [ 183886]     0 183886     1282       33    45056       32             0 blkmapd
2024-02-13T13:36:31.807156+01:00 pve kernel: [12367942.964006] [ 183887]     0 183887      812       33    40960       64             0 rpc.idmapd
2024-02-13T13:36:31.807158+01:00 pve kernel: [12367942.964009] [ 183888]     0 183888     1366       64    49152       64             0 nfsdcld
2024-02-13T13:36:31.807158+01:00 pve kernel: [12367942.964012] [ 183946]   100 183946     2113       32    49152       96          -900 dbus-daemon
2024-02-13T13:36:31.807160+01:00 pve kernel: [12367942.964015] [ 183951]     0 183951     7523      128    98304     2144             0 networkd-dispat
2024-02-13T13:36:31.807180+01:00 pve kernel: [12367942.964019] [ 183953]   101 183953    55601        0    77824      224             0 rsyslogd
2024-02-13T13:36:31.807182+01:00 pve kernel: [12367942.964023] [ 183956]     0 183956     3836       32    69632      256             0 systemd-logind
2024-02-13T13:36:31.807183+01:00 pve kernel: [12367942.964026] [ 184028]     0 184028     1284       66    45056       64             0 rpc.mountd
2024-02-13T13:36:31.807184+01:00 pve kernel: [12367942.964029] [ 184027]   111 184027     1194       96    53248       64             0 rpc.statd
2024-02-13T13:36:31.807185+01:00 pve kernel: [12367942.964032] [ 184081]     0 184081      972       64    49152       32             0 cron
2024-02-13T13:36:31.807186+01:00 pve kernel: [12367942.964035] [ 207991]     0 207991     1830       64    53248      225             0 cron
2024-02-13T13:36:31.807186+01:00 pve kernel: [12367942.964038] [ 207992]     0 207992      723       32    45056        0             0 sh
2024-02-13T13:36:31.807187+01:00 pve kernel: [12367942.964041] [ 207993]     0 207993     4121       96    73728     1056             0 script.py
2024-02-13T13:36:31.807188+01:00 pve kernel: [12367942.964044] [ 184089]     0 184089      705        0    40960       32             0 agetty
2024-02-13T13:36:31.807199+01:00 pve kernel: [12367942.964047] [ 184346]     0 184346     4170      128    69632      288             0 systemd
2024-02-13T13:36:31.807200+01:00 pve kernel: [12367942.964050] [ 184347]     0 184347     5352      128    77824      768             0 (sd-pam)
2024-02-13T13:36:31.807201+01:00 pve kernel: [12367942.964054] [ 184091]     0 184091     1916       32    49152      192             0 login
2024-02-13T13:36:31.807201+01:00 pve kernel: [12367942.964056] [ 184352]     0 184352     1379       96    49152      352             0 bash
2024-02-13T13:36:31.807202+01:00 pve kernel: [12367942.964059] [ 197512]     0 197512     4376       96    77824     1120             0 script.py
2024-02-13T13:36:31.807203+01:00 pve kernel: [12367942.964062] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/lxc/108,task_memcg=/lxc/108/ns/system.slice/networkd-dispatcher.service,task=networkd-dispat,pid=183951,uid=0
2024-02-13T13:36:31.807204+01:00 pve kernel: [12367942.964082] Memory cgroup out of memory: Killed process 183951 (networkd-dispat) total-vm:30092kB, anon-rss:256kB, file-rss:256kB, shmem-rss:0kB, UID:0 pgtables:96kB oom_score_adj:0
2024-02-13T13:36:33.842612+01:00 pve kernel: [12367944.996580] oom_reaper: reaped process 183951 (networkd-dispat), now anon-rss:256kB, file-rss:256kB, shmem-rss:0kB
2024-02-13T13:36:36.846665+01:00 pve kernel: [12367948.000430] kworker/u16:8 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
2024-02-13T13:36:36.846679+01:00 pve kernel: [12367948.000436] CPU: 2 PID: 198323 Comm: kworker/u16:8 Tainted: P           O       6.2.16-12-pve #1
2024-02-13T13:36:36.846681+01:00 pve kernel: [12367948.000439] Hardware name: Dell Inc. OptiPlex 7050/0XHGV1, BIOS 1.20.0 03/23/2022
2024-02-13T13:36:36.846682+01:00 pve kernel: [12367948.000441] Workqueue: writeback wb_workfn (flush-cifs-130)
2024-02-13T13:36:36.846683+01:00 pve kernel: [12367948.000446] Call Trace:
2024-02-13T13:36:36.846684+01:00 pve kernel: [12367948.000448]  <TASK>
2024-02-13T13:36:36.846684+01:00 pve kernel: [12367948.000450]  dump_stack_lvl+0x48/0x70
2024-02-13T13:36:36.846685+01:00 pve kernel: [12367948.000455]  dump_stack+0x10/0x20
2024-02-13T13:36:36.846686+01:00 pve kernel: [12367948.000457]  dump_header+0x50/0x290
2024-02-13T13:36:36.846687+01:00 pve kernel: [12367948.000461]  oom_kill_process+0x10d/0x1c0
2024-02-13T13:36:36.846688+01:00 pve kernel: [12367948.000464]  out_of_memory+0x23c/0x570
2024-02-13T13:36:36.846689+01:00 pve kernel: [12367948.000467]  mem_cgroup_out_of_memory+0x145/0x170
2024-02-13T13:36:36.846689+01:00 pve kernel: [12367948.000470]  try_charge_memcg+0x731/0x820
2024-02-13T13:36:36.846690+01:00 pve kernel: [12367948.000474]  ? __mod_memcg_state+0x47/0xd0
2024-02-13T13:36:36.846691+01:00 pve kernel: [12367948.000477]  mem_cgroup_charge_skmem+0x40/0xf0
2024-02-13T13:36:36.846692+01:00 pve kernel: [12367948.000480]  __sk_mem_raise_allocated+0xcc/0x4f0
2024-02-13T13:36:36.846692+01:00 pve kernel: [12367948.000483]  __sk_mem_schedule+0x30/0x50
2024-02-13T13:36:36.846693+01:00 pve kernel: [12367948.000485]  tcp_wmem_schedule+0x41/0x90
2024-02-13T13:36:36.846694+01:00 pve kernel: [12367948.000488]  tcp_sendmsg_locked+0x5ae/0xc90
2024-02-13T13:36:36.846695+01:00 pve kernel: [12367948.000493]  tcp_sendmsg+0x2c/0x50
2024-02-13T13:36:36.846696+01:00 pve kernel: [12367948.000496]  inet_sendmsg+0x42/0x80
2024-02-13T13:36:36.846697+01:00 pve kernel: [12367948.000498]  sock_sendmsg+0x66/0x70
2024-02-13T13:36:36.846697+01:00 pve kernel: [12367948.000503]  smb_send_kvec+0x84/0x190 [cifs]
2024-02-13T13:36:36.846698+01:00 pve kernel: [12367948.000566]  ? release_sock+0x8f/0xb0
2024-02-13T13:36:36.846699+01:00 pve kernel: [12367948.000570]  __smb_send_rqst+0x2ef/0x7b0 [cifs]
2024-02-13T13:36:36.846699+01:00 pve kernel: [12367948.000624]  smb_send_rqst+0x19b/0x200 [cifs]
2024-02-13T13:36:36.846700+01:00 pve kernel: [12367948.000680]  cifs_call_async+0x144/0x330 [cifs]
2024-02-13T13:36:36.846701+01:00 pve kernel: [12367948.000731]  ? __pfx_smb2_writev_callback+0x10/0x10 [cifs]
2024-02-13T13:36:36.846702+01:00 pve kernel: [12367948.000789]  smb2_async_writev+0x4bf/0x740 [cifs]
2024-02-13T13:36:36.846702+01:00 pve kernel: [12367948.000842]  ? __pfx_cifs_writedata_release+0x10/0x10 [cifs]
2024-02-13T13:36:36.846703+01:00 pve kernel: [12367948.000895]  cifs_writepages+0xa4c/0x1230 [cifs]
2024-02-13T13:36:36.846704+01:00 pve kernel: [12367948.000942]  ? cifs_writepages+0xa4c/0x1230 [cifs]
2024-02-13T13:36:36.846704+01:00 pve kernel: [12367948.000991]  do_writepages+0xcf/0x1e0
2024-02-13T13:36:36.846705+01:00 pve kernel: [12367948.000995]  __writeback_single_inode+0x44/0x370
2024-02-13T13:36:36.846706+01:00 pve kernel: [12367948.000998]  ? fprop_fraction_percpu+0x34/0x90
2024-02-13T13:36:36.846706+01:00 pve kernel: [12367948.001002]  writeback_sb_inodes+0x21b/0x500
2024-02-13T13:36:36.846707+01:00 pve kernel: [12367948.001006]  __writeback_inodes_wb+0x54/0x100
2024-02-13T13:36:36.846708+01:00 pve kernel: [12367948.001008]  ? queue_io+0x115/0x120
2024-02-13T13:36:36.846709+01:00 pve kernel: [12367948.001010]  wb_writeback+0x20e/0x300
2024-02-13T13:36:36.846709+01:00 pve kernel: [12367948.001013]  wb_workfn+0x334/0x4d0
2024-02-13T13:36:36.846710+01:00 pve kernel: [12367948.001015]  ? try_to_wake_up+0x6c/0x600
2024-02-13T13:36:36.846711+01:00 pve kernel: [12367948.001019]  process_one_work+0x222/0x430
2024-02-13T13:36:36.846712+01:00 pve kernel: [12367948.001022]  worker_thread+0x50/0x3e0
2024-02-13T13:36:36.846713+01:00 pve kernel: [12367948.001025]  ? __pfx_worker_thread+0x10/0x10
2024-02-13T13:36:36.846714+01:00 pve kernel: [12367948.001028]  kthread+0xe6/0x110
2024-02-13T13:36:36.846714+01:00 pve kernel: [12367948.001031]  ? __pfx_kthread+0x10/0x10
2024-02-13T13:36:36.846715+01:00 pve kernel: [12367948.001035]  ret_from_fork+0x29/0x50
2024-02-13T13:36:36.846716+01:00 pve kernel: [12367948.001039]  </TASK>
2024-02-13T13:36:36.846717+01:00 pve kernel: [12367948.001040] memory: usage 524288kB, limit 524288kB, failcnt 21738
2024-02-13T13:36:36.846717+01:00 pve kernel: [12367948.001042] swap: usage 31840kB, limit 524288kB, failcnt 0
2024-02-13T13:36:36.846718+01:00 pve kernel: [12367948.001044] Memory cgroup stats for /lxc/108:
2024-02-13T13:36:36.846719+01:00 pve kernel: [12367948.001060] anon 0
2024-02-13T13:36:36.846719+01:00 pve kernel: [12367948.001060] file 518873088
2024-02-13T13:36:36.846720+01:00 pve kernel: [12367948.001060] kernel 17805312
2024-02-13T13:36:36.846721+01:00 pve kernel: [12367948.001060] kernel_stack 491520
2024-02-13T13:36:36.846722+01:00 pve kernel: [12367948.001060] pagetables 1482752
2024-02-13T13:36:36.846722+01:00 pve kernel: [12367948.001060] sec_pagetables 0
2024-02-13T13:36:36.846723+01:00 pve kernel: [12367948.001060] percpu 701080
2024-02-13T13:36:36.846724+01:00 pve kernel: [12367948.001060] sock 135168
2024-02-13T13:36:36.846725+01:00 pve kernel: [12367948.001060] vmalloc 487424
2024-02-13T13:36:36.846725+01:00 pve kernel: [12367948.001060] shmem 0
2024-02-13T13:36:36.846726+01:00 pve kernel: [12367948.001060] zswap 0
2024-02-13T13:36:36.846727+01:00 pve kernel: [12367948.001060] zswapped 0
2024-02-13T13:36:36.846728+01:00 pve kernel: [12367948.001060] file_mapped 0
2024-02-13T13:36:36.846729+01:00 pve kernel: [12367948.001060] file_dirty 0
2024-02-13T13:36:36.846730+01:00 pve kernel: [12367948.001060] file_writeback 518848512
2024-02-13T13:36:36.846730+01:00 pve kernel: [12367948.001060] swapcached 0
2024-02-13T13:36:36.846731+01:00 pve kernel: [12367948.001060] anon_thp 0
2024-02-13T13:36:36.846732+01:00 pve kernel: [12367948.001060] file_thp 0
2024-02-13T13:36:36.846732+01:00 pve kernel: [12367948.001060] shmem_thp 0
2024-02-13T13:36:36.846733+01:00 pve kernel: [12367948.001060] inactive_anon 0
2024-02-13T13:36:36.846734+01:00 pve kernel: [12367948.001060] active_anon 0
2024-02-13T13:36:36.846734+01:00 pve kernel: [12367948.001060] inactive_file 518914048
2024-02-13T13:36:36.846735+01:00 pve kernel: [12367948.001060] active_file 8192
2024-02-13T13:36:36.846736+01:00 pve kernel: [12367948.001060] unevictable 0
2024-02-13T13:36:36.846737+01:00 pve kernel: [12367948.001060] slab_reclaimable 11699240
2024-02-13T13:36:36.846737+01:00 pve kernel: [12367948.001060] slab_unreclaimable 2323000
2024-02-13T13:36:36.846738+01:00 pve kernel: [12367948.001060] slab 14022240
2024-02-13T13:36:36.846739+01:00 pve kernel: [12367948.001066] Tasks state (memory values in pages):
2024-02-13T13:36:36.846739+01:00 pve kernel: [12367948.001067] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-02-13T13:36:36.846740+01:00 pve kernel: [12367948.001069] [ 183713]     0 183713     4589       96    77824      704             0 systemd
2024-02-13T13:36:36.846741+01:00 pve kernel: [12367948.001074] [ 184092]     0 184092      705       32    45056        0             0 agetty
2024-02-13T13:36:36.846742+01:00 pve kernel: [12367948.001078] [ 184194]     0 184194    10308       36    73728      160             0 master
2024-02-13T13:36:36.846742+01:00 pve kernel: [12367948.001081] [ 184195]   102 184195    10389       64    69632       96             0 pickup
2024-02-13T13:36:36.846743+01:00 pve kernel: [12367948.001083] [ 184196]   102 184196    10400       64    73728      128             0 qmgr
2024-02-13T13:36:36.846744+01:00 pve kernel: [12367948.001086] [ 183828]     0 183828     7835       32    81920      256          -250 systemd-journal
2024-02-13T13:36:36.846745+01:00 pve kernel: [12367948.001089] [ 183866]   105 183866     4031       32    73728      256             0 systemd-network
2024-02-13T13:36:36.846745+01:00 pve kernel: [12367948.001091] [ 183879]   110 183879     2026       96    53248       64             0 rpcbind
2024-02-13T13:36:36.846746+01:00 pve kernel: [12367948.001094] [ 183882]   106 183882     6384       87    94208     1056             0 systemd-resolve
2024-02-13T13:36:36.846747+01:00 pve kernel: [12367948.001096] [ 183886]     0 183886     1282       33    45056       32             0 blkmapd
2024-02-13T13:36:36.846747+01:00 pve kernel: [12367948.001099] [ 183887]     0 183887      812       33    40960       64             0 rpc.idmapd
2024-02-13T13:36:36.846748+01:00 pve kernel: [12367948.001102] [ 183888]     0 183888     1366       64    49152       64             0 nfsdcld
2024-02-13T13:36:36.846749+01:00 pve kernel: [12367948.001105] [ 183946]   100 183946     2113       32    49152       96          -900 dbus-daemon
2024-02-13T13:36:36.846750+01:00 pve kernel: [12367948.001108] [ 183951]     0 183951     7523      128    98304        0             0 networkd-dispat
2024-02-13T13:36:36.846751+01:00 pve kernel: [12367948.001110] [ 183953]   101 183953    55601        0    77824      224             0 rsyslogd
2024-02-13T13:36:36.846752+01:00 pve kernel: [12367948.001113] [ 183956]     0 183956     3836       32    69632      256             0 systemd-logind
2024-02-13T13:36:36.846752+01:00 pve kernel: [12367948.001116] [ 184028]     0 184028     1284       66    45056       64             0 rpc.mountd
2024-02-13T13:36:36.846753+01:00 pve kernel: [12367948.001119] [ 184027]   111 184027     1194       96    53248       64             0 rpc.statd
2024-02-13T13:36:36.846754+01:00 pve kernel: [12367948.001121] [ 184081]     0 184081      972       64    49152       32             0 cron
2024-02-13T13:36:36.846754+01:00 pve kernel: [12367948.001124] [ 207991]     0 207991     1830       64    53248      225             0 cron
2024-02-13T13:36:36.846755+01:00 pve kernel: [12367948.001126] [ 207992]     0 207992      723       32    45056        0             0 sh
2024-02-13T13:36:36.846756+01:00 pve kernel: [12367948.001128] [ 207993]     0 207993     4121       96    73728     1056             0 script.py
2024-02-13T13:36:36.846757+01:00 pve kernel: [12367948.001130] [ 184089]     0 184089      705        0    40960       32             0 agetty
2024-02-13T13:36:36.846757+01:00 pve kernel: [12367948.001134] [ 184346]     0 184346     4170      128    69632      288             0 systemd
2024-02-13T13:36:36.846758+01:00 pve kernel: [12367948.001136] [ 184347]     0 184347     5352      128    77824      768             0 (sd-pam)
2024-02-13T13:36:36.846759+01:00 pve kernel: [12367948.001139] [ 184091]     0 184091     1916       32    49152      192             0 login
2024-02-13T13:36:36.846759+01:00 pve kernel: [12367948.001141] [ 184352]     0 184352     1379       96    49152      352             0 bash
2024-02-13T13:36:36.846760+01:00 pve kernel: [12367948.001143] [ 197512]     0 197512     4376       96    77824     1120             0 script.py
2024-02-13T13:36:36.846761+01:00 pve kernel: [12367948.001145] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/lxc/108,task_memcg=/lxc/108/ns/user.slice/user-0.slice/session-65982.scope,task=script.py,pid=197512,uid=0
2024-02-13T13:36:36.846762+01:00 pve kernel: [12367948.001157] Memory cgroup out of memory: Killed process 197512 (script.py) total-vm:17504kB, anon-rss:0kB, file-rss:384kB, shmem-rss:0kB, UID:0 pgtables:76kB oom_score_adj:0
2024-02-13T13:36:38.866615+01:00 pve kernel: [12367950.020038] oom_reaper: reaped process 197512 (script.py), now anon-rss:0kB, file-rss:384kB, shmem-rss:0kB
2024-02-13T13:36:42.850657+01:00 pve kernel: [12367954.004132] kworker/u16:8 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
2024-02-13T13:36:42.850668+01:00 pve kernel: [12367954.004138] CPU: 2 PID: 198323 Comm: kworker/u16:8 Tainted: P           O       6.2.16-12-pve #1
2024-02-13T13:36:42.850670+01:00 pve kernel: [12367954.004141] Hardware name: Dell Inc. OptiPlex 7050/0XHGV1, BIOS 1.20.0 03/23/2022
2024-02-13T13:36:42.850671+01:00 pve kernel: [12367954.004143] Workqueue: writeback wb_workfn (flush-cifs-130)
2024-02-13T13:36:42.850672+01:00 pve kernel: [12367954.004148] Call Trace:
2024-02-13T13:36:42.850673+01:00 pve kernel: [12367954.004150]  <TASK>
2024-02-13T13:36:42.850674+01:00 pve kernel: [12367954.004152]  dump_stack_lvl+0x48/0x70
2024-02-13T13:36:42.850675+01:00 pve kernel: [12367954.004156]  dump_stack+0x10/0x20
2024-02-13T13:36:42.850676+01:00 pve kernel: [12367954.004158]  dump_header+0x50/0x290
2024-02-13T13:36:42.850676+01:00 pve kernel: [12367954.004161]  oom_kill_process+0x10d/0x1c0
2024-02-13T13:36:42.850677+01:00 pve kernel: [12367954.004164]  out_of_memory+0x23c/0x570
2024-02-13T13:36:42.850678+01:00 pve kernel: [12367954.004167]  mem_cgroup_out_of_memory+0x145/0x170
2024-02-13T13:36:42.850679+01:00 pve kernel: [12367954.004170]  try_charge_memcg+0x731/0x820
2024-02-13T13:36:42.850679+01:00 pve kernel: [12367954.004174]  ? __mod_memcg_state+0x47/0xd0
2024-02-13T13:36:42.850680+01:00 pve kernel: [12367954.004177]  mem_cgroup_charge_skmem+0x40/0xf0
2024-02-13T13:36:42.850681+01:00 pve kernel: [12367954.004179]  __sk_mem_raise_allocated+0xcc/0x4f0
2024-02-13T13:36:42.850682+01:00 pve kernel: [12367954.004182]  __sk_mem_schedule+0x30/0x50
2024-02-13T13:36:42.850682+01:00 pve kernel: [12367954.004184]  tcp_wmem_schedule+0x41/0x90
2024-02-13T13:36:42.850683+01:00 pve kernel: [12367954.004187]  tcp_sendmsg_locked+0x5ae/0xc90
2024-02-13T13:36:42.850684+01:00 pve kernel: [12367954.004192]  tcp_sendmsg+0x2c/0x50
2024-02-13T13:36:42.850685+01:00 pve kernel: [12367954.004194]  inet_sendmsg+0x42/0x80
2024-02-13T13:36:42.850686+01:00 pve kernel: [12367954.004196]  sock_sendmsg+0x66/0x70
2024-02-13T13:36:42.850687+01:00 pve kernel: [12367954.004200]  smb_send_kvec+0x84/0x190 [cifs]
2024-02-13T13:36:42.850688+01:00 pve kernel: [12367954.004255]  ? release_sock+0x8f/0xb0
2024-02-13T13:36:42.850688+01:00 pve kernel: [12367954.004258]  __smb_send_rqst+0x2ef/0x7b0 [cifs]
2024-02-13T13:36:42.850689+01:00 pve kernel: [12367954.004309]  smb_send_rqst+0x19b/0x200 [cifs]
2024-02-13T13:36:42.850690+01:00 pve kernel: [12367954.004359]  cifs_call_async+0x144/0x330 [cifs]
2024-02-13T13:36:42.850690+01:00 pve kernel: [12367954.004406]  ? __pfx_smb2_writev_callback+0x10/0x10 [cifs]
2024-02-13T13:36:42.850691+01:00 pve kernel: [12367954.004457]  smb2_async_writev+0x4bf/0x740 [cifs]
2024-02-13T13:36:42.850692+01:00 pve kernel: [12367954.004507]  ? __pfx_cifs_writedata_release+0x10/0x10 [cifs]
2024-02-13T13:36:42.850692+01:00 pve kernel: [12367954.004554]  cifs_writepages+0xa4c/0x1230 [cifs]
2024-02-13T13:36:42.850693+01:00 pve kernel: [12367954.004599]  ? cifs_writepages+0xa4c/0x1230 [cifs]
2024-02-13T13:36:42.850694+01:00 pve kernel: [12367954.004647]  do_writepages+0xcf/0x1e0
2024-02-13T13:36:42.850694+01:00 pve kernel: [12367954.004650]  __writeback_single_inode+0x44/0x370
2024-02-13T13:36:42.850695+01:00 pve kernel: [12367954.004652]  ? fprop_fraction_percpu+0x34/0x90
2024-02-13T13:36:42.850696+01:00 pve kernel: [12367954.004656]  writeback_sb_inodes+0x21b/0x500
2024-02-13T13:36:42.850697+01:00 pve kernel: [12367954.004660]  __writeback_inodes_wb+0x54/0x100
2024-02-13T13:36:42.850697+01:00 pve kernel: [12367954.004662]  ? queue_io+0x115/0x120
2024-02-13T13:36:42.850698+01:00 pve kernel: [12367954.004664]  wb_writeback+0x20e/0x300
2024-02-13T13:36:42.850699+01:00 pve kernel: [12367954.004667]  wb_workfn+0x334/0x4d0
2024-02-13T13:36:42.850699+01:00 pve kernel: [12367954.004669]  ? try_to_wake_up+0x6c/0x600
2024-02-13T13:36:42.850700+01:00 pve kernel: [12367954.004673]  process_one_work+0x222/0x430
2024-02-13T13:36:42.850701+01:00 pve kernel: [12367954.004676]  worker_thread+0x50/0x3e0
2024-02-13T13:36:42.850702+01:00 pve kernel: [12367954.004679]  ? __pfx_worker_thread+0x10/0x10
2024-02-13T13:36:42.850702+01:00 pve kernel: [12367954.004681]  kthread+0xe6/0x110
2024-02-13T13:36:42.850703+01:00 pve kernel: [12367954.004684]  ? __pfx_kthread+0x10/0x10
2024-02-13T13:36:42.850704+01:00 pve kernel: [12367954.004687]  ret_from_fork+0x29/0x50
2024-02-13T13:36:42.850704+01:00 pve kernel: [12367954.004690]  </TASK>
2024-02-13T13:36:42.850705+01:00 pve kernel: [12367954.004691] memory: usage 524288kB, limit 524288kB, failcnt 21874
2024-02-13T13:36:42.850706+01:00 pve kernel: [12367954.004693] swap: usage 27488kB, limit 524288kB, failcnt 0
2024-02-13T13:36:42.850707+01:00 pve kernel: [12367954.004694] Memory cgroup stats for /lxc/108:
2024-02-13T13:36:42.850708+01:00 pve kernel: [12367954.004740] anon 0
2024-02-13T13:36:42.850709+01:00 pve kernel: [12367954.004740] file 518873088
2024-02-13T13:36:42.850710+01:00 pve kernel: [12367954.004740] kernel 17805312
2024-02-13T13:36:42.850710+01:00 pve kernel: [12367954.004740] kernel_stack 491520
2024-02-13T13:36:42.850711+01:00 pve kernel: [12367954.004740] pagetables 1482752
2024-02-13T13:36:42.850712+01:00 pve kernel: [12367954.004740] sec_pagetables 0
2024-02-13T13:36:42.850712+01:00 pve kernel: [12367954.004740] percpu 701080
2024-02-13T13:36:42.850713+01:00 pve kernel: [12367954.004740] sock 135168
2024-02-13T13:36:42.850714+01:00 pve kernel: [12367954.004740] vmalloc 487424
2024-02-13T13:36:42.850715+01:00 pve kernel: [12367954.004740] shmem 0
2024-02-13T13:36:42.850715+01:00 pve kernel: [12367954.004740] zswap 0
2024-02-13T13:36:42.850716+01:00 pve kernel: [12367954.004740] zswapped 0
2024-02-13T13:36:42.850717+01:00 pve kernel: [12367954.004740] file_mapped 0
2024-02-13T13:36:42.850718+01:00 pve kernel: [12367954.004740] file_dirty 0
2024-02-13T13:36:42.850718+01:00 pve kernel: [12367954.004740] file_writeback 518848512
2024-02-13T13:36:42.850719+01:00 pve kernel: [12367954.004740] swapcached 0
2024-02-13T13:36:42.850720+01:00 pve kernel: [12367954.004740] anon_thp 0
2024-02-13T13:36:42.850720+01:00 pve kernel: [12367954.004740] file_thp 0
2024-02-13T13:36:42.850721+01:00 pve kernel: [12367954.004740] shmem_thp 0
2024-02-13T13:36:42.850722+01:00 pve kernel: [12367954.004740] inactive_anon 0
2024-02-13T13:36:42.850722+01:00 pve kernel: [12367954.004740] active_anon 0
2024-02-13T13:36:42.850723+01:00 pve kernel: [12367954.004740] inactive_file 335196160
2024-02-13T13:36:42.850724+01:00 pve kernel: [12367954.004740] active_file 183250944
2024-02-13T13:36:42.850725+01:00 pve kernel: [12367954.004740] unevictable 0
2024-02-13T13:36:42.850726+01:00 pve kernel: [12367954.004740] slab_reclaimable 11699240
2024-02-13T13:36:42.850727+01:00 pve kernel: [12367954.004740] slab_unreclaimable 2323000
2024-02-13T13:36:42.850727+01:00 pve kernel: [12367954.004740] slab 14022240
2024-02-13T13:36:42.850728+01:00 pve kernel: [12367954.004744] Tasks state (memory values in pages):
2024-02-13T13:36:42.850729+01:00 pve kernel: [12367954.004745] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-02-13T13:36:42.850730+01:00 pve kernel: [12367954.004747] [ 183713]     0 183713     4589       96    77824      704             0 systemd
2024-02-13T13:36:42.850730+01:00 pve kernel: [12367954.004750] [ 184092]     0 184092      705       32    45056        0             0 agetty
2024-02-13T13:36:42.850731+01:00 pve kernel: [12367954.004754] [ 184194]     0 184194    10308       36    73728      160             0 master
2024-02-13T13:36:42.850732+01:00 pve kernel: [12367954.004757] [ 184195]   102 184195    10389       64    69632       96             0 pickup
2024-02-13T13:36:42.850732+01:00 pve kernel: [12367954.004759] [ 184196]   102 184196    10400       64    73728      128             0 qmgr
2024-02-13T13:36:42.850733+01:00 pve kernel: [12367954.004761] [ 183828]     0 183828     7835       32    81920      256          -250 systemd-journal
2024-02-13T13:36:42.850734+01:00 pve kernel: [12367954.004764] [ 183866]   105 183866     4031       32    73728      256             0 systemd-network
2024-02-13T13:36:42.850734+01:00 pve kernel: [12367954.004767] [ 183879]   110 183879     2026       96    53248       64             0 rpcbind
2024-02-13T13:36:42.850735+01:00 pve kernel: [12367954.004770] [ 183882]   106 183882     6384       87    94208     1056             0 systemd-resolve
2024-02-13T13:36:42.850736+01:00 pve kernel: [12367954.004772] [ 183886]     0 183886     1282       33    45056       32             0 blkmapd
2024-02-13T13:36:42.850736+01:00 pve kernel: [12367954.004775] [ 183887]     0 183887      812       33    40960       64             0 rpc.idmapd
2024-02-13T13:36:42.850737+01:00 pve kernel: [12367954.004777] [ 183888]     0 183888     1366       64    49152       64             0 nfsdcld
2024-02-13T13:36:42.850738+01:00 pve kernel: [12367954.004780] [ 183946]   100 183946     2113       32    49152       96          -900 dbus-daemon
2024-02-13T13:36:42.850739+01:00 pve kernel: [12367954.004782] [ 183951]     0 183951     7523      128    98304        0             0 networkd-dispat
2024-02-13T13:36:42.850740+01:00 pve kernel: [12367954.004785] [ 183953]   101 183953    55601        0    77824      224             0 rsyslogd
2024-02-13T13:36:42.850740+01:00 pve kernel: [12367954.004787] [ 183956]     0 183956     3836       32    69632      256             0 systemd-logind
2024-02-13T13:36:42.850741+01:00 pve kernel: [12367954.004790] [ 184028]     0 184028     1284       66    45056       64             0 rpc.mountd
2024-02-13T13:36:42.850742+01:00 pve kernel: [12367954.004793] [ 184027]   111 184027     1194       96    53248       64             0 rpc.statd
2024-02-13T13:36:42.850743+01:00 pve kernel: [12367954.004796] [ 184081]     0 184081      972       64    49152       32             0 cron
2024-02-13T13:36:42.850744+01:00 pve kernel: [12367954.004798] [ 207991]     0 207991     1830       64    53248      225             0 cron
2024-02-13T13:36:42.850744+01:00 pve kernel: [12367954.004800] [ 207992]     0 207992      723       32    45056        0             0 sh
2024-02-13T13:36:42.850745+01:00 pve kernel: [12367954.004802] [ 207993]     0 207993     4121       96    73728     1056             0 script.py
2024-02-13T13:36:42.850746+01:00 pve kernel: [12367954.004804] [ 184089]     0 184089      705        0    40960       32             0 agetty
2024-02-13T13:36:42.850747+01:00 pve kernel: [12367954.004807] [ 184346]     0 184346     4170      128    69632      288             0 systemd
2024-02-13T13:36:42.850748+01:00 pve kernel: [12367954.004809] [ 184347]     0 184347     5352      128    77824      768             0 (sd-pam)
2024-02-13T13:36:42.850748+01:00 pve kernel: [12367954.004811] [ 184091]     0 184091     1916       32    49152      192             0 login
2024-02-13T13:36:42.850749+01:00 pve kernel: [12367954.004814] [ 184352]     0 184352     1379       96    49152      352             0 bash
2024-02-13T13:36:42.850750+01:00 pve kernel: [12367954.004816] [ 197512]     0 197512     4376       96    77824       13             0 script.py
2024-02-13T13:36:42.850750+01:00 pve kernel: [12367954.004818] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/lxc/108,task_memcg=/lxc/108/ns/system.slice/cron.service,task=script.py,pid=207993,uid=0
2024-02-13T13:36:42.850751+01:00 pve kernel: [12367954.004829] Memory cgroup out of memory: Killed process 207993 (script.py) total-vm:16484kB, anon-rss:128kB, file-rss:256kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:0
2024-02-13T13:36:44.886612+01:00 pve kernel: [12367956.039955] oom_reaper: reaped process 207993 (script.py), now anon-rss:128kB, file-rss:256kB, shmem-rss:0kB
2024-02-13T13:36:48.502664+01:00 pve kernel: [12367959.655876] systemd-journal invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-250
2024-02-13T13:36:48.502677+01:00 pve kernel: [12367959.655881] CPU: 5 PID: 183828 Comm: systemd-journal Tainted: P           O       6.2.16-12-pve #1
2024-02-13T13:36:48.502678+01:00 pve kernel: [12367959.655884] Hardware name: Dell Inc. OptiPlex 7050/0XHGV1, BIOS 1.20.0 03/23/2022
2024-02-13T13:36:48.502679+01:00 pve kernel: [12367959.655886] Call Trace:
2024-02-13T13:36:48.502680+01:00 pve kernel: [12367959.655888]  <TASK>
2024-02-13T13:36:48.502681+01:00 pve kernel: [12367959.655890]  dump_stack_lvl+0x48/0x70
2024-02-13T13:36:48.502683+01:00 pve kernel: [12367959.655895]  dump_stack+0x10/0x20
2024-02-13T13:36:48.502683+01:00 pve kernel: [12367959.655898]  dump_header+0x50/0x290
2024-02-13T13:36:48.502684+01:00 pve kernel: [12367959.655901]  oom_kill_process+0x10d/0x1c0
2024-02-13T13:36:48.502685+01:00 pve kernel: [12367959.655904]  out_of_memory+0x23c/0x570
2024-02-13T13:36:48.502686+01:00 pve kernel: [12367959.655907]  mem_cgroup_out_of_memory+0x145/0x170
2024-02-13T13:36:48.502687+01:00 pve kernel: [12367959.655911]  try_charge_memcg+0x731/0x820
2024-02-13T13:36:48.502687+01:00 pve kernel: [12367959.655914]  ? xas_load+0x4c/0x60
2024-02-13T13:36:48.502688+01:00 pve kernel: [12367959.655918]  charge_memcg+0x3e/0x100
2024-02-13T13:36:48.502689+01:00 pve kernel: [12367959.655921]  mem_cgroup_swapin_charge_folio+0x5a/0x130
2024-02-13T13:36:48.502690+01:00 pve kernel: [12367959.655923]  __read_swap_cache_async+0x1fd/0x290
2024-02-13T13:36:48.502690+01:00 pve kernel: [12367959.655926]  swap_cluster_readahead+0x250/0x3a0
2024-02-13T13:36:48.502691+01:00 pve kernel: [12367959.655929]  ? __rq_qos_issue+0x26/0x50
2024-02-13T13:36:48.502692+01:00 pve kernel: [12367959.655933]  swapin_readahead+0x5e/0x520
2024-02-13T13:36:48.502693+01:00 pve kernel: [12367959.655936]  do_swap_page+0x1aa/0xc20
2024-02-13T13:36:48.502693+01:00 pve kernel: [12367959.655938]  ? update_load_avg+0x82/0x810
2024-02-13T13:36:48.502694+01:00 pve kernel: [12367959.655941]  __handle_mm_fault+0x8f7/0x1070
2024-02-13T13:36:48.502695+01:00 pve kernel: [12367959.655946]  handle_mm_fault+0x119/0x330
2024-02-13T13:36:48.502696+01:00 pve kernel: [12367959.655948]  ? lock_mm_and_find_vma+0x43/0x230
2024-02-13T13:36:48.502696+01:00 pve kernel: [12367959.655950]  do_user_addr_fault+0x194/0x620
2024-02-13T13:36:48.502697+01:00 pve kernel: [12367959.655953]  exc_page_fault+0x80/0x1b0
2024-02-13T13:36:48.502698+01:00 pve kernel: [12367959.655956]  asm_exc_page_fault+0x27/0x30
2024-02-13T13:36:48.502699+01:00 pve kernel: [12367959.655959] RIP: 0010:__put_user_nocheck_4+0x3/0x10
2024-02-13T13:36:48.502700+01:00 pve kernel: [12367959.655962] Code: c9 0f 01 ca c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 01 cb <89> 01 31 c9 0f 01 ca c3 cc cc cc cc 90 90 90 90 90 90 90 90 90 90
2024-02-13T13:36:48.502701+01:00 pve kernel: [12367959.655964] RSP: 0018:ffff9d5793b53d98 EFLAGS: 00050202
2024-02-13T13:36:48.502701+01:00 pve kernel: [12367959.655967] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 00005646fb991db0
2024-02-13T13:36:48.502702+01:00 pve kernel: [12367959.655968] RDX: 0000000000000000 RSI: 00005646fb98e090 RDI: 0000000000000000
2024-02-13T13:36:48.502703+01:00 pve kernel: [12367959.655969] RBP: ffff9d5793b53e70 R08: 0000000000000000 R09: 0000000000000000
2024-02-13T13:36:48.502703+01:00 pve kernel: [12367959.655970] R10: 0000000000000000 R11: 0000000000000000 R12: 00005646fb991db0
2024-02-13T13:36:48.502704+01:00 pve kernel: [12367959.655971] R13: ffff9d5793b53de0 R14: ffff8cdf4b614f98 R15: ffff8cdf4b614f80
2024-02-13T13:36:48.502705+01:00 pve kernel: [12367959.655975]  ? do_epoll_wait+0x308/0x830
2024-02-13T13:36:48.502706+01:00 pve kernel: [12367959.655980]  ? __pfx_ep_autoremove_wake_function+0x10/0x10
2024-02-13T13:36:48.502707+01:00 pve kernel: [12367959.655983]  __x64_sys_epoll_wait+0x6f/0x110
2024-02-13T13:36:48.502708+01:00 pve kernel: [12367959.655986]  do_syscall_64+0x58/0x90
2024-02-13T13:36:48.502708+01:00 pve kernel: [12367959.655989]  ? exit_to_user_mode_prepare+0xa5/0x190
2024-02-13T13:36:48.502709+01:00 pve kernel: [12367959.655992]  ? irqentry_exit_to_user_mode+0x9/0x20
2024-02-13T13:36:48.502710+01:00 pve kernel: [12367959.655994]  ? irqentry_exit+0x43/0x50
2024-02-13T13:36:48.502711+01:00 pve kernel: [12367959.655996]  ? exc_page_fault+0x91/0x1b0
2024-02-13T13:36:48.502711+01:00 pve kernel: [12367959.655998]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
2024-02-13T13:36:48.502712+01:00 pve kernel: [12367959.656000] RIP: 0033:0x7fa2e1125e2e
2024-02-13T13:36:48.502713+01:00 pve kernel: [12367959.656005] Code: Unable to access opcode bytes at 0x7fa2e1125e04.
2024-02-13T13:36:48.502713+01:00 pve kernel: [12367959.656006] RSP: 002b:00007fffdd26e830 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
2024-02-13T13:36:48.502714+01:00 pve kernel: [12367959.656008] RAX: ffffffffffffffda RBX: 00005646fb98e020 RCX: 00007fa2e1125e2e
2024-02-13T13:36:48.502715+01:00 pve kernel: [12367959.656009] RDX: 000000000000001e RSI: 00005646fb991db0 RDI: 0000000000000007
2024-02-13T13:36:48.502716+01:00 pve kernel: [12367959.656010] RBP: 0000000000000032 R08: 0000000000000000 R09: 0000000000000000
2024-02-13T13:36:48.502716+01:00 pve kernel: [12367959.656011] R10: 00000000ffffffff R11: 0000000000000293 R12: 00000000000000e6
2024-02-13T13:36:48.502717+01:00 pve kernel: [12367959.656013] R13: 000000000000001e R14: 0000000000000017 R15: 00005646fb98e1b0
2024-02-13T13:36:48.502718+01:00 pve kernel: [12367959.656016]  </TASK>
2024-02-13T13:36:48.502718+01:00 pve kernel: [12367959.656017] memory: usage 524288kB, limit 524288kB, failcnt 21984
2024-02-13T13:36:48.502719+01:00 pve kernel: [12367959.656018] swap: usage 22880kB, limit 524288kB, failcnt 0
2024-02-13T13:36:48.502720+01:00 pve kernel: [12367959.656019] Memory cgroup stats for /lxc/108:
2024-02-13T13:36:48.502721+01:00 pve kernel: [12367959.656077] anon 0
2024-02-13T13:36:48.502721+01:00 pve kernel: [12367959.656077] file 518873088
2024-02-13T13:36:48.502722+01:00 pve kernel: [12367959.656077] kernel 17805312
2024-02-13T13:36:48.502723+01:00 pve kernel: [12367959.656077] kernel_stack 491520
2024-02-13T13:36:48.502723+01:00 pve kernel: [12367959.656077] pagetables 1482752
2024-02-13T13:36:48.502724+01:00 pve kernel: [12367959.656077] sec_pagetables 0
2024-02-13T13:36:48.502725+01:00 pve kernel: [12367959.656077] percpu 701080
2024-02-13T13:36:48.502725+01:00 pve kernel: [12367959.656077] sock 135168
2024-02-13T13:36:48.502726+01:00 pve kernel: [12367959.656077] vmalloc 487424
2024-02-13T13:36:48.502727+01:00 pve kernel: [12367959.656077] shmem 0
2024-02-13T13:36:48.502728+01:00 pve kernel: [12367959.656077] zswap 0
2024-02-13T13:36:48.502728+01:00 pve kernel: [12367959.656077] zswapped 0
2024-02-13T13:36:48.502729+01:00 pve kernel: [12367959.656077] file_mapped 0
2024-02-13T13:36:48.502730+01:00 pve kernel: [12367959.656077] file_dirty 0
2024-02-13T13:36:48.502730+01:00 pve kernel: [12367959.656077] file_writeback 518848512
2024-02-13T13:36:48.502731+01:00 pve kernel: [12367959.656077] swapcached 0
2024-02-13T13:36:48.502732+01:00 pve kernel: [12367959.656077] anon_thp 0
2024-02-13T13:36:48.502732+01:00 pve kernel: [12367959.656077] file_thp 0
2024-02-13T13:36:48.502733+01:00 pve kernel: [12367959.656077] shmem_thp 0
2024-02-13T13:36:48.502734+01:00 pve kernel: [12367959.656077] inactive_anon 0
2024-02-13T13:36:48.502734+01:00 pve kernel: [12367959.656077] active_anon 0
2024-02-13T13:36:48.502735+01:00 pve kernel: [12367959.656077] inactive_file 11386880
2024-02-13T13:36:48.502736+01:00 pve kernel: [12367959.656077] active_file 507150336
2024-02-13T13:36:48.502737+01:00 pve kernel: [12367959.656077] unevictable 0
2024-02-13T13:36:48.502738+01:00 pve kernel: [12367959.656077] slab_reclaimable 11699240
2024-02-13T13:36:48.502738+01:00 pve kernel: [12367959.656077] slab_unreclaimable 2323000
2024-02-13T13:36:48.502739+01:00 pve kernel: [12367959.656077] slab 14022240
2024-02-13T13:36:48.502740+01:00 pve kernel: [12367959.656082] Tasks state (memory values in pages):
2024-02-13T13:36:48.502741+01:00 pve kernel: [12367959.656082] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-02-13T13:36:48.502741+01:00 pve kernel: [12367959.656084] [ 183713]     0 183713     4589       96    77824      704             0 systemd
2024-02-13T13:36:48.502742+01:00 pve kernel: [12367959.656088] [ 184092]     0 184092      705       32    45056        0             0 agetty
2024-02-13T13:36:48.502743+01:00 pve kernel: [12367959.656091] [ 184194]     0 184194    10308       36    73728      160             0 master
2024-02-13T13:36:48.502743+01:00 pve kernel: [12367959.656093] [ 184195]   102 184195    10389       64    69632       96             0 pickup
2024-02-13T13:36:48.502744+01:00 pve kernel: [12367959.656095] [ 184196]   102 184196    10400       64    73728      128             0 qmgr
2024-02-13T13:36:48.502745+01:00 pve kernel: [12367959.656098] [ 183828]     0 183828     7835       32    81920      256          -250 systemd-journal
2024-02-13T13:36:48.502746+01:00 pve kernel: [12367959.656100] [ 183866]   105 183866     4031       32    73728      256             0 systemd-network
2024-02-13T13:36:48.502746+01:00 pve kernel: [12367959.656103] [ 183879]   110 183879     2026       96    53248       64             0 rpcbind
2024-02-13T13:36:48.502747+01:00 pve kernel: [12367959.656105] [ 183882]   106 183882     6384       87    94208     1056             0 systemd-resolve
2024-02-13T13:36:48.502748+01:00 pve kernel: [12367959.656108] [ 183886]     0 183886     1282       33    45056       32             0 blkmapd
2024-02-13T13:36:48.502749+01:00 pve kernel: [12367959.656110] [ 183887]     0 183887      812       33    40960       64             0 rpc.idmapd
2024-02-13T13:36:48.502750+01:00 pve kernel: [12367959.656113] [ 183888]     0 183888     1366       64    49152       64             0 nfsdcld
2024-02-13T13:36:48.502751+01:00 pve kernel: [12367959.656115] [ 183946]   100 183946     2113       32    49152       96          -900 dbus-daemon
2024-02-13T13:36:48.502752+01:00 pve kernel: [12367959.656117] [ 183951]     0 183951     7523      128    98304        0             0 networkd-dispat
2024-02-13T13:36:48.502752+01:00 pve kernel: [12367959.656119] [ 183953]   101 183953    55601        0    77824      224             0 rsyslogd
2024-02-13T13:36:48.502753+01:00 pve kernel: [12367959.656122] [ 183956]     0 183956     3836       32    69632      256             0 systemd-logind
2024-02-13T13:36:48.502754+01:00 pve kernel: [12367959.656124] [ 184028]     0 184028     1284       66    45056       64             0 rpc.mountd
2024-02-13T13:36:48.502754+01:00 pve kernel: [12367959.656126] [ 184027]   111 184027     1194       96    53248       64             0 rpc.statd
2024-02-13T13:36:48.502755+01:00 pve kernel: [12367959.656129] [ 184081]     0 184081      972       64    49152       32             0 cron
2024-02-13T13:36:48.502756+01:00 pve kernel: [12367959.656131] [ 207991]     0 207991     1830       64    53248      225             0 cron
2024-02-13T13:36:48.502756+01:00 pve kernel: [12367959.656133] [ 207992]     0 207992      723       32    45056        0             0 sh
2024-02-13T13:36:48.502757+01:00 pve kernel: [12367959.656135] [ 207993]     0 207993     4121       96    73728        0             0 script.py
2024-02-13T13:36:48.502758+01:00 pve kernel: [12367959.656138] [ 184089]     0 184089      705        0    40960       32             0 agetty
2024-02-13T13:36:48.502759+01:00 pve kernel: [12367959.656141] [ 184346]     0 184346     4170      128    69632      288             0 systemd
2024-02-13T13:36:48.502759+01:00 pve kernel: [12367959.656143] [ 184347]     0 184347     5352      128    77824      768             0 (sd-pam)
2024-02-13T13:36:48.502760+01:00 pve kernel: [12367959.656145] [ 184091]     0 184091     1916       32    49152      192             0 login
2024-02-13T13:36:48.502761+01:00 pve kernel: [12367959.656147] [ 184352]     0 184352     1379       96    49152      352             0 bash
2024-02-13T13:36:48.502762+01:00 pve kernel: [12367959.656149] [ 197512]     0 197512     4376       96    77824       13             0 script.py
2024-02-13T13:36:48.502763+01:00 pve kernel: [12367959.656151] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0,oom_memcg=/lxc/108,task_memcg=/lxc/108/ns/system.slice/systemd-resolved.service,task=systemd-resolve,pid=183882,uid=106
2024-02-13T13:36:48.502768+01:00 pve kernel: [12367959.656167] Memory cgroup out of memory: Killed process 183882 (systemd-resolve) total-vm:25536kB, anon-rss:92kB, file-rss:256kB, shmem-rss:0kB, UID:106 pgtables:92kB oom_score_adj:0
2024-02-13T13:36:50.514626+01:00 pve kernel: [12367961.667985] oom_reaper: reaped process 183882 (systemd-resolve), now anon-rss:92kB, file-rss:256kB, shmem-rss:0kB
2024-02-13T13:36:51.318669+01:00 pve kernel: [12367962.474192] in:imklog invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
2024-02-13T13:36:51.318681+01:00 pve kernel: [12367962.474198] CPU: 1 PID: 183958 Comm: in:imklog Tainted: P           O       6.2.16-12-pve #1
2024-02-13T13:36:51.318682+01:00 pve kernel: [12367962.474201] Hardware name: Dell Inc. OptiPlex 7050/0XHGV1, BIOS 1.20.0 03/23/2022
2024-02-13T13:36:51.318683+01:00 pve kernel: [12367962.474203] Call Trace:
2024-02-13T13:36:51.318685+01:00 pve kernel: [12367962.474205]  <TASK>
2024-02-13T13:36:51.318685+01:00 pve kernel: [12367962.474207]  dump_stack_lvl+0x48/0x70
2024-02-13T13:36:51.318686+01:00 pve kernel: [12367962.474212]  dump_stack+0x10/0x20
2024-02-13T13:36:51.318687+01:00 pve kernel: [12367962.474215]  dump_header+0x50/0x290
2024-02-13T13:36:51.318688+01:00 pve kernel: [12367962.474218]  oom_kill_process+0x10d/0x1c0
2024-02-13T13:36:51.318689+01:00 pve kernel: [12367962.474221]  out_of_memory+0x23c/0x570
2024-02-13T13:36:51.318689+01:00 pve kernel: [12367962.474223]  mem_cgroup_out_of_memory+0x145/0x170
2024-02-13T13:36:51.318690+01:00 pve kernel: [12367962.474227]  try_charge_memcg+0x731/0x820
2024-02-13T13:36:51.318691+01:00 pve kernel: [12367962.474230]  ? xas_load+0x4c/0x60
2024-02-13T13:36:51.318691+01:00 pve kernel: [12367962.474234]  charge_memcg+0x3e/0x100
2024-02-13T13:36:51.318692+01:00 pve kernel: [12367962.474236]  mem_cgroup_swapin_charge_folio+0x5a/0x130
2024-02-13T13:36:51.318693+01:00 pve kernel: [12367962.474238]  __read_swap_cache_async+0x1fd/0x290
2024-02-13T13:36:51.318694+01:00 pve kernel: [12367962.474242]  swap_cluster_readahead+0x190/0x3a0
2024-02-13T13:36:51.318695+01:00 pve kernel: [12367962.474246]  swapin_readahead+0x5e/0x520
2024-02-13T13:36:51.318696+01:00 pve kernel: [12367962.474250]  do_swap_page+0x1aa/0xc20
2024-02-13T13:36:51.318696+01:00 pve kernel: [12367962.474253]  ? update_load_avg+0x82/0x810
2024-02-13T13:36:51.318697+01:00 pve kernel: [12367962.474255]  __handle_mm_fault+0x8f7/0x1070
2024-02-13T13:36:51.318698+01:00 pve kernel: [12367962.474259]  handle_mm_fault+0x119/0x330
2024-02-13T13:36:51.318699+01:00 pve kernel: [12367962.474261]  ? lock_mm_and_find_vma+0x43/0x230
2024-02-13T13:36:51.318700+01:00 pve kernel: [12367962.474264]  do_user_addr_fault+0x194/0x620
2024-02-13T13:36:51.318700+01:00 pve kernel: [12367962.474267]  exc_page_fault+0x80/0x1b0
2024-02-13T13:36:51.318701+01:00 pve kernel: [12367962.474271]  asm_exc_page_fault+0x27/0x30
2024-02-13T13:36:51.318702+01:00 pve kernel: [12367962.474273] RIP: 0010:copy_user_short_string+0xd/0x50
2024-02-13T13:36:51.318702+01:00 pve kernel: [12367962.474276] Code: 89 d0 0f 01 ca c3 cc cc cc cc 01 ca eb df 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 89 d1 83 e2 07 c1 e9 03 74 12 4c 8b 06 <4c> 89 07 48 8d 76 08 48 8d 7f 08 ff c9 75 ee 21 d2 74 10 89 d1 8a
2024-02-13T13:36:51.318703+01:00 pve kernel: [12367962.474278] RSP: 0018:ffff9d5793ccbbc0 EFLAGS: 00050202
2024-02-13T13:36:51.318704+01:00 pve kernel: [12367962.474280] RAX: 0000000000000021 RBX: ffff8cdf57395400 RCX: 0000000000000004
2024-02-13T13:36:51.318705+01:00 pve kernel: [12367962.474281] RDX: 0000000000000001 RSI: ffff8cdf57395400 RDI: 00007f57e1155c60
2024-02-13T13:36:51.318706+01:00 pve kernel: [12367962.474283] RBP: ffff9d5793ccbbc8 R08: 363332315b3e343c R09: 0000000000000000
2024-02-13T13:36:51.318706+01:00 pve kernel: [12367962.474284] R10: 3a6563617254206c R11: 617254206c6c6143 R12: 0000000000000021
2024-02-13T13:36:51.318707+01:00 pve kernel: [12367962.474285] R13: 0000000000001fa0 R14: 00007f57e1155c60 R15: 0000000000000000
2024-02-13T13:36:51.318708+01:00 pve kernel: [12367962.474289]  ? _copy_to_user+0x25/0x60
2024-02-13T13:36:51.318709+01:00 pve kernel: [12367962.474291]  syslog_print+0x262/0x390
2024-02-13T13:36:51.318710+01:00 pve kernel: [12367962.474295]  ? __pfx_autoremove_wake_function+0x10/0x10
2024-02-13T13:36:51.318711+01:00 pve kernel: [12367962.474300]  do_syslog+0x1c8/0x3e0
2024-02-13T13:36:51.318712+01:00 pve kernel: [12367962.474303]  kmsg_read+0x43/0x60
2024-02-13T13:36:51.318712+01:00 pve kernel: [12367962.474306]  proc_reg_read+0x2e/0xa0
2024-02-13T13:36:51.318713+01:00 pve kernel: [12367962.474309]  vfs_read+0xa9/0x330
2024-02-13T13:36:51.318714+01:00 pve kernel: [12367962.474312]  ? wake_up_q+0x50/0xa0
2024-02-13T13:36:51.318714+01:00 pve kernel: [12367962.474314]  ? __fget_light+0xa5/0x120
2024-02-13T13:36:51.318715+01:00 pve kernel: [12367962.474317]  ksys_read+0x73/0x100
2024-02-13T13:36:51.318716+01:00 pve kernel: [12367962.474319]  __x64_sys_read+0x19/0x30
2024-02-13T13:36:51.318716+01:00 pve kernel: [12367962.474321]  do_syscall_64+0x58/0x90
2024-02-13T13:36:51.318717+01:00 pve kernel: [12367962.474324]  ? __secure_computing+0x89/0xf0
2024-02-13T13:36:51.318718+01:00 pve kernel: [12367962.474327]  ? exit_to_user_mode_prepare+0x39/0x190
2024-02-13T13:36:51.318718+01:00 pve kernel: [12367962.474329]  ? syscall_exit_to_user_mode+0x29/0x50
2024-02-13T13:36:51.318719+01:00 pve kernel: [12367962.474332]  ? do_syscall_64+0x67/0x90
2024-02-13T13:36:51.318720+01:00 pve kernel: [12367962.474334]  ? exc_page_fault+0x91/0x1b0
2024-02-13T13:36:51.318721+01:00 pve kernel: [12367962.474336]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
2024-02-13T13:36:51.318721+01:00 pve kernel: [12367962.474338] RIP: 0033:0x7f57e191481c
2024-02-13T13:36:51.318722+01:00 pve kernel: [12367962.474343] Code: Unable to access opcode bytes at 0x7f57e19147f2.
2024-02-13T13:36:51.318723+01:00 pve kernel: [12367962.474344] RSP: 002b:00007f57e1155420 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
2024-02-13T13:36:51.318724+01:00 pve kernel: [12367962.474346] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f57e191481c
2024-02-13T13:36:51.318724+01:00 pve kernel: [12367962.474348] RDX: 0000000000001fa0 RSI: 00007f57e1155c60 RDI: 0000000000000006
2024-02-13T13:36:51.318725+01:00 pve kernel: [12367962.474349] RBP: 000055de2017c180 R08: 0000000000000000 R09: 000055de20184248
2024-02-13T13:36:51.318726+01:00 pve kernel: [12367962.474350] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f57e1155c60
2024-02-13T13:36:51.318726+01:00 pve kernel: [12367962.474351] R13: 0000000000001fa0 R14: 00007f57e1155cbd R15: 00007f57e168e438
2024-02-13T13:36:51.318727+01:00 pve kernel: [12367962.474354]  </TASK>
2024-02-13T13:36:51.318728+01:00 pve kernel: [12367962.474355] memory: usage 524288kB, limit 524288kB, failcnt 22041
2024-02-13T13:36:51.318728+01:00 pve kernel: [12367962.474356] swap: usage 18552kB, limit 524288kB, failcnt 0
2024-02-13T13:36:51.318729+01:00 pve kernel: [12367962.474357] Memory cgroup stats for /lxc/108:
2024-02-13T13:36:51.318730+01:00 pve kernel: [12367962.474390] anon 0
2024-02-13T13:36:51.318731+01:00 pve kernel: [12367962.474390] file 518873088
2024-02-13T13:36:51.318732+01:00 pve kernel: [12367962.474390] kernel 17805312
2024-02-13T13:36:51.318732+01:00 pve kernel: [12367962.474390] kernel_stack 491520
2024-02-13T13:36:51.318733+01:00 pve kernel: [12367962.474390] pagetables 1482752
2024-02-13T13:36:51.318734+01:00 pve kernel: [12367962.474390] sec_pagetables 0
2024-02-13T13:36:51.318735+01:00 pve kernel: [12367962.474390] percpu 701080
2024-02-13T13:36:51.318735+01:00 pve kernel: [12367962.474390] sock 135168
2024-02-13T13:36:51.318736+01:00 pve kernel: [12367962.474390] vmalloc 487424
2024-02-13T13:36:51.318737+01:00 pve kernel: [12367962.474390] shmem 0
2024-02-13T13:36:51.318737+01:00 pve kernel: [12367962.474390] zswap 0
2024-02-13T13:36:51.318738+01:00 pve kernel: [12367962.474390] zswapped 0
2024-02-13T13:36:51.318739+01:00 pve kernel: [12367962.474390] file_mapped 0
2024-02-13T13:36:51.318739+01:00 pve kernel: [12367962.474390] file_dirty 0
2024-02-13T13:36:51.318740+01:00 pve kernel: [12367962.474390] file_writeback 518848512
2024-02-13T13:36:51.318741+01:00 pve kernel: [12367962.474390] swapcached 0
2024-02-13T13:36:51.318742+01:00 pve kernel: [12367962.474390] anon_thp 0
2024-02-13T13:36:51.318742+01:00 pve kernel: [12367962.474390] file_thp 0
2024-02-13T13:36:51.318743+01:00 pve kernel: [12367962.474390] shmem_thp 0
2024-02-13T13:36:51.318744+01:00 pve kernel: [12367962.474390] inactive_anon 0
2024-02-13T13:36:51.318745+01:00 pve kernel: [12367962.474390] active_anon 0
2024-02-13T13:36:51.318746+01:00 pve kernel: [12367962.474390] inactive_file 500600832
2024-02-13T13:36:51.318746+01:00 pve kernel: [12367962.474390] active_file 17936384
2024-02-13T13:36:51.318747+01:00 pve kernel: [12367962.474390] unevictable 0
2024-02-13T13:36:51.318748+01:00 pve kernel: [12367962.474390] slab_reclaimable 11699240
2024-02-13T13:36:51.318748+01:00 pve kernel: [12367962.474390] slab_unreclaimable 2323000
2024-02-13T13:36:51.318749+01:00 pve kernel: [12367962.474390] slab 14022240
2024-02-13T13:36:51.318750+01:00 pve kernel: [12367962.474394] Tasks state (memory values in pages):
2024-02-13T13:36:51.318750+01:00 pve kernel: [12367962.474394] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-02-13T13:36:51.318751+01:00 pve kernel: [12367962.474396] [ 183713]     0 183713     4589       96    77824      704             0 systemd
2024-02-13T13:36:51.318752+01:00 pve kernel: [12367962.474400] [ 184092]     0 184092      705       32    45056        0             0 agetty
2024-02-13T13:36:51.318753+01:00 pve kernel: [12367962.474403] [ 184194]     0 184194    10308       36    73728      160             0 master
2024-02-13T13:36:51.318753+01:00 pve kernel: [12367962.474406] [ 184195]   102 184195    10389       64    69632       96             0 pickup
2024-02-13T13:36:51.318754+01:00 pve kernel: [12367962.474408] [ 184196]   102 184196    10400       64    73728      128             0 qmgr
2024-02-13T13:36:51.318755+01:00 pve kernel: [12367962.474411] [ 183828]     0 183828     7835       32    81920      256          -250 systemd-journal
2024-02-13T13:36:51.318756+01:00 pve kernel: [12367962.474413] [ 183866]   105 183866     4031       32    73728      256             0 systemd-network
2024-02-13T13:36:51.318756+01:00 pve kernel: [12367962.474415] [ 183879]   110 183879     2026       96    53248       64             0 rpcbind
2024-02-13T13:36:51.318757+01:00 pve kernel: [12367962.474418] [ 183882]   106 183882     6384       87    94208       14             0 systemd-resolve
2024-02-13T13:36:51.318758+01:00 pve kernel: [12367962.474420] [ 183886]     0 183886     1282       33    45056       32             0 blkmapd
2024-02-13T13:36:51.318759+01:00 pve kernel: [12367962.474422] [ 183887]     0 183887      812       33    40960       64             0 rpc.idmapd
2024-02-13T13:36:51.318760+01:00 pve kernel: [12367962.474424] [ 183888]     0 183888     1366       64    49152       64             0 nfsdcld
2024-02-13T13:36:51.318761+01:00 pve kernel: [12367962.474426] [ 183946]   100 183946     2113       32    49152       96          -900 dbus-daemon
2024-02-13T13:36:51.318761+01:00 pve kernel: [12367962.474428] [ 183951]     0 183951     7523      128    98304        0             0 networkd-dispat
2024-02-13T13:36:51.318762+01:00 pve kernel: [12367962.474431] [ 183953]   101 183953    55601        0    77824      224             0 rsyslogd
2024-02-13T13:36:51.318763+01:00 pve kernel: [12367962.474433] [ 183956]     0 183956     3836       32    69632      256             0 systemd-logind
2024-02-13T13:36:51.318763+01:00 pve kernel: [12367962.474436] [ 184028]     0 184028     1284       66    45056       64             0 rpc.mountd
2024-02-13T13:36:51.318765+01:00 pve kernel: [12367962.474438] [ 184027]   111 184027     1194       96    53248       64             0 rpc.statd
2024-02-13T13:36:51.318765+01:00 pve kernel: [12367962.474440] [ 184081]     0 184081      972       64    49152       32             0 cron
2024-02-13T13:36:51.318766+01:00 pve kernel: [12367962.474443] [ 207991]     0 207991     1830       64    53248      225             0 cron
2024-02-13T13:36:51.318767+01:00 pve kernel: [12367962.474445] [ 207992]     0 207992      723       32    45056        0             0 sh
2024-02-13T13:36:51.318768+01:00 pve kernel: [12367962.474447] [ 207993]     0 207993     4121       96    73728        0             0 script.py
2024-02-13T13:36:51.318769+01:00 pve kernel: [12367962.474449] [ 184089]     0 184089      705        0    40960       32             0 agetty
2024-02-13T13:36:51.318770+01:00 pve kernel: [12367962.474451] [ 184346]     0 184346     4170      128    69632      288             0 systemd
2024-02-13T13:36:51.318770+01:00 pve kernel: [12367962.474453] [ 184347]     0 184347     5352      128    77824      768             0 (sd-pam)
2024-02-13T13:36:51.318771+01:00 pve kernel: [12367962.474456] [ 184091]     0 184091     1916       32    49152      192             0 login
2024-02-13T13:36:51.318772+01:00 pve kernel: [12367962.474458] [ 184352]     0 184352     1379       96    49152      352             0 bash
2024-02-13T13:36:51.318772+01:00 pve kernel: [12367962.474460] [ 197512]     0 197512     4376       96    77824       13             0 script.py
2024-02-13T13:36:51.318778+01:00 pve kernel: [12367962.474462] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0,oom_memcg=/lxc/108,task_memcg=/lxc/108/ns/user.slice/user-0.slice/user@0.service,task=(sd-pam),pid=184347,uid=0
2024-02-13T13:36:51.318779+01:00 pve kernel: [12367962.474479] Memory cgroup out of memory: Killed process 184347 ((sd-pam)) total-vm:21408kB, anon-rss:384kB, file-rss:128kB, shmem-rss:0kB, UID:0 pgtables:76kB oom_score_adj:0
2024-02-13T13:36:53.330633+01:00 pve kernel: [12367964.483750] oom_reaper: reaped process 184347 ((sd-pam)), now anon-rss:384kB, file-rss:128kB, shmem-rss:0kB
2024-02-13T13:36:55.494662+01:00 pve kernel: [12367966.650223] kworker/u16:8 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
2024-02-13T13:36:55.494673+01:00 pve kernel: [12367966.650229] CPU: 7 PID: 198323 Comm: kworker/u16:8 Tainted: P           O       6.2.16-12-pve #1
2024-02-13T13:36:55.494675+01:00 pve kernel: [12367966.650232] Hardware name: Dell Inc. OptiPlex 7050/0XHGV1, BIOS 1.20.0 03/23/2022
2024-02-13T13:36:55.494676+01:00 pve kernel: [12367966.650234] Workqueue: writeback wb_workfn (flush-cifs-130)
2024-02-13T13:36:55.494677+01:00 pve kernel: [12367966.650239] Call Trace:
2024-02-13T13:36:55.494678+01:00 pve kernel: [12367966.650241]  <TASK>
2024-02-13T13:36:55.494679+01:00 pve kernel: [12367966.650243]  dump_stack_lvl+0x48/0x70
2024-02-13T13:36:55.494680+01:00 pve kernel: [12367966.650247]  dump_stack+0x10/0x20
2024-02-13T13:36:55.494681+01:00 pve kernel: [12367966.650250]  dump_header+0x50/0x290
2024-02-13T13:36:55.494682+01:00 pve kernel: [12367966.650253]  oom_kill_process+0x10d/0x1c0
2024-02-13T13:36:55.494682+01:00 pve kernel: [12367966.650256]  out_of_memory+0x23c/0x570
2024-02-13T13:36:55.494683+01:00 pve kernel: [12367966.650258]  mem_cgroup_out_of_memory+0x145/0x170
2024-02-13T13:36:55.494684+01:00 pve kernel: [12367966.650262]  try_charge_memcg+0x731/0x820
2024-02-13T13:36:55.494685+01:00 pve kernel: [12367966.650266]  ? __mod_memcg_state+0x47/0xd0
2024-02-13T13:36:55.494686+01:00 pve kernel: [12367966.650269]  mem_cgroup_charge_skmem+0x40/0xf0
2024-02-13T13:36:55.494686+01:00 pve kernel: [12367966.650271]  __sk_mem_raise_allocated+0xcc/0x4f0
2024-02-13T13:36:55.494687+01:00 pve kernel: [12367966.650274]  __sk_mem_schedule+0x30/0x50
2024-02-13T13:36:55.494688+01:00 pve kernel: [12367966.650276]  tcp_wmem_schedule+0x41/0x90
2024-02-13T13:36:55.494689+01:00 pve kernel: [12367966.650278]  tcp_sendmsg_locked+0x5ae/0xc90
2024-02-13T13:36:55.494689+01:00 pve kernel: [12367966.650282]  tcp_sendmsg+0x2c/0x50
2024-02-13T13:36:55.494690+01:00 pve kernel: [12367966.650284]  inet_sendmsg+0x42/0x80
2024-02-13T13:36:55.494691+01:00 pve kernel: [12367966.650287]  sock_sendmsg+0x66/0x70
2024-02-13T13:36:55.494692+01:00 pve kernel: [12367966.650291]  smb_send_kvec+0x84/0x190 [cifs]
2024-02-13T13:36:55.494693+01:00 pve kernel: [12367966.650346]  ? release_sock+0x8f/0xb0
2024-02-13T13:36:55.494694+01:00 pve kernel: [12367966.650348]  __smb_send_rqst+0x2ef/0x7b0 [cifs]
2024-02-13T13:36:55.494694+01:00 pve kernel: [12367966.650400]  smb_send_rqst+0x19b/0x200 [cifs]
2024-02-13T13:36:55.494695+01:00 pve kernel: [12367966.650450]  cifs_call_async+0x144/0x330 [cifs]
2024-02-13T13:36:55.494696+01:00 pve kernel: [12367966.650498]  ? __pfx_smb2_writev_callback+0x10/0x10 [cifs]
2024-02-13T13:36:55.494696+01:00 pve kernel: [12367966.650549]  smb2_async_writev+0x4bf/0x740 [cifs]
2024-02-13T13:36:55.494697+01:00 pve kernel: [12367966.650599]  ? __pfx_cifs_writedata_release+0x10/0x10 [cifs]
2024-02-13T13:36:55.494698+01:00 pve kernel: [12367966.650646]  cifs_writepages+0xa4c/0x1230 [cifs]
2024-02-13T13:36:55.494699+01:00 pve kernel: [12367966.650691]  ? cifs_writepages+0xa4c/0x1230 [cifs]
2024-02-13T13:36:55.494700+01:00 pve kernel: [12367966.650738]  do_writepages+0xcf/0x1e0
2024-02-13T13:36:55.494700+01:00 pve kernel: [12367966.650741]  __writeback_single_inode+0x44/0x370
2024-02-13T13:36:55.494701+01:00 pve kernel: [12367966.650743]  ? fprop_fraction_percpu+0x34/0x90
2024-02-13T13:36:55.494702+01:00 pve kernel: [12367966.650747]  writeback_sb_inodes+0x21b/0x500
2024-02-13T13:36:55.494703+01:00 pve kernel: [12367966.650751]  __writeback_inodes_wb+0x54/0x100
2024-02-13T13:36:55.494704+01:00 pve kernel: [12367966.650753]  ? queue_io+0x115/0x120
2024-02-13T13:36:55.494704+01:00 pve kernel: [12367966.650754]  wb_writeback+0x20e/0x300
2024-02-13T13:36:55.494705+01:00 pve kernel: [12367966.650757]  wb_workfn+0x334/0x4d0
2024-02-13T13:36:55.494706+01:00 pve kernel: [12367966.650759]  ? try_to_wake_up+0x6c/0x600
2024-02-13T13:36:55.494707+01:00 pve kernel: [12367966.650763]  process_one_work+0x222/0x430
2024-02-13T13:36:55.494707+01:00 pve kernel: [12367966.650766]  worker_thread+0x50/0x3e0
2024-02-13T13:36:55.494708+01:00 pve kernel: [12367966.650769]  ? __pfx_worker_thread+0x10/0x10
2024-02-13T13:36:55.494709+01:00 pve kernel: [12367966.650771]  kthread+0xe6/0x110
2024-02-13T13:36:55.494709+01:00 pve kernel: [12367966.650774]  ? __pfx_kthread+0x10/0x10
2024-02-13T13:36:55.494710+01:00 pve kernel: [12367966.650777]  ret_from_fork+0x29/0x50
2024-02-13T13:36:55.494711+01:00 pve kernel: [12367966.650781]  </TASK>
2024-02-13T13:36:55.494711+01:00 pve kernel: [12367966.650785] memory: usage 524288kB, limit 524288kB, failcnt 22123
2024-02-13T13:36:55.494712+01:00 pve kernel: [12367966.650787] swap: usage 17276kB, limit 524288kB, failcnt 0
2024-02-13T13:36:55.494713+01:00 pve kernel: [12367966.650788] Memory cgroup stats for /lxc/108:
2024-02-13T13:36:55.494714+01:00 pve kernel: [12367966.650865] anon 0
2024-02-13T13:36:55.494714+01:00 pve kernel: [12367966.650865] file 518873088
2024-02-13T13:36:55.494715+01:00 pve kernel: [12367966.650865] kernel 17805312
2024-02-13T13:36:55.494716+01:00 pve kernel: [12367966.650865] kernel_stack 491520
2024-02-13T13:36:55.494717+01:00 pve kernel: [12367966.650865] pagetables 1482752
2024-02-13T13:36:55.494717+01:00 pve kernel: [12367966.650865] sec_pagetables 0
2024-02-13T13:36:55.494718+01:00 pve kernel: [12367966.650865] percpu 701080
2024-02-13T13:36:55.494719+01:00 pve kernel: [12367966.650865] sock 135168
2024-02-13T13:36:55.494719+01:00 pve kernel: [12367966.650865] vmalloc 487424
2024-02-13T13:36:55.494720+01:00 pve kernel: [12367966.650865] shmem 0
2024-02-13T13:36:55.494721+01:00 pve kernel: [12367966.650865] zswap 0
2024-02-13T13:36:55.494722+01:00 pve kernel: [12367966.650865] zswapped 0
2024-02-13T13:36:55.494722+01:00 pve kernel: [12367966.650865] file_mapped 0
2024-02-13T13:36:55.494723+01:00 pve kernel: [12367966.650865] file_dirty 0
2024-02-13T13:36:55.494724+01:00 pve kernel: [12367966.650865] file_writeback 518848512
2024-02-13T13:36:55.494725+01:00 pve kernel: [12367966.650865] swapcached 0
2024-02-13T13:36:55.494725+01:00 pve kernel: [12367966.650865] anon_thp 0
2024-02-13T13:36:55.494726+01:00 pve kernel: [12367966.650865] file_thp 0
2024-02-13T13:36:55.494727+01:00 pve kernel: [12367966.650865] shmem_thp 0
2024-02-13T13:36:55.494728+01:00 pve kernel: [12367966.650865] inactive_anon 0
2024-02-13T13:36:55.494729+01:00 pve kernel: [12367966.650865] active_anon 0
2024-02-13T13:36:55.494729+01:00 pve kernel: [12367966.650865] inactive_file 283521024
2024-02-13T13:36:55.494730+01:00 pve kernel: [12367966.650865] active_file 234885120
2024-02-13T13:36:55.494731+01:00 pve kernel: [12367966.650865] unevictable 0
2024-02-13T13:36:55.494731+01:00 pve kernel: [12367966.650865] slab_reclaimable 11699240
2024-02-13T13:36:55.494732+01:00 pve kernel: [12367966.650865] slab_unreclaimable 2323000
2024-02-13T13:36:55.494733+01:00 pve kernel: [12367966.650865] slab 14022240
2024-02-13T13:36:55.494733+01:00 pve kernel: [12367966.650870] Tasks state (memory values in pages):
2024-02-13T13:36:55.494734+01:00 pve kernel: [12367966.650870] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-02-13T13:36:55.494735+01:00 pve kernel: [12367966.650873] [ 183713]     0 183713     4589       96    77824      704             0 systemd
2024-02-13T13:36:55.494736+01:00 pve kernel: [12367966.650877] [ 184092]     0 184092      705       32    45056        0             0 agetty
2024-02-13T13:36:55.494736+01:00 pve kernel: [12367966.650881] [ 184194]     0 184194    10308       36    73728      160             0 master
2024-02-13T13:36:55.494737+01:00 pve kernel: [12367966.650883] [ 184195]   102 184195    10389       64    69632       96             0 pickup
2024-02-13T13:36:55.494738+01:00 pve kernel: [12367966.650885] [ 184196]   102 184196    10400       64    73728      128             0 qmgr
2024-02-13T13:36:55.494738+01:00 pve kernel: [12367966.650889] [ 183828]     0 183828     7835       32    81920      256          -250 systemd-journal
2024-02-13T13:36:55.494739+01:00 pve kernel: [12367966.650892] [ 183866]   105 183866     4031       32    73728      256             0 systemd-network
2024-02-13T13:36:55.494740+01:00 pve kernel: [12367966.650895] [ 183879]   110 183879     2026       96    53248       64             0 rpcbind
2024-02-13T13:36:55.494741+01:00 pve kernel: [12367966.650897] [ 183882]   106 183882     6384       87    94208       14             0 systemd-resolve
2024-02-13T13:36:55.494742+01:00 pve kernel: [12367966.650899] [ 183886]     0 183886     1282       33    45056       32             0 blkmapd
2024-02-13T13:36:55.494743+01:00 pve kernel: [12367966.650902] [ 183887]     0 183887      812       33    40960       64             0 rpc.idmapd
2024-02-13T13:36:55.494744+01:00 pve kernel: [12367966.650904] [ 183888]     0 183888     1366       64    49152       64             0 nfsdcld
2024-02-13T13:36:55.494744+01:00 pve kernel: [12367966.650907] [ 183946]   100 183946     2113       32    49152       96          -900 dbus-daemon
2024-02-13T13:36:55.494745+01:00 pve kernel: [12367966.650909] [ 183951]     0 183951     7523      128    98304        0             0 networkd-dispat
2024-02-13T13:36:55.494746+01:00 pve kernel: [12367966.650912] [ 183953]   101 183953    55601        0    77824      224             0 rsyslogd
2024-02-13T13:36:55.494747+01:00 pve kernel: [12367966.650915] [ 183956]     0 183956     3836       32    69632      256             0 systemd-logind
2024-02-13T13:36:55.494748+01:00 pve kernel: [12367966.650917] [ 184028]     0 184028     1284       66    45056       64             0 rpc.mountd
2024-02-13T13:36:55.494748+01:00 pve kernel: [12367966.650920] [ 184027]   111 184027     1194       96    53248       64             0 rpc.statd
2024-02-13T13:36:55.494749+01:00 pve kernel: [12367966.650923] [ 184081]     0 184081      972       64    49152       32             0 cron
2024-02-13T13:36:55.494750+01:00 pve kernel: [12367966.650925] [ 207991]     0 207991     1830       64    53248      225             0 cron
2024-02-13T13:36:55.494750+01:00 pve kernel: [12367966.650927] [ 207992]     0 207992      723       32    45056        0             0 sh
2024-02-13T13:36:55.494751+01:00 pve kernel: [12367966.650929] [ 207993]     0 207993     4121       96    73728        0             0 script.py
2024-02-13T13:36:55.494752+01:00 pve kernel: [12367966.650932] [ 184089]     0 184089      705        0    40960       32             0 agetty
2024-02-13T13:36:55.494752+01:00 pve kernel: [12367966.650934] [ 184346]     0 184346     4170      128    69632      288             0 systemd
2024-02-13T13:36:55.494753+01:00 pve kernel: [12367966.650936] [ 184347]     0 184347     5352      128    77824        0             0 (sd-pam)
2024-02-13T13:36:55.494754+01:00 pve kernel: [12367966.650939] [ 184091]     0 184091     1916       32    49152      192             0 login
2024-02-13T13:36:55.494754+01:00 pve kernel: [12367966.650941] [ 184352]     0 184352     1379       96    49152      352             0 bash
2024-02-13T13:36:55.494755+01:00 pve kernel: [12367966.650943] [ 197512]     0 197512     4376       96    77824       13             0 script.py
2024-02-13T13:36:55.494756+01:00 pve kernel: [12367966.650945] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/lxc/108,task_memcg=/lxc/108/ns/init.scope,task=systemd,pid=183713,uid=0
2024-02-13T13:36:55.494757+01:00 pve kernel: [12367966.650958] Memory cgroup out of memory: Killed process 183713 (systemd) total-vm:18356kB, anon-rss:0kB, file-rss:384kB, shmem-rss:0kB, UID:0 pgtables:76kB oom_score_adj:0
2024-02-13T13:36:57.522627+01:00 pve kernel: [12367968.675779] oom_reaper: reaped process 183713 (systemd), now anon-rss:0kB, file-rss:92kB, shmem-rss:0kB
2024-02-13T13:36:58.630699+01:00 pve kernel: [12367969.786682] systemd-journal invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-250
2024-02-13T13:36:58.630710+01:00 pve kernel: [12367969.786688] CPU: 6 PID: 183828 Comm: systemd-journal Tainted: P           O       6.2.16-12-pve #1
2024-02-13T13:36:58.630711+01:00 pve kernel: [12367969.786691] Hardware name: Dell Inc. OptiPlex 7050/0XHGV1, BIOS 1.20.0 03/23/2022
2024-02-13T13:36:58.630712+01:00 pve kernel: [12367969.786692] Call Trace:
2024-02-13T13:36:58.630714+01:00 pve kernel: [12367969.786694]  <TASK>
2024-02-13T13:36:58.630715+01:00 pve kernel: [12367969.786696]  dump_stack_lvl+0x48/0x70
2024-02-13T13:36:58.630716+01:00 pve kernel: [12367969.786701]  dump_stack+0x10/0x20
2024-02-13T13:36:58.630717+01:00 pve kernel: [12367969.786703]  dump_header+0x50/0x290
2024-02-13T13:36:58.630717+01:00 pve kernel: [12367969.786706]  oom_kill_process+0x10d/0x1c0
2024-02-13T13:36:58.630718+01:00 pve kernel: [12367969.786708]  out_of_memory+0x23c/0x570
2024-02-13T13:36:58.630719+01:00 pve kernel: [12367969.786711]  mem_cgroup_out_of_memory+0x145/0x170
2024-02-13T13:36:58.630720+01:00 pve kernel: [12367969.786714]  try_charge_memcg+0x731/0x820
2024-02-13T13:36:58.630720+01:00 pve kernel: [12367969.786718]  ? xas_load+0x4c/0x60
2024-02-13T13:36:58.630721+01:00 pve kernel: [12367969.786721]  charge_memcg+0x3e/0x100
2024-02-13T13:36:58.630722+01:00 pve kernel: [12367969.786723]  mem_cgroup_swapin_charge_folio+0x5a/0x130
2024-02-13T13:36:58.630723+01:00 pve kernel: [12367969.786725]  __read_swap_cache_async+0x1fd/0x290
2024-02-13T13:36:58.630724+01:00 pve kernel: [12367969.786729]  swap_cluster_readahead+0x250/0x3a0
2024-02-13T13:36:58.630725+01:00 pve kernel: [12367969.786731]  ? __rq_qos_issue+0x26/0x50
2024-02-13T13:36:58.630726+01:00 pve kernel: [12367969.786735]  swapin_readahead+0x5e/0x520
2024-02-13T13:36:58.630726+01:00 pve kernel: [12367969.786738]  do_swap_page+0x1aa/0xc20
2024-02-13T13:36:58.630727+01:00 pve kernel: [12367969.786740]  ? update_load_avg+0x82/0x810
2024-02-13T13:36:58.630728+01:00 pve kernel: [12367969.786743]  __handle_mm_fault+0x8f7/0x1070
2024-02-13T13:36:58.630728+01:00 pve kernel: [12367969.786747]  handle_mm_fault+0x119/0x330
2024-02-13T13:36:58.630729+01:00 pve kernel: [12367969.786749]  ? lock_mm_and_find_vma+0x43/0x230
2024-02-13T13:36:58.630730+01:00 pve kernel: [12367969.786751]  do_user_addr_fault+0x194/0x620
2024-02-13T13:36:58.630731+01:00 pve kernel: [12367969.786754]  exc_page_fault+0x80/0x1b0
2024-02-13T13:36:58.630731+01:00 pve kernel: [12367969.786757]  asm_exc_page_fault+0x27/0x30
2024-02-13T13:36:58.630732+01:00 pve kernel: [12367969.786759] RIP: 0010:__put_user_nocheck_4+0x3/0x10
2024-02-13T13:36:58.630733+01:00 pve kernel: [12367969.786762] Code: c9 0f 01 ca c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 01 cb <89> 01 31 c9 0f 01 ca c3 cc cc cc cc 90 90 90 90 90 90 90 90 90 90
2024-02-13T13:36:58.630733+01:00 pve kernel: [12367969.786764] RSP: 0018:ffff9d5793b53d98 EFLAGS: 00050202
2024-02-13T13:36:58.630734+01:00 pve kernel: [12367969.786766] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 00005646fb991db0
2024-02-13T13:36:58.630735+01:00 pve kernel: [12367969.786767] RDX: 0000000000000000 RSI: 00005646fb98e090 RDI: 0000000000000000
2024-02-13T13:36:58.630736+01:00 pve kernel: [12367969.786768] RBP: ffff9d5793b53e70 R08: 0000000000000000 R09: 0000000000000000
2024-02-13T13:36:58.630736+01:00 pve kernel: [12367969.786769] R10: 0000000000000000 R11: 0000000000000000 R12: 00005646fb991db0
2024-02-13T13:36:58.630737+01:00 pve kernel: [12367969.786771] R13: ffff9d5793b53de0 R14: ffff8cdf4b614f98 R15: ffff8cdf4b614f80
2024-02-13T13:36:58.630738+01:00 pve kernel: [12367969.786774]  ? do_epoll_wait+0x308/0x830
2024-02-13T13:36:58.630739+01:00 pve kernel: [12367969.786778]  ? __pfx_ep_autoremove_wake_function+0x10/0x10
2024-02-13T13:36:58.630740+01:00 pve kernel: [12367969.786781]  __x64_sys_epoll_wait+0x6f/0x110
2024-02-13T13:36:58.630741+01:00 pve kernel: [12367969.786784]  do_syscall_64+0x58/0x90
2024-02-13T13:36:58.630741+01:00 pve kernel: [12367969.786787]  ? exit_to_user_mode_prepare+0xa5/0x190
2024-02-13T13:36:58.630742+01:00 pve kernel: [12367969.786789]  ? irqentry_exit_to_user_mode+0x9/0x20
2024-02-13T13:36:58.630743+01:00 pve kernel: [12367969.786791]  ? irqentry_exit+0x43/0x50
2024-02-13T13:36:58.630744+01:00 pve kernel: [12367969.786793]  ? exc_page_fault+0x91/0x1b0
2024-02-13T13:36:58.630744+01:00 pve kernel: [12367969.786796]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
2024-02-13T13:36:58.630745+01:00 pve kernel: [12367969.786797] RIP: 0033:0x7fa2e1125e2e
2024-02-13T13:36:58.630746+01:00 pve kernel: [12367969.786802] Code: Unable to access opcode bytes at 0x7fa2e1125e04.
2024-02-13T13:36:58.630746+01:00 pve kernel: [12367969.786803] RSP: 002b:00007fffdd26e830 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
2024-02-13T13:36:58.630747+01:00 pve kernel: [12367969.786804] RAX: ffffffffffffffda RBX: 00005646fb98e020 RCX: 00007fa2e1125e2e
2024-02-13T13:36:58.630748+01:00 pve kernel: [12367969.786806] RDX: 000000000000001e RSI: 00005646fb991db0 RDI: 0000000000000007
2024-02-13T13:36:58.630749+01:00 pve kernel: [12367969.786807] RBP: 0000000000000032 R08: 0000000000000000 R09: 0000000000000000
2024-02-13T13:36:58.630749+01:00 pve kernel: [12367969.786807] R10: 00000000ffffffff R11: 0000000000000293 R12: 00000000000000e6
2024-02-13T13:36:58.630750+01:00 pve kernel: [12367969.786809] R13: 000000000000001e R14: 0000000000000017 R15: 00005646fb98e1b0
2024-02-13T13:36:58.630751+01:00 pve kernel: [12367969.786811]  </TASK>
2024-02-13T13:36:58.630751+01:00 pve kernel: [12367969.786812] memory: usage 524288kB, limit 524288kB, failcnt 22182
2024-02-13T13:36:58.630752+01:00 pve kernel: [12367969.786813] swap: usage 14460kB, limit 524288kB, failcnt 0
2024-02-13T13:36:58.630753+01:00 pve kernel: [12367969.786814] Memory cgroup stats for /lxc/108:
2024-02-13T13:36:58.630754+01:00 pve kernel: [12367969.786840] anon 0
2024-02-13T13:36:58.630754+01:00 pve kernel: [12367969.786840] file 518873088
2024-02-13T13:36:58.630755+01:00 pve kernel: [12367969.786840] kernel 17805312
2024-02-13T13:36:58.630756+01:00 pve kernel: [12367969.786840] kernel_stack 491520
2024-02-13T13:36:58.630756+01:00 pve kernel: [12367969.786840] pagetables 1482752
2024-02-13T13:36:58.630757+01:00 pve kernel: [12367969.786840] sec_pagetables 0
2024-02-13T13:36:58.630758+01:00 pve kernel: [12367969.786840] percpu 701080
2024-02-13T13:36:58.630759+01:00 pve kernel: [12367969.786840] sock 135168
2024-02-13T13:36:58.630760+01:00 pve kernel: [12367969.786840] vmalloc 487424
2024-02-13T13:36:58.630761+01:00 pve kernel: [12367969.786840] shmem 0
2024-02-13T13:36:58.630761+01:00 pve kernel: [12367969.786840] zswap 0
2024-02-13T13:36:58.630762+01:00 pve kernel: [12367969.786840] zswapped 0
2024-02-13T13:36:58.630763+01:00 pve kernel: [12367969.786840] file_mapped 0
2024-02-13T13:36:58.630763+01:00 pve kernel: [12367969.786840] file_dirty 0
2024-02-13T13:36:58.630764+01:00 pve kernel: [12367969.786840] file_writeback 518848512
2024-02-13T13:36:58.630765+01:00 pve kernel: [12367969.786840] swapcached 0
2024-02-13T13:36:58.630765+01:00 pve kernel: [12367969.786840] anon_thp 0
2024-02-13T13:36:58.630766+01:00 pve kernel: [12367969.786840] file_thp 0
2024-02-13T13:36:58.630767+01:00 pve kernel: [12367969.786840] shmem_thp 0
2024-02-13T13:36:58.630767+01:00 pve kernel: [12367969.786840] inactive_anon 0
2024-02-13T13:36:58.630768+01:00 pve kernel: [12367969.786840] active_anon 0
2024-02-13T13:36:58.630769+01:00 pve kernel: [12367969.786840] inactive_file 262795264
2024-02-13T13:36:58.630770+01:00 pve kernel: [12367969.786840] active_file 255856640
2024-02-13T13:36:58.630770+01:00 pve kernel: [12367969.786840] unevictable 0
2024-02-13T13:36:58.630771+01:00 pve kernel: [12367969.786840] slab_reclaimable 11699240
2024-02-13T13:36:58.630772+01:00 pve kernel: [12367969.786840] slab_unreclaimable 2323000
2024-02-13T13:36:58.630773+01:00 pve kernel: [12367969.786840] slab 14022240
2024-02-13T13:36:58.630774+01:00 pve kernel: [12367969.786844] Tasks state (memory values in pages):
2024-02-13T13:36:58.630774+01:00 pve kernel: [12367969.786844] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-02-13T13:36:58.630775+01:00 pve kernel: [12367969.786846] [ 183713]     0 183713     4589       23    77824        0             0 systemd
2024-02-13T13:36:58.630776+01:00 pve kernel: [12367969.786849] [ 184092]     0 184092      705       32    45056        0             0 agetty
2024-02-13T13:36:58.630777+01:00 pve kernel: [12367969.786852] [ 184194]     0 184194    10308       36    73728      160             0 master
2024-02-13T13:36:58.630777+01:00 pve kernel: [12367969.786854] [ 184195]   102 184195    10389       64    69632       96             0 pickup
2024-02-13T13:36:58.630778+01:00 pve kernel: [12367969.786856] [ 184196]   102 184196    10400       64    73728      128             0 qmgr
2024-02-13T13:36:58.630779+01:00 pve kernel: [12367969.786858] [ 183828]     0 183828     7835       32    81920      256          -250 systemd-journal
2024-02-13T13:36:58.630779+01:00 pve kernel: [12367969.786860] [ 183866]   105 183866     4031       32    73728      256             0 systemd-network
2024-02-13T13:36:58.630780+01:00 pve kernel: [12367969.786862] [ 183879]   110 183879     2026       96    53248       64             0 rpcbind
2024-02-13T13:36:58.630781+01:00 pve kernel: [12367969.786864] [ 183882]   106 183882     6384       87    94208       14             0 systemd-resolve
2024-02-13T13:36:58.630781+01:00 pve kernel: [12367969.786866] [ 183886]     0 183886     1282       33    45056       32             0 blkmapd
2024-02-13T13:36:58.630782+01:00 pve kernel: [12367969.786868] [ 183887]     0 183887      812       33    40960       64             0 rpc.idmapd
2024-02-13T13:36:58.630783+01:00 pve kernel: [12367969.786870] [ 183888]     0 183888     1366       64    49152       64             0 nfsdcld
2024-02-13T13:36:58.630784+01:00 pve kernel: [12367969.786872] [ 183946]   100 183946     2113       32    49152       96          -900 dbus-daemon
2024-02-13T13:36:58.630785+01:00 pve kernel: [12367969.786874] [ 183951]     0 183951     7523      128    98304        0             0 networkd-dispat
2024-02-13T13:36:58.630785+01:00 pve kernel: [12367969.786876] [ 183953]   101 183953    55601        0    77824      224             0 rsyslogd
2024-02-13T13:36:58.630786+01:00 pve kernel: [12367969.786879] [ 183956]     0 183956     3836       32    69632      256             0 systemd-logind
2024-02-13T13:36:58.630787+01:00 pve kernel: [12367969.786881] [ 184028]     0 184028     1284       66    45056       64             0 rpc.mountd
2024-02-13T13:36:58.630788+01:00 pve kernel: [12367969.786883] [ 184027]   111 184027     1194       96    53248       64             0 rpc.statd
2024-02-13T13:36:58.630789+01:00 pve kernel: [12367969.786885] [ 184081]     0 184081      972       64    49152       32             0 cron
2024-02-13T13:36:58.630790+01:00 pve kernel: [12367969.786887] [ 207991]     0 207991     1830       64    53248      225             0 cron
2024-02-13T13:36:58.630790+01:00 pve kernel: [12367969.786888] [ 207992]     0 207992      723       32    45056        0             0 sh
2024-02-13T13:36:58.630791+01:00 pve kernel: [12367969.786890] [ 207993]     0 207993     4121       96    73728        0             0 script.py
2024-02-13T13:36:58.630792+01:00 pve kernel: [12367969.786892] [ 184089]     0 184089      705        0    40960       32             0 agetty
2024-02-13T13:36:58.630792+01:00 pve kernel: [12367969.786894] [ 184346]     0 184346     4170      128    69632      288             0 systemd
2024-02-13T13:36:58.630793+01:00 pve kernel: [12367969.786896] [ 184347]     0 184347     5352      128    77824        0             0 (sd-pam)
2024-02-13T13:36:58.630794+01:00 pve kernel: [12367969.786898] [ 184091]     0 184091     1916       32    49152      192             0 login
2024-02-13T13:36:58.630794+01:00 pve kernel: [12367969.786900] [ 184352]     0 184352     1379       96    49152      352             0 bash
2024-02-13T13:36:58.630795+01:00 pve kernel: [12367969.786901] [ 197512]     0 197512     4376       96    77824       13             0 script.py
2024-02-13T13:36:58.630796+01:00 pve kernel: [12367969.786903] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0,oom_memcg=/lxc/108,task_memcg=/lxc/108/ns/user.slice/user-0.slice/session-65982.scope,task=bash,pid=184352,uid=0
2024-02-13T13:36:58.630797+01:00 pve kernel: [12367969.786918] Memory cgroup out of memory: Killed process 184352 (bash) total-vm:5516kB, anon-rss:128kB, file-rss:256kB, shmem-rss:0kB, UID:0 pgtables:48kB oom_score_adj:0
2024-02-13T13:37:00.662622+01:00 pve kernel: [12367971.815607] oom_reaper: reaped process 184352 (bash), now anon-rss:128kB, file-rss:256kB, shmem-rss:0kB
2024-02-13T13:37:01.339207+01:00 pve pvestatd[1236]: status update time (30.827 seconds)
2024-02-13T13:37:01.446719+01:00 pve kernel: [12367972.599809] fwbr108i0: port 2(veth108i0) entered disabled state
2024-02-13T13:37:01.446727+01:00 pve kernel: [12367972.600342] device veth108i0 left promiscuous mode
2024-02-13T13:37:01.446728+01:00 pve kernel: [12367972.600345] fwbr108i0: port 2(veth108i0) entered disabled state
2024-02-13T13:37:01.698792+01:00 pve kernel: [12367972.853988] kauditd_printk_skb: 3 callbacks suppressed
2024-02-13T13:37:01.698802+01:00 pve kernel: [12367972.853990] audit: type=1400 audit(1707827821.691:616): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-108_</var/lib/lxc>" pid=213317 comm="apparmor_parser"
2024-02-13T13:37:01.752249+01:00 pve pvestatd[1236]: unable to get PID for CT 108 (not running?)

script.py ist dabei das Script welches läuft.
Wenn ich mir allerdings SWAP und RAM Auslastung sowohl des Hosts, des Gasts und des Zieles anschaue ist keines auch nur über 50%

Warum das Problem auftritt und wie ich es lösen könnte verstehe ich allerdings immer noch nicht.

UnBreakable · 13. Februar 2024

Ich habe nun überall Memory erhöht. Resultat ist, dass zwar der Container nicht mehr abstürzt, aber das Script gekillt wird.
Hier auch diese Meldung:

Bash:

2024-02-13T14:43:39.434700+01:00 pve kernel: [12371970.567115] script.py invoked oom-killer: gfp_mask=0x8cc0(GFP_KERNEL|__GFP_NOFAIL), order=0, oom_score_adj=0
2024-02-13T14:43:39.434716+01:00 pve kernel: [12371970.567120] CPU: 4 PID: 261372 Comm: script.py Tainted: P           O       6.2.16-12-pve #1
2024-02-13T14:43:39.434719+01:00 pve kernel: [12371970.567122] Hardware name: Dell Inc. OptiPlex 7050/0XHGV1, BIOS 1.20.0 03/23/2022
2024-02-13T14:43:39.434720+01:00 pve kernel: [12371970.567123] Call Trace:
2024-02-13T14:43:39.434721+01:00 pve kernel: [12371970.567124]  <TASK>
2024-02-13T14:43:39.434722+01:00 pve kernel: [12371970.567127]  dump_stack_lvl+0x48/0x70
2024-02-13T14:43:39.434723+01:00 pve kernel: [12371970.567131]  dump_stack+0x10/0x20
2024-02-13T14:43:39.434725+01:00 pve kernel: [12371970.567133]  dump_header+0x50/0x290
2024-02-13T14:43:39.434742+01:00 pve kernel: [12371970.567136]  oom_kill_process+0x10d/0x1c0
2024-02-13T14:43:39.434744+01:00 pve kernel: [12371970.567139]  out_of_memory+0x23c/0x570
2024-02-13T14:43:39.434745+01:00 pve kernel: [12371970.567141]  mem_cgroup_out_of_memory+0x145/0x170
2024-02-13T14:43:39.434746+01:00 pve kernel: [12371970.567144]  try_charge_memcg+0x731/0x820
2024-02-13T14:43:39.434747+01:00 pve kernel: [12371970.567148]  ? tcp_stream_alloc_skb+0x2d/0x150
2024-02-13T14:43:39.434748+01:00 pve kernel: [12371970.567150]  ? __kmalloc_node_track_caller+0x51/0xe0
2024-02-13T14:43:39.434750+01:00 pve kernel: [12371970.567153]  mem_cgroup_charge_skmem+0x40/0xf0
2024-02-13T14:43:39.434751+01:00 pve kernel: [12371970.567155]  sk_forced_mem_schedule+0xae/0x100
2024-02-13T14:43:39.434752+01:00 pve kernel: [12371970.567157]  tcp_stream_alloc_skb+0xb1/0x150
2024-02-13T14:43:39.434753+01:00 pve kernel: [12371970.567159]  tcp_sendmsg_locked+0x4b4/0xc90
2024-02-13T14:43:39.434754+01:00 pve kernel: [12371970.567165]  inet_sendmsg+0x42/0x80
2024-02-13T14:43:39.434755+01:00 pve kernel: [12371970.567167]  sock_sendmsg+0x66/0x70
2024-02-13T14:43:39.434755+01:00 pve kernel: [12371970.567171]  smb_send_kvec+0x84/0x190 [cifs]
2024-02-13T14:43:39.434756+01:00 pve kernel: [12371970.567213]  ? release_sock+0x8f/0xb0
2024-02-13T14:43:39.434757+01:00 pve kernel: [12371970.567216]  __smb_send_rqst+0x441/0x7b0 [cifs]
2024-02-13T14:43:39.434758+01:00 pve kernel: [12371970.567250]  ? ttwu_do_activate+0x7c/0x100
2024-02-13T14:43:39.434759+01:00 pve kernel: [12371970.567256]  smb_send_rqst+0x19b/0x200 [cifs]
2024-02-13T14:43:39.434760+01:00 pve kernel: [12371970.567293]  cifs_call_async+0x144/0x330 [cifs]
2024-02-13T14:43:39.434761+01:00 pve kernel: [12371970.567326]  ? __pfx_smb3_handle_read_data+0x10/0x10 [cifs]
2024-02-13T14:43:39.434762+01:00 pve kernel: [12371970.567363]  ? __pfx_smb2_readv_callback+0x10/0x10 [cifs]
2024-02-13T14:43:39.434763+01:00 pve kernel: [12371970.567399]  ? __pfx_cifs_readv_receive+0x10/0x10 [cifs]
2024-02-13T14:43:39.434764+01:00 pve kernel: [12371970.567434]  smb2_async_readv+0x1ef/0x3a0 [cifs]
2024-02-13T14:43:39.434765+01:00 pve kernel: [12371970.567472]  cifs_readahead+0x574/0xca0 [cifs]
2024-02-13T14:43:39.434766+01:00 pve kernel: [12371970.567504]  read_pages+0x95/0x2a0
2024-02-13T14:43:39.434767+01:00 pve kernel: [12371970.567507]  page_cache_ra_unbounded+0x135/0x1d0
2024-02-13T14:43:39.434767+01:00 pve kernel: [12371970.567510]  page_cache_ra_order+0x1cd/0x360
2024-02-13T14:43:39.434768+01:00 pve kernel: [12371970.567512]  ? __folio_mark_dirty+0x1c2/0x2d0
2024-02-13T14:43:39.434769+01:00 pve kernel: [12371970.567514]  ? xas_load+0x4c/0x60
2024-02-13T14:43:39.434770+01:00 pve kernel: [12371970.567516]  ondemand_readahead+0x1e7/0x4b0
2024-02-13T14:43:39.434771+01:00 pve kernel: [12371970.567519]  page_cache_sync_ra+0x8a/0xa0
2024-02-13T14:43:39.434772+01:00 pve kernel: [12371970.567521]  filemap_get_pages+0x10e/0x5d0
2024-02-13T14:43:39.434773+01:00 pve kernel: [12371970.567523]  ? cifs_write_end+0xec/0x490 [cifs]
2024-02-13T14:43:39.434774+01:00 pve kernel: [12371970.567555]  filemap_read+0xe4/0x410
2024-02-13T14:43:39.434784+01:00 pve kernel: [12371970.567560]  generic_file_read_iter+0xe1/0x170
2024-02-13T14:43:39.434785+01:00 pve kernel: [12371970.567562]  cifs_strict_readv+0x164/0x180 [cifs]
2024-02-13T14:43:39.434786+01:00 pve kernel: [12371970.567594]  generic_file_splice_read+0xda/0x1b0
2024-02-13T14:43:39.434787+01:00 pve kernel: [12371970.567598]  do_splice_to+0x7d/0xe0
2024-02-13T14:43:39.434788+01:00 pve kernel: [12371970.567600]  splice_direct_to_actor+0xbd/0x230
2024-02-13T14:43:39.434789+01:00 pve kernel: [12371970.567602]  ? __pfx_direct_splice_actor+0x10/0x10
2024-02-13T14:43:39.434790+01:00 pve kernel: [12371970.567605]  do_splice_direct+0x8a/0xe0
2024-02-13T14:43:39.434791+01:00 pve kernel: [12371970.567607]  do_sendfile+0x376/0x510
2024-02-13T14:43:39.434792+01:00 pve kernel: [12371970.567610]  __x64_sys_sendfile64+0x69/0xd0
2024-02-13T14:43:39.434793+01:00 pve kernel: [12371970.567612]  do_syscall_64+0x58/0x90
2024-02-13T14:43:39.434793+01:00 pve kernel: [12371970.567615]  ? irqentry_exit_to_user_mode+0x9/0x20
2024-02-13T14:43:39.434794+01:00 pve kernel: [12371970.567617]  ? irqentry_exit+0x43/0x50
2024-02-13T14:43:39.434795+01:00 pve kernel: [12371970.567619]  ? exc_page_fault+0x91/0x1b0
2024-02-13T14:43:39.434796+01:00 pve kernel: [12371970.567621]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
2024-02-13T14:43:39.434797+01:00 pve kernel: [12371970.567623] RIP: 0033:0x7efdf9b1917e
2024-02-13T14:43:39.434798+01:00 pve kernel: [12371970.567627] Code: Unable to access opcode bytes at 0x7efdf9b19154.
2024-02-13T14:43:39.434799+01:00 pve kernel: [12371970.567627] RSP: 002b:00007ffcdc824ea8 EFLAGS: 00000206 ORIG_RAX: 0000000000000028
2024-02-13T14:43:39.434800+01:00 pve kernel: [12371970.567630] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007efdf9b1917e
2024-02-13T14:43:39.434801+01:00 pve kernel: [12371970.567631] RDX: 00007ffcdc824eb8 RSI: 0000000000000003 RDI: 0000000000000004
2024-02-13T14:43:39.434802+01:00 pve kernel: [12371970.567632] RBP: 0000560b876fd6d0 R08: 0000000000000000 R09: 0000000000000000
2024-02-13T14:43:39.434803+01:00 pve kernel: [12371970.567633] R10: 0000001258ff4000 R11: 0000000000000206 R12: 0000000000000003
2024-02-13T14:43:39.434804+01:00 pve kernel: [12371970.567634] R13: 00007ffcdc824eb8 R14: 0000001258ff4000 R15: 0000560b876033d0
2024-02-13T14:43:39.434805+01:00 pve kernel: [12371970.567637]  </TASK>
2024-02-13T14:43:39.434805+01:00 pve kernel: [12371970.567638] memory: usage 262144kB, limit 262144kB, failcnt 38463
2024-02-13T14:43:39.434806+01:00 pve kernel: [12371970.567639] swap: usage 26876kB, limit 262144kB, failcnt 0
2024-02-13T14:43:39.434807+01:00 pve kernel: [12371970.567640] Memory cgroup stats for /lxc/108:
2024-02-13T14:43:39.434808+01:00 pve kernel: [12371970.567684] anon 0
2024-02-13T14:43:39.434809+01:00 pve kernel: [12371970.567684] file 256684032
2024-02-13T14:43:39.434810+01:00 pve kernel: [12371970.567684] kernel 11747328
2024-02-13T14:43:39.434811+01:00 pve kernel: [12371970.567684] kernel_stack 425984
2024-02-13T14:43:39.434812+01:00 pve kernel: [12371970.567684] pagetables 1261568
2024-02-13T14:43:39.434813+01:00 pve kernel: [12371970.567684] sec_pagetables 0
2024-02-13T14:43:39.434814+01:00 pve kernel: [12371970.567684] percpu 700440
2024-02-13T14:43:39.434815+01:00 pve kernel: [12371970.567684] sock 4096
2024-02-13T14:43:39.434815+01:00 pve kernel: [12371970.567684] vmalloc 487424
2024-02-13T14:43:39.434816+01:00 pve kernel: [12371970.567684] shmem 0
2024-02-13T14:43:39.434817+01:00 pve kernel: [12371970.567684] zswap 0
2024-02-13T14:43:39.434818+01:00 pve kernel: [12371970.567684] zswapped 0
2024-02-13T14:43:39.434819+01:00 pve kernel: [12371970.567684] file_mapped 0
2024-02-13T14:43:39.434820+01:00 pve kernel: [12371970.567684] file_dirty 655360
2024-02-13T14:43:39.434821+01:00 pve kernel: [12371970.567684] file_writeback 255520768
2024-02-13T14:43:39.434821+01:00 pve kernel: [12371970.567684] swapcached 0
2024-02-13T14:43:39.434822+01:00 pve kernel: [12371970.567684] anon_thp 0
2024-02-13T14:43:39.434823+01:00 pve kernel: [12371970.567684] file_thp 0
2024-02-13T14:43:39.434824+01:00 pve kernel: [12371970.567684] shmem_thp 0
2024-02-13T14:43:39.434825+01:00 pve kernel: [12371970.567684] inactive_anon 0
2024-02-13T14:43:39.434841+01:00 pve kernel: [12371970.567684] active_anon 0
2024-02-13T14:43:39.434842+01:00 pve kernel: [12371970.567684] inactive_file 128405504
2024-02-13T14:43:39.434843+01:00 pve kernel: [12371970.567684] active_file 128278528
2024-02-13T14:43:39.434844+01:00 pve kernel: [12371970.567684] unevictable 0
2024-02-13T14:43:39.434845+01:00 pve kernel: [12371970.567684] slab_reclaimable 6167328
2024-02-13T14:43:39.434846+01:00 pve kernel: [12371970.567684] slab_unreclaimable 2125488
2024-02-13T14:43:39.434847+01:00 pve kernel: [12371970.567684] slab 8292816
2024-02-13T14:43:39.434848+01:00 pve kernel: [12371970.567687] Tasks state (memory values in pages):
2024-02-13T14:43:39.434849+01:00 pve kernel: [12371970.567688] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-02-13T14:43:39.434850+01:00 pve kernel: [12371970.567689] [ 254983]     0 254983    41454      192    90112      736             0 systemd
2024-02-13T14:43:39.434851+01:00 pve kernel: [12371970.567693] [ 255298]     0 255298      705       64    45056        0             0 agetty
2024-02-13T14:43:39.434852+01:00 pve kernel: [12371970.567696] [ 255401]     0 255401    10308      100    73728      128             0 master
2024-02-13T14:43:39.434853+01:00 pve kernel: [12371970.567698] [ 255402]   102 255402    10389      128    69632      128             0 pickup
2024-02-13T14:43:39.434853+01:00 pve kernel: [12371970.567700] [ 255403]   102 255403    10400      128    69632       96             0 qmgr
2024-02-13T14:43:39.434854+01:00 pve kernel: [12371970.567702] [ 255081]     0 255081     7834      128    86016      224          -250 systemd-journal
2024-02-13T14:43:39.434855+01:00 pve kernel: [12371970.567705] [ 255121]   105 255121     4031      128    69632      224             0 systemd-network
2024-02-13T14:43:39.434857+01:00 pve kernel: [12371970.567707] [ 255131]   110 255131     2026       64    57344       64             0 rpcbind
2024-02-13T14:43:39.434858+01:00 pve kernel: [12371970.567709] [ 255136]     0 255136     1282       65    49152       32             0 blkmapd
2024-02-13T14:43:39.434859+01:00 pve kernel: [12371970.567711] [ 255137]     0 255137      812       97    45056       32             0 rpc.idmapd
2024-02-13T14:43:39.434860+01:00 pve kernel: [12371970.567713] [ 255138]     0 255138     1366       96    49152       64             0 nfsdcld
2024-02-13T14:43:39.434861+01:00 pve kernel: [12371970.567715] [ 255181]   100 255181     2113       96    49152       96          -900 dbus-daemon
2024-02-13T14:43:39.434862+01:00 pve kernel: [12371970.567718] [ 255188]   101 255188    55601        0    86016      224             0 rsyslogd
2024-02-13T14:43:39.434862+01:00 pve kernel: [12371970.567720] [ 255191]     0 255191     3836       96    69632      256             0 systemd-logind
2024-02-13T14:43:39.434863+01:00 pve kernel: [12371970.567723] [ 255233]     0 255233     1284      130    49152       32             0 rpc.mountd
2024-02-13T14:43:39.434865+01:00 pve kernel: [12371970.567725] [ 255232]   111 255232     1194      135    53248       32             0 rpc.statd
2024-02-13T14:43:39.434866+01:00 pve kernel: [12371970.567727] [ 255288]     0 255288      972      128    45056       32             0 cron
2024-02-13T14:43:39.434867+01:00 pve kernel: [12371970.567729] [ 255296]     0 255296      705       64    45056        0             0 agetty
2024-02-13T14:43:39.434867+01:00 pve kernel: [12371970.567731] [ 260552]   106 260552     6384      153    90112      992             0 systemd-resolve
2024-02-13T14:43:39.434868+01:00 pve kernel: [12371970.567733] [ 255620]     0 255620     4170      128    69632      288             0 systemd
2024-02-13T14:43:39.434869+01:00 pve kernel: [12371970.567735] [ 255621]     0 255621     5351       99    77824      800             0 (sd-pam)
2024-02-13T14:43:39.434870+01:00 pve kernel: [12371970.567737] [ 255297]     0 255297     1916      128    57344      160             0 login
2024-02-13T14:43:39.434871+01:00 pve kernel: [12371970.567739] [ 255626]     0 255626     1379      128    49152      320             0 bash
2024-02-13T14:43:39.434871+01:00 pve kernel: [12371970.567741] [ 261372]     0 261372     4376      160    73728     1056             0 script.py
2024-02-13T14:43:39.434872+01:00 pve kernel: [12371970.567743] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0,oom_memcg=/lxc/108,task_memcg=/lxc/108/ns/user.slice/user-0.slice/session-66027.scope,task=script.py,pid=261372,uid=0
2024-02-13T14:43:39.434873+01:00 pve kernel: [12371970.567754] Memory cgroup out of memory: Killed process 261372 (script.py) total-vm:17504kB, anon-rss:128kB, file-rss:512kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:0

Erstmal verstehe ich nicht, wieso es was hilft Memory zu erhöhen, obwohl nicht alles genutzt wurde.
Nun verstehe ich aber auch nicht, wieso das Script gekillt wird, obwohl nicht alles genutzt wird.
Auch google hilft mir nicht weiter. Das Problem scheinen zwar mehrere zu haben, aber eine Lösung habe ich nicht gefunden.

EDIT: Ok, jetzt ist der Container wieder abgestürzt, die Erhöhung des Arbeitsspeichers hat also doch nichts gebracht.

riversource · 13. Februar 2024

UnBreakable schrieb:
Ich habe nun überall Memory erhöht.

Im zweiten Fall steht dem Prozess aber weniger Speicher zur Verfügung, als im ersten Fall. Halb so viel, um präzise zu sein.

Wie dem auch sei, der OOM (Out of Memory) Killer schlägt zu. Wenn die ganze VM gekillt wird, könnte es auch sein, dass auch der OOM Killer des Host-Systems zuschlägt.
Ich vermute, dein Script hat ein Speicherleck oder missachtet das Memory Management. Behalte doch mal die Entwicklung der Speichernutzung während der Nutzung des Scripts mit htop oder btop im Auge. Oder zeige uns das Script.

frabron · 13. Februar 2024

UnBreakable schrieb:
2024-02-13T14:43:39.434805+01:00 pve kernel: [12371970.567638] memory: usage 262144kB, limit 262144kB, failcnt 38463 2024-02-13T14:43:39.434806+01:00 pve kernel: [12371970.567639] swap: usage 26876kB, limit 262144kB, failcnt 0

Jo, Zeile 65/66 zum Beispiel

Ubuntu LXC einfach aus

Lt. Junior Grade

Lieutenant Pro

Captain

Lt. Junior Grade

Captain

Lt. Junior Grade

Captain

Lt. Junior Grade

Captain

Lt. Junior Grade

Captain

Lt. Junior Grade

Fleet Admiral

Captain

Captain

Lt. Junior Grade

Lt. Junior Grade

Lt. Junior Grade

Captain

Ensign Pro

Ähnliche Themen

Passend zum Thema