I just did an in-place upgrade to 7.0.10 using the guide and it went very smoothly indeed. Great job!
One change/anomaly I noticed after upgrading PVE:
A container running a single instance of elastic stack worked fine under pve-6.4. After upgrading PVE, the elastic process (java) in that container gets consistently oom killed immediately on startup. I retested with more memory made available to the container, same result.
I then set an explicit java heap size in jvm.options for elasticsearch, and the process ran successfully. This had not been necessary under pve-6.4 and the available memory had, presumably, been correctly detected within the container and java heap size set accordingly.
One change/anomaly I noticed after upgrading PVE:
A container running a single instance of elastic stack worked fine under pve-6.4. After upgrading PVE, the elastic process (java) in that container gets consistently oom killed immediately on startup. I retested with more memory made available to the container, same result.
I then set an explicit java heap size in jvm.options for elasticsearch, and the process ran successfully. This had not been necessary under pve-6.4 and the available memory had, presumably, been correctly detected within the container and java heap size set accordingly.
Code:
#/etc/pve/lxc/108.conf
arch: amd64
cores: 2
hostname: elk
memory: 8192
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.0.1,hwaddr=8A:2A:4F:C3:64:57,ip=192.168.0.8/24,tag=10,type=veth
onboot: 1
ostype: ubuntu
rootfs: vmdisks:vm-108-disk-0,mountoptions=noatime,size=32G
startup: order=1,up=60
swap: 0
unprivileged: 1
Code:
Aug 07 11:51:16 pve kernel: GC Thread#1 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Aug 07 11:51:16 pve kernel: CPU: 0 PID: 51497 Comm: GC Thread#1 Tainted: P O 5.11.22-3-pve #1
Aug 07 11:51:16 pve kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z77X-UD5H, BIOS F16j 11/14/2017
Aug 07 11:51:16 pve kernel: Call Trace:
Aug 07 11:51:16 pve kernel: dump_stack+0x70/0x8b
Aug 07 11:51:16 pve kernel: dump_header+0x4f/0x1f6
Aug 07 11:51:16 pve kernel: oom_kill_process.cold+0xb/0x10
Aug 07 11:51:16 pve kernel: out_of_memory+0x1cf/0x520
Aug 07 11:51:16 pve kernel: mem_cgroup_out_of_memory+0x139/0x150
Aug 07 11:51:16 pve kernel: try_charge+0x750/0x7b0
Aug 07 11:51:16 pve kernel: mem_cgroup_charge+0x8a/0x280
Aug 07 11:51:16 pve kernel: handle_mm_fault+0xf9e/0x1a70
Aug 07 11:51:16 pve kernel: do_user_addr_fault+0x1a3/0x450
Aug 07 11:51:16 pve kernel: ? exit_to_user_mode_prepare+0x37/0x190
Aug 07 11:51:16 pve kernel: exc_page_fault+0x6c/0x150
Aug 07 11:51:16 pve kernel: ? asm_exc_page_fault+0x8/0x30
Aug 07 11:51:16 pve kernel: asm_exc_page_fault+0x1e/0x30
Aug 07 11:51:16 pve kernel: RIP: 0033:0x7f9b89928a40
Aug 07 11:51:16 pve kernel: Code: Unable to access opcode bytes at RIP 0x7f9b89928a16.
Aug 07 11:51:16 pve kernel: RSP: 002b:00007f9b44deddf0 EFLAGS: 00010206
Aug 07 11:51:16 pve kernel: RAX: 0000000426800000 RBX: 00007f9b8b13d848 RCX: 0000000000000000
Aug 07 11:51:16 pve kernel: RDX: 0000000000001000 RSI: 00000005e4c00000 RDI: 00000005e4ad0000
Aug 07 11:51:16 pve kernel: RBP: 00007f9b44deddf0 R08: 0000000000000000 R09: 00000000ffffffff
Aug 07 11:51:16 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007f9b8b13d830
Aug 07 11:51:16 pve kernel: R13: 00007f9b44dede2c R14: 00000000ffffffff R15: 00007f9b84079480
Aug 07 11:51:16 pve kernel: memory: usage 8388608kB, limit 8388608kB, failcnt 56081
Aug 07 11:51:16 pve kernel: swap: usage 0kB, limit 0kB, failcnt 0
Aug 07 11:51:16 pve kernel: Memory cgroup stats for /lxc/108:
Aug 07 11:51:16 pve kernel: anon 8550727680
file 405504
kernel_stack 2113536
pagetables 28790784
percpu 0
sock 0
shmem 0
file_mapped 135168
file_dirty 0
file_writeback 0
anon_thp 0
file_thp 0
shmem_thp 0
inactive_anon 8550862848
active_anon 0
inactive_file 57344
active_file 40960
unevictable 0
slab_reclaimable 2687352
slab_unreclaimable 3804976
slab 6492328
workingset_refault_anon 0
workingset_refault_file 46959
workingset_activate_anon 0
workingset_activate_file 0
workingset_restore_anon 0
workingset_restore_file 0
workingset_nodereclaim 0
pgfault 7408302
pgmajfault 5478
pgrefill 121121
pgscan 268608
pgsteal 183389
pgactivate 106953
pgdeactivate 106351
pglazyfree 0
pglazyfreed 0
thp_fault_alloc 0
thp_collapse_alloc 0
Aug 07 11:51:16 pve kernel: Tasks state (memory values in pages):
Aug 07 11:51:16 pve kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Aug 07 11:51:16 pve kernel: [ 44826] 100000 44826 56192 454 208896 0 0 systemd
Aug 07 11:51:16 pve kernel: [ 45104] 100000 45104 3258 33 69632 0 0 agetty
Aug 07 11:51:16 pve kernel: [ 45106] 100000 45106 3258 33 73728 0 0 agetty
Aug 07 11:51:16 pve kernel: [ 45299] 100000 45299 16847 122 114688 0 0 master
Aug 07 11:51:16 pve kernel: [ 45301] 100104 45301 18455 122 131072 0 0 pickup
Aug 07 11:51:16 pve kernel: [ 45302] 100104 45302 18467 123 131072 0 0 qmgr
Aug 07 11:51:16 pve kernel: [ 44988] 100000 44988 35829 147 294912 0 0 systemd-journal
Aug 07 11:51:16 pve kernel: [ 44998] 100101 44998 17931 127 172032 0 0 systemd-network
Aug 07 11:51:16 pve kernel: [ 45078] 100102 45078 17625 116 184320 0 0 systemd-resolve
Aug 07 11:51:16 pve kernel: [ 45081] 100100 45081 47834 138 143360 0 0 rsyslogd
Aug 07 11:51:16 pve kernel: [ 45082] 100000 45082 17610 153 180224 0 0 systemd-logind
Aug 07 11:51:16 pve kernel: [ 45083] 100000 45083 7092 67 98304 0 0 cron
Aug 07 11:51:16 pve kernel: [ 45084] 100000 45084 41864 1934 229376 0 0 networkd-dispat
Aug 07 11:51:16 pve kernel: [ 45085] 100103 45085 12515 150 143360 0 0 dbus-daemon
Aug 07 11:51:16 pve kernel: [ 45088] 100000 45088 71162 210 192512 0 0 accounts-daemon
Aug 07 11:51:16 pve kernel: [ 45126] 100000 45126 18078 181 184320 0 0 sshd
Aug 07 11:51:16 pve kernel: [ 45091] 100109 45091 154165 7542 1495040 0 0 node
Aug 07 11:51:16 pve kernel: [ 45467] 100109 45467 282791 57739 7553024 0 0 node
Aug 07 11:51:16 pve kernel: [ 45093] 100000 45093 46076 1971 266240 0 0 unattended-upgr
Aug 07 11:51:16 pve kernel: [ 45105] 100000 45105 3258 34 69632 0 0 agetty
Aug 07 11:51:16 pve kernel: [ 45728] 100999 45728 1065615 180577 2146304 0 0 java
Aug 07 11:51:16 pve kernel: [ 46002] 100000 46002 19101 204 180224 0 0 systemd
Aug 07 11:51:16 pve kernel: [ 46003] 100000 46003 27243 461 245760 0 0 (sd-pam)
Aug 07 11:51:16 pve kernel: [ 45999] 100000 45999 26999 255 249856 0 0 sshd
Aug 07 11:51:16 pve kernel: [ 46033] 100000 46033 4922 388 77824 0 0 bash
Aug 07 11:51:16 pve kernel: [ 51314] 100000 51314 15463 100 155648 0 0 systemctl
Aug 07 11:51:16 pve kernel: [ 51315] 100000 51315 15425 98 163840 0 0 systemd-tty-ask
Aug 07 11:51:16 pve kernel: [ 51316] 100108 51316 4392044 1832139 14905344 0 0 java
Aug 07 11:51:16 pve kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0,oom_memcg=/lxc/108,task_memcg=/lxc/108/ns,task=java,pid=51316,uid=100108
Aug 07 11:51:16 pve kernel: Memory cgroup out of memory: Killed process 51316 (java) total-vm:17568176kB, anon-rss:7328556kB, file-rss:0kB, shmem-rss:0kB, UID:100108 pgtables:14556kB oom_score_adj:0
Last edited:
