Cannot open /proc/stat: Transport endpoint is not connected

[SPQRInc]

This is great news for all users.

But why aren't much users affected? I counted only 2 in this forum.

 

[windinternet]

It is contingent upon a realloc failing. You have to catch the process at a moment it is just somehow not able to quickly allocate a new memory page that is large enough.

This probably happens mostly with containers that have very large tasklists and a lot of systemd activity with cgroups. Perhaps some service that is employing cgroup limiting within the container? It surprises me also quite a bit that you should encounter it with a system that apparently has lots of available RAM memory. But if the realloc fix in lxcfs solves it, then apparently that is happening.

 

[wbumiller]

It's not about allocating it quickly enough, but realizing that a new batch has to be allocated at all. The fact only 2 users were affected puzzles me. On the other hand if you confine programs into containers you typically have multiple smaller containers rather than fewer big ones so reaching the number of processes to cause it probably isn't very likely in practice. (Though that's only one way to cause it. I suppose for most other /proc files the initial buffer is large enough anyway.)

 

[hregis]

i have a new error message today :

an 11 11:25:02 ns338919 lxcfs[21524]: lxcfs: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.

Jan 11 11:25:02 ns338919 systemd[1]: lxcfs.service: main process exited, code=killed, status=6/ABRT

Jan 11 11:25:02 ns338919 systemd[1]: Unit lxcfs.service entered failed state.

Jan 11 11:25:03 ns338919 systemd[1]: lxcfs.service holdoff time over, scheduling restart.

Jan 11 11:25:03 ns338919 systemd[1]: Stopping FUSE filesystem for LXC...

Jan 11 11:25:03 ns338919 systemd[1]: Starting FUSE filesystem for LXC...

Jan 11 11:25:03 ns338919 systemd[1]: Started FUSE filesystem for LXC.

Jan 11 11:25:03 ns338919 lxcfs[26988]: hierarchies: 1: name=systemd

Jan 11 11:25:03 ns338919 lxcfs[26988]: 2: cpuset

Jan 11 11:25:03 ns338919 lxcfs[26988]: 3: cpu,cpuacct

Jan 11 11:25:03 ns338919 lxcfs[26988]: 4: blkio

Jan 11 11:25:03 ns338919 lxcfs[26988]: 5: memory

Jan 11 11:25:03 ns338919 lxcfs[26988]: 6: devices

Jan 11 11:25:03 ns338919 lxcfs[26988]: 7: freezer

Jan 11 11:25:03 ns338919 lxcfs[26988]: 8: net_cls,net_prio

Jan 11 11:25:03 ns338919 lxcfs[26988]: 9: perf_event

Jan 11 11:25:03 ns338919 lxcfs[26988]: 10: hugetlb

 

[wbumiller]

@hregis: With which lxcfs package version?

 

[hregis]

@wbumiller

Version : 0.13-pve2

 

[wbumiller]

The fixes are in lxcfs >= 0.13-pve3 so try updating
Note that the running containers will still be hooked up to the older running process so you might want to restart them if you can after the upgrade.

 

[hregis]

i try with entreprise repo but nothing !

 

[hregis]

@wbumiller
where is this update ?

 

[tom]

@wbumiller
where is this update ?

currently in the pvetest and the pve-no-subscription repository.

 

[hregis]

thank you @tom

 

[Dura]

I still have this issue on proxmox 5 today, no idea why, same error when trying to open htop (and top doesn't work either)

 

[mstgeo]

Hi Guys,

I found this thread and it seems I started to have a similiar problem on a Proxmox v6 with LXC.

After some time after run of a core server We're getting in VM while top / htop the message:

Cannot open /proc/stat: Transport endpoint is not connected

A stop / start for VM helps for a short time and a problem comes back...

The VM is debian 10 while the Proxmox is based on:

Linux server 5.4.174-2-pve #1 SMP PVE 5.4.174-2 (Thu, 10 Mar 2022 15:58:44 +0100) x86_64 GNU/Linux

Anyone has any ideas what could it be and how to get rid of this problem.

Thanks in advance !

regards,
G.