Spontaneous reboot

[windinternet]

Hello,

We are having problems with spontaneous reboots on version 4 stable on supermicro hardware with ZFS mirror (Installed with installer from ISO cd). Version 4 proxmox with 4.2.2-1-pve and 4.2.3-2-pve kernel both. Same kinds of installs on 3.4 and 4 stable on mdraid do not show this behaviour. There is usually nothing on the console or in the syslog at the time of crash, it just blinks off and starts rebooting, although onetime we were lucky to catch some stacktraces of hung_tasks 15 minutes before seemingly pointing to problems offloading pages to swap, because they contained: _swap_writepage. Also, the servers do not reboot until there is at least some usage of the swap file.

zpool history
 zfs create -V 32505856K -b 4K rpool/swap

We have one server running now with:

swapoff -a

and it has not crashed since.

 

[t.lamprecht]

Hi,

swap on ZFS is not really recommended, AFAIK, and i saw already some problems in the wild but I'm not still clear what causes the problems.

You could try the following:
* set the ZVOL block size to match the system page size, which can be obtained by the getconf PAGESIZE command (default on x86_64 is 4KiB). (this must be done)
* disable caching the swap ZVOL data.

 

[t.lamprecht]

See also this bug report, it could have some helpful infos https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199189

 

[dietmar]

Also, please update your system, and test with latest packages/kernel.

 

[windinternet]

Hello Thomas,

The system is running with the default ZFS settings made by the installer. The rpool/swap has volblocksize=4K which matches the system page size. The total rpool has caching turned off.

The freebsd bug report may be helpful. Maybe we can find some comparable sysctl setting.

Candidates', I think:

vm.min_free_kbytes
vm.user_reserve_kbytes

 

[t.lamprecht]

Also turn the swappiness down, if you do not over commit ram you may make it really small towards 1 or even zero, at first try it with 10.
Swappiness of 0 results in swap only to avoid an out of memory condition.

sysctl -w vm.swappiness=10

 

[windinternet]

Also turn the swappiness down, if you do not over commit ram you may make it really small towards 1 or even zero, at first try it with 10.
Swappiness of 0 results in swap only to avoid an out of memory condition.

sysctl -w vm.swappiness=10

Yes, it is clear that avoiding swapping will avoid the reboot.

If the poster Marcus Reid on the freebsd bug is right, then vm.min_free_kbytes may avoid the problem altogether even while swapping. I notice that it currently is only about one third of default Ubuntu server settings. Can't test this in front of the weekend though. At least with swap off it doesn't crash and reboot.

BTW, Is there any reason to believe that pve-kernel-4.2.6-1-pve solves this?


Regards,
Gerrit

 

[windinternet]

I can report some progress on this issue.

I have been able to reproduce the sudden reboots on a Virtualbox with simulated Sata controller and 2 disks in a ZFS raid 0 configuration installed with the 4.0 installer and updated to the 4.2.6-1-pve kernel and other updates with the no subscription repository.

To reproduce I first make a tar of the filesystem and then proceed to untar it inside a subdir multiple times. After a bit of swapping the machine reboots suddenly.

I also tried the same with a proxmox 3.4 install, but cannot succeed in reproducing the reboot, nor did we ever see it in a production machine running on that version. ZFS settings of the rpool/swap ZVOL all look the same.

After this I tried tuning vm.min_free_kbytes. But increasing this setting does not prevent the reboots at all.

Finally I changed the settings on the swap ZVOL. I turned of all additional functions like compression, caching, sync and checksum. This does have a notable effect, especially it seems setting zfs set sync=disabled rpool/swap. After doing this, my simple untarring does not cause a sudden reboot anymore. I can still get a spontaneous reboot or a kernel hung_tasks message, but only by starting 8+ untars concurrently.