This is my first message - hello everyone.
I have encountered a problem and I have no clue what caused it or how to prevent it happening. Let me describe the situation in details.
Given: Dell 3020 box, 16GB RAM, 250GB SSD. Running Proxmox 8.1 without subscribtion. Under Proxmox is a single HAOS VM with 4GB RAM and 60GB drive space. HAOS was installed fresh, using great step-by-step manual.
After installing HAOS I addeed 4 USB devices as pass-thru: built-in Bluetooth (pass-thru as a device ID), ZooZ 800 series stick, Sonoff Zigbee stick, SD-RTL stick (pass-thru as ports). Did Setup of all sensors/devices/dashboards. The whole work took almost a week (close to 60 devices across all these 4 tech/radios) but everything worked - all USB devices working without a problem on HAOS VM.
I also have habit to take backup and snapshot every single time I added new stick and configured devices, so 3 weeks ago - when I finished everything, I took final backup and snapshot.
I subscribed to Nabu Casa for remote access of HAOS instance: this setup is located at a remote property, it is behind CGNAT and I don't have means to access Proxmox GUI/command line remotely. After confirming that remote access worked, that I left everything working and it continued working for 10 days - I saw timestamp updates from devices and got notifications on my HA app as well.
Couple of days ago, when logged on to the system remotely, I see that all devices from Zigbee, Z-wave and SD-RTL are 'unavailable' in the dashboard. All these three add-ons are not starting. Starting/restarting them manually did not work. Reboot of the HAOS remotely did not work either - I saw in supervisor logs that 'can't start add-on'. I did not try to remove/add add-on. Also when going to system->hardware I saw that all three USB devices have disappeared from hardware list.
Yesterday I managed to go back to the site. I rebooted the computer - proxmox and HAOS. This did not fix the situation.
I had very limited time, so I decided to restore HAOS VM from the snapshot. It bright the state back from 2 weeks ago. Obviously I lost all the data since the snapshot (and need to spend several hours to add missing/updated dashboards), however it brought all the devices from all three radios back and everything is working as it was 2 weeks ago.
So... I have no idea what happened and why.
Here is list of my questions after spending several hours of researching:
I have encountered a problem and I have no clue what caused it or how to prevent it happening. Let me describe the situation in details.
Given: Dell 3020 box, 16GB RAM, 250GB SSD. Running Proxmox 8.1 without subscribtion. Under Proxmox is a single HAOS VM with 4GB RAM and 60GB drive space. HAOS was installed fresh, using great step-by-step manual.
After installing HAOS I addeed 4 USB devices as pass-thru: built-in Bluetooth (pass-thru as a device ID), ZooZ 800 series stick, Sonoff Zigbee stick, SD-RTL stick (pass-thru as ports). Did Setup of all sensors/devices/dashboards. The whole work took almost a week (close to 60 devices across all these 4 tech/radios) but everything worked - all USB devices working without a problem on HAOS VM.
I also have habit to take backup and snapshot every single time I added new stick and configured devices, so 3 weeks ago - when I finished everything, I took final backup and snapshot.
I subscribed to Nabu Casa for remote access of HAOS instance: this setup is located at a remote property, it is behind CGNAT and I don't have means to access Proxmox GUI/command line remotely. After confirming that remote access worked, that I left everything working and it continued working for 10 days - I saw timestamp updates from devices and got notifications on my HA app as well.
Couple of days ago, when logged on to the system remotely, I see that all devices from Zigbee, Z-wave and SD-RTL are 'unavailable' in the dashboard. All these three add-ons are not starting. Starting/restarting them manually did not work. Reboot of the HAOS remotely did not work either - I saw in supervisor logs that 'can't start add-on'. I did not try to remove/add add-on. Also when going to system->hardware I saw that all three USB devices have disappeared from hardware list.
Yesterday I managed to go back to the site. I rebooted the computer - proxmox and HAOS. This did not fix the situation.
I had very limited time, so I decided to restore HAOS VM from the snapshot. It bright the state back from 2 weeks ago. Obviously I lost all the data since the snapshot (and need to spend several hours to add missing/updated dashboards), however it brought all the devices from all three radios back and everything is working as it was 2 weeks ago.
So... I have no idea what happened and why.
Here is list of my questions after spending several hours of researching:
- Have you seen anything similar happening with Proxmox/HAOS pair in past?
- How to prevent this happening again? I currently added rebooting host at 2AM but not sure it will help. I am also planning to add backup on OneDrive, but again, not sure how it will help...
- After reading number of threads, there were mentioning that daily periodic updates might disable pass-thru of USB devices. If this is the case, how do I disable all updates/checks?
- What was interesting, that only devices that were configured as pass-thru via ports disappeared, but the bluetooth device that is part of Wifi/blutooth M2 card - did not.
- As mentioned, I don't hace access to Proxmox directly remotely, but I can ssh to it via terminal from HAOS. Is there a way to do reset/re-scanning of USB devices via command line so they disappear/re-appear within HAOS?
- Where within Proxmox logs can I dig to understand what is happeneing?
- Anything else?