command IOTOP = ksoftirqd/0 99%

I change few versions of kernel to find last good version.

This versions HAVE problem with ksoftirqd 99% IO
Code:
pve-kernel-4.13.13-1-pve/stable,now 4.13.13-31 amd64 [installed,automatic]
  The Proxmox PVE Kernel Image

pve-kernel-4.13.3-1-pve/stable,now 4.13.3-2 amd64 [installed]
  The Proxmox PVE Kernel Image

pve-kernel-4.13.4-1-pve/stable,now 4.13.4-26 amd64 [installed]
  The Proxmox PVE Kernel Image


Last version WITHOUT problem: pve-kernel-4.10.17-5-pve

My stand now WITHOUT this problem:
Code:
# pveversion -v
proxmox-ve: 5.1-31 (running kernel: 4.10.17-5-pve)
pve-manager: 5.1-40 (running version: 5.1-40/ea05b379)
pve-kernel-4.13.3-1-pve: 4.13.3-2
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.10.17-5-pve: 4.10.17-25
pve-kernel-4.13.13-1-pve: 4.13.13-31
libpve-http-server-perl: 2.0-8
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-19
qemu-server: 5.0-18
pve-firmware: 2.0-3
libpve-common-perl: 5.0-25
libpve-guest-common-perl: 2.0-14
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-17
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-3
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-5
pve-container: 2.0-18
pve-firewall: 3.0-5
pve-ha-manager: 2.0-4
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.1-2
lxcfs: 2.0.8-1
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9

Somebody can tell me, can I use pve-kernel-4.10.17-5-pve with proxmox-ve: 5.1-31?

As I see in page https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.0, proxmox 5.1. working with pve-kernel-4.13.
 
Last edited:
I change few versions of kernel to find last good version.

This versions HAVE problem with ksoftirqd 99% IO
Code:
pve-kernel-4.13.13-1-pve/stable,now 4.13.13-31 amd64 [installed,automatic]
  The Proxmox PVE Kernel Image

pve-kernel-4.13.3-1-pve/stable,now 4.13.3-2 amd64 [installed]
  The Proxmox PVE Kernel Image

pve-kernel-4.13.4-1-pve/stable,now 4.13.4-26 amd64 [installed]
  The Proxmox PVE Kernel Image


Last version WITHOUT problem: pve-kernel-4.10.17-5-pve

My stand now WITHOUT this problem:
Code:
# pveversion -v
proxmox-ve: 5.1-31 (running kernel: 4.10.17-5-pve)
pve-manager: 5.1-40 (running version: 5.1-40/ea05b379)
pve-kernel-4.13.3-1-pve: 4.13.3-2
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.10.17-5-pve: 4.10.17-25
pve-kernel-4.13.13-1-pve: 4.13.13-31
libpve-http-server-perl: 2.0-8
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-19
qemu-server: 5.0-18
pve-firmware: 2.0-3
libpve-common-perl: 5.0-25
libpve-guest-common-perl: 2.0-14
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-17
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-3
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-5
pve-container: 2.0-18
pve-firewall: 3.0-5
pve-ha-manager: 2.0-4
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.1-2
lxcfs: 2.0.8-1
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9

Somebody can tell me, can I use pve-kernel-4.10.17-5-pve with proxmox-ve: 5.1-31?

As I see in page https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.0, proxmox 5.1. working with pve-kernel-4.13.


I going to install it now but do this have kworker issue?
 
I can verify if it has the ksoftirqd 99% problem, however could you guys please give me some details on what is needed to install a kernel and to say to the system that it has got to use this one?
 
ksoftirqd 99% io issue present in kernels starting from pve-kernel-4.13.3-1-pve version

I revers my reply it seems pve-kernel-4.10.17-5-pve kernel have issue as well


TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
3829 be/4 root 3.56 K/s 0.00 B/s 0.00 % 99.99 % [kworker/u16:4]
6629 be/0 root 3.93 M/s 118.71 M/s 0.00 % 91.26 % [loop25]
28022 be/4 498 976.43 K/s 60.58 K/s 0.00 % 68.16 % nginx: worker process
28017 be/4 498 363.49 K/s 35.64 K/s 0.00 % 35.17 % nginx: worker process
3971 be/4 root 1270.60 K/s 0.00 B/s 0.00 % 90.62 % perl -T /usr/bin/pvesr run --mail 1
29928 be/4 root 8.91 M/s 9.12 M/s 0.00 % 79.77 % perl /home/admin/web/sc/mixdata1l.pl
28017 be/4 498 316.77 K/s 7.04 K/s 0.00 % 70.58 % nginx: worker process
10651 be/0 root 4.51 M/s 3.52 K/s 0.00 % 58.39 % [loop8]
5042 be/0 root 4.98 M/s 0.00 B/s 0.00 % 54.04 % [loop3]
28024 be/4 498 175.98 K/s 7.04 K/s 0.00 % 39.19 % nginx: worker process
 
I revers my reply it seems pve-kernel-4.10.17-5-pve kernel have issue as well


TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
3829 be/4 root 3.56 K/s 0.00 B/s 0.00 % 99.99 % [kworker/u16:4]
6629 be/0 root 3.93 M/s 118.71 M/s 0.00 % 91.26 % [loop25]
28022 be/4 498 976.43 K/s 60.58 K/s 0.00 % 68.16 % nginx: worker process
28017 be/4 498 363.49 K/s 35.64 K/s 0.00 % 35.17 % nginx: worker process
3971 be/4 root 1270.60 K/s 0.00 B/s 0.00 % 90.62 % perl -T /usr/bin/pvesr run --mail 1
29928 be/4 root 8.91 M/s 9.12 M/s 0.00 % 79.77 % perl /home/admin/web/sc/mixdata1l.pl
28017 be/4 498 316.77 K/s 7.04 K/s 0.00 % 70.58 % nginx: worker process
10651 be/0 root 4.51 M/s 3.52 K/s 0.00 % 58.39 % [loop8]
5042 be/0 root 4.98 M/s 0.00 B/s 0.00 % 54.04 % [loop3]
28024 be/4 498 175.98 K/s 7.04 K/s 0.00 % 39.19 % nginx: worker process

We talking about ksoftirqd, NOT kworker
 
Hi,

please all who have the error send me the output of the following command

Code:
cat /proc/interrupts
 
Hi,

please all who have the error send me the output of the following command

Code:
cat /proc/interrupts

there is another kworker % CPU and I/O issue as well but not shows for the moment it come every few days and gone after reboot and back.

Code:
root@lxx:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
  0:         15          0          0          0          0          0          0          0  IR-IO-APIC   2-edge      timer
  1:          2          0          0          0          0          0          0          0  IR-IO-APIC   1-edge      i8042
  8:          1          0          0          0          0          0          0          0  IR-IO-APIC   8-edge      rtc0
  9:          0          0          0          0          0          0          0          0  IR-IO-APIC   9-fasteoi   acpi
 12:          4          0          0          0          0          0          0          0  IR-IO-APIC  12-edge      i8042
 16:         68          0          0          0          0          0          0          0  IR-IO-APIC  16-fasteoi   ehci_hcd:usb1
 18:          0          0          0          0          0          0          0          0  IR-IO-APIC  18-fasteoi   i801_smbus
 23:         29          0          0          0          0          0          0          0  IR-IO-APIC  23-fasteoi   ehci_hcd:usb2
 24:          0          0          0          0          0          0          0          0  DMAR-MSI   0-edge      dmar0
 25:          0          0          0          0          0          0          0          0  IR-PCI-MSI 16384-edge      PCIe PME
 26:          0          0          0          0          0          0          0          0  IR-PCI-MSI 18432-edge      PCIe PME
 27:          0          0          0          0          0          0          0          0  IR-PCI-MSI 458752-edge      PCIe PME
 28:          0          0          0          0          0          0          0          0  IR-PCI-MSI 466944-edge      PCIe PME
 29:          0          0          0          0          0          0          0          0  IR-PCI-MSI 471040-edge      PCIe PME
 30:    2750802          0          0          0          0          0          0          0  IR-PCI-MSI 1048576-edge      megasas
 31:          0    2784780          0          0          0          0          0          0  IR-PCI-MSI 1048577-edge      megasas
 32:          0          0    3119695          0          0          0          0          0  IR-PCI-MSI 1048578-edge      megasas
 33:          0          0          0    2998779          0          0          0          0  IR-PCI-MSI 1048579-edge      megasas
 34:          0          0          0          0    3073363          0          0          0  IR-PCI-MSI 1048580-edge      megasas
 35:          0          0          0          0          0    2815404          0          0  IR-PCI-MSI 1048581-edge      megasas
 36:          0          0          0          0          0          0    2747985          0  IR-PCI-MSI 1048582-edge      megasas
 37:          0          0          0          0          0          0          0    2761370  IR-PCI-MSI 1048583-edge      megasas
 38:       1494          0          0          0          0          0          0  124666149  IR-PCI-MSI 2097152-edge      eth0-rx-0
 39:         79          0          0          0          0          0   62449279          0  IR-PCI-MSI 2097153-edge      eth0-tx-0
 40:          2          0          0          0          0          6          0          0  IR-PCI-MSI 2097154-edge      eth0
NMI:       2083       2553       2245       2281       2125       2137       2227       2179   Non-maskable interrupts
LOC:  163361887  143834688  156131936  155709216  168675507  165858736  154821656  145033636   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:       2083       2553       2245       2281       2125       2137       2227       2179   Performance monitoring interrupts
IWI:          0          2          0          1          0          0          0          0   IRQ work interrupts
RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
RES:   10581093    6718556    5731702    4772588    5936586    4830031    4705634    4425105   Rescheduling interrupts
CAL:   16678294   16299882   18018944   17410308   18020071   17914995   17258983    8366409   Function call interrupts
TLB:      94948     126328     192186     146869     111228      96196     103146      78656   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:        232        232        232        232        232        232        232        232   Machine check polls
ERR:          0
MIS:          0
PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
NPI:          0          0          0          0          0          0          0          0   Nested posted-interrupt event
PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event
root@xx:~#
 
Hi,

please all who have the error send me the output of the following command

Code:
cat /proc/interrupts


Sorry for the delay.

Hypervisor with the problem (PVE 5.1-36):
Code:
root@hypve1:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7      
  0:         22          0          0          0          0          0          0          0  IR-IO-APIC   2-edge      timer
  8:          0          0          1          0          0          0          0          0  IR-IO-APIC   8-edge      rtc0
  9:          0          0          0          0          1          0          0          0  IR-IO-APIC   9-fasteoi   acpi
 16:          0          0          0          0          0          0          0          0  IR-IO-APIC  16-fasteoi   i801_smbus
 24:          0          0          0          0          0          0          0          0  DMAR-MSI   0-edge      dmar0
 25:          0          0          0          0          0          0          0          0  IR-PCI-MSI 16384-edge      PCIe PME
 26:          0          0          0          0          0          0          0          0  IR-PCI-MSI 18432-edge      PCIe PME
 27:          0          0          0          0          0          0          0          0  IR-PCI-MSI 458752-edge      PCIe PME
 28:          0          0          0          0          0          0          0          0  IR-PCI-MSI 460800-edge      PCIe PME
 29:          0          0          0          0          0          0          0          0  IR-PCI-MSI 462848-edge      PCIe PME
 30:          0          0          0          0          0          0          0          0  IR-PCI-MSI 464896-edge      PCIe PME
 31:          0          0          0          0          0          0          0          0  IR-PCI-MSI 471040-edge      PCIe PME
 32:          0          0          0          0          0          0          0          0  IR-PCI-MSI 475136-edge      PCIe PME
 33:         15          6          1          6          6          7          7          3  IR-PCI-MSI 327680-edge      xhci_hcd
 34:   21708439    8630612    5645924    3883124   18374969   12824425    8828467    6267107  IR-PCI-MSI 1048576-edge      enp2s0f0
 35:    2385364     931244     540533     334545    3641986    2060948    1218520     762360  IR-PCI-MSI 376832-edge      ahci[0000:00:17.0]
 38:  172943662   68153404   43742248   29517704  154625040  108480699   74240742   52500371  IR-PCI-MSI 1050624-edge      enp2s0f1
 39:  451132480  172582965  108931720   72821909  568374206  345916599  224398010  152844206  IR-PCI-MSI 2621440-edge      eno3
 40:  416437704  161323520  102875166   69450965  550743741  324886144  208969639  142009041  IR-PCI-MSI 3145728-edge      eno4
NMI:      86101      85933      85899      86131      85989      84452      84059      84138   Non-maskable interrupts
LOC: 2287758530 2212705319 2167305218 2152216518 2315264583 2264788424 2224390005 2186609622   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:      86101      85933      85899      86131      85989      84452      84059      84138   Performance monitoring interrupts
IWI:          1          0          0          0          1          0          1          2   IRQ work interrupts
RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
RES:  496684972  391563735  338019514  313596947  358337040  344365690  332003882  324231243   Rescheduling interrupts
CAL:  587660596  589382579  588555372  586675301  584771806  591418702  593824233  593563404   Function call interrupts
TLB:   22087594   22300143   22348975   22547999   22123164   22227538   22069069   22314721   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:       6867       6867       6867       6867       6867       6867       6867       6867   Machine check polls
ERR:         12
MIS:          0
PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
NPI:          0          0          0          0          0          0          0          0   Nested posted-interrupt event
PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event

Hypervisor w/out the problem (PVE 4.4-12):
Code:
root@pve1:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7      
  0:         19          0          0          0          0          0          0          0  IR-IO-APIC   2-edge      timer
  8:          0          1          0          0          0          0          0          0  IR-IO-APIC   8-edge      rtc0
  9:          0          0          0          0          0          0          0          0  IR-IO-APIC   9-fasteoi   acpi
 16:          6          0          0          0          1          0          0         18  IR-IO-APIC  16-fasteoi   ehci_hcd:usb1
 18:          0          0          0          0          0          0          0          0  IR-IO-APIC  18-fasteoi   i801_smbus
 23:         18          0          1          1          5          0          0          4  IR-IO-APIC  23-fasteoi   ehci_hcd:usb2
 24:          0          0          0          0          0          0          0          0  DMAR-MSI   0-edge      dmar0
 25:          0          0          0          0          0          0          0          0  IR-PCI-MSI 16384-edge      PCIe PME
 26:          0          0          0          0          0          0          0          0  IR-PCI-MSI 18432-edge      PCIe PME
 27:          0          0          0          0          0          0          0          0  IR-PCI-MSI 458752-edge      PCIe PME
 28:          0          0          0          0          0          0          0          0  IR-PCI-MSI 460800-edge      PCIe PME
 29:          0          0          0          0          0          0          0          0  IR-PCI-MSI 466944-edge      PCIe PME
 30:         30          1          5          0          6          3         11         25  IR-PCI-MSI 327680-edge      xhci_hcd
 31:  145410221   45631817   40208815   18598449   94833732   34385932   31964255   16709673  IR-PCI-MSI 409600-edge      eth1
 32:          0          1          0          0          0          0          0          0  IR-PCI-MSI 2621440-edge      eth0
 33:  148253592   52842201   32395129   20605668  124936174   40812422   22714279   16389421  IR-PCI-MSI 2621441-edge      eth0-TxRx-0
 34:   60763242   16077678   11286026    8123924   22361021   10216491    8141935    6131104  IR-PCI-MSI 1048576-edge      eth2
 35:    4469512    2526122    1504518     738350    3868388    1439171     765723     365947  IR-PCI-MSI 512000-edge      0000:00:1f.2
 36:   50983971   17178204   11499027    7981702   23435926   10788724    7915800    5659434  IR-PCI-MSI 1050624-edge      eth3
 37:  278178806   34436902   29487840   23493934   49902378   29512872   23789693   17611363  IR-PCI-MSI 3145728-edge      vfio-msi[0](0000:06:00.0)
 38:  218463265   29978057   26348381   20394859   42443987   24523268   20525374   14817951  IR-PCI-MSI 3147776-edge      vfio-msi[0](0000:06:00.1)
NMI:      16879      15650      15901      15750       7999       7788       7740       7871   Non-maskable interrupts
LOC: 2187546410 2066227964 1999338694 2094205683  700615295  796644635  657500762  886223001   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:      16879      15650      15901      15750       7999       7788       7740       7871   Performance monitoring interrupts
IWI:      16879      15650      15901      15751       7999       7788       7740       7871   IRQ work interrupts
RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
RES:  109714827  103368431   93666866   84649236   88823132   75073493   73619777   65345925   Rescheduling interrupts
CAL:   68166782   70565549   69652860   66068090   44897487   36095693   34466693   29055814   Function call interrupts
TLB:   11472384   11636009   11517536   11507251    8024224    7884322    7346694    7635440   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:       7087       7087       7087       7087       7087       7087       7087       7087   Machine check polls
ERR:          0
MIS:          0
PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event
 
root@pve:~# apt-get update
Get:1 http://security.debian.org stretch/updates InRelease [63.0 kB]
Ign:2 http://ftp.br.debian.org/debian stretch InRelease
Get:3 http://ftp.br.debian.org/debian stretch Release [118 kB]
Get:4 http://ftp.br.debian.org/debian stretch Release.gpg [2,434 B]
Get:5 http://security.debian.org stretch/updates/main amd64 Packages [252 kB]
Get:6 http://security.debian.org stretch/updates/main Translation-en [110 kB]
Get:7 http://security.debian.org stretch/updates/contrib amd64 Packages [1,352 B]
Get:8 http://ftp.br.debian.org/debian stretch/main amd64 Packages [7,123 kB]
Get:9 http://security.debian.org stretch/updates/contrib Translation-en [1,023 B]
Get:10 http://ftp.br.debian.org/debian stretch/main Translation-en [5,393 kB]
Get:11 http://ftp.br.debian.org/debian stretch/contrib amd64 Packages [50.9 kB]
Get:12 http://ftp.br.debian.org/debian stretch/contrib Translation-en [45.9 kB]
Ign:13 https://enterprise.proxmox.com/debian/pve stretch InRelease
Ign:14 https://enterprise.proxmox.com/debian/pve stretch Release
Ign:15 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Packages
Ign:16 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Packages
Ign:17 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en
Ign:18 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en_US
Ign:15 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Packages
Ign:16 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Packages
Ign:17 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en
Ign:18 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en_US
Ign:15 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Packages
Ign:16 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Packages
Ign:17 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en
Ign:18 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en_US
Ign:15 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Packages
Ign:16 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Packages
Ign:17 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en
Ign:18 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en_US
Ign:15 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Packages
Ign:16 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Packages
Ign:17 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en
Ign:18 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en_US
Err:15 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Packages
401 Unauthorized
Ign:16 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Packages
Ign:17 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en
Ign:18 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translation-en_US
Fetched 13.2 MB in 5s (2,341 kB/s)
Reading package lists... Done
W: The repository 'https://enterprise.proxmox.com/debian/pve stretch Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/stretch/pve-enterprise/binary-amd64/Packages 401 Unauthorized
E: Some index files failed to download. They have been ignored, or old ones used instead.
root@pve:~#