Thread: New Proxmox VE kernels 2.6.18, 2.6.32 and 2.6.35 released to stable

We just moved our latest 2.6.18, 2.6.32 and 2.6.35 Kernel from pvetest to the stable repository. A big thanks to all beta testers!

Release notes:

- pve-kernel-2.6.18 (2.6.18-15)

• update to ovzkernel-2.6.18-238.9.1.el5.028stab089.1.src.rpm

-pve-kernel-2.6.32 (2.6.32-33)

• update to Debian Squeeze kernel 2.6.32-34

-pve-kernel-2.6.35 (2.6.35-11)

• update to Ubuntu-2.6.35-29.51

About our kernel branches:
Proxmox VE Kernels
__________________
Best regards,
Martin Maurer

Hi Martin,

I can not update to the new kernel. Aptitude update/safe-upgrade do not show me any pending updates. I tried also to install the latest proxmox-ve-2.6.35.
Not sure what the problem is. Can I force the update ?

Kind regards.

Current pveversion:

Code:

BigOne:~# pveversion -v
pve-manager: 1.8-17 (pve-manager/1.8/5948)
running kernel: 2.6.35-1-pve
proxmox-ve-2.6.35: 1.8-10
pve-kernel-2.6.35-1-pve: 2.6.35-10
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.26-1pve4
vzdump: 1.2-11
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.0-3
ksm-control-daemon: 1.0-5

any error when you run aptitude update? try 'aptitude full-upgrade'

Hi Tom,

Output aptitude update:

Code:

BigOne:~# aptitude update
Hit http://security.debian.org lenny/updates Release.gpg
Hit http://security.debian.org lenny/updates Release
Ign http://security.debian.org lenny/updates/main Packages/DiffIndex
Ign http://security.debian.org lenny/updates/contrib Packages/DiffIndex
Hit http://security.debian.org lenny/updates/main Packages
Hit http://security.debian.org lenny/updates/contrib Packages
Hit http://ftp.de.debian.org lenny Release.gpg
Hit http://ftp.de.debian.org lenny Release
Hit http://download.proxmox.com lenny Release.gpg
Hit http://download.proxmox.com lenny Release
Ign http://ftp.de.debian.org lenny/main Packages/DiffIndex
Ign http://download.proxmox.com lenny/pve Packages/DiffIndex
Ign http://ftp.de.debian.org lenny/contrib Packages/DiffIndex
Ign http://download.proxmox.com lenny/pve Packages
Hit http://download.proxmox.com lenny/pve Packages
Hit http://ftp.de.debian.org lenny/main Packages
Hit http://ftp.de.debian.org lenny/contrib Packages
Reading package lists... Done

Output aptitude full-upgrade:

Code:

BigOne:~# aptitude full-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Reading task descriptions... Done
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0B of archives. After unpacking 0B will be used.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Reading task descriptions... Done

I don not see any errors...

Kind regards

strange. download the missing packages manually and install them manually:

Code:

wget http://download.proxmox.com/debian/dists/lenny/pve/binary-amd64/proxmox-ve-2.6.35_1.8-11_all.deb http://download.proxmox.com/debian/dists/lenny/pve/binary-amd64/proxmox-ve-2.6.35_1.8-11_all.deb http://download.proxmox.com/debian/dists/lenny/pve/binary-amd64/vzdump_1.2-12_all.deb

and install:

Code:

dpkg - i ......

Thank you very much Tom - that worked.

Does the 2.6.35 kernel have support for PCI_DEVICE_ID_ARECA_1880 ? (Areca ARC-1880-ix cards). From what I see we'd need to wait until .37 for main debian kernels.

Could anyone suggest a good 12 or 16-port raid card known to work well?

Great thanks!... I have been looking for release notes on these kernels ... are there any? What improvements can we expect?

Originally Posted by tmo

Does the 2.6.35 kernel have support for PCI_DEVICE_ID_ARECA_1880 ? (Areca ARC-1880-ix cards). From what I see we'd need to wait until .37 for main debian kernels.

NO, I cant see that symbol in the sources.

The Adaptec unified raid-controllers really rock, terrible fast and absolutely reliable.
We use eg: Adaptec RAID 51645

Originally Posted by tmo

Does the 2.6.35 kernel have support for PCI_DEVICE_ID_ARECA_1880 ? (Areca ARC-1880-ix cards). From what I see we'd need to wait until .37 for main debian kernels.

Could anyone suggest a good 12 or 16-port raid card known to work well?

Removed post into a new thread.

I had a test server with an Areca 1261 / ARC1280ML that worked well. (I don't remember which version it was - "lspci" and the silkscreen on the PCB didn't agree with each other, my notes are a little vague) That particular machine was running the stock PVE 1.7 / OpenVZ kernel, quite a bit older. The 1261 is 16-ports, the 1280 is 24.

Our new servers are running Dell PERC H700 cards, which are reworked LSI MegaSAS 9260 controllers. They're only 8 ports, but I'd imagine the larger cards in the series would work fine. (Kernel 2.6.32-4-pve, megaraid_sas module)

Cool, thanks for the feedback. We were able to build a custom kernel and get it working... but now another question:

Anyone try the (PMC) Adaptec 6805/6405 series? We got a few new servers with this card based on Adaptec saying the card used the same drivers, but this is false. Got it working in a debian install using the drivers from adaptec, but having trouble in proxmox.

Will this driver (Adaptec 6 series) be included in a near release of proxmox? (it flies with the BBU under debian - faster and cheaper than the 5405Z/5805z, and included cable as well!)

till now we don´t get the new series of Adaptec in our test labs - still looking for a sponsor. are they really faster than the 5er series? they are much cheaper and not proposed for high performance on the Adaptec website.

what trouble do you get on Proxmox VE with these cards?

Why do you say they are not proposed for high performance at adaptec? I saw they still say: "Series 5 still delivers high performance in all server-related workloads with a high number of I/O operations like OLTP and file servers, the Series 6 offers enhanced bandwidth with its 6Gb/s SAS 2.0 and PCIe Gen2 interfaces"...But I find that interesting, as with 6.0Gb/s 10k drives in a raid 1E we were getting about 4k fsyncs/sec (using the proxmox pveperf tool in debian). I can send the output when I am back at the office.

I will talk to my boss on Monday, but I am sure we can sponsor a card and the zmcp (bbu). Can I have him email you?

I am referring to this page:
http://www.adaptec.com/en-us/product.../?channel=tab1

the first row shows 'performance' controllers, the second 'value' controllers. and the value controllers are also cheaper, using another raid chip - so I assume they do not perform such good as the 5er series but as I wrote we do not benchmark the 6 series so I do not know it really.

And yes, would be great if you want to sponsor a card for our test lab here. just contact martin@...

Hello !

Do you have any plan to include 2.6.38 kernel to pvetest repo ?
I have deeply tested pure kvm performance on 2.6.32, 2.6.35 and 2.6.38, and the cpu usage for the same task is just better

• nic:e1000 kernel:2.6.32 --- WinCPU:90% ProxCPU:100%
• nic:e1000 kernel:2.6.35 --- WinCPU:30% ProxCPU:80%
• nic:e1000 kernel:2.6.38 --- WinCPU:25% ProxCPU:75%

currently there is no plan to include 2.6.38.

regarding cpu usage and network: use virtio, should be less cpu intensive.

Thanks for the response, I will wait a few year and ask again

About virtio, It always crash with intensive network usage, and I tested it with 1.6 and 2.0 dll, over those 3 kernels branch. (2.6.32 / .35 / .38)
Crash = no more network packet.

if you see a case where the network crashes please open a new thread and describe a way to reproduce the issue.

just to mention, no test card (or money) arrived here in our test labs. so looks like your boss decided to not supporting our test labs?

rofl

I will shoot him an email and find out. He said he would, so likely after he gets back from abroad. Did he contact you at all?

yes, he contacted Martin but we still wait for the card/money.

Thanks Tom. You should expect something later in the day, from what Urs said.

Anyway, we have replaced the cards with 5405Z/5805Zs, but are getting about 2200 fsyncs/sec with 10k SAS drives in a raid 10, with the .32 kernel (new 1.8 build). We've have the aligned the stripes and re-initialized the array, and are not seeing better results. With 4 RE4's in raid 10, we don't do much better, at about 2400 fsyncs/sec.

At the time, there are no vms running:

pveperf /var/lib/vz
CPU BOGOMIPS: 100802.84
REGEX/SECOND: 721044
HD SIZE: 969.05 GB (/dev/mapper/pve-data)
BUFFERED READS: 418.48 MB/sec
AVERAGE SEEK TIME: 5.89 ms
FSYNCS/SECOND: 2107.16
DNS EXT: 54.61 ms
DNS INT: 14.53 ms (xxxx.com)

dd if=/dev/zero of=/var/lib/vz/temp bs=4k count=4000000 conv=fdatasync
4000000+0 records in
4000000+0 records out
16384000000 bytes (16 GB) copied, 86.1892 s, 190 MB/s

hdparm -Tt /dev/sda

/dev/sda:
Timing cached reads: 5996 MB in 2.00 seconds = 2999.98 MB/sec
Timing buffered disk reads: 1438 MB in 3.00 seconds = 479.02 MB/sec

pveversion -v
pve-manager: 1.8-17 (pve-manager/1.8/5948)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-33
pve-kernel-2.6.32-4-pve: 2.6.32-33
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.26-1pve4
vzdump: 1.2-12
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.0-3
ksm-control-daemon: 1.0-5

What should we typically see for these cards? Searching through the forums, I saw yesterday some are reporting in the 3000 range. Are there any other tweaks you would recommend/suggest looking at?

Additionally, I have tried the .35 kernel, and results are nearly identical.

your results looks not bad.

here are some results (all with 2.6.32)

box1: 5805Z with Raid10 (4 x WD1002FBYS)

Code:

CPU BOGOMIPS:      8534.56
REGEX/SECOND:      684729
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    175.59 MB/sec
AVERAGE SEEK TIME: 9.78 ms
FSYNCS/SECOND:     2433.91
DNS EXT:           26.96 ms
DNS INT:           1.12 ms (proxmox.com)

box2: 5805z with Raid10 (4 x WD5001ABYS)

Code:

CPU BOGOMIPS:      19151.97
REGEX/SECOND:      762016
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    112.59 MB/sec
AVERAGE SEEK TIME: 9.47 ms
FSYNCS/SECOND:     2443.00
DNS EXT:           28.87 ms
DNS INT:           0.70 ms (proxmox.com)

box3: 5805 with Raid10 (6 x ST3250824NS)

Code:

CPU BOGOMIPS:      19151.60
REGEX/SECOND:      772463
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    273.32 MB/sec
AVERAGE SEEK TIME: 9.17 ms
FSYNCS/SECOND:     3748.74
DNS EXT:           25.97 ms
DNS INT:           0.71 ms (proxmox.com)

pls note, pveperf or hdparm are just very basic benchmarks. if you want more details you should do real benchmarks. e.g. with phoronix-test-suite or others - benchmarking is quite complex.

Thanks Tom, we will be doing some additional benchmarking with bonnie++ and also awaiting another areca card (seems like one of the active users here has a lot of success with arecas).

Still curious as to why with the pveperf tool running under a Debian build with the 6405 + bbu, but same drives was just under 4k fsyncs/sec. Our the SAS drives are SASII, so 6Gb/s. Perhaps that does make a difference?

Anyway, it may be nice to have a wiki article about controller + disk + cpu/ram + raid level, with basic performance benchmarks so others can get a better feel for building systems. I see there is one about controller cards, but is pretty slim in terms of content. From my experience seems like disk + controller are key factors here for maximizing real-world performance, preventing IO bottlenecks.

Feel free to add a wiki article. But hardware is outdated soon so keeping this up2date is quite challenging.

Originally Posted by tmo

Still curious as to why with the pveperf tool running under a Debian build with the 6405 + bbu, but same drives was just under 4k fsyncs/sec. Our the SAS drives are SASII, so 6Gb/s. Perhaps that does make a difference?

You may want to check what (read, write) caching your controller has set. There are also caches you can set per drive attached to the controller card. Setting all this can make a really big difference. Also, if you have no BBU, the controller may disable all caches.

Unfortunately, each hardware RAID card has a different (vendor) tool to do it; some do not allow manipulating drive caches, so your results may be very different.

Originally Posted by tmo

Cool, thanks for the feedback. We were able to build a custom kernel and get it working... but now another question:

Anyone try the (PMC) Adaptec 6805/6405 series? We got a few new servers with this card based on Adaptec saying the card used the same drivers, but this is false. Got it working in a debian install using the drivers from adaptec, but having trouble in proxmox.

Will this driver (Adaptec 6 series) be included in a near release of proxmox? (it flies with the BBU under debian - faster and cheaper than the 5405Z/5805z, and included cable as well!)

Here we go: http://forum.proxmox.com/threads/6521-New-Proxmox-VE-1.8-ISO-with-latest-packages

Adaptec 6 series works out of the box, waiting for your feedback and test results.

Very Cool! How is the performance? Did you have a chance to run some benchmarks?

Originally Posted by tmo

Very Cool! How is the performance? Did you have a chance to run some benchmarks?

not really as I do not have free 6 GB/s sata drives here. looking for a sponsor ;-)

but I assume it will be identical to the tests you did with Debian and the driver from Adaptec.

Originally Posted by tom

looking for a sponsor ;-)

LOL .. I will talk to my boss again during our staff meetings tomorrow ;-)

Also, I assume this driver will be in future releases of all Proxmox releases, correct? And, any chance that the Areca 1880 series will be included? That one blows the adaptec away! (we built a custom kernel, but are not looking forward rebuilding on each kernel update)

Thanks again!

Originally Posted by tmo

LOL .. I will talk to my boss again during our staff meetings tomorrow ;-)

Also, I assume this driver will be in future releases of all Proxmox releases, correct?

works in the current 2.6.32 kernel branch for Proxmox 1.x, based on squeeze (does not work in the 2.6.18er and the current 2.6.35er.)

Originally Posted by tmo

And, any chance that the Areca 1880 series will be included? That one blows the adaptec away! (we built a custom kernel, but are not looking forward rebuilding on each kernel update)

Thanks again!

as far as I see the needed module is all kernels later 2.6.37/38. the current arcmsr module in squeeze does not support it (yet).

Originally Posted by tom

Here we go: http://forum.proxmox.com/threads/6521-New-Proxmox-VE-1.8-ISO-with-latest-packages

Adaptec 6 series works out of the box, waiting for your feedback and test results.

I have an Adaptec 6405 running for a couple weeks now. Maybe I'm being a knucklehead and overlooking something but, according to Adaptec documentation, LVM partitioning isn't supported with the 6 series cards under Debian. My understanding is the bare-metal installer creates LVM partitions by default. Wouldn't the installer fail? I attempted a plain Debian 5 install with LVM on the raid array and, while the install looked successful, the machine wouldn't boot.

The way I got everything working was by installing Debian 5 on ext3 and then installing Proxmox VE from the repository. Obviously, I don't have any of the advantages that LVM would provide with this configuration. I turned on a snapshot backup in addition to a sleep/resume backup. The snapshot never shows an error, and the size looks correct, but I really don't know if the snapshot would work if I tried to restore which is why I do both.

Do you have a recommendation of a better way to set this up or is my understanding of the bare-metal installer, LVM, and snapshot backups wrong?

Originally Posted by SpecZero

I have an Adaptec 6405 running for a couple weeks now. Maybe I'm being a knucklehead and overlooking something but, according to Adaptec documentation, LVM partitioning isn't supported with the 6 series cards under Debian. ...

pls point me to this docu, I can´t believe that. we got a 6805 in our test lab, works here.

Originally Posted by tom

pls point me to this docu, I can´t believe that. we got a 6805 in our test lab, works here.

http://download.adaptec.com/pdfs/readme/series-6-controller_readme_3_2011.pdf

Section 2.3 starting on the last line of page 2.

thanks for the link, you are right, quite strange. but probably this points to an old driver (aacraid 1.1.5). I installed LVM partitioning (just ISO installer from today) without problems.

Originally Posted by tom

thanks for the link, you are right, quite strange. but probably this points to an old driver (aacraid 1.1.5). I installed LVM partitioning (just ISO installer from today) without problems.

Thanks for the follow up and confirmation that this works fully. I'll be using the new ISO and attempting a bare-metal install with the 6405 this weekend.

pls report your results, would be nice if you can also post the results of pveperf (include your hardware details).

Originally Posted by SpecZero

I have an Adaptec 6405 ... according to Adaptec documentation, LVM partitioning isn't supported with the 6 series cards under Debian.

Absolutely it works under debian. We had it running there with LVM, just not Proxmox. After bribing, they have kindly added support for it in the latest ISO. We are setting up a box with one now, and so far install is going smoothly. Will post results sometime tomorrow when the raid 10 is finished building.

Areca should work with this kernel - http://forum.proxmox.com/threads/6527-New-Proxmox-VE-kernel-2.6.32-(pvetest)

Originally Posted by tom

Areca should work with this kernel - http://forum.proxmox.com/threads/6527-New-Proxmox-VE-kernel-2.6.32-(pvetest)

Awesome!!!! Thank you! Can't wait to try it out!

Originally Posted by tmo

Will post results sometime tomorrow when the raid 10 is finished building.

Update: raid initialization is complete with the 6405 with bbu/zmm, in raid 10, using 4 sasII (6Gbit/s) 10K drives

pve24:~# pveperf /var/lib/vz
CPU BOGOMIPS: 100802.43
REGEX/SECOND: 719498
HD SIZE: 969.05 GB (/dev/mapper/pve-data)
BUFFERED READS: 408.83 MB/sec
AVERAGE SEEK TIME: 5.10 ms
FSYNCS/SECOND: 1485.37
DNS EXT: 62.97 ms
DNS INT: 17.05 ms (example.com)

Not sure if the partition alignment has anything to do with the pveperf i/o results or if this is the driver, but they are lower than we expected, and significantly lower than on native debian during our tests (we had ~4K fsync/sec with 8 sasII drives. (more drives, better i/o for this card)).

Anyway, fsyncs/sec is a good 25% lower with the Adaptec 6 series compared with the adaptec 5 series, with same exact hardware configuration (results for 5 series posted above).

Originally Posted by tom

Here we go: http://forum.proxmox.com/threads/6521-New-Proxmox-VE-1.8-ISO-with-latest-packages

Adaptec 6 series works out of the box, waiting for your feedback and test results.

The 6 series works great with 2.6.32 but 2.6.35 won't boot.

Originally Posted by tmo

Does the 2.6.35 kernel have support for PCI_DEVICE_ID_ARECA_1880 ? (Areca ARC-1880-ix cards). From what I see we'd need to wait until .37 for main debian kernels.

Could anyone suggest a good 12 or 16-port raid card known to work well?

I've got a new Adaptec Serie 6 RAID card which is fantastic with PVE.

Originally Posted by martin

Areca should work with this kernel - http://forum.proxmox.com/threads/652...6.32-(pvetest)

Any chance to get a Proxmox install .iso which includes the Areca 1880-series driver?
The controllers are now out since nearly a year and a capable kernel in the repository makes no sense when you are not able to install Proxmox on it....

there are some issue with this kernel, preventing a release to stable and iso.

Tried to do a bare metal install on system with adaptec 6805. The system did not boot. Got the message "Unable to create data volume at /usr/bin/proxinstall line 570". I read on the forum what others did to get their 6805 working. I confirmed with Adaptec support that the drivers in the kernel 2.6.39 support LVM on the 6805. My question is can I install Proxmox 1.8 build 6070-5 and upgrade the kernel to 2.6.39 or should I install the driver separately on a sata HD? Thanks in advance.

"Unable to create data volume at /usr/bin/proxinstall line 570" - if disk partition >2Tb and label - gpt
Proxmox 1.8 build 6070-5 - dont support hdd >2Tb, dont use parted from create slice and dont support gpt partition

if need install Proxmox 1.8 build 6070-5 on big hdd, then install system on other sata hdd and mount raid to /var/lib/vz
or makelabel - msdos and use only 2Tb from raid...

Im install Proxmox 1.8 build 6070-5 on big drive:
- boot from liveCD OpenSuSe 12.1 (2.6.39) - create slice (vgcreate pve and lv - swap, root, data) and install system, install grub 2
- deleta all data from pve-root
- unpack from Proxmox 1.8 build 6070-5 iso - /usr/bin proxinstall and edit he (comment create slice and vgcreate), edited proxinstall copy to iso and burn CD
- boot from this CD and install Proxmox 1.8 build 6070-5 on system

Thanks FAN

I was not able to get system to boot with a volume larger the 2TB. I rebuild the raid 10 volume on the Adaptec 6805 and changed the size to 2TB and the Proxmox 1.8 build 6070-5 installed and utilized the 2TB volume without any trouble.

Thanks again