Are people running large KVM Clusters?

B

blackpaw

Renowned Member
Nov 1, 2013
302
22
83
Just curious to see the size of KVM proxmox installs and what sort of backing hardware people are using. Any large farms out there? how are you finding proxmox as a management tool? do you rely on the built in backup or roll your own?

Looking at a smallish setup my self.
Two of the following servers:
- Xenon E5-2620
- 32GB ECC RAM
- Server Board with twin nics, GB Ethenet
- 128GB SSD, for OS only

VM Storage:
QNAP NAS
- 6TB (Raid 10)
- NFS Shares

Running around 5 Windows Servers, low load.
 
Take a look at this thread:
http://forum.proxmox.com/threads/16106-Volunteers-Wanted-to-test-Virtual-Machine-Cloud-Platform!

We have about 73 Beta Tester testing Proxmox cloud platform remotely. Besides 73 VMs, we also have about 27 VMs running for various purposes. All being managed through Proxmox WebGUI. All VMs are stored on CEPH Shared Storage. We are trying to break Proxmox system by making it over work for weeks, but still not breaking. Although there are some minor issues here and there, mostly because i am constantly tweaking this, changing that etc. Whole platform running on 4 Proxmox Nodes and 2 CEPH Nodes. None of the Proxmox Nodes has any "Server Motherboard".

For backup i use Proxmox built-in backup for VM and BackupPC for granular file backup inside VM. NFS Shared Storage is currently used for Backup. But i am thinking of creating another Ceph Cluster just for backup purpose.
 
Last edited:
I suspect you are not counting on using a Qnap for production purposes?
For backup it is fine - I also use a Qnap for backup - but for VM's it is way to slow. Bad throughput and low IOPS.
 
mir said:
I suspect you are not counting on using a Qnap for production purposes?
For backup it is fine - I also use a Qnap for backup - but for VM's it is way to slow. Bad throughput and low IOPS.
Click to expand...

Alas, we are, and yes not happy with the performance, proving a bottleneck.

Any suggestions for a replacement?
 
I have no experience with Ceph or GlusterFS since I prefer ZFS. I have a ZFS box populated with SATA disks in a RAID10 with the following performance numbers (All tests was made while serving 10 running VM's and 2 running CT's from the same storage):
Code: 
write 40.96 GB via dd, please wait...
time dd if=/dev/zero of=/vMotion/dd.tst bs=2048000 count=20000

20000+0 records in
20000+0 records out

real       30.8
user        0.0
sys        28.0

40.96 GB in 30.8s = [B]1329.87 MB/s Write[/B]

wait 20 s
[HR][/HR]read 40.96 GB via dd, please wait...
time dd if=/vMotion/dd.tst of=/dev/null bs=2048000

20000+0 records in
20000+0 records out

real       20.9
user        0.0
sys        20.6

40.96 GB in 20.9s = [B]1959.81 MB/s Read


[/B]
Running fio inside VM mount via iSCSI
Code: 
[global]
blocksize=16k
rwmixread=75 # 3:1 read:write
ramp_time=30
runtime=600
time_based
buffered=1
# size = free-ram * 80% / 5
size=100m
ioengine=libaio
directory=/media/vdd
[sla-0]
readwrite=randrw:2
[sla-1]
readwrite=randrw:2
[sla-2]
readwrite=randrw
[sla-3]
readwrite=randrw
[sla-4]
readwrite=randrw


sla-0: (groupid=0, jobs=1): err= 0: pid=3317
  read : io=84704KB, bw=56064KB/s, iops=3504 , runt=600008msec
    slat (usec): min=0 , max=439686 , avg=20.68, stdev=405.13
    clat (usec): min=0 , max=21570 , avg= 5.06, stdev=114.63
     lat (usec): min=1 , max=439693 , avg=24.60, stdev=414.58
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    6], 99.50th=[   13], 99.90th=[   45], 99.95th=[  106],
     | 99.99th=[ 6368]
    bw (KB/s)  : min= 8662, max=382303, per=21.42%, avg=57587.57, stdev=23625.46
  write: io=2962.4MB, bw=19037KB/s, iops=1189 , runt=600008msec
    slat (usec): min=0 , max=387845 , avg=755.62, stdev=8701.42
    clat (usec): min=0 , max=23866 , avg= 4.95, stdev=112.09
     lat (usec): min=3 , max=387857 , avg=757.36, stdev=8701.39
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    8], 99.50th=[   14], 99.90th=[   41], 99.95th=[   88],
     | 99.99th=[ 6176]
    bw (KB/s)  : min=    0, max=60054, per=20.91%, avg=19153.37, stdev=7317.48
    lat (usec) : 2=0.11%, 4=59.00%, 10=40.00%, 20=0.65%, 50=0.15%
    lat (usec) : 100=0.04%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.91%, sys=7.84%, ctx=24310, majf=0, minf=0
  IO depths    : 1=103.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2102446/w=0/d=713881, short=r=0/w=0/d=0
sla-1: (groupid=0, jobs=1): err= 0: pid=3318
  read : io=3117.3MB, bw=54245KB/s, iops=3390 , runt=600093msec
    slat (usec): min=0 , max=443170 , avg=21.20, stdev=413.62
    clat (usec): min=0 , max=28326 , avg= 5.10, stdev=115.81
     lat (usec): min=1 , max=443179 , avg=25.41, stdev=429.57
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    7], 99.50th=[   14], 99.90th=[   45], 99.95th=[  101],
     | 99.99th=[ 6496]
    bw (KB/s)  : min= 7905, max=463553, per=20.79%, avg=55893.99, stdev=24637.10
  write: io=2646.4MB, bw=18494KB/s, iops=1155 , runt=600093msec
    slat (usec): min=1 , max=562347 , avg=776.52, stdev=8928.90
    clat (usec): min=0 , max=24037 , avg= 5.22, stdev=119.24
     lat (usec): min=5 , max=562354 , avg=780.13, stdev=8940.93
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   10], 99.50th=[   15], 99.90th=[   44], 99.95th=[  106],
     | 99.99th=[ 7008]
    bw (KB/s)  : min=    0, max=54649, per=20.31%, avg=18600.23, stdev=7210.38
    lat (usec) : 2=0.12%, 4=53.68%, 10=45.27%, 20=0.69%, 50=0.15%
    lat (usec) : 100=0.04%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.95%, sys=7.67%, ctx=23786, majf=0, minf=0
  IO depths    : 1=103.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2034512/w=0/d=693634, short=r=0/w=0/d=0
sla-2: (groupid=0, jobs=1): err= 0: pid=3319
  read : io=3092.5MB, bw=54200KB/s, iops=3387 , runt=600121msec
    slat (usec): min=0 , max=436433 , avg=21.12, stdev=434.51
    clat (usec): min=0 , max=28042 , avg= 5.10, stdev=114.12
     lat (usec): min=1 , max=436441 , avg=25.06, stdev=417.26
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    7], 99.50th=[   14], 99.90th=[   46], 99.95th=[  112],
     | 99.99th=[ 6688]
    bw (KB/s)  : min= 1399, max=478336, per=20.80%, avg=55900.39, stdev=24451.73
  write: io=2645.9MB, bw=18493KB/s, iops=1155 , runt=600121msec
    slat (usec): min=1 , max=481743 , avg=776.67, stdev=8834.30
    clat (usec): min=0 , max=24032 , avg= 4.96, stdev=102.75
     lat (usec): min=4 , max=481750 , avg=779.83, stdev=8855.88
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    8], 99.50th=[   15], 99.90th=[   47], 99.95th=[  112],
     | 99.99th=[ 5664]
    bw (KB/s)  : min=    0, max=53088, per=20.31%, avg=18603.79, stdev=7013.24
    lat (usec) : 2=0.16%, 4=54.96%, 10=43.98%, 20=0.66%, 50=0.14%
    lat (usec) : 100=0.04%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.94%, sys=7.72%, ctx=24012, majf=0, minf=0
  IO depths    : 1=103.8%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2032899/w=0/d=693620, short=r=0/w=0/d=0
sla-3: (groupid=0, jobs=1): err= 0: pid=3320
  read : io=1805.4MB, bw=52012KB/s, iops=3250 , runt=600036msec
    slat (usec): min=0 , max=300945 , avg=21.29, stdev=340.15
    clat (usec): min=0 , max=32037 , avg= 5.16, stdev=123.40
     lat (usec): min=2 , max=68096 , avg=25.54, stdev=293.04
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    7], 99.50th=[   14], 99.90th=[   46], 99.95th=[  114],
     | 99.99th=[ 6944]
    bw (KB/s)  : min= 1072, max=287052, per=20.01%, avg=53785.44, stdev=20463.57
  write: io=2209.3MB, bw=17750KB/s, iops=1109 , runt=600036msec
    slat (usec): min=2 , max=472733 , avg=808.91, stdev=9073.45
    clat (usec): min=0 , max=20847 , avg= 5.19, stdev=123.08
     lat (usec): min=6 , max=472740 , avg=815.51, stdev=9116.50
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    8], 99.50th=[   15], 99.90th=[   42], 99.95th=[   94],
     | 99.99th=[ 6304]
    bw (KB/s)  : min=    0, max=43200, per=19.47%, avg=17834.41, stdev=6276.52
    lat (usec) : 2=0.12%, 4=57.87%, 10=41.11%, 20=0.66%, 50=0.14%
    lat (usec) : 100=0.04%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.87%, sys=7.43%, ctx=23949, majf=0, minf=0
  IO depths    : 1=104.2%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1950552/w=0/d=665677, short=r=0/w=0/d=0
sla-4: (groupid=0, jobs=1): err= 0: pid=3321
  read : io=1983.1MB, bw=52317KB/s, iops=3269 , runt=600012msec
    slat (usec): min=0 , max=436463 , avg=21.60, stdev=413.35
    clat (usec): min=0 , max=21554 , avg= 5.19, stdev=119.71
     lat (usec): min=1 , max=436472 , avg=26.12, stdev=438.81
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[    7], 99.50th=[   14], 99.90th=[   45], 99.95th=[  106],
     | 99.99th=[ 7072]
    bw (KB/s)  : min=13540, max=465727, per=20.15%, avg=54166.15, stdev=23269.63
  write: io=2259.9MB, bw=17837KB/s, iops=1114 , runt=600012msec
    slat (usec): min=1 , max=463201 , avg=803.43, stdev=9125.13
    clat (usec): min=0 , max=20527 , avg= 5.35, stdev=124.75
     lat (usec): min=5 , max=463206 , avg=808.99, stdev=9155.50
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   10], 99.50th=[   16], 99.90th=[   49], 99.95th=[  113],
     | 99.99th=[ 7520]
    bw (KB/s)  : min=    0, max=49212, per=19.61%, avg=17959.51, stdev=6688.46
    lat (usec) : 2=0.13%, 4=55.02%, 10=43.94%, 20=0.67%, 50=0.14%
    lat (usec) : 100=0.04%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.86%, sys=7.56%, ctx=24405, majf=0, minf=0
  IO depths    : 1=104.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1961926/w=0/d=668919, short=r=0/w=0/d=0


Run status group 0 (all jobs):
   READ: io=157536MB, aggrb=268808KB/s, minb=52011KB/s, maxb=56064KB/s, mint=600008msec, maxt=600121msec
  WRITE: io=53683MB, aggrb=91601KB/s, minb=17750KB/s, maxb=19036KB/s, mint=600008msec, maxt=600121msec


Disk stats (read/write):
  vdd: ios=24001/2238264, merge=0/5106, ticks=23632/91526664, in_queue=91555692, util=99.78%
Running fio inside VM mount via NFS raw image
Code: 
[global]
blocksize=16k
rwmixread=75 # 3:1 read:write
ramp_time=30
runtime=600
time_based
buffered=1
# size = free-ram * 80% / 5
size=100m
ioengine=libaio
directory=/media/vdd
[sla-0]
readwrite=randrw:2
[sla-1]
readwrite=randrw:2
[sla-2]
readwrite=randrw
[sla-3]
readwrite=randrw
[sla-4]
readwrite=randrw

sla-0: (groupid=0, jobs=1): err= 0: pid=3654
  read : io=1187.6MB, bw=71924KB/s, iops=4495 , runt=600064msec
    slat (usec): min=0 , max=276209 , avg=13.55, stdev=319.52
    clat (usec): min=0 , max=16048 , avg= 0.94, stdev=39.35
     lat (usec): min=0 , max=50446 , avg=11.05, stdev=159.92
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    1], 99.50th=[    1], 99.90th=[   13], 99.95th=[   26],
     | 99.99th=[  112]
    bw (KB/s)  : min=10231, max=219296, per=21.72%, avg=73825.44, stdev=27311.30
  write: io=1898.7MB, bw=24209KB/s, iops=1513 , runt=600064msec
    slat (usec): min=4 , max=1068.1K, avg=630.56, stdev=8126.64
    clat (usec): min=0 , max=9408 , avg= 1.03, stdev=38.86
     lat (usec): min=4 , max=1068.1K, avg=622.14, stdev=7989.79
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    2], 99.50th=[    3], 99.90th=[   14], 99.95th=[   28],
     | 99.99th=[  111]
    bw (KB/s)  : min=    0, max=70784, per=21.45%, avg=24417.89, stdev=9116.15
    lat (usec) : 2=99.33%, 4=0.40%, 10=0.11%, 20=0.11%, 50=0.04%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=1.00%, sys=5.55%, ctx=21621, majf=0, minf=0
  IO depths    : 1=101.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2697445/w=0/d=907942, short=r=0/w=0/d=0
sla-1: (groupid=0, jobs=1): err= 0: pid=3655
  read : io=3151.8MB, bw=61303KB/s, iops=3831 , runt=600001msec
    slat (usec): min=0 , max=361477 , avg=21.52, stdev=640.43
    clat (usec): min=0 , max=14560 , avg= 0.93, stdev=38.40
     lat (usec): min=0 , max=245770 , avg=16.11, stdev=380.24
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    1], 99.50th=[    2], 99.90th=[   13], 99.95th=[   25],
     | 99.99th=[  112]
    bw (KB/s)  : min=    0, max=184885, per=18.23%, avg=61958.36, stdev=25621.35
  write: io=3768.3MB, bw=20412KB/s, iops=1275 , runt=600001msec
    slat (usec): min=0 , max=4249.9K, avg=742.71, stdev=9996.47
    clat (usec): min=0 , max=9661 , avg= 0.95, stdev=32.75
     lat (usec): min=0 , max=549499 , avg=728.71, stdev=8707.34
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    2], 99.50th=[    3], 99.90th=[   14], 99.95th=[   27],
     | 99.99th=[   99]
    bw (KB/s)  : min=  264, max=58128, per=18.19%, avg=20704.30, stdev=8425.21
    lat (usec) : 2=99.24%, 4=0.44%, 10=0.15%, 20=0.11%, 50=0.04%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=0.97%, sys=5.59%, ctx=22436, majf=0, minf=0
  IO depths    : 1=100.3%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2298864/w=0/d=765455, short=r=0/w=0/d=0
sla-2: (groupid=0, jobs=1): err= 0: pid=3656
  read : io=3669.9MB, bw=69171KB/s, iops=4323 , runt=600060msec
    slat (usec): min=0 , max=274919 , avg=15.11, stdev=360.88
    clat (usec): min=0 , max=16047 , avg= 0.95, stdev=39.11
     lat (usec): min=0 , max=51152 , avg=11.64, stdev=167.13
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    1], 99.50th=[    1], 99.90th=[   13], 99.95th=[   27],
     | 99.99th=[  115]
    bw (KB/s)  : min= 5358, max=203947, per=20.73%, avg=70465.06, stdev=26292.60
  write: io=1317.5MB, bw=23218KB/s, iops=1451 , runt=600060msec
    slat (usec): min=0 , max=611120 , avg=658.86, stdev=8209.86
    clat (usec): min=0 , max=15201 , avg= 1.07, stdev=42.35
     lat (usec): min=1 , max=611124 , avg=647.95, stdev=8099.56
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    2], 99.50th=[    3], 99.90th=[   14], 99.95th=[   29],
     | 99.99th=[  131]
    bw (KB/s)  : min=    0, max=66651, per=20.58%, avg=23426.80, stdev=8891.53
    lat (usec) : 2=99.29%, 4=0.41%, 10=0.13%, 20=0.11%, 50=0.04%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=1.06%, sys=5.53%, ctx=22527, majf=0, minf=0
  IO depths    : 1=101.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2594165/w=0/d=870750, short=r=0/w=0/d=0
sla-3: (groupid=0, jobs=1): err= 0: pid=3657
  read : io=3142.3MB, bw=68277KB/s, iops=4267 , runt=600001msec
    slat (usec): min=0 , max=214706 , avg=15.99, stdev=345.65
    clat (usec): min=0 , max=12756 , avg= 0.92, stdev=36.11
     lat (usec): min=0 , max=49357 , avg=11.86, stdev=164.75
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    1], 99.50th=[    1], 99.90th=[   13], 99.95th=[   27],
     | 99.99th=[  110]
    bw (KB/s)  : min= 9660, max=220281, per=20.51%, avg=69718.26, stdev=26482.17
  write: io=1107.3MB, bw=22861KB/s, iops=1428 , runt=600001msec
    slat (usec): min=0 , max=611136 , avg=667.53, stdev=8191.93
    clat (usec): min=0 , max=11254 , avg= 0.98, stdev=33.14
     lat (usec): min=1 , max=611139 , avg=657.57, stdev=8138.34
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    2], 99.50th=[    3], 99.90th=[   14], 99.95th=[   28],
     | 99.99th=[   95]
    bw (KB/s)  : min=    0, max=70750, per=20.25%, avg=23050.52, stdev=8889.60
    lat (usec) : 2=99.29%, 4=0.40%, 10=0.13%, 20=0.11%, 50=0.04%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=1.09%, sys=5.55%, ctx=23284, majf=0, minf=0
  IO depths    : 1=100.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2560397/w=0/d=857295, short=r=0/w=0/d=0
sla-4: (groupid=0, jobs=1): err= 0: pid=3658
  read : io=3716.5MB, bw=69250KB/s, iops=4328 , runt=600060msec
    slat (usec): min=0 , max=175368 , avg=15.53, stdev=313.54
    clat (usec): min=0 , max=14790 , avg= 0.96, stdev=40.12
     lat (usec): min=0 , max=52490 , avg=11.82, stdev=166.82
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    1], 99.50th=[    1], 99.90th=[   13], 99.95th=[   27],
     | 99.99th=[  110]
    bw (KB/s)  : min= 9031, max=183195, per=20.78%, avg=70625.30, stdev=26691.10
  write: io=1282.9MB, bw=23159KB/s, iops=1447 , runt=600060msec
    slat (usec): min=0 , max=1043.8K, avg=659.50, stdev=8192.59
    clat (usec): min=0 , max=11494 , avg= 0.97, stdev=33.82
     lat (usec): min=0 , max=1043.9K, avg=648.92, stdev=8114.02
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    2], 99.50th=[    3], 99.90th=[   14], 99.95th=[   29],
     | 99.99th=[  110]
    bw (KB/s)  : min=    0, max=62814, per=20.53%, avg=23374.93, stdev=9054.85
    lat (usec) : 2=99.30%, 4=0.40%, 10=0.13%, 20=0.11%, 50=0.04%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=1.04%, sys=5.58%, ctx=23192, majf=0, minf=0
  IO depths    : 1=100.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2597147/w=0/d=868534, short=r=0/w=0/d=0


Run status group 0 (all jobs):
   READ: io=199188MB, aggrb=339910KB/s, minb=61302KB/s, maxb=71924KB/s, mint=600001msec, maxt=600064msec
  WRITE: io=66718MB, aggrb=113853KB/s, minb=20412KB/s, maxb=24209KB/s, mint=600001msec, maxt=600064msec


Disk stats (read/write):
  vdb: ios=33061/2685548, merge=11/6459, ticks=71644/89475240, in_queue=89549124, util=99.62%
OS: Omnios (Solaris based, Illumos) with napp-it
CPU: A4 3300
MB: Asrock A75 Extreme 6
RAM: 16G
Disks (Storage): WD Blue WD10EZEX 1 TB (only one platter so extremely low latency)
Disks (OS): 2 x Seagate Barracuda 7200.10 250G in RAID1
HBA: LSI 1068E in IT mode
Nics: 2 x Intel 82574L

For at larger setup I would recommend the following changes:
CPU: Opteron 3350 or bigger
RAM: 32 GB
Disks (Storage): SAS 2/3 disks
HBA: LSI 2008
 
Last edited:
blackpaw said:
... Any large farms out there? how are you finding proxmox as a management tool? do you rely on the built in backup or roll your own?
...
Click to expand...
Hi,
how you defined "large"?

I manage three cluster (3/3/5 hosts) with 3 - 9 CPUs (up to 68 cores + 336GB RAM in one cluster) - is this large? I think not - but it's an environment which is good for most things.

Proxmox as managing tool is good - I can do all things I want (but also use ofter the cli).

Backup: I use the builtin-backup only for disaster recovery - the daily backup is done by bacula (I have some VMs with some TB-disks).

Udo
 
mir said:
...
Code: 
write 40.96 GB via dd, please wait...
time dd if=/dev/zero of=/vMotion/dd.tst bs=2048000 count=20000

20000+0 records in
20000+0 records out

real       30.8
user        0.0
sys        28.0

40.96 GB in 30.8s = [B]1329.87 MB/s Write[/B]

wait 20 s
[HR][/HR]read 40.96 GB via dd, please wait...
time dd if=/vMotion/dd.tst of=/dev/null bs=2048000

20000+0 records in
20000+0 records out

real       20.9
user        0.0
sys        20.6

40.96 GB in 20.9s = [B]1959.81 MB/s Read


[/B]
...
Click to expand...
Hi,
and how looks the values with sync/and without caches?:
Code: 
time dd if=/dev/zero of=/vMotion/dd.tst bs=2048000 count=20000 conv=fdatasync[I]
# delete cache
echo 3[/I] > /proc/sys/vm/drop_caches
time dd if=/vMotion/dd.tst of=/dev/null bs=2048000
Udo
 
udo said:
Hi,
and how looks the values with sync/and without caches?:
Code: 
time dd if=/dev/zero of=/vMotion/dd.tst bs=2048000 count=20000 conv=fdatasync[I]
# delete cache
echo 3[/I] > /proc/sys/vm/drop_caches
time dd if=/vMotion/dd.tst of=/dev/null bs=2048000
Udo
Click to expand...
# time dd if=/dev/zero of=/vMotion/dd.tst bs=2048000 count=20000 conv=fdatasync
20000+0 records in
20000+0 records out
40960000000 bytes (41 GB) copied, 31.271 s, 1.3 GB/s


real 0m31.283s
user 0m0.058s
sys 0m29.363s

# sync

# time dd if=/vMotion/dd.tst of=/dev/null bs=2048000
20000+0 records in
20000+0 records out
40960000000 bytes (41 GB) copied, 20.6754 s, 2.0 GB/s


real 0m20.679s
user 0m0.048s
sys 0m20.518s
 
And the numbers using Qcow2 over NFS (cache=writethrough. With cache=none performance will increase by 50-100%):
Code: 
sla-0: (groupid=0, jobs=1): err= 0: pid=3678
  read : io=1100.1MB, bw=29835KB/s, iops=1864 , runt=600112msec
    slat (usec): min=0 , max=308036 , avg=27.72, stdev=486.48
    clat (usec): min=0 , max=20369 , avg= 5.25, stdev=102.54
     lat (usec): min=5 , max=308043 , avg=26.94, stdev=419.13
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   13], 99.50th=[   19], 99.90th=[   53], 99.95th=[  100],
     | 99.99th=[ 5920]
    bw (KB/s)  : min= 2456, max=301392, per=20.79%, avg=31303.32, stdev=16210.23
  write: io=1946.4MB, bw=10310KB/s, iops=644 , runt=600112msec
    slat (usec): min=5 , max=775660 , avg=1416.12, stdev=13835.21
    clat (usec): min=0 , max=15506 , avg= 5.28, stdev=98.38
     lat (usec): min=9 , max=775669 , avg=1463.56, stdev=14065.44
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   14], 99.50th=[   20], 99.90th=[   56], 99.95th=[  108],
     | 99.99th=[ 5728]
    bw (KB/s)  : min=    0, max=36666, per=20.33%, avg=10442.67, stdev=4680.36
    lat (usec) : 2=0.04%, 4=33.52%, 10=64.91%, 20=1.05%, 50=0.37%
    lat (usec) : 100=0.07%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.14%, sys=5.03%, ctx=19900, majf=0, minf=0
  IO depths    : 1=106.4%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1119034/w=0/d=386712, short=r=0/w=0/d=0
sla-1: (groupid=0, jobs=1): err= 0: pid=3679
  read : io=2070.3MB, bw=31495KB/s, iops=1968 , runt=600001msec
    slat (usec): min=6 , max=113475 , avg=28.48, stdev=439.57
    clat (usec): min=0 , max=15457 , avg= 4.94, stdev=87.28
     lat (usec): min=9 , max=113480 , avg=25.81, stdev=273.59
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   13], 99.50th=[   17], 99.90th=[   52], 99.95th=[   90],
     | 99.99th=[ 4832]
    bw (KB/s)  : min= 1030, max=316901, per=21.84%, avg=32883.20, stdev=16921.05
  write: io=2209.5MB, bw=10761KB/s, iops=672 , runt=600001msec
    slat (usec): min=10 , max=1066.3K, avg=1379.96, stdev=13435.94
    clat (usec): min=0 , max=19899 , avg= 5.10, stdev=95.42
     lat (usec): min=14 , max=1066.3K, avg=1400.95, stdev=13591.73
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   13], 99.50th=[   18], 99.90th=[   55], 99.95th=[  101],
     | 99.99th=[ 4256]
    bw (KB/s)  : min=    0, max=34413, per=21.18%, avg=10879.80, stdev=4909.55
    lat (usec) : 2=0.03%, 4=38.23%, 10=60.31%, 20=1.00%, 50=0.33%
    lat (usec) : 100=0.06%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=1.22%, sys=5.15%, ctx=20853, majf=0, minf=0
  IO depths    : 1=104.2%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1181074/w=0/d=403548, short=r=0/w=0/d=0
sla-2: (groupid=0, jobs=1): err= 0: pid=3680
  read : io=965264KB, bw=29567KB/s, iops=1847 , runt=600077msec
    slat (usec): min=0 , max=125310 , avg=31.57, stdev=502.02
    clat (usec): min=0 , max=18185 , avg= 5.27, stdev=101.74
     lat (usec): min=1 , max=119820 , avg=27.13, stdev=317.55
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   13], 99.50th=[   19], 99.90th=[   57], 99.95th=[  126],
     | 99.99th=[ 5536]
    bw (KB/s)  : min= 6042, max=236512, per=20.33%, avg=30599.81, stdev=14011.00
  write: io=1814.7MB, bw=10086KB/s, iops=630 , runt=600077msec
    slat (usec): min=3 , max=588111 , avg=1479.61, stdev=13888.56
    clat (usec): min=0 , max=14820 , avg= 5.14, stdev=89.13
     lat (usec): min=7 , max=588119 , avg=1495.16, stdev=14029.23
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   14], 99.50th=[   19], 99.90th=[   57], 99.95th=[  110],
     | 99.99th=[ 4768]
    bw (KB/s)  : min=    0, max=29628, per=19.85%, avg=10195.08, stdev=4223.96
    lat (usec) : 2=0.04%, 4=35.43%, 10=62.99%, 20=1.05%, 50=0.37%
    lat (usec) : 100=0.07%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=1.18%, sys=4.90%, ctx=21900, majf=0, minf=0
  IO depths    : 1=103.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1108905/w=0/d=378280, short=r=0/w=0/d=0
sla-3: (groupid=0, jobs=1): err= 0: pid=3681
  read : io=949584KB, bw=29541KB/s, iops=1846 , runt=600075msec
    slat (usec): min=0 , max=307604 , avg=32.73, stdev=602.53
    clat (usec): min=0 , max=20009 , avg= 5.13, stdev=92.41
     lat (usec): min=0 , max=307610 , avg=27.73, stdev=447.93
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   13], 99.50th=[   18], 99.90th=[   56], 99.95th=[  107],
     | 99.99th=[ 5216]
    bw (KB/s)  : min= 3597, max=200672, per=20.34%, avg=30614.39, stdev=14037.75
  write: io=1782.8MB, bw=10032KB/s, iops=626 , runt=600075msec
    slat (usec): min=0 , max=698050 , avg=1487.99, stdev=14000.71
    clat (usec): min=0 , max=15718 , avg= 5.10, stdev=88.03
     lat (usec): min=5 , max=698057 , avg=1501.56, stdev=14108.46
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   14], 99.50th=[   18], 99.90th=[   56], 99.95th=[  109],
     | 99.99th=[ 5088]
    bw (KB/s)  : min=    0, max=30716, per=19.76%, avg=10152.74, stdev=4375.22
    lat (usec) : 2=0.04%, 4=33.63%, 10=64.84%, 20=1.02%, 50=0.35%
    lat (usec) : 100=0.07%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.19%, sys=4.89%, ctx=21491, majf=0, minf=0
  IO depths    : 1=103.3%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1107925/w=0/d=376236, short=r=0/w=0/d=0
sla-4: (groupid=0, jobs=1): err= 0: pid=3682
  read : io=1261.7MB, bw=30115KB/s, iops=1882 , runt=600001msec
    slat (usec): min=6 , max=124223 , avg=32.96, stdev=508.43
    clat (usec): min=0 , max=23230 , avg= 5.22, stdev=101.67
     lat (usec): min=10 , max=92732 , avg=27.09, stdev=301.95
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   13], 99.50th=[   19], 99.90th=[   55], 99.95th=[  112],
     | 99.99th=[ 5792]
    bw (KB/s)  : min= 3206, max=181550, per=20.59%, avg=30998.59, stdev=13916.44
  write: io=1871.3MB, bw=10184KB/s, iops=636 , runt=600001msec
    slat (usec): min=10 , max=914196 , avg=1473.45, stdev=13936.44
    clat (usec): min=0 , max=19914 , avg= 5.09, stdev=95.95
     lat (usec): min=14 , max=914209 , avg=1479.06, stdev=14025.33
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
     | 99.00th=[   14], 99.50th=[   19], 99.90th=[   55], 99.95th=[   96],
     | 99.99th=[ 4128]
    bw (KB/s)  : min=    0, max=30611, per=20.07%, avg=10307.84, stdev=4437.44
    lat (usec) : 2=0.04%, 4=36.67%, 10=61.79%, 20=1.02%, 50=0.36%
    lat (usec) : 100=0.07%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.20%, sys=4.98%, ctx=22927, majf=0, minf=0
  IO depths    : 1=102.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1129322/w=0/d=381903, short=r=0/w=0/d=0


Run status group 0 (all jobs):
   READ: io=88223MB, aggrb=150538KB/s, minb=29540KB/s, maxb=31495KB/s, mint=600001msec, maxt=600112msec
  WRITE: io=30104MB, aggrb=51368KB/s, minb=10031KB/s, maxb=10761KB/s, mint=600001msec, maxt=600112msec


Disk stats (read/write):
  vdc: ios=35966/1239335, merge=5/2790, ticks=60780/91516964, in_queue=91586228, util=99.89%
 
Last edited:
From within a VM
Code: 
root@nas-test:~# time dd if=/dev/zero of=/media/vdc/dd.tst bs=2048000 count=20000 conv=fdatasync 
20000+0 records in
20000+0 records out
40960000000 bytes (41 GB) copied, 410.186 s, 99.9 MB/s


real	6m50.191s
user	0m0.092s
sys	0m49.051s
root@nas-test:~# echo 3 > /proc/sys/vm/drop_caches
root@nas-test:~# time dd if=/media/vdc/dd.tst of=/dev/null bs=2048000
20000+0 records in
20000+0 records out
40960000000 bytes (41 GB) copied, 381.625 s, 107 MB/s


real	6m21.779s
user	0m0.096s
sys	0m37.766s
The bottleneck is the network. Almost completely saturate the gigabit network;-) Higher numbers will require 10GB nics or fibre channel.
 
I have 3 servers (Intel(R) Xeon(R) CPU X3323 @ 2.50GHz 4 cores) running 25 KVM systems converted from Citrix XenServer 2 weeks ago.

Storage is currently a Promise m610i iSCSI, but I'll soon have my old XenServer storage moved over, which is a DRBD system with RAID-6 arrays on GigE.

Gerald
 
udo said:
Hi,
how you defined "large"?

I manage three cluster (3/3/5 hosts) with 3 - 9 CPUs (up to 68 cores + 336GB RAM in one cluster) - is this large? I think not - but it's an environment which is good for most things.
Click to expand...

Its all rellative I guess, certainly larger than our current environment, but if it all works out we will certainl expand the nodes.

udo said:
Proxmox as managing tool is good - I can do all things I want (but also use ofter the cli).
Click to expand...

Good to know. Quite comfortable with the cmd line myself. And anothe reason to rpefer proxmox over the likes of ovirt etc for me - much prefer debian.


udo said:
Backup: I use the builtin-backup only for disaster recovery - the daily backup is done by bacula (I have some VMs with some TB-disks).
Click to expand...

Yah, only looking at it for DR myself. We're installing online backup siftware for our actual data.
 
gbr said:
I have 3 servers (Intel(R) Xeon(R) CPU X3323 @ 2.50GHz 4 cores) running 25 KVM systems converted from Citrix XenServer 2 weeks ago.

Storage is currently a Promise m610i iSCSI, but I'll soon have my old XenServer storage moved over, which is a DRBD system with RAID-6 arrays on GigE.

Gerald
Click to expand...


How did you go with converting the XenServer VM's? Linux? Windows?

Currently trying to test migrate our AD/File server and its giving me real headaches.
 
My largest single cluster is 20 physical servers of varying setup, AMD and Intel.
Smallest node has 16GB RAM with 6 core CPU, largest has 128GB RAM with dual octo core Intel Xeon.
We use DRBD for most of our storage (16 nodes), the rest are local storage.
10G Infiniband is used for the Proxmox cluster/DRBD networks. 900MB/sec disk writes on DRBD, 600MB/sec live migrations.
All servers have RAID cards, most with BBU, some use 7200RPM SATA or SAS disks, some use SSD.
Running about 115 virtual servers with room for many more.

Just started testing CEPH in our lab, looks promising.

We backup some data in the VM itself such as databases and utilize the Proxmox backup for full full vm backups.
A hook script for vzdump is used to mount/unmount our encrypted backup disks, the hooks allow for enough customization to adjust the Proxmox backup to most users needs.

The only complaint I have about the GUI is sometimes it seems a little sluggish when using it from lower end workstations/laptops.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back