install ELK on debian, below KVM and LXC ?

[lapapa]

is it possible to install Elasticsearch (or ELK) directly on debian (not KVM or LXC) for perf reason ? That combination would be perfect for us, elk on HW, then lot of services in vm-s and containers.

 

[ness1602]

You mean directly on PVE host? Yes, it is possible, same as every package. What are you perf reasons that you want to evade VM or LXC?

 

[Dunuin]

My ELK stack is running inside a LXC. I don't see why running it on the host should give you way better performance compared to a privileged LXC.

 

[lapapa]

Hi,

thanks for answers, sorry for delay.

our elk is heavy on IO, around 0,5TB input per day, ~ 250 TB in ES. Hot+warm ES nodes, SSD + spinning disks. We had problems with perf. https://discuss.elastic.co/t/warm-n...ling-out-of-cluster-during-recovery/250078/14 . So, that's way i'm trying to avoid anything not necessary between ES and disks. ES and logstash on PVE, almost everything else on LXC and exceptions on vm.

 

[ness1602]

Yeah, okay this is serious ES cluster, in my opinion logstash is really resource hungry, sometimes even more that Elasticsearch.
Usually i only give 30% of host RAM to Elasticsearch, so i recommend you do the same on PVE nodes.

 

[lapapa]

yes, logstash is heavy cpu user with grok, netflow ... Our setup: dell 740xd, 12C/256 RAM, only one redhat, 2 ES nodes (2 *31GB heap), 3 logstash instances (12g heap), and lots of RAM for cache

 

[ness1602]

Regarding netflow(since i work in netflow company) , do you really need raw data, or do you aggregate data in 3 hours/days/monts?
Our competitor Elatstiflow stores raw data,and it is really inefficient in the long run.

 

[proxmox_novice]

I was looking around some example on ES on Proxmox. Didn't find much out there. I have finally setup 10xR630 Proxmox cluster and two LXC on a node for first five system. which has dual NVME. using one nvme per LXC (bind mount) as direct mount or pass through does not give enough write performance.

Writing directly to host and LXC gives me same write/read performance.

10xProxmox cluster, 5x has NVME, 5 are as is

10xLXC for data
3x LXC for master
3xLXC for logstash
2xLXC for kibana