Community Help Request, Purchasing Suggestions and Options (All Opinions Welcome)

Upon review I suppose it is using ZFS, though there's no mirror or stripe going on for the pool, just a single large virtual disk.... Come to think of it I remember now I had to disable write sync in the VM and set the disk image in proxmox to "write back" to get decent write performance to TrueNAS in this implementation. It's been about a year since I configured that so had completely forgotten. Either way, with those adjustments it works pretty well, but I wonder if there's room for improvement with a "stripe" of virtual disks on separate threads? hmmmm... tbd...

With that said, Ceph is agnostic to what the VM does with the virtual disk. Data is data. You could encrypt the entire contents of the disk and ceph would not know any difference and would still perform scrub operations just the same looking for data integrity issues. Data that doesn't match across replications is discovered and can be repaired either way.
 
I'm selling all my gear, and whatever I deem unneeded and starting fresh.
Thanks for the price tag on your build AllanM.

I'm Canadian so $3-4g doesn't exactly translate into the same Canadian.

I'm going to develop a game plan then.

Selling all my gaming gear should fetch me about 6-10 grand Canadian if I'm lucky.

Of course that's only 4.7-7.8 USD

Once I have the cash in hand I'll let you know how much I can spend and what the server market looks like on ebay.

It might take a month or two, so hopefully this thread does not got locked.
 
I'm selling all my gear, and whatever I deem unneeded and starting fresh.
Thanks for the price tag on your build AllanM.

I'm Canadian so $3-4g doesn't exactly translate into the same Canadian.

I'm going to develop a game plan then.

Selling all my gaming gear should fetch me about 6-10 grand Canadian if I'm lucky.

Of course that's only 4.7-7.8 USD

Once I have the cash in hand I'll let you know how much I can spend and what the server market looks like on ebay.

It might take a month or two, so hopefully this thread does not got locked.
You might want to keep the i9 rig to use as a starting point for the hackintosh stuff. For a "workstation" hosting machine, a fast workstation/gaming computer is still a good thing to maximize application performance. It's just not a good arrangement for hosting server VM's from.
 
I don't know if you have any way to "import" a purchase over the boarder (maybe a friend who could act as a middle-man?), but I found something on ebay that might be of interest:

https://www.ebay.com/itm/SuperMicro...315714?hash=item2ae3228102:g:7J4AAOSwL7leYX40

It's basically a proxmox cluster waiting to happen....

Here's an approximate parts list to get the gears turning...

"barebones" F627R3-RTB ~$600
4 X AOC-CTG-i2T 10Gb X540 micoLP ethernet cards https://www.ebay.com/sch/i.html?_from=R40&_nkw=AOC-CTG-i2T (~$210 on ebay)
8 X CBL-0231L for microLP card support readiness (~$40)
16-32 X 16GB 2RX4 DDR3 ECC RDIMM's, 1333MT/s or faster. (~$400-800 on ebay)
8 X E5-2667 V2 or 2670/2680/2690 V2 (optional CPU upgrade, could wait till later, ~$600-1000 on ebay)
4 X PCIE to 2 X M.2 adapter (1 X NVME 1 X SATA) (https://www.ebay.com/itm/M-2-NGFF-t...ual-SSD-PCI-Express-Adapter-Card/124338877312 ($40 on ebay)
8-10 X 3.5" to 2.5" sleds.... MCP-220-93801-0B ($150) (8+2 spares)
8-10 X 500GB 2.5" SATA SSDs... something with full data path power protection would be nice if possible. (8+ 2 spares, ~$400-900)
4-5 X 500-1000GB M.2 NVME SSDs.... something with full data path power protection would be nice if possible (4+1 spare ~$250-750)
4-5 X 64-256GB M.2 SATA SSDs... (boot drives, ~$150)
12-24 X 8TB ST8000NM0055 HDDs (~$2000-4000)
Adjustable depth 4 post rack on wheels (something tallish to get it all up off the floor): https://www.ebay.com/itm/Sysracks-3...rver-Rack-Adjustable-Depth-24-36/224265333891 (~$200 on ebay)
1-2 20-52 port managed switches with 4X10G uplinks (use 10G for ceph network, and if you have 2 switches with 10G uplinks, use the other 10G for the LAN trunks on the other 10G port on those microLP cards. (~$200-500 on ebay, don't forget the sfp+ to rj45 trancievers)

Resulting configuration would be a cluster with 48-80 cores, 256-512GB RAM, 6-7.5TB SSD pool (/3=2-2.5TB "usable"), 96-192TB HDD pool (/3=32-64TB "usable"), 10Gb storage network.

Total cost ~ $4640 - $9340 USD
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!