3 Node Proxmox 10GB interfaces without 10GB switch

[Digitaldaz]

I have a three node cluster with three dual 10GB in each plus 4 x 1GB ethernet.

There is no 10GB switch. We just use DAC cables.

Each node has 2 x 10GB connected to the other two for ceph.
Each node has 2 x 10GB connected to the other two for cluster replication.
Each node has 1 x 10GB connected to a backup server.

This setup has been working just fine and had about a 750 day uptime and I have been very pleased with it.

We have had a problem arise where one of the dual port ceph 10 GB NICs has failed on one server. The cluster didn't handle it very well as the box was still up and running just fine. Once I managed to kill the box through the IPMI, HA kicked in, moved the VMs and now everything is running fine again just across two servers.

I have got the replacement parts on the way right now.

My two questions are:

Can I just replace the broken NIC and just bring back up the third box. I think the answer to this is yes.

Also, can I just add another dual 10GB NIC to each box and then just add the ports to the existing bond I have for ceph, thus giving them extra redundancy for the future?

Thanks
Daz

 

[LnxBil]

Can I just replace the broken NIC and just bring back up the third box. I think the answer to this is yes.

If it's the same slot and the same number of ports, it should be a drop-in replacement. Otherwise, you need to check the slot-dependent naming scheme for network interconnects (the names of the network devices could change).

Also, can I just add another dual 10GB NIC to each box and then just add the ports to the existing bond I have for ceph, thus giving them extra redundancy for the future?

Always a good idea.

There is no 10GB switch. We just use DAC cables.

The big downside of this approach is that if a node goes down, the port on the remote end goes down aswell. Therefore, multiple cluster software vendors (e.g. Oracle RAC, also OCFS2) require a switch so that only the "up" or "down" state can be enough to trigger a host reboot.

 

[Digitaldaz]

Thanks, I should be good with the replacement card as they are the onboard integrated things so hopefully all the identifiers will be the same.

Even the extra nics will probably be the same as they'll be going in the same slot on each server.

With regards to the statement:

The big downside of this approach is that if a node goes down, the port on the remote end goes down aswell. Therefore, multiple cluster software vendors (e.g. Oracle RAC, also OCFS2) require a switch so that only the "up" or "down" state can be enough to trigger a host reboot.

This doesn't relate to Proxmox though does it?

 

[LnxBil]

This doesn't relate to Proxmox though does it?

Depends on what you use. Some cluster filesystem could need it. But if you bonded your connection together, you will be fine.

Have you connected the boxes via twinax or GBIC+FC-cable?

 

[Digitaldaz]

Depends on what you use. Some cluster filesystem could need it. But if you bonded your connection together, you will be fine.

Have you connected the boxes via twinax or GBIC+FC-cable?

Twinax

I think one of the connections is GBIC+FC but not on the Ceph. I didn't actually realize this made a difference. I would have only done that because I was short of a twinax DAC cable but I can very easily rectify it if needed.

I can swap them all out to whatever is best. I hope its twinax as I've just order another three Intel X520-DA2s with twinax to use as redundancy

Thanks
Daz

 

[LnxBil]

I can swap them all out to whatever is best. I hope its twinax as I've just order another three Intel X520-DA2s with twinax to use as redundancy

I'd also go with twinax. It is so much cheaper and as good as the FC stuff with respect to speed. You will IMHO not have galvanic separation, but the servers are most of the time connected to the same power rails or copper based 1 GBE ethernet switch.