1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

10GbaseT Switch Recommendation

Discussion in 'Proxmox VE: Networking and Firewall' started by Gurn_Blanston, Feb 16, 2017.

  1. Gurn_Blanston

    Gurn_Blanston Member
    Proxmox VE Subscriber

    Joined:
    May 13, 2016
    Messages:
    42
    Likes Received:
    0
    Greetings,

    We just had a weird event happen when shutting down our third "quorum" host, which is being replaced. We use IPoIB on some old Mellanox IB cards and a Mellanox IB switch. The result was that one of the two remaining hosts went into kernel panic and had to be powercycled. In order to have faster VM migrations, we moved the cluster network over to IB several months ago, which I understand is no longer necessary as of PVE 4.4. At any rate, this sudden and unexpected outage has resulted in enough of a human panic to justify scrapping IB for 10GBaseT. The reason we were on IB in the first place is that 10G switches are super expensive, at least the managed ones from the likes of Dell/Cisco, etc.

    Are any of you out there using the cheaper Netgear switches? Such as
    NETGEAR ProSAFE 8-Port 10-Gigabit Ethernet Web Managed Switch (XS708E-200NES)

    This switch costs around $700 USD and we could make due with 8 ports. I have found mixed reviews regarding cheap 10G switches. Any positive or negative experiences and with what hardware?

    Regarding NICs, we have had success with Intel X540T2 to connect two hosts without any switch so I imagine we will stick with those adapters.

    Thanks!
     
  2. tschanness

    tschanness Member

    Joined:
    Oct 30, 2016
    Messages:
    143
    Likes Received:
    7
    The small Netgears are not Stackable (or at least: were). Apart from that I found them really easy to use perform quiet good.

    Jonas
     
  3. tom

    tom Proxmox Staff Member
    Staff Member

    Joined:
    Aug 29, 2006
    Messages:
    11,669
    Likes Received:
    126
    A bit more expensive, Cisco SG350XG-2F10 12p 10GBase-T Stack Switch - we use one in our testlab, works very good.

    We also got an cheaper Netgear since two year, also working without problems but this one is quite noisy.
     
  4. Nikolay

    Nikolay New Member

    Joined:
    Jul 16, 2016
    Messages:
    16
    Likes Received:
    1
  5. Gurn_Blanston

    Gurn_Blanston Member
    Proxmox VE Subscriber

    Joined:
    May 13, 2016
    Messages:
    42
    Likes Received:
    0
    Thank you all for your time and attention! We splurged and went with

    NETGEAR ProSAFE 8-Port 10-Gigabit Ethernet Smart Managed Switch (XS708T-100NES)

    It was around $1000 USD and has the word "managed" in the description. There is no serial console so it isn't as managed as you might hope but so far it is able to keep up with our spindle based zpools. We don't have anything that can write steadily at more than 500 MB/s but it seems to handle that easily. It is a single point of failure and has only a single power supply so we don't push cluster traffic on it. It is primarily used to connect our three hosts to an old re-purposed Precision T7500 stuffed with RAM and cheap 5 TB disks for backups.

    I think it is worth mentioning that all of our NFS timeouts and surginess has gone away since abandoning Infiniband. I don't know if it was due to our old eBay IB hardware or driver problems but we never did get IPoIB to run in a trustworthy fashion.

    Thanks again for the advice,

    GB
     

Share This Page