Storinator & 10Gbit Networking

From 45 Drives Technical Information Wiki
Jump to: navigation, search

Due to falling costs of components, relative ease of setup and an increasing demand of bandwidth, 10GbE networking is poised to enter mainstream adoption. This report details the basic components needed for your own 10GbE network as well as our own experience with 10GbE here at 45 Drives

Intro/What is 10GbE?

The IEEE standard for 10 Gigabit Ethernet (10GbE) was approved eight years ago and almost immediately, large enterprises started installing 10GbE in their corporate servers and datacentres to support high-bandwidth, mission critical applications. However due to the cost needed to replace the old networking infrastructure and the lack of necessity for such high transfer speeds smaller users outside the enterprise world never saw the need to implement 10GbE. But over the few years, improvements in 10GbE infrastructure price and performance have extended its reach beyond enterprise data centers to medium sized businesses and users requiring much higher bandwidth capabilities such as video production companies.

With companies such as Netgear making more affordable 10GbE switches (~$100/port) and the price of 10GbE Network Interface Cards (NIC) ranging from $250 to $1000, implementing a 10GbE network is becoming more realistic.

This report will walk through the pieces needed to construct a basic 10GbE network as well as document the setup of our own 10GbE network here in the 45 Drives lab.

What Infrastructure do I need for 10GbE?

Keeping things simple, there are four main pieces to 10GbE network, (1) Cables, (2) Switches and (3) Network Interface Card(s). Here’s a quick description of each:

10GbE Cables

You have two options when it comes to cable material, (a) Copper and (b) Optical Fiber.


(a) To choose the right copper Ethernet cable for a 10GbE network one must consider three options: Cat6, Cat6a and Cat7. Cat6 is the norm right now and it’s specified to be able to handle 1GbE over a 100m run, it can also support 10GbE over a 56m run however is very susceptible to crosstalk from nearby Ethernet cables dude to the high frequency of the transmission. We have verified in our lab that Cat6 can indeed function on a 10GbE network however should only be used in pinch as Cat6a is much more suited for 10GbE then Cat6. Cat6a has improved performance in the areas of reduced crosstalk at high frequencies and the ability to run 10GbE across 100m of cable. Cat7 is the newest of the Ethernet cables and has improved crosstalk reduction than that of Cat6a. Besides that however there is no real noticeable difference between Cat6a and Cat 7. Therefore Cat6a is the ideal choice for 10GbE copper cable due to its specification of 10Gb/s across 100m runs and its price point (~$0.25/ft compared to Cat7 ~$0.65/ft). Keep in mind that not all Ethernet cables are created equal and while the spec says one thing it depends on the quality of the manufacturer. Always verify the cable manufacturer before you buy a bulk amount of cable as sometimes the price is too good to be true.

(b) The main advantage of using fiber over copper is that you are no longer restricted by cable length. Single mode fiber (SMF) is used in very long runs (>400m) and multimode fiber (MMF) is used for shorter runs. There are 4 types of MMF, OM1-4 but the ones that interest us are OM3 & OM4 as they are the cables that can support 10GbE up to 300m and 400m respectively. OM4 is to OM3 as Cat6A is to Cat6, both do the job however OM4 can reach longer and is considered future-proof as it can support both 40GbE and 100GbE up to 150m and is back compatible with OM3. There is a lot more to consider when choosing the ideal fiber solution for your 10GbE network, such as laser type and what kind of transceivers to use. This however is outside the scope of this report.

Switches

When considering a switch for your 10GbE network, price per port is likely going to be your main concern. If you are building a enterprise level network chances are you already know a fair bit about 10GbE networking and will require a heavy duty switch, typical vendors are Cisco, Arista, Dell, etc… these are quite expensive though and range anywhere from $400/port to $1000/port. Although, with these switches you’re also paying for full-support and attention of the vendor you bought it from.

Companies like Netgear, however are making 10GbE switches cheaper and more available to everyday consumers. These switches are in range of ~$100/port but you don’t receive the around the clock support you would get out of paying for an enterprise level switch. So when buying a switch, go with the cheapest price per port and at this point of time Netgear has the cheapest 10GbE switch (XS708E) priced at $850 for 8ports unmanaged. We have been using a 10GbE XS708E Netgear switch in our lab and it preforms wonderfully.


Network Interface Cards

A 10GbE NIC is a PCIe expansion card for your server/computer that allows it to send and receive across the network at 10Gb/s provided you have the necessary 10GbE components (cables, switch). There are many manufacturers of 10GbE NICs out there however in our lab we have been using a two port “Intel X540-AT2” they have been performing great and very simple to use, no low level configurationuring needs to be done to use them and as of now these NICs are ~$500. Cheaper cards are available from manufacturers such as Femrice.

If you are planning a 10GbE network using copper cables make sure the NIC you choose terminates with RJ45 ports rather SFP+, as you would then need to buy a special SFP+ to RJ45 adapter.

Our Experience

So far I have explained the parts needed for your own basic 10GbE network but you may be wondering “That’s cool and all, but what’s the benefit really?” or “Yea, those parts are getting cheaper but it must be difficult to actually set them up and use them”. To address these concerns we are going to show how easy it is to set up your own 10GbE network as well as how much faster we can move files on 10GbE network rather than 1GbE.

Our Setup

In the 45Drives lab we have built our own basic 10GbE network using Cat6 & Cat7 cables, a Netgear 8port 10GbE switch(XS708E), 10GbE Intel NICs (X540-T2) and 4 Storinators. See a full list of what’s in each Storinator in the section below. This network coexists with our current 1GbE network as well. The diagram below illustrates the superposition of the two networks.

Lab setup.png
Figure 1 Network Diagram of our lab setup.

Each Storinator has two active network connections, a 10GbE connection between the five machines and a 1GbE connection with themselves and the internet. These two networks each have separate IP ranges as they are two completely different networks. Each machine on the 10GbE network have IP statically set in 10.0.0.(1-5) and the 1Gb/s network is set up as DHCP and dynamically assigns IPs to each Storinator.

I first started by setting up the Host Storinator by inserting the 10GbE Intel NIC into the HOST Storinator and booted up. I statically set the IP of the 10Gbit ports to “10.0.0.1”. The HOST then can then be located on the 10GbE network at that IP. I then did this for each Storinator Client by doing the same steps but incrementing the IP address for each.

I didn’t do anything to set up the 1GbE network except plug the Ethernet cord into the onboard 1GbE port as our DHCP server handles the IP assignments.

Once each Storinator had a 10.0.0.x IP I connected the Ethernet cables to the appropriate ports on the Storinator(s) as well as the 10GbE switch and pinged each address from the host to make sure everybody was connected. In this scenario I disabled the firewall on each machine by running the command “service iptables stop”, but in real world applications this is not the safest method and exceptions should be added into your firewall rules rather than simply turning it off, however in this case study I accepted this risk.

Now that the HOST Storinator and the three Client Storinators are set up on a 10GbE network we are ready to transfer files.

What’s in Each Storinator?

HOST STORINATOR (45 pod):

  • 10gbe Intel NIC
  • 45x 1TB Seagate Constellation ES.3 in a RAID0 configuration
  • CentOS 6.6
  • SuperMicro X9 SCM-F Motherboard
  • Intel i3-3240 CPU
  • 24GB RAM
  • Cat 7 Ethernet Cable

CLIENT 1 STORINATOR (45 pod):

  • 10gbe Intel NIC
  • 45x 1TB Seagate Constellation ES.3 in a RAID0 configuration
  • CentOS 6.6
  • SuperMicro X9 SCM-F Motherboard
  • Intel i3-3240 CPU
  • 8GB RAM
  • Cat 7 Ethernet Cable

CLIENT 2 STORINATOR (30 pod):

  • 10gbe Intel NIC
  • 30x 1TB Seagate Constellation ES.3 in a RAID0 configuration
  • CentOS 6.6
  • SuperMicro X9 SCM-F Motherboard
  • Intel E3-1275 CPU
  • 8GB RAM
  • Cat6 Ethernet Cable

CLIENT 3 STORINATOR (30 pod):

  • 10gbe Intel NIC
  • 30x 1TB Seagate Constellation ES.3 in a RAID0 configuration
  • CentOS 6.6
  • SuperMicro X9 SCM-F Motherboard
  • Intel i3-3240 CPU
  • 8GB RAM
  • Cat 6 Ethernet Cable

Our Experiment

To demonstrate the speed increase mentioned above we are going to transfer a 206GB folder of files from our HOST Storinator to our Client Storinators, first on a 1GbE network then on a 10GbE network.

1GbE

In the first case, transferring the files via 1GbE, I will transfer from the HOST to a single Client rather than all three. I chose to do this simply based on how long the transfer would take if all three pods shared the 1GbE connection, and the fact that a single Client Storinator was more than capable of saturating the 1GbE network.

So I opened a FTP connection between the HOST and Client 1 and transferred the folder of files.

The file transferred at 110MB/s, which is the approximately the maximum a 1GbE line can handle considering file system overhead. It took about 30 minutes to complete the transfer of the 206GB files from the HOST to the one Client.

If I would’ve done this transfer to all three Clients simultaneously the bandwidth would be shared between them and the file would only transfer at a third of rate that did with only one client, therefore would’ve taken approximately 1.5 hours.

1gig transfer.jpg
Figure 2 File transfer across 1GbE network

10GbE

In the second case, transferring files via 10GbE, I will transfer from the HOST to all three clients simultaneously. I chose to do this for two reasons, first to show you how much faster we can move files across the 10GbE pipe and second to highlight that the HOST Storinator is capable of saturating a 10GbE line.

So like the 1GbE transfer I opened a FTP connection between the HOST and the clients and transferred the same folder of files to each client simultaneously.

Now each connection from HOST to client does not share the whole 10GbE pipe equally, however the sum of each links transfer speed will add up to ~1.10GB/s which is approximately the maximum a 10GbE line can handle considering file system overhead. It took just over 10 minutes to complete the transfers of 206GB files from HOST to each of the clients simultaneously.

10Gig transfer.jpg
Figure 3 File transfer(s) across 10GbE network

Comparisons

It is clear that 10 GbE allows for much quicker transfer speeds due to the higher bandwidth than 1 GbE. It took just over 10 minutes to transfer three 206GB files to three machines on a 10GbE network instead of the agonizing 1.5 hours it would’ve taken to do the exact same thing on a 1 GbE network. This experiment also shows that our Storinators are very capable machines that have the bandwidth to saturate a 10 GbE line.

Conclusions

Until now 10GbE networking has been confined to large enterprises and their corporate servers and datacenters. However due to the small to medium sized businesses increasing need for higher bandwidth, combined with the fact manufacturers are producing more and more lower cost 10GbE components, we here at 45 Drives feel that 10Gbit networking will quickly become industry standard.

In this report we showed you all the pieces your need to get your own 10GbE network up and running, how simple it is to set up and how much faster it is than 1GbE networking. In the amount of time it took to transfer our file across a 1GbE network to three clients we could do the same transfer on a 10GbE network 9 times.

And last but not least we got to showcase the beast that is the Storinator; it has the bandwidth capabilities to saturate a 10GbE line and enough storage for a lifetime. That coupled with our very modular design I’m confident in saying our Storinators are future proof!