Build your own workstation

17 April 2018

A recent endeavour of mine has been to build a Linux workstation, for my work in the Data Science and Machine Learning space. The primary technical requirement for this machine is full support for two GTX 1080 Ti graphics cards.

What do I mean by full support? The answer to that question relates to several aspects of building a well-balanced system, but primarily comes down to bandwidth.

PCI Express bandwidth

While many motherboards do support multiple graphics cards, your typical consumer-level motherboard / CPU will have very limited PCI Express bandwidth, which may result in a bottleneck between the chipset/CPU and PCI Express devices.

This bandwidth is measured in ‘lanes’, and at time of writing, a typical motherboard will support 16 lanes of PCI Express 3.0 bandwidth to/from the CPU, and another 8 lanes of PCI Express 2.0 bandwidth to/from the chipset. This is shared between the devices in the system. So in this case, installing two GTX 1080 Ti graphics would cause each card to run at 8x (8 lanes) instead of 16x.

Choosing a motherboard and CPU

Choosing a motherboard and CPU that can run two GTX 1080 Ti slots at 16x was an interesting exercise in studying technical specs. There were a number of high end gamer motherboards that would do the job, and some very expensive workstation motherboards (such as the ASUS X99-E WS, which supports up to 4 GPUs at 8x).

Ultimately, I chose the lesser ASUS X99-E, since I have no intention of installing more than two GPUs. This board supports up to 40 lanes of PCI Express 3.0 bandwidth from the CPU. In a stroke of good fortune, I was even able to source that motherboard second-hand, which did a lot to keep my budget in check.

The choice of CPU was easy. The Intel i7-6850K was the most affordable 40-lane CPU that would fit the LGA 2011-3 socket on the ASUS X99-E.

GTX 1080 Ti

Another interesting consideration is the choice of graphics card. Why not a Titan X?

With 12GB of GDDR5X RAM and 3584 CUDA cores, NVIDIA’s Titan X made a name for itself as the graphics card for machine learning. There are many workstation builds featuring two (or even four) Titan X GPUs.

However, NVIDIA has since released the GTX 1080 Ti, a more affordable variant with the same GP102 GPU used in the Titan X. Both cards are based on NVIDIA’s Pascal architecture, with the primary difference being that the GTX 1080 Ti has ‘just’ 11GB of GDDR5X RAM.

Benchmarks show that the performance of these cards is very similar, which meant that the GTX 1080 Ti would be a good balance between memory, performance and price. GTX 1080 Ti cards were also more readily available, both new and second-hand.

Bill of parts

These are the new parts that eventually came together:

Intel i7-6850K CPU
ASUS X99-E motherboard
32GB of Corsair Vengence DDR4 2400Mhz RAM
EVGA GTX 1080 Ti SC2
Corsair RM1000x 1000W power supply
Noctua NH-U9S CPU cooler

The RAM was chosen as a kit of 4x 8GB, so that it could run in quad-channel mode. Kits that run at a higher frequency are available, but the i7-6850K is limited to 2400Mhz without some form of over-clocking.

There were also a few parts that I re-used:

Lian Li case
Western Digital 4TB NAS HDD
Patriot Torqx 128GB SSD
D-Link DWA-566 Dual Band Wireless adapter

The Torqx SSD is one of the parts that I was unsure about. First purchased in 2009 to breathe new life into a Macbook Pro, it is now a bit less impressive and glamorous than some of the modern M.2 SSDs. But it has served me so well over the years, that it seemed appropriate to use it as the system disk in this build. (More pragmatically, it already had my working Ubuntu installation on it, so that would save me the hassle of installing from scratch.)

The 4TB NAS drive was salvaged from a NAS unit that I found dumped on the street.

Installation and first boot

The build was pretty straight-forward, if you’ve ever built a PC. However booting up Linux was more difficult. The motherboard came with Secure Boot enabled, which prevented my the custom-compiled NVIDIA driver module from being loaded. This is easy enough to disable via the UEFI.

The only other UEFI-related change I made, was to allow the CPU fan speed to be adjusted automatically, using variable voltage control. This ensures that, when idle, the system is very quiet.

Is it actually a workstation?

While building this machine, it occured to me that prevalance of powerful PCs with high-end graphics cards might make the term ‘workstation’ obsolete. Skimming the Wikipedia article on Workstations, I found a list of criteria that can be used to justify use of the term:

Support for ECC memory
Larger number of memory sockets which use registered (buffered) modules
Multiple processor sockets, powerful CPUs
Multiple displays
Run reliable operating system with advanced features
High performance, reliable graphics card

The ASUS X99-e motherboard supports ECC memory, has eight memory sockets, and allows up to 128GB of RAM running in quad-channel mode. The board was chosen primarily for the Intel X99 chipset which, with 40 lanes of PCI Express bandwidth, can provide a x16 links for two graphics cards.

The CPU is an Intel i7-6850K. This too was chosen for its 40 lanes of PCI Express bandwidth, but is a damn fine processor on its other merits. The combination of six cores, 15MB of L3 cache, and a base frequency of 3.6 Ghz leave me hopeful that this processor will have a useful life measured in decades.

Last but not least is the graphics card…

The EVGA GTX 1080 Ti SC2 has 11GB of GDDR5X RAM, and 3584 CUDA cores. Despite being marketed as a gaming card, these specs make it comparable to the Titan X (which has 12GB) while being much more affordable. At some point in the not-too-distant future, when prices come down and/or my budget allows for it, I plan to install a second graphics card.

I think this system makes the cut.