The main component of your deep learning rig is the GPU. GPU's enable highly parallelized matrix operations, which are very important for deep learning operations. It would be ideal to have roughly 20GB of GPU memory for larger models that you are trying to run.
The TITAN GeForce is near top of the line for deep learning applications and research. You should have no problem running neural networks at blazing frames per second with these, totaling ~24 GB of GDDR5X RAM. It comes with an incredible 3584 cores per gpu, enabling fast parallelization of linear algebra operations.
On the slightly less expensive side, there is the GTX 1080 Ti. This is based on the new Pascal and has shown to have increased performance and benchmarks for deep learning experiments, often times matching or surpassing the Titan X. However, the catch is that the 1080 Ti only has 11GB of memory compared to the TITAN X's 12GB per GPU. However, I think this is the more cost effective option at the moment, expecially with cryptocurrencies driving up GPU prices.
For CPU's you want something that can load and augment data fast, especially if your dataset can't fit in RAM. You don't want your workflows to be hindered and slowed down by the CPU.
As for CPU, you will want something like an Intel i7 processor. The Intel i7 4.00 GHz Quad Core Skylake should do a great job at keeping your system fast at preprocessing and augmenting your data on the fly.
One important thing to remember here is that you want to ensure that you have enough PCIe lanes. As failing to have enough of these in your CPU can be a bottleneck for multi-GPU builds.
Another good option is the slightly slower Intel i5 3.50 GHz Skylake
If you really want to go all out, the AMD Ryzen Threadripper 1950X is about the best that money can buy currently. It is great value for 16 cores and 32 threads of processing power at 4.0 GHz, all while boasting 64 PCIe Gen3 Lanes.
Here you want sufficient space to keep your small/medium sized datasets in memory, which can help speed things up quite a bit. I've heard good things about this 32GB kit from Ballistix.
Here you might even want to opt for two of these, especially if you are doing computer vision related tasks. You can improve training and inference speed by increasing your batch size. To do so takes more RAM typically.
Any modern high performance computing environment needs solid state memory. Seriously. If you sum up all the time required to launch IDE's, notebooks, copy and write data, you can save hours of your time by using an SSD. I recommend a Samsung Internal SSD.
I do a lot of programming and video editing in my free time. I prefer an operating system that has a linux/unix style command line interface, with high performance while looking good. The Dell XPS 13 8th Gen fulfills all these needs. It has an infinity edge display with almost no bezels, a solid state hard drive, 8GB or 16GB of RAM, and the new 8th generation intel CPU. I plan on installing Ubuntu or Debian on the laptop, as it comes with Windows 10 installed. I may have to pounce on this laptop during the holiday season.
I will be continually adding things here as time goes on and I discover new parts.
Until next time,