{"id":5282,"date":"2018-10-23T12:14:21","date_gmt":"2018-10-23T10:14:21","guid":{"rendered":"https:\/\/new.cloudandheat.com\/gpu-cloud-showcase-neural-style-transfer-parameter-study-using-multiple-nvidia-p100-gpus\/"},"modified":"2023-03-08T16:04:23","modified_gmt":"2023-03-08T15:04:23","slug":"gpu-cloud-showcase-neural-style-transfer-parameter-study-using-multiple-nvidia-p100-gpus","status":"publish","type":"post","link":"https:\/\/www.cloudandheat.com\/en\/gpu-cloud-showcase-neural-style-transfer-parameter-study-using-multiple-nvidia-p100-gpus\/","title":{"rendered":"GPU cloud showcase: Neural Style Transfer parameter study using multiple Nvidia P100 GPUs"},"content":{"rendered":"

As a showcase for Cloud&Heat\u2019s upcoming GPU cloud, this blog post summarizes the results of a hackathon organized by the IT team at Cloud&Heat. We set up and ran a Deep Learning application on four Nvidia Tesla P100 GPUs which we have in our lab for testing purposes.<\/span><\/p>\n

Introduction<\/span><\/h2>\n

Deep Learning is one branch of the current Machine Learning buzz that has a multitude of diverse applications as well as a hunger for computational resources. It is used for image and speech recognition, translation or autonomous driving, just to name a few. In terms of hardware, GPUs remain the workhorse of Deep Learning, since both training and inference make use of matrix operations which can be accelerated on these devices. Despite Google\u2019s TPU cloud, GPUs have a far greater application domain in other computationally demanding applications.<\/span><\/p>\n

The application that we will use here is termed Neural Style Transfer.<\/span><\/p>\n

\"Cloud<\/p>\n

It uses deep neural networks to combine the content information of one image (left) with the artistic style of another (artwork in the middle) in order to make it look as if it was painted by the same artist1<\/sup><\/a>.<\/p>\n

The idea to do this with deep learning was kicked off in 2015 by a paper called A Neural Algorithm of Artistic Style<\/a> by Gatys et al.<\/p>\n

One basic ingredient is the usage of pre-trained deep convolutional neural networks such as the VGG19 network<\/a>, which was trained for the ImageNet Large Scale Visual Recognition Challenge (ILSVRC)<\/a> to categorize images into 1000 different classes. Since the trained networks have developed an abstract representation of objects as well as their visual representation, one can use that to interrogate the network when feeding it an image. Gatys et al. showed how to extract the essential content information (e.g. the features of a face), as well the artistic style (e.g. brush strokes). Now one can use an optimization algorithm starting from a white noise image (or the content image) with the objective that the final image shall have both the desired content and style.<\/p>\n

This optimization-based approach has the advantage that it can combine any given content with any style image. A faster approach was given by Johnson et al.<\/a> and later refined by Ulyanov et al.<\/a> where they train a second \u201ctransfer network\u201d to solve the optimization in one forward pass. Once trained, the method is very fast, in fact one can even transfer videos in real time. However, the style image and all free parameters of the transfer are fixed. Also, to train the transfer network, one needs a large database of content images. Since we need a method to quickly load our GPUs, we go with Gatys\u2019s method, were we only need one content and one style image.<\/p>\n

There are many implementations of Gatys\u2019 method such as this from Keras<\/a>, this from Johnson<\/a> or this from Athalye<\/a>. We chose the latter TensorFlow<\/a> implementation<\/a> since it has a couple of parameters to play with. We use this as a nice showcase to illustrate a problem that everybody<\/em> has to solve: parameter tuning. There are at least six parameters in our chosen implementation that influence the generated image. The ratio in which to weight style and content is just one example (hint: 50:50 is not it). With the computational power we have available, we can run many style transfer calculations in an orderly fashion to scan for the best parameters. Note that also the fast method of Johnson from above has to do this first in order to define the objective function for training the transfer network.<\/p>\n

There are also newer advanced methods for Style Transfer based on Generative Adversarial Networks (CycleGANs<\/a>) with impressive results which we won\u2019t use here, however.<\/p>\n

Before we go on, we\u2019ll tell you about the hardware and software that we have used. If that is not your cup of tea then skip ahead \u2014 we also have Jupyter notebooks down there!<\/p>\n

Hardware and Operating System<\/h2>\n

Our test machine is composed of a Supermicro chassis, a Supermicro X10DRG-OT+-CPU motherboard, 24 GB RAM and 24 CPU cores in a dual socket Intel Xeon E5-2650 setup. The case houses 4 Nvidia Tesla P100 GPUs (Pascal architecture), connected via PCIe (no NVLink). These cards are datacenter-grade devices, packing 16 GB of RAM per card and having 5, 11 and 21 Tflops peak performance<\/a> in double (64bit), single (32bit) and half (16bit) precision mode, respectively.<\/p>\n

We run Ubuntu 16.04 LTS on the bare metal host. In order to showcase the cloud usage scenario, we spin up a VM using 20 GB of RAM and 20 vCPUs, also running an Ubuntu 16.04 image. The machine resides in our lab and is not integrated into our OpenStack cloud<\/a>, which is why we run it \u201cby hand\u201d using QEM<\/a> in KVM<\/a> mode driven by libvirt<\/a>]<\/sup> and remote controlled by virt-manager<\/a>. Each of the four P100s is attached as PCI device to the VM (GPU pass-through). We also pass all host CPU features through to the VM for maximal performance.<\/p>\n

TensorFlow and CUDA<\/h2>\n

There are TensorFlow Docker images<\/a> available, but we show here that it is easy to install TensorFlow along with all dependencies. After all, installing stuff is what our DevOps engineers do all day!<\/p>\n

What we need to install for TensorFlow to work with the GPUs is<\/p>\n