{"id":5391,"date":"2022-11-10T11:53:17","date_gmt":"2022-11-10T10:53:17","guid":{"rendered":"https:\/\/new.cloudandheat.com\/machine-learning-benchmark\/"},"modified":"2025-09-16T14:40:46","modified_gmt":"2025-09-16T12:40:46","slug":"machine-learning-benchmark","status":"publish","type":"post","link":"https:\/\/www.cloudandheat.com\/en\/machine-learning-benchmark\/","title":{"rendered":"Machine Learning Benchmark"},"content":{"rendered":"
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Cloud&Heat operates a wide variety of different hardware setups and takes care of the provisioning of computing resources via the security hardened cloud operating system SecuStack<\/a>. We also continuously optimize our infrastructure in regards to the execution of Machine Learning (ML) workloads. Depending on the use case, we provide a variety of hardware setups by configuring i.e. V100, A100, A10 or T4 graphics cards in conjunction with NVMe block storage, HDD block storage, or with a local SSD. In order to compare different hardware configurations, we use two separate MLCommons benchmarks. With these benchmarks, not only individual components, but rather the entire setup can be tested for its overall performance. The first benchmark is a training benchmark, which trains an ML model for image segmentation. The second is an inference benchmark, which determines the turnaround time of an ML process for image classification and recognition.<\/p>

Training benchmark<\/h2>

The benchmark was tested with Linux Ubuntu 20.4 and represents a 3D medical image segmentation task. The model used is a variant of the U-Net3D model, based on the \u00a0paper \"No New-Net<\/a> . The dataset (KiTS19<\/a>) from the Kidney Tumour Segmentation Challenge 2019<\/a> is used to train the model.<\/p>

\u00a0<\/p>

Overview of the implementation of the benchmark:<\/h4>

First, the Nvidia driver, Nvidia Container Toolkit, Nvidia Docker2<\/a> and Docker<\/a>\u00a0 must be installed.<\/p>

The corresponding repository must be downloaded from GitHub<\/p>

git clone https:\/\/github.com\/mmarcinkiewicz\/training.git<\/a><\/pre>
The image for the Docker container is created from an existing Dockerfile<\/p>
docker build unet-3d .<\/pre>
Download the dataset KiTS19<\/a>.<\/p>
The data should then be structured as follows:<\/p>
data\n|__ case_00000\n<\/span>|        |__ imaging.nii.gz\n<\/span>|        |__ segmentation.nii.gz\n<\/span>|__ case_00001\n<\/span>|        |__ imaging.nii.gz\n<\/span>|        |__ segmentation.nii.gz\n...\n<\/span>|__ case_00209\n<\/span>|        |__ imaging.nii.gz\n<\/span>|        |__ segmentation.nii.gz\n<\/span>|__ kits.json<\/span><\/code><\/pre>
Then<\/p>
sudo docker run<\/pre>
is used to start an interactive session in the container.<\/p>
Various directories from the Shell are mounted in the container so that there is access to the previously downloaded data.<\/p>
The previously downloaded data is prepared in the container for the ML process and stored as a NumPy array.<\/p>
Finally, the ML model will train in the container<\/p>
bash run_and_time.sh<\/pre>
This command should be executed for seeds in the range {1..9}, the target accuracy should converge to 0.908.<\/p>
Quality metric is mean (composite) DICE score for classes 1 (kidney) and 2 (kidney tumour).<\/p>
Afterwards we measure training time for comparison of the different hardware setups.<\/p>
Inference Benchmark<\/h2>
The benchmark was tested with Linux Ubuntu 20.4. The COCO (Common Objects in Context) dataset, based on the paper \"Microsoft COCO: Common objects in context\"<\/a>, is used. The model is ssd-resnet34, which assigns images from the dataset to a category, e.g. \"bottle\". The ssd-resnet-34-1200-onnx model is a multiscale SSD based on the ResNet-34 backbone network and is intended to perform object detection. The model has been trained from the COCO image dataset. This model is pre-trained in the PyTorch framework and converted to ONNX format.<\/p>
\u00a0<\/p>
Overview of the implementation of the benchmark:<\/h4>