Building NBA

Supported Platforms

Currently NBA is only tested on Linux x86_64 3.10 or newer kernels, and the Ubuntu 14.04/16.04 LTS distribution.

Step-by-step Guide

Installing dependencies

Ensure that you have a C/C++ compiler (e.g., g++ 4.8 or newer). The compiler must support the C++11 standard.

Check out the latest DPDK source tree:

$ git clone git://dpdk.org/dpdk
$ cd dpdk
$ $EDITOR configs/common_linuxapp
$ make install T=x86_64-native-linuxapp-gcc

See details on what you need to edit on configs/common_linuxapp at DPDK configurations for network cards.

Note

You need to install the kernel header/source packages first.

Install library dependencies on your system:

$ sudo apt-get install libev-dev libssl-dev libpapi-dev

Install Python 3.5 on your system. You may use the system package manager such as apt-get. In that case, ensure that you also have development package as well:

$ sudo apt-get install python3.5 libpython3.5-dev

Then install our Python dependencies:

$ pip3 install --user snakemake

Note

We recommend using a separate Python environment contained inside the user directory. See pyenv for more details.

Compilation

Clone the project source code:

$ git clone https://github.com/anlab-kaist/NBA nba

Install our 3rd-party libraries, the Click configuration parser:

$ cd nba
$ git submodule init && git submodule update

It will be automatically built along with NBA when you first build NBA.

Set the environment variable as follows:

$ export NBA_DPDK_PATH=/home/userid/dpdk/x86_64-native-linuxapp-gcc
$ export USE_CUDA=0  # for testing CPU-only version without CUDA installation

Finally, run:

$ snakemake -j

If all is well, the executable is located in bin/main.

DPDK Configs for Network Cards

Intel X520 Series (82599 chipset)

You just need to bind the PCI addresses of network cards to igb_uio using tools/dpdk_nic_bind.py script provided by DPDK.

Attention

When using ixgbe driver with vectorized PMD enabled, you should fix the IO batch size to be 32, whereas you may change the computation batch size as you want.

In our experiements, the IO batch size 32 and the computation batch size 64 performs best. We have already set those as the default values. (see config/default.py)

Mellanox ConnectX Series

You need to install the OFED toolchain provided by Mellanox because DPDK’s mlx4 poll-mode driver uses Mellanox’s kernel Infiniband driver to control the hardware and perform DMA. We recommend to use version 3.0 or later, as these new versions have much better performance and includes firmware updates.

To use mlx4_pmd on DPDK, turn on it inside DPDK’s configuration:

CONFIG_RTE_LIBRTE_MLX4_PMD=y

To increase throughputs, set the following in the same config:

CONFIG_RTE_LIBRTE_MLX4_SGE_WR_N=1

For maximum throughputs, turn off the followings:

  • blueflame [1]: sudo ethtool --set-priv-flags ethXX blueflame off
  • rx/tx auto-negotiation for flow control: sudo ethtool -A ethXX rx off tx off

Note that above settings must be done in packet generators as well.

Warning

We recommend to turn off blueflame when loading the mlx4_core kernel module as module parameters, instead of using ethtool afterwards.

You do not need to explicitly bind the PCI addresses of Mellanox cards to igb_uio because mlx4_pmd automatically detects them using the kernel driver.

To use mlx4 in NBA, set the following environment variable and rebuild:

$ export NBA_PMD=mlx4
$ snakemake clean && snakemake -j
[1]“blueflame” is a Mellanox-specific feature that uses PCI BAR for tranferring descriptors of small packets instead of using DMA on RX/TX rings. It is known to have lower latency, but causes throughput degradation with NBA.

Optional Installations

NVIDIA CUDA

If you want to use GPU acceleration, install NVIDIA CUDA 7.0 or newer. We recommend to download the latest version of .bin package from the NVIDIA website instead of using system packages.

Note

A small daemon is required to “pin” GPU’s interrupts to specific cores. See details in our gist.

Make CUDA binaries accessible from your shell:

$ echo 'export PATH="$PATH:/usr/local/cuda/bin"' >> ~/.profile
$ sudo sh -c 'echo /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda.conf'
$ sudo ldconfig

To use CUDA in NBA, do:

$ export USE_CUDA=1
$ snakemake clean && snakemake -j

Intel Xeon Phi

If you want to use Xeon Phi acceleration, install the latest Intel MPSS (many-core platform software stack) by visiting the official website.

CPU statistics

To run experiment scripts, install sysstat package (or any package that offers mpstat command).

Customizing Your Build

Our build script offers a few configurable parameters as environment variables:

  • NBA_DPDK_PATH: specifies the path to Intel DPDK (required)
  • NBA_RANDOM_PORT_ACCESS: randomizes the RX queue scanning order for each worker thread (default: false)
  • NBA_OPENSSL_PATH: specifies the path of OpenSSL library (default: /usr)
  • DEBUG: build without compiler optimization (default: 0)
  • USE_CUDA: activates NVIDIA CUDA support (default: 1)
  • USE_KNAPP: activates Knapp-based Intel Xeon Phi support (default: 0)
  • USE_PHI: activates OpenCL-based Intel Xeon Phi support (default: 0, not implemented)
  • USE_NVPROF: activates nvprof API calls to track GPU-related timings (default: 0)
  • USE_OPENSSL_EVP: determines whether to use EVP API for OpenSSL that enables AES-NI support (default: 1)
  • NBA_NO_HUGE: determines whether to use huge-pages (default: 1)
  • NBA_PMD: determines what poll-mode driver to use (default: ixgbe)

Note

1 means true and 0 means false for boolean options.