2. VirtIO-crypto

The dao-virtio-crypto application is a sample application that shows how to use the DAO virtio_crypto library. The application is designed to work with the VirtIO crypto device, which is a virtualized crypto device that can be used to offload crypto operations to a hardware crypto engine. The application maps dao_virtio_cryptodev queues to rte_cryptodev queues. Only crypto_cn10k PMD is allowed to be used with the this application, as crypto sessions cannot be shared between different PMDs.

The application is dependent on below libraries for its functionality:

DPDK dmadev library to use DPI HW and transfer data between Host and Octeon memory.
DPDK cryptodev library to receive and send crypto operations to Octeon CPT engine.
DAO virtio_crypto library to receive / send crypto operations from / to host.

Application creates lcores as below:

One lcore as service core to do dao_virtio_cryptodev_desc_manage() API call per virtio dev.
One or more lcores as worker cores to do dao_virtio_crypto_host_rx() on virtio device and enqueue crypto ops to crypto device using rte_cryptodev_enqueue_burst().
One or more lcores as worker cores to do rte_cryptodev_dequeue_burst() on crypto device to dequeue crypto ops and send them to Host using dao_virtio_crypto_host_tx().

2.1. Setting up EP environment

2.1.2. Setup huge pages for DPDK application

Setup enough hugepages and a mount point for the same in order for the dao-virtio-crypto application to run.

echo 8 > /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages

2.1.3. Bind required DMA devices to vfio-pci

dao-virtio-crypto application needs two DMA devices per lcore one for DEV2MEM and another for MEM2DEV and two more for control lcore. Control lcore is created by virtio library to handle control commands. Below is sample code to bind DMA VF’s to vfio-pci.

DPI_PF=`lspci -d :a080 | awk -e '{print $1}'`

# Enhance DPI engine FIFO size and MRRS
echo 0x10101010 > /sys/module/octeontx2_dpi/parameters/eng_fifo_buf
echo 512 > /sys/module/octeontx2_dpi/parameters/mrrs
echo 256 > /sys/module/octeontx2_dpi/parameters/mps

echo $DPI_PF > /sys/bus/pci/devices/$DPI_PF/driver/unbind
echo octeontx2-dpi > /sys/bus/pci/devices/$DPI_PF/driver_override
echo $DPI_PF > /sys/bus/pci/drivers_probe

echo 32 >/sys/bus/pci/devices/$DPI_PF/sriov_numvfs
DPI_VF=`lspci -d :a081 | awk -e '{print $1}' | head -22`
dpdk-devbind.py -b vfio-pci $DPI_VF

2.1.4. Bind required CPT VF’s to vfio-pci

Application maps dao_virtio_cryptodev device to rte_cryptodev.

Sample code to map CN10K cryptodev’s to vfio-pci.

CPT_PF=0002:20:00.0
VF_CNT=1

dpdk-devbind.py -b rvu_cptpf $CPT_PF
echo $VF_CNT > /sys/bus/pci/devices/$CPT_PF/sriov_numvfs

CPT_VF=`lspci -d :a0f3 | awk -e '{print $1}'`

dpdk-devbind.py -u $CPT_VF
dpdk-devbind.py -b vfio-pci $CPT_VF

2.1.5. Bind PEM BAR4 and DPI BAR0 platform devices to vfio-platform

VirtIO library uses pem0-bar4-mem and dpi_sdp_regs platform devices via vfio-platform. Hence enable vfio-platform in kernel build. If the platform devices are not available, skip this section.

Use vfio-platform.reset_required=0 in kernel command line if vfio-platform is inbuilt kernel or pass reset_required=0 as module parameter while doing loading vfio-platform kernel module.
Bind pem0-bar4-mem and dpi_sdp_regs to vfio-platform.

Sample code to bind platform devices to vfio-platform.

# Platform device suffixes to search for
pem_sfx="pem0-bar4-mem"
sdp_sfx="dpi_sdp_regs"

# Loop through devices
for dev_path in /sys/bus/platform/devices/*; do
    if [[ -d "$dev_path" && "$dev_path" =~ $pem_sfx || "$dev_path" =~ $sdp_sfx ]]; then
        # Get device name from path
        dev_name=$(basename "$dev_path")

        # Bind the device to vfio-platform driver
        echo "vfio-platform" | tee "$dev_path/driver_override" > /dev/null
        echo "$dev_name" | tee "/sys/bus/platform/drivers/vfio-platform/bind" > /dev/null

        echo "Device $dev_name configured."
    fi
done

2.1.6. Bind required SDP RVU devices to vfio-pci

VirtIO library uses RVU SDP devices when platform devices are not present.

dpdk-devbind.py -b vfio-pci 0002:18:00.0
dpdk-devbind.py -b vfio-pci 0002:19:00.0

2.2. Running the EP firmware application

The application has number of command line options.

dao-virtio-crypto [EAL Options] -- [VC offload options]
VC offload options:
  -h, --help
  -v, --virtio-mask=<VIRTO_MASK_L[,VIRTIO_MASK_H]> Hexadecimal bitmask of virtio devices
  -c, --crypto-mask=<CRYPTO_MASK_L[,CRYPTO_MASK_H]> Hexadecimal bitmask of crypto devices
  -C, --crypto-config=(dev,lcore_mask)[,(dev,lcore_mask)] : Crypto enq lcore mapping
  -n, --nb_cryptodev_desc=NB_DESC : Number of descriptors (in range 1024 to 16384)
  -q, --virtio-q-lcore-map=(lcore_id, vdev_id, vq_id)[, (lcore_id, vdev_id, vq_id1, vq_id2)] : Lcore and virtio-queue id map
  -b, --buffer-size=<BUFFER_SIZE> : Virtio mempool buffer size [64, 9216]

2.2.1. EAL Options

The following are the EAL command-line options that can be used in conjunction with the dao-virtio-l2fwd application. See the DPDK Getting Started Guides for more information on these options.

-c <COREMASK> or -l <CORELIST>

Set the hexadecimal bitmask of the cores to run on. The corelist is a list of cores to use.

2.2.2. Application Options

The following are the application command-line options:

-v VIRTIO_MASK_L[,VIRTIO_MASK_H]

Hexadecimal bitmask of virtio-crypto devices to configure. Comma separated 64 bit mask to support 128 virtio-crypto devices. This is a mandatory option.
-p CRYPTO_MASK_L[,CRYPTO_MASK_H]

Hexadecimal bitmask of rte_cryptodev devices to configure. Comma separated 64 bit mask to support up to 128 crypto devices. This is mandatory option.
--crypto-config (dev,lcore_mask)[,(dev,lcore_mask)]

Config to indicate on which lcores crypto enqueue would happen for a given rte_cryptodev device. By default all the configured cryptodev devices would be enqueued on all lcore’s that are detected and available excluding 1 service lcore.
--nb_cryptodev_desc <NB_DESC>
Number of descriptors to be used for each rte_cryptodev device. The number of descriptors should be in the range 1024 to 16384. Default is 8192.
--virtio-q-lcore-map (lcore_id, vdev_id, vq_id)[, (lcore_id, vdev_id, vq_id1, vq_id2)]
Lcore and virtio-queue id map. This is used to configure the lcore id and virtio queue id mapping. The default is to use the first available lcore for each virtio queue. The lcore id and virtio queue id are separated by a comma. Multiple lcore and virtio queue id mapping can be specified by separating them with a comma. The default is to use the first available lcore for each virtio queue.
--buffer-size <BUFFER_SIZE>
This is the size of the buffer used for the virtio mempool. The buffer size should be in the range 64 to 9216. Default is 2048.
--help
Display the help message and exit.

2.2.3. Example EP firmware command

Example to command to run dao-virtio-crypto on CN10K with one virtio device and one crypto device.

DPI_ALLOW='-a 0000:06:00.1 -a 0000:06:00.2 -a 0000:06:00.3 -a 0000:06:00.4 -a 0000:06:00.5 -a 0000:06:00.6 -a 0000:06:00.7 -a 0000:06:01.0 -a 0000:06:01.1 -a 0000:06:01.2 -a 0000:06:01.3 -a 0000:06:01.4 -a 0000:06:01.5 -a 0000:06:01.6 -a 0000:06:01.7 -a 0000:06:02.0 -a 0000:06:02.1 -a 0000:06:02.2 -a 0000:06:02.3 -a 0000:06:02.4 -a 0000:06:02.5 -a 0000:06:02.6'

dao-virtio-crypto -l 0,4,5,6 -a 0002:20:00.1 $DPI_ALLOW --  -v 0x1 -c 0x1 --crypto-config "(0,0x10)" --virtio-q-lcore-map "(4,0,0)"

If dao-virtio-crypto is not build with static linking to DPDK, we need to explicitly load node library and PMD libraries for the application to function.

DPI_ALLOW='-a 0000:06:00.1 -a 0000:06:00.2 -a 0000:06:00.3 -a 0000:06:00.4 -a 0000:06:00.5 -a 0000:06:00.6 -a 0000:06:00.7 -a 0000:06:01.0 -a 0000:06:01.1 -a 0000:06:01.2 -a 0000:06:01.3 -a 0000:06:01.4 -a 0000:06:01.5 -a 0000:06:01.6 -a 0000:06:01.7 -a 0000:06:02.0 -a 0000:06:02.1 -a 0000:06:02.2 -a 0000:06:02.3 -a 0000:06:02.4 -a 0000:06:02.5 -a 0000:06:02.6'

dao-virtio-crypto -d librte_node.so -d librte_crypto_cnxk.so -d librte_mempool_cnxk.so -d librte_dma_cnxk.so -d librte_mempool_ring.so -l 2-7 -a 0002:02:00.1 -l 0,4,5,6 -a 0002:20:00.1 $DPI_ALLOW --  -v 0x1 -c 0x1 --crypto-config "(0,0x10)" --virtio-q-lcore-map "(4,0,0)"

2.3. Setting up Host environment

For host setup, refer to the following guide which provides detailed steps on how to set up the host for VirtIO solutions:

Steps to setup up host for VirtIO solutions

2.4. Running DPDK crypto-perf on host virtio device

To run DPDK dpdk-test-crypto-perf application on the host virtio device, you can use the following command:

dpdk-test-crypto-perf -c 0x3 --socket-mem 1024 --proc-type auto --file-prefix=virtio-user0 --no-pci --vdev=crypto_virtio_user0,path=/dev/vhost-vdpa-0,queue_size=2048 --log-level="pmd.crypto.virtio,info" -- --devtype crypto_virtio_user --optype rsa  --pool-sz 16384 --total-ops 100000 --burst-sz 32 --buffer-sz 20 --ptest throughput --asym-op sign --rsa-priv-keytype qt --rsa-modlen 1024

This command will benchmark the crypto performance on the virtio device with the specified parameters. Make sure to adjust the parameters according to your requirements. For more information on the available options and how to use the dpdk-test-crypto-perf application, refer to the DPDK documentation.