Running Linux with Quad-core SMP in LiteX/VexRiscv on Arty A7

FPGAs can be seen as a bridge between the domains of hardware and software as their hardware operation can be reprogrammed using code, giving developers unprecedented flexibility. In advanced products that Antmicro helps its customers build, this flexibility to process parallel data often needs to be complemented with a more standard, sequential processing capability provided by a CPU, which can run control software and interface with the user or other devices.

This need for heterogeneous computing has been at the root of the success of FPGA SoC technologies like Xilinx’ Zynq/UltraScale+ or Intel’s Cyclone/Stratix, and is the reason behind the excitement about PolarFire SoC, world’s first Linux-capable RISC-V FPGA SoC. The upcoming Icicle Development kit for PFSoC is now also available as a platform in our simulation framework, Renode, but we’ll discuss this in a dedicated note soon.

Softcores

Another approach to providing standard CPU processing power to an FPGA design involves building a so-called softcore - a CPU synthesized inside an FPGA. While they offer high flexibility, they are not normally associated with high performance or software capability, certainly not on par with the Linux-capable dual or quad core CPUs found in ‘real’ FPGA SoCs.

With the recent advancements in open source ISAs, HDLs, soft SoC frameworks and FPGA development flows in general that Antmicro is spearheading, that is about to change - read on how we enhanced VexRiscv, an open and flexible 32-bit RISC-V implementation to be a quad-core, Linux capable CPU to offer more computing power and versatility for software-intensive workloads running entirely in an FPGA.

Antmicro’s use of LiteX and VexRiscv

A growing number of our FPGA projects involve a Python-based soft SoC generator called LiteX, which enables building highly flexible SoCs in FPGAs for a variety of applications. While it offers many CPU options (covering architectures such as RISC-V, OpenRISC, lm32 or - since recently - POWER, it is most effective when paired with a flexible and high-performance, Linux-capable FPGA-optimized CPU like VexRiscv.

Originally developed and maintained by Charles Papon (the inventor of SpinalHDL, a high-level Scala-based HDL akin to Chisel, also used for the CPU itself), VexRiscv is a high-performance 32-bit Linux-capable Spinal HDL-based soft RISC-V core optimized for FPGAs. Its high configurability, which allows FPGAs to be used as a sort of custom silicon, makes it a Swiss army knife among CPUs.

Multi-core VexRiscv

In order to increase VexRiscv’s capabilities even further, we have collaborated with Charles to add a scalable multi-core configuration to the CPU, thus delivering the first FPGA RISC-V implementation that offers high performance and is resource optimized for FPGAs at the same time. Using VexRiscv in your FPGA design - together with the multitude of peripheral IPs such as DRAM memory controller, USB, Ethernet, PCIe etc. available in LiteX - results in what could be called a “soft FPGA SoC”.
Depending on your need for I/Os and processing power, and how much space you need for custom IP and accelerators in the FPGA, this “soft FPGA SoC” can be scaled up and down adding cores or peripherals as needed.

To showcase one practical use case, we created a full-blown quad-core SMP system running Linux (of course, it could run an RTOS as well if needed - we have also added VexRiscv support to e.g. Zephyr). To do that, we needed to implement communication and synchronization between the cores, including:

• inter-processor communication, enabled by inter-processor interrupts used by the schedulers in the OS (Linux in this case) to ensure correct scheduling and interrupting of tasks,
• cache-coherence mechanisms, to avoid the software having to manually invalidate cache lines,
• bus interconnections, which allow all the cores to access the memory, including out of order access to maximize the effectivity of the CPUs and memory resources,
• cpu consistency mechanisms, to ensure the ordering of load and store instructions from different cores.

In collaboration with the creators of LiteX and SpinalHDL, we implemented a test design on Digilent’s Arty A7 board. Taking only 70% of a 35 KLUT FPGA - the Artix A35T - VexRiscv runs at 100 MHz and boots Linux in about 4 seconds. It has also been verified to run and pass the Dhrystone CPU benchmark on all cores simultaneously. Machine mode exceptions are handled by openSBI. The CPU’s multi-core capability has also been tested on the much bigger 200T FPGA, where it occupies a much smaller chunk of the circuit’s resources.

On top of that, this implementation is now also supported in Renode - Antmicro’s framework for hardware simulation and HW/SW co-development. The RISC-V
support in Renode gives you the possibility to co-simulate your LiteX SoC with a custom FPGA design using Verilator, providing a full development / debug environment for RISC-V SoC prototyping projects.

Summary

RISC-V on FPGA has reached an important milestone with VexRiscv becoming the first multi-core capable FPGA-optimized CPU on the extensive RISC-V cores list, which we are helping to maintain for the RISC-V International. Antmicro has been delivering advanced customer projects in the FPGA space using the flexible SoC framework as well as refining the relevant tooling and implementing workflow improvements that can make your FPGA development much more portable, software-centric and future-proof. If you are working on an FPGA-related project, reach out to us to take advantage of Antmicro’s expertise in this area.