Introduction

Rattan is a fast and extensible Internet path emulator framework.

We provide a simple and easy-to-use API to create and manage network emulations. Rattan is designed to be used in a wide range of scenarios, from testing network applications to debugging complex network performance issues.

Our modular design makes it easy to extend Rattan with different network effects or conditions. We provide a set of built-in modules that can be used to emulate different network conditions, such as bandwidth, latency, packet loss, ISP policies and etc.

We support Linux only at the moment. Currently, kernel version v5.4, v5.15, v6.8 and v6.11 are tested.

Design Targets

• High Performance. Rattan provides both high peak performance and execution efficiency.
• Flexible. Rattan tries to be agnostic of the underlying path emulation model.
• Extensible. Rattan provides rich features out-of-the-box, meanwhile tends to be easily extensible for custom conditions.
• User-Friendly. Rattan aims to provide a simple and intuitive interface for quick usage on common cases and ensure complete controllability under the hood to cover the corner as well.

Basic Concepts

Run rattan will generate some network namespaces and veth pairs to emulate the network environment. The network topology is like:

   ns-left                                                           ns-right
+-----------+ [Internet]                               [Internet] +-----------+
|    vL0    |    |                 ns-rattan                 |    |    vR0    |
|  external | <--+          +---------------------+          +--> |  external |
|   vL1-L   |               |  vL1-R  [P]   vR1-L |               |   vR1-R   |
| .1.1.x/32 | <-----------> |.1.1.2/32   .2.1.2/32| <-----------> | .2.1.x/32 |
|   vL2-L   |               |  vL2-R        vR2-L |               |   vR2-R   |
| .1.2.y/32 | <-----------> |.1.2.2/32   .2.2.2/32| <-----------> | .2.2.y/32 |
~    ...    ~   Veth pairs  ~  ...           ...  ~   Veth pairs  ~    ...    ~
+-----------+               +---------------------+               +-----------+

Then Rattan will build and link the cells according to the configuration file. Each cell emulates some network characteristics, such as bandwidth, delay, loss, etc.

ATTENTION: Check firewall settings before running Rattan CLI. Please make sure you allow the following addresses:

• 10.1.1.0/24
• 10.2.1.0/24

For example, you can run the following commands if using ufw:

• ufw allow from 10.1.1.0/24
• ufw allow from 10.2.1.0/24

Contributing

Rattan is free and open source. You can find the source code on GitHub and issues and feature requests can be posted on the GitHub issue tracker. Rattan relies on the community to fix bugs and add features: if you'd like to contribute, please read the CONTRIBUTING guide and consider opening a pull request.

Installation

The Rattan project contains a CLI tools (named rattan) for common use cases and a Rust library (named rattan-core) for you to build your own tools on top of.

There are multiple ways to install the Rattan CLI tool. Choose any one of the methods below that best suit your needs.

Pre-compiled binaries

Executable binaries are available for download on the GitHub Releases page. Download the binary and extract the archive. The archive contains an rattan executable which you can run to start your distributed platform.

To make it easier to run, put the path to the binary into your PATH or install it in a directory that is already in your PATH. Note that, you have to grant the binary the necessary capabilities to run. For example, you can run the following commands on Linux:

# install the binary
sudo install -m 755 rattan /usr/local/bin/rattan
# grant the necessary capabilities
sudo setcap 'cap_dac_override,cap_dac_read_search,cap_sys_ptrace,cap_net_admin,cap_sys_admin,cap_net_raw+ep' /usr/local/bin/rattan

Besides, for certain operating systems that enable systemd-networkd, you have to configure it to not change MAC address of veth interfaces. You can do this by adding the following lines to /lib/systemd/network/80-rattan.link:

[Match]
OriginalName=ns*-v*-*
Driver=veth

[Link]
MACAddressPolicy=none

And then restart the systemd-networkd service:

sudo systemctl daemon-reload
sudo systemctl restart systemd-networkd.service

We also contain a install script scripts/install.sh in the repository, which you can use to install the binary on Ubuntu:

./scripts/install.sh rattan

Build from source using Rust

Dependencies

We recommend developers to install the following dependencies for better testing and development experience. And you have to install some of the dependencies to build rattan with specific features:

sudo apt install ethtool, iputils-ping, iperf3, pkg-config, m4, clang, llvm, libelf-dev, libpcap-dev, gcc-multilib

Installing with Cargo

To build the rattan executable from source, you will first need to install Rust and Cargo. Follow the instructions on the Rust installation page.

Once you have installed Rust, the following command can be used to build and install rattan:

cargo install rattan

This will automatically download rattan from crates.io, build it, and install it in Cargo's global binary directory (~/.cargo/bin/ by default).

You can run cargo install rattan again whenever you want to update to a new version. That command will check if there is a newer version, and re-install rattan if a newer version is found.

To uninstall, run the command cargo uninstall rattan.

Installing the latest git version with Cargo

The version published to crates.io will ever so slightly be behind the version hosted on GitHub. If you need the latest version you can build the git version of rattan yourself. Cargo makes this super easy!

cargo install --git https://github.com/stack-rs/rattan.git rattan

Again, make sure to add the Cargo bin directory to your PATH.

Building from source

If you want to build the binary from source, you can clone the repository and build it using Cargo.

git clone https://github.com/stack-rs/rattan.git
cd rattan
cargo build --release

Then you can find the binary in target/release/rattan and install or run it as you like (e.g., run ./scripts/install.sh target/release/rattan).

Modifying and contributing

If you are interested in making modifications to Rattan itself, check out the Contributing Guide for more information.

Bundled CLI Tool

The Rattan project provides a CLI tool named rattan for common use cases. The tool is built on top of the rattan-core library, which you can use to build your own tools.

We provide different features under different subcommands:

• rattan link: Run a templated channel with command line arguments. We include a set of predefined cells and templated channels in the tool to help you get started quickly. You can emulate a simple network path with just a few commands.
• rattan run: Run the instance according to the configuration. For more complex configurations or reproduction purpose, you can use the flexible configuration file to define your own channel with specific cells, network paths and even routes.

There are also global options that you can use to customize the behavior of the tool:

• --generate and --generate-path are used to generate a configuration file from your settings instead of running an instance.
• --packet-log is used to enable compressed packet log and specify the file to store the records.
• --file-log and --file-log-path are used to enable logging to a file and specify the path to store the log.

For more detailed usage, you can run rattan --help to see the available commands and options:

Usage: rattan [OPTIONS] <COMMAND>

Commands:
  link  Run a templated channel with command line arguments
  run   Run the instance according to the config
  help  Print this message or the help of the given subcommand(s)

Options:
      --generate              Generate config file instead of running a instance
      --generate-path <File>  Generate config file to the specified path instead of stdout
      --packet-log <File>     The file to store compressed packet log (overwrite config) (default: None)
      --file-log              Enable logging to file
      --file-log-path <File>  File log path, default to $CACHE_DIR/rattan/core.log
  -h, --help                  Print help (see more with '--help')
  -V, --version               Print version

Predefined Cells and Templated Channels

The templated channels consist of a set of cells defined in the rattan-core library, which you can use to quickly set up a network path for testing. You can use these channels through the rattan link subcommand with just a few command line arguments.

For now, we only provide a simple bidirectional channel, with each direction comprising three cells: bandwidth, delay and loss. This channel is suitable for most common network emulation scenarios.

You can configure the parameters of each cell by passing the arguments to the subcommand.

For example, to create a channel with 10Mbps bandwidth, infinite queue, 50ms latency and 1% loss rate in both directions, you can run the following command:

rattan link uplink-bandwidth 10Mbps uplink-delay 50ms uplink-loss 0.1 downlink-bandwidth 10Mbps downlink-delay 50ms downlink-loss 0.1

Then you will be prompted to a shell with the network path set up.

For detailed usage, you can run rattan link --help to see the available cells and options:

Run a templated channel with command line arguments

Usage: rattan link [OPTIONS] [-- <COMMAND>...]

Arguments:
  [COMMAND]...  Command to run. Only used when in compatible mode

Options:
      --uplink-loss <Loss>              Uplink packet loss
      --downlink-loss <Loss>            Downlink packet loss
      --generate                        Generate config file instead of running a instance
      --uplink-delay <Delay>            Uplink delay
      --downlink-delay <Delay>          Downlink delay
      --generate-path <File>            Generate config file to the specified path instead of stdout
      --packet-log <File>               The file to store compressed packet log (overwrite config) (default: None)
      --uplink-bandwidth <Bandwidth>    Uplink bandwidth
      --file-log                        Enable logging to file
      --uplink-trace <File>             Uplink trace file
      --file-log-path <File>            File log path, default to $CACHE_DIR/rattan/core.log
      --uplink-queue <Queue Type>       Uplink queue type [possible values: infinite, droptail, drophead, codel]
      --uplink-queue-args <JSON>        Uplink queue arguments
      --downlink-bandwidth <Bandwidth>  Downlink bandwidth
      --downlink-trace <File>           Downlink trace file
      --downlink-queue <Queue Type>     Downlink queue type [possible values: infinite, droptail, drophead, codel]
      --downlink-queue-args <JSON>      Downlink queue arguments
  -s, --shell <SHELL>                   Shell used to run if no command is specified. Only used when in compatible mode [default: default] [possible values: default, sh, bash, zsh, fish]
  -h, --help                            Print help (see more with '--help')
  -V, --version                         Print version

Flexible Configuration

Generally, our specialized network emulations necessitate a different number of cells or more complex channels, and they may even require the configuration of routes, such as a multi-path network path with ISP QoS policies. Our CLI tool achieves flexibility and configurability through the use of TOML config files. The configuration file primarily consists of six sections.

We provide an example config file config.example.toml as reference. You can just copy and modify it to suit your needs.

For detailed usage, you can run rattan config --help to see the available cells and options:

Run the instance according to the config

Usage: rattan run [OPTIONS] --config <Config File>

Options:
  -c, --config <Config File>        Use config file to run a instance
      --left [<LEFT_COMMAND>...]    Command to run in left ns. Can be used in compatible and isolated mode
      --generate                    Generate config file instead of running a instance
      --right [<RIGHT_COMMAND>...]  Command to run in right ns. Only used in isolated mode
      --generate-path <File>        Generate config file to the specified path instead of stdout
      --packet-log <File>           The file to store compressed packet log (overwrite config) (default: None)
      --file-log                    Enable logging to file
      --file-log-path <File>        File log path, default to $CACHE_DIR/rattan/core.log
  -h, --help                        Print help (see more with '--help')
  -V, --version                     Print version

General Section

This section determines how to run the rattan itself in general.

• packet_log. Now we only have this option in the section, which is used to enable compressed packet log and specify the file to store the records. The meta data of flows will be stored in the file of name packet_log.flows

Environment Section

This section determines how to build the environment.

• mode. You can configure rattan to function in Compatible mode or Isolated mode. The ns-left side is always a new network namespace. The difference of the two modes lies in whether ns-right side is the host network namespace (the namespace run rattan) or a new network namespace. The default is Compatible.
• left_veth_count and right_veth_count. You can configure the number of veth pairs to create in the ns-left (i.e., between ns-left and ns-rattan) and ns-right (i.e., between ns-right and ns-rattan) network namespaces. The default is 1.

HTTP Section

This section configures Rattan's HTTP control server.

• enable. You can enable or disable the HTTP control server. The default is false.
• port. You can configure the port number of the HTTP control server. The default is 8086.

Cells Section

This section describes the emulation cells. It is a table of cells, where each cell has a unique string ID, defined like [cells.<ID>].

Each cell MUST have a "type" field, which determines the cell type and cell configuration fields. Possible cell types:

• Bw: Fixed bandwidth
• BwReplay: Bandwidth trace replayer
• Delay: Fixed delay
• Loss: Loss pattern
• Custom: Custom cell, only specify the cell ID used in link section, and the cell must be built by the user

For example, the following is a cell configuration for a fixed bandwidth cell:

[cells.up_1]
type = "Bw"
bw_type = "NetworkLayer"
bandwidth = "1Gbps 200Mbps"
queue = "DropTail"
queue_config = { packet_limit = 60 }

For detailed parameters, you can check the documentation of rattan-core or the example config file config.example.toml.

Links Section

This section describes how to link the cells.

"left" and "right" are preserved cell IDs, which respectively represent the ns-left side veth (named vL1-R) and the ns-right side veth (named vR1-L). But if a network namespace has more than one veth NICs, ns-left side for example, "left1", "left2" etc. will be defined.

The other cell IDs are defined by user in the cells section.

Each line describes a one-way link from a cell egress to a cell ingress.

For example, to emulate a network topology with two network paths like following:

+-------+  -------->  shadow --->  up_1  --->  up_2  ---> +-------+
| left1 | <==+           ^                                | right |
+-------+    }= router <-|------- down_2 <--- down_1 <--- +-------+
| left2 | <==+           |
+-------+  --------------+

Note: The double line "<===" from router is connected in Cells Section

You can configure the link section like this:

[links]
left1 = "shadow"
left2 = "shadow"
shadow = "up_1"
up_1 = "up_2"
up_2 = "right"
right = "down_1"
down_1 = "down_2"
down_2 = "router"
# We do not need to link the Router Cell to other cells anymore, as we should do it in Cells Section

Resource Section

This section describes how rattan use system resource

The section is preserved for future use but not used now.

• cores. The CPU cores that rattan is allowed to use.

Commands Section

This section describes the commands to run in rattan.

When in Compatible mode, only the left and shell options are used, and the commands will run in the ns-left network namespace.

When in Isolated mode, only the left and right options are used, and the commands will run in the ns-left and ns-right network namespace, respectively.

• left and right. The commands to run in the ns-left and ns-right network namespace, respectively. Should be an array of strings.
• shell. The shell used to run if no command is specified. Only used when in compatible mode. The default is Default (i.e., the default shell defined in environment variable $SHELL).

Write a New Cell

In Rattan, a "cell" is a unit that emulates various network effects for your network path, including delay, packet loss, reordering, and so forth. As the fundamental building block of Rattan, cells follow the actor concurrency programming model. Each cell runs as an self-contained coroutine that processes events through several well-defined asynchronous interfaces (i.e., Ingress, Egress, ControlInterface).

Users can combine different cells to emulate a variety of network scenarios. We've bundled several built-in cells in Rattan to support common network effects, including the Delay cell, Loss cell, among others.

However, there may be instances where users wish to implement additional effects or need to cater to specific requirements within a cell. In such cases, they can follow this guide to create their own custom cell.

Basic Structure

A cell in Rattan is primarily composed of the following components: Ingress, Egress, and ControlInterface. To implement your own cell, you need to implement these three traits, along with an additional Cell trait to combine them together.

In the design of Rattan, each Cell receives packets via Ingress and sends packets via Egress. The Cell settings can be adjusted at runtime through the ControlInterface. The Rattan core handles the forwarding of packets from the Egress of one Cell to the Ingress of another.

In the following sections, we will use a DropPacketCell as an example, which drops one packet every loss_interval packets, to illustrate the basic structure and implementation process of a cell.

Ingress

The Ingress is used to receive packets, which are dequeued by other cells and forwarded by the rattan core. It serves as the entry point for our cell, hence the need for us to implement an enqueue method to pass forwarded packets into the cell.

Typically, we use a channel to forward packets within the cell. As such, we can incorporate an UnboundedSender member within Ingress and use its send method to transmit packets (similarly, we can include an UnboundedReceiver member within Egress and use its recv method to receive packets).

A single Ingress is shared across multiple forwarding threads to allow flexible composition, so it can only have an immutable reference to itself in the enqueue method and must implement the Clone trait. It is worth attention that mutable operations on packets are possible within the method.

Here is an example of an Ingress implementation, which we set the timestamp of the packet to the current time before sending it to the Egress:

#![allow(unused)]
fn main() {
pub struct DropPacketCellIngress<P>
where
    P: Packet,
{
    // Send packets to Egress
    ingress: mpsc::UnboundedSender<P>,
}

impl<P> Clone for DropPacketCellIngress<P>
where
    P: Packet,
{
    fn clone(&self) -> Self {
        Self {
            ingress: self.ingress.clone(),
        }
    }
}

impl<P> Ingress<P> for DropPacketCellIngress<P>
where
    P: Packet + Send,
{
    fn enqueue(&self, mut packet: P) -> Result<(), Error> {
        packet.set_timestamp(Instant::now());
        self.ingress
            .send(packet)
            .map_err(|_| Error::ChannelError("Data channel is closed.".to_string()))?;
        Ok(())
    }
}
}

Egress

The Egress is used to dequeue packets from the Cell, which are then sent to another cell by the rattan core. It serves as the export of the cell.

As we mentioned above, we typically use a channel to forward packets within the cell. Therefore, we can include an UnboundedReceiver member within Egress and use its recv method to receive packets for further process.

The Egress component is designed to be exclusively held by a single thread, allowing its dequeue method to obtain a mutable reference to itself. This design allows for stateful operations within this method. Network effects often necessitate maintaining mutable internal states.

For instance, the Gilbert-Elliott packet loss model requires internal state transitions along with a varying packet loss probability.

Consequently, we usually implement the majority of packet handling strategies, such as delaying or dropping a packet, and modify the internal states within the dequeue method.

We always include a state variable (typically, an AtomicI32) in the Egress struct to control the cell's state, which is informed by Rattan runtime through method change_state. Currently, three values are supported: 0 (drop), 1 (pass-through), and 2 (normal operation). Also, we include a configuration receiver (the specific type depends on the way you implement trait ControlInterface) to receive configuration information from the ControlInterface.

An example is shown below:

#![allow(unused)]
fn main() {
pub struct DropPacketCellEgress<P>
where
    P: Packet,
{
    // Receive packets from Ingress
    egress: mpsc::UnboundedReceiver<P>,
    // Internal states
    inner_loss_interval: usize,
    counter: usize,
    loss_interval: Arc<AtomicUsize>,
    state: AtomicI32,
}
}

Here, loss_interval is used to receive the configuration information from the ControlInterface, inner_loss_interval is used to store the received parameters, counter is used to count the number of packets since last packet loss, and state is used to control the state of the cell.

For the Egress trait, you must implement the dequeue method and might override the change_state and reset method (both are blank implementations by default). The dequeue method is where you implement the functional logic of your cell. For example, if you need a delay function, you can write your dequeue method to return a packet after a specified delay time; if you need to drop packets, you can write your dequeue method to return a packet or drop it with a certain probability. The change_state method is used to control the state of your cell. The reset method is used to reset the internal state of your cell, often useful in calibrating the internal timer, which will always be called by Rattan runtime before each run.

Here is an example of an Egress implementation, which drops one packet every loss_interval packets:

#![allow(unused)]
fn main() {
#[async_trait]
impl<P> Egress<P> for DropPacketCellEgress<P>
where
    P: Packet + Send + Sync,
{
    async fn dequeue(&mut self) -> Option<P> {
        // Receive packets from Ingress
        let packet = match self.egress.recv().await {
            Some(packet) => packet,
            None => return None,
        };
        // Check state of your cell
        match self.state.load(std::sync::atomic::Ordering::Acquire) {
            0 => {
                // Drop
                return None;
            }
            1 => {
                // Pass-through
                return Some(packet);
            }
            _ => {}
        }
        // The control logic of your own cell
        self.counter += 1;
        self.inner_loss_interval = self
            .loss_interval
            .load(std::sync::atomic::Ordering::Acquire);
        if self.counter >= self.inner_loss_interval && self.inner_loss_interval != 0 {
            self.counter = 0;
            None
        } else {
            Some(packet)
        }
    }

    fn change_state(&self, state: i32) {
        self.state
            .store(state, std::sync::atomic::Ordering::Release);
    }
}
}

ControlInterface

The ControlInterface trait is only required when you need to modify the configuration (or exchange information) of your cell at runtime. It is used to pass configuration information to the Egress component. Developers are responsible for

Developers are responsible for embedding communication mechanisms at both ends (i.e., ControlInterface and Egress). This can be achieved using atomic types, channels, or shared memory, depending on the specific requirements of the cell. There is also a requirement to modify Egress to read or listen for pertinent information. The only method that this trait requires is the set_config method, which is designed to execute the desired communication mechanism or other logical code.

The example provided in this guide utilizes atomic types and is intended solely for reference purposes:

#![allow(unused)]
fn main() {
pub struct DropPacketCellControlInterface {
    loss_interval: Arc<AtomicUsize>,
}

impl ControlInterface for DropPacketCellControlInterface {
    type Config = DropPacketCellConfig;

    fn set_config(&self, config: Self::Config) -> Result<(), Error> {
        self.loss_interval
            .store(config.loss_interval, std::sync::atomic::Ordering::Release);
        Ok(())
    }
}
}

Note that the code above includes a DropPacketCellConfig struct, which is the configuration struct defined for the example cell. When implementing the ControlInterface trait, we also need to define a struct for our cell's configuration. This struct typically contains the parameters that you want to use within the cell. It will also be helpful if you want to read settings or parameters from some configuration files (e.g., TOML).

#![allow(unused)]
fn main() {
#[derive(Debug, Default, Clone)]
pub struct DropPacketCellConfig {
    pub loss_interval: usize,
}

impl DropPacketCellConfig {
    pub fn new<T: Into<usize>>(loss_interval: T) -> Self {
        Self {
            loss_interval: loss_interval.into(),
        }
    }
}
}

Cell

Up until now, we have implemented the three necessary traits for creating a custom cell: Ingress, Egress, and ControlInterface, along with the struct for the cell's configuration information. However, these structs are just parts of the cell. In order to make them function properly in Rattan, we need to assemble them into a single struct. This is why we need to implement the Cell trait.

The Cell trait has four methods that must be implemented: sender, receiver, into_receiver, and control_interface:

• The sender method returns a cloned Arc of the Ingress
• The receiver method returns a mutable reference to the Egress
• The into_receiver method returns the Egress
• The control_interface method returns an Arc of the ControlInterface

Refer to the example below:

#![allow(unused)]
fn main() {
pub struct DropPacketCell<P: Packet> {
    ingress: Arc<DropPacketCellIngress<P>>,
    egress: DropPacketCellEgress<P>,
    control_interface: Arc<DropPacketCellControlInterface>,
}

impl<P> Cell<P> for DropPacketCell<P>
where
    P: Packet + Send + Sync + 'static,
{
    type IngressType = DropPacketCellIngress<P>;
    type EgressType = DropPacketCellEgress<P>;
    type ControlInterfaceType = DropPacketCellControlInterface;

    fn sender(&self) -> Arc<Self::IngressType> {
        self.ingress.clone()
    }

    fn receiver(&mut self) -> &mut Self::EgressType {
        &mut self.egress
    }

    fn into_receiver(self) -> Self::EgressType {
        self.egress
    }

    fn control_interface(&self) -> Arc<Self::ControlInterfaceType> {
        Arc::clone(&self.control_interface)
    }
}

impl<P> DropPacketCell<P>
where
    P: Packet,
{
    pub fn new<L: Into<usize>>(loss_interval: L) -> Result<DropPacketCell<P>, Error> {
        let loss_interval = loss_interval.into();
        let (rx, tx) = mpsc::unbounded_channel();
        let loss_interval = Arc::new(AtomicUsize::new(loss_interval));
        Ok(DropPacketCell {
            ingress: Arc::new(DropPacketCellIngress { ingress: rx }),
            egress: DropPacketCellEgress {
                egress: tx,
                loss_interval: loss_interval.clone(),
                inner_loss_interval: 0,
                state: AtomicI32::new(0),
                counter: 0,
            },
            control_interface: Arc::new(DropPacketCellControlInterface { loss_interval }),
        })
    }
}
}

Integrate Cell to Rattan

In the guide above, we have created a new Cell. To integrate it into Rattan, you also have to implement the CellFactory trait, which serves as the builder function of the Cell. The CellFactory trait is a type alias for a closure that takes a reference to a tokio runtime Handle and returns a Cell instance. You should call the build_cell method of RattanRadix to build your cell.

It is recommended to implement a method for a struct to return the closure like the example below:

#![allow(unused)]
fn main() {
pub type DropPacketCellBuildConfig = drop_packet::DropPacketCellConfig;

impl DropPacketCellBuildConfig {
    pub fn into_factory<P: Packet>(self) -> impl CellFactory<drop_packet::DropPacketCell<P>> {
        move |handle| {
            let _guard = handle.enter();
            // Create the closure to build the cell
            drop_packet::DropPacketCell::new(self.loss_interval)
        }
    }
}
}

If you want to contribute back to the official Rattan repository or modify the source code, you can also add your build configuration type to the enum CellBuildConfig. In this way, you can easily read its configuration from a TOML file through the load_cells_config method (which internally calls build_cell as well) of RattanRadix. Remember to also derive the Deserialize and Serialize traits for your configuration struct in such cases.

#![allow(unused)]
fn main() {
pub enum CellBuildConfig<P: Packet> {
    /* Other cells */
    DropPacket(DropPacketCellBuildConfig),
    Custom,
}
}

Don't forget to modify load_cells_config in RattanRadix to add your cell configuration to the match statement.

Now, you can add your cell to a TOML file and use Rattan to try parsing it! There is an example for you to refer to to write your TOML file:

# Other Sections
# ...

# ----- Cells Section -----
# DropPacket Cell Example
[cells.my_drop]
type = "DropPacket"
loss_interval = 2

# Other Cells
# ...
# ----- Cells Section End -----

Test Your New Cell

This section is optional when you are writing a new cell. Tests are only required when you wish to integrate your cell into Rattan's official repository and the corresponding CLI tool.

We require every cell to have unit tests and integration tests to check the correctness of the internal cell implementations and external interactions.

Unit Tests

The goal of unit tests is to check whether the internal state of your cell transitions correctly and whether its functionality behaves as expected. Typically, when writing unit tests, you create an instance of your cell, manually call its enqueue and dequeue methods, and observe the results to check its correctness.

To run the unit tests, you can first run cargo nextest list --workspace to view the list of unit tests, find the test(s) you want to run, and then run cargo nextest run <the name of your test> --release --all-features --workspace to execute your test(s).

Below are two examples of unit tests for DropPacketCell for reference:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use tracing::{info, span, Level};

    use crate::cells::StdPacket;

    use super::*;

    #[test_log::test]
    fn test_drop_packet_cell() -> Result<(), Error> {
        let _span = span!(Level::INFO, "test_drop_packet_cell").entered();
        let rt = tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()?;
        let _guard = rt.enter();

        info!("Creating cell with loss_interval 3");
        let cell = DropPacketCell::new(3_usize)?;
        let ingress = cell.sender();
        let mut egress = cell.into_receiver();
        egress.reset();
        egress.change_state(2);

        info!("Testing loss for drop packet cell of loss 3");
        let mut received_packets = vec![false; 100];

        for i in 0..100 {
            let mut test_packet = StdPacket::from_raw_buffer(&[0; 256]);
            test_packet.set_flow_id(i);
            ingress.enqueue(test_packet)?;
            let received = rt.block_on(async { egress.dequeue().await });

            if let Some(content) = received {
                assert!(content.length() == 256);
                received_packets[content.get_flow_id() as usize] = true;
            }
        }
        info!("Tested loss sequence: {:?}", received_packets);
        for (i, item) in received_packets.iter().enumerate().take(100) {
            if (i + 1) % 3 == 0 {
                assert!(!item);
            } else {
                assert!(item);
            }
        }
        Ok(())
    }

    #[test_log::test]
    fn test_drop_packet_cell_config_update() -> Result<(), Error> {
        let _span = span!(Level::INFO, "test_drop_packet_cell_config_update").entered();
        let rt = tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()?;
        let _guard = rt.enter();

        info!("Creating cell with loss_interval 5");
        let cell = DropPacketCell::new(5_usize)?;
        let config_changer = cell.control_interface();
        let ingress = cell.sender();
        let mut egress = cell.into_receiver();
        egress.reset();
        egress.change_state(2);

        let mut received_packets = vec![false; 100];

        info!("Testing loss for drop packet cell of loss 5");

        for i in 0..48 {
            let mut test_packet = StdPacket::from_raw_buffer(&[0; 256]);
            test_packet.set_flow_id(i);
            ingress.enqueue(test_packet)?;
            let received = rt.block_on(async { egress.dequeue().await });

            if let Some(content) = received {
                assert!(content.length() == 256);
                received_packets[content.get_flow_id() as usize] = true;
            }
        }
        for (i, item) in received_packets.iter().enumerate().take(48) {
            if (i + 1) % 5 == 0 {
                assert!(!item);
            } else {
                assert!(item);
            }
        }

        info!("Changing loss_interval to 3");
        config_changer.set_config(DropPacketCellConfig::new(3_usize))?;

        for i in 48..100 {
            let mut test_packet = StdPacket::from_raw_buffer(&[0; 256]);
            test_packet.set_flow_id(i);
            ingress.enqueue(test_packet)?;
            let received = rt.block_on(async { egress.dequeue().await });

            if let Some(content) = received {
                assert!(content.length() == 256);
                received_packets[content.get_flow_id() as usize] = true;
            }
        }
        info!("Tested loss sequence: {:?}", received_packets);
        for i in 0..=51 {
            if i % 3 == 0 {
                assert!(!received_packets[i + 48]);
            } else {
                assert!(received_packets[i + 48]);
            }
        }
        Ok(())
    }
}
}

Integration Tests

The purpose of integration tests is to check whether your cell interacts correctly with Rattan's runtime. When writing integration tests, you need to first create your cell in the rattan runtime, and then execute commands such as ping between network namespaces to observe whether your cell behaves as expected.

To run the integration tests, you can first run cargo nextest list --workspace to view the list of integration tests, find the test you want to run, and then run cargo nextest run <the name of your test> --release --all-features --workspace to execute your test.

Here is an example of an integration test for reference:

#![allow(unused)]
fn main() {
#[instrument]
#[test_log::test]
fn test_drop_packet() {
    let mut config = RattanConfig::<StdPacket> {
        env: StdNetEnvConfig {
            mode: StdNetEnvMode::Isolated,
            client_cores: vec![1],
            server_cores: vec![3],
            ..Default::default()
        },
        ..Default::default()
    };
    config.cells.insert(
        "up_drop".to_string(),
        CellBuildConfig::DropPacket(DropPacketCellConfig::new(0_usize)),
    );
    config.cells.insert(
        "down_drop".to_string(),
        CellBuildConfig::DropPacket(DropPacketCellConfig::new(0_usize)),
    );
    config.links = HashMap::from([
        ("left".to_string(), "up_drop".to_string()),
        ("up_drop".to_string(), "right".to_string()),
        ("right".to_string(), "down_drop".to_string()),
        ("down_drop".to_string(), "left".to_string()),
    ]);
    let mut radix = RattanRadix::<AfPacketDriver>::new(config).unwrap();
    radix.spawn_rattan().unwrap();
    radix.start_rattan().unwrap();

    // Wait for AfPacketDriver to be ready
    std::thread::sleep(std::time::Duration::from_millis(100));

    // Before set the LossCell, the average loss rate should be 0%
    {
        let _span = span!(Level::INFO, "ping_no_loss").entered();
        info!("try to ping with no loss");
        let right_ip = radix.right_ip(1).to_string();
        let left_handle = radix
            .left_spawn(None, move || {
                let handle = std::process::Command::new("ping")
                    .args([&right_ip, "-c", "20", "-i", "0.3"])
                    .stdout(std::process::Stdio::piped())
                    .spawn()
                    .unwrap();
                Ok(handle.wait_with_output())
            })
            .unwrap();
        let output = left_handle.join().unwrap().unwrap().unwrap();
        let stdout = String::from_utf8(output.stdout).unwrap();

        let re = Regex::new(r"(\d+)% packet loss").unwrap();
        let loss_percentage = re
            .captures(&stdout)
            .map(|cap| cap[1].parse::<u64>())
            .unwrap()
            .unwrap();
        info!("loss_percentage: {}", loss_percentage);
        assert!(loss_percentage == 0);
    }

    // After set the loss_interval of DropPacketCell to 2, the average loss rate should be 50%
    {
        let _span = span!(Level::INFO, "ping_with_loss").entered();
        info!("try to ping with loss interval set to 2");
        radix
            .op_block_exec(RattanOp::ConfigCell(
                "up_drop".to_string(),
                serde_json::to_value(DropPacketCellConfig::new(2_usize)).unwrap(),
            ))
            .unwrap();

        let right_ip = radix.right_ip(1).to_string();
        let left_handle = radix
            .left_spawn(None, move || {
                let handle = std::process::Command::new("ping")
                    .args([&right_ip, "-c", "50", "-i", "0.3"])
                    .stdout(std::process::Stdio::piped())
                    .spawn()
                    .unwrap();
                Ok(handle.wait_with_output())
            })
            .unwrap();
        let output = left_handle.join().unwrap().unwrap().unwrap();
        let stdout = String::from_utf8(output.stdout).unwrap();

        let re = Regex::new(r"(\d+)% packet loss").unwrap();
        let loss_percentage = re
            .captures(&stdout)
            .map(|cap| cap[1].parse::<u64>())
            .unwrap()
            .unwrap();
        info!("loss_percentage: {}", loss_percentage);
        assert!((loss_percentage == 50));
    }
}
}

Write Your Tailored Tool (TODO)

Packet I/O

Currently, we support two packet I/O mechanisms: AF_PACKET and AF_XDP. The former is a socket-based mechanism, while the latter is a driver-based mechanism. The AF_PACKET version is more portable, while the AF_XDP version is more performant.

Example

By default, the AF_PACKET version is used. To use the AF_XDP version, you need to enable the camellia feature.

cargo run --example channel --release # AF_PACKET version
cargo run --example channel-xdp --release --features="camellia" # AF_XDP version

Flamegraph

To generate a flamegraph, you can use the flamegraph tool. First, install it:

cargo install flamegraph

Then, run the following command:

cargo flamegraph --root --example channel

This will generate a flamegraph, which you can open with a browser to inspect hot spot.

Packet Log Spec

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       LH.length       | LH.ty.|   GPH.length  |GPH.ac.|GPH.ty.|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          GP.timestamp                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           GP.length           |       PRH.length      |PRH.ty.|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          PRT (custom)                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Generated with protocol, where:

• LH is short for log entry header
• GPH is short for general packet entry header
• GP is short for general packet entry (the body part)
• PRH is short for protocol entry header
• PRT is short for protocol entry (the body part)
• ty. is short for type
• ac. is short for action

protocol "LH.length:12,LH.ty.:4,GPH.length:8,GPH.ac.:4,GPH.ty.:4,GP.timestamp:32,GP.length:16,PRH.length:12,PRH.ty.:4,PRT (custom):32"

We currently provide TCP log spec (a variant of the protocol entry body part, i.e. PRT (custom)):

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          tcp.flow_id                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            tcp.seq                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            tcp.ack                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             ip.id             |         ip.frag_offset        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          ip.checksum          |   tcp.flags   |    padding    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

protocol "tcp.flow_id:32,tcp.seq:32,tcp.ack:32,ip.id:16,ip.frag_offset:16,ip.checksum:16,tcp.flags:8,padding:8"