Testing AWS network performance

Our customers often ask us about different aspects of network performance in AWS, how architecture or configurations can affect it, what to expect and how to optimize it.

In this blog post I will explain basic components influencing a network performance, complete several tests and demonstrate outcomes.

Bandwidth is the maximum rate of transfer over the network, defined in bits per second (abbreviated Bps, Mbps, Gbps, etc.). Network bandwidth defines the maximum bandwidth rate, but the actual user or application transfer rate will also be affected by latency, protocol, and packet loss.

Latency is the delay between two points in a network. Latency can be measured in one-way delay or Round-Trip Time (RTT) between two points. Ping is a common way to test RTT delay. Delays include propagation delays for signals to travel across different mediums such as copper or fiber optics, often at speeds close to the speed of light. There are also processing delays for packets to move through physical or virtual network devices, such as the Amazon Virtual Private Cloud (Amazon VPC) virtual router. Network drivers and operating systems can be optimized to minimize processing latency on the host system as well.

Jitter is the variation in inter-packet delays. Jitter is caused by a variance in delay over time between two points in the network. Jitter is often caused by variations in processing delays and queueing delays in the network, which increase with higher network load. For example, if the one-way delay between two systems varies from 10 ms to 100 ms, then there is 90 ms of jitter. This type of varying delay causes issues with voice and real-time systems that process media because the systems have to decide to buffer data longer or continue without the data.

Throughput is the rate of successful data transferred, measured in bits per second. Bandwidth, latency, and packet loss affect the throughput rate. The bandwidth will define the maximum rate possible. Latency affects the bandwidth of protocols like Transmission Control Protocol (TCP) with round-trip handshakes.

Packet loss is typically stated in terms of the percentage of packets that are dropped in a flow or on a circuit. Packet loss will affect applications differently. TCP applications are generally sensitive to loss due to congestion control.

Packets per second refers to how many packets are processed in one second. Packets per second are a common bottleneck in network performance testing. All processing points in the network must process each packet, requiring computing resources. Particularly for small packets, per-packet processing can limit throughput before bandwidth limits are reached.

The Maximum Transmission Unit (MTU) defines the largest packet that can be sent over the network. The maximum on most Internet and Wide Area Networks (WANs) is 1,500 bytes. Jumbo frames are packets larger than 1,500 bytes. AWS supports 9,001 byte jumbo frames within a VPC. VPC peering and traffic leaving a VPC support up to 1,500 byte packets, including Internet and AWS Direct Connect traffic. Increasing the MTU increases throughput when the packet per second processing rate is the performance bottleneck.

Enhanced networking uses single root I/O virtualization (SR-IOV) as a method of device virtualization that provides higher I/O performance and lower CPU utilization when compared to traditional virtualized network interfaces. Enhanced networking provides higher bandwidth, higher packet per second (PPS) performance, and consistently lower inter-instance latencies.

You can check whether enhanced networking is enabled or not using “ethtool” for Linux OS.

### Debian 8.1 ###
#  ethtool -i eth0
driver: vif
version:
firmware-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Driver “vif” means that enhanced networking is disabled.

### Amazon Linux 2 ###
# ethtool -i eth0
driver: ixgbevf
version: 4.1.0-k
firmware-version: 
expansion-rom-version: 
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

### Debian ###
# ethtool -i eth0
driver: ena
version: 1.0.0
firmware-version: 
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Drivers “ixgbevf” and “ena” means that enhanced networking is enabled. Driver depends on instance type and OS version.

The first test has been completed between Ireland (eu-west-1) and US Northern Virginia (us-east-1) for EC2 instances with enabled and disabled enhanced networking. EC2 type is r4.large with 2 vCPUs 15.25 Gb RAM and network bandwidth “Up to 10 Gbps”

Ping has been used for testing latency, Iperf3 has been used for testing bandwidth and jitter for TCP and UDP traffic. VPCs were not peered, therefore EC2 instances used public IPs and AWS Internet Gateways.

# iperf3 -s -p 5001