CERN-RAL Network Tests 30 Jun / 1 Jul 05

This is Work in Progress - comments welcome ! The aim is to understand and compare data transfer rates and packet loss and packet re-ordering between CERN and RAL using a direct UKLight path. Details of the end hosts and Networks used are:

oplapro71 oplapro72 CERN	dual 1.5 GHz IA-64 Itanium 2 Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet
gftp0440 gftp0441 RAL	2.8 GHz Xeon (Hyperthreading off) Intel 1000/pro copper (82541E Gigabit Ethernet Controller)

UDPmon was used to measure the network performance:

Request - response latency.
Memory-memory transfer rates.
packet loss
packet re-ordering
Inter packet Jitter.
1-way delay.
cpu load & number of interrupts

Summary (so far)

UDP throughput tests used 1500 byte MTU with bulk data flowing from CERN to RAL.

CERN to RAL showed good behaviour: a reasonably flat section then 1/t. Achieved ~995 Mbit/s with full 1472 byte packets.
No re-ordering for any packet size.
There was considerable packet loss, ~0.25%s for full 1472 byte packets, and lower levels 0.001 to 0.01% at packet spacings >15 us(790Mbit/s), 20 us (600Mbit/s) and even 40 us (300 Mbit/s).
The Packet Loss scatter plot shows that packets reported lost by udpmon were recorded as "InDiscards" i.e. lost in the receiving host stack due to buffers being full.

Made host-host tests over the LAN at RAL

RAL gftp0440 to gftp0441 showed good behaviour: a reasonably flat section then 1/t. Achieved ~995 Mbit/s with full 1472 byte packets.
No re-ordering for any packet size.
Packet loss 0.001 - 0.01% for most packet sizes mainly at spacings < 20 us.
Both CPUs are used for the transfers Hyperthreading was off. RAL systems seem to require less CPU that CERN IA-64 Itanium 2 i.e. in kernel mode 30 + 20% for 1472 byte packets vs 50 + 30% for Itanium
The Packet Loss scatter plot indicates losses occur in receiveing stack - CPU busy or insufficient power?

Made host-host tests over the LAN at CERN

CERN oplapro71 to oplapro72 showed reasonable behaviour: a reasonably flat section (more variable for packets <800 bytes) then 1/t. Achieved ~995 Mbit/s with full 1472 byte packets.
No re-ordering for any packet size.
Considerable packet loss 0.01 - 1% for all packet sizes and spacings. Small packets >=400 bytes had over 10% loss for spacings less than 10 us.
The Packet Loss scatter plot indicates losses occur in receiveing stack - CPU busy or insufficient power?

First UDP throughput tests 8-9 May are shown here. They used 1500 byte MTU and 1472 bytes of user data between CERN & RAL

CERN to RAL showed ~950 Mbit/s. but has a little packet loss (0.02%s). No re-ordering.
RAL to CERN ~950 Mbit/s only for packets >800 bytes. Showed large losses ~40-50% loss for packets <600 bytes and spacing <5 us.
Significant loss to 1% for all sizes - would kill TCP. No re-ordering

Topology of the Tests

UDPmon CERN-RAL Throughput, Packet loss & CPU load 1 Jul 05

MTU 1500 bytes. There was no packet re-ordering.

cern op71 --> RAL gftp0440 1Jul05
1,000,000 packets

Data taken: 08:20 - 10:45

Looks reasonable / very good

More loss than expected to ~0.25% for large packets >1400 bytes at line speed.
Surprising loss 0.001-0.01% most packet sizes and spacing >18 us.

More loss than tests host-host on RAL LAN

Packet re-ordering

None - As expected using UKLight

% CPU use in kernel mode sending node CERN Itanium 2

% CPU use in kernel mode receiving node

The other 2 cpus did no work - Hyperthreading was off

Packet Loss plot

For both packet sizes, packets reported lost by udpmon were recorded as "InDiscards" i.e. lost in the receiving host stack due to buffers being full.

UDPmon Host-LAN-Host Throughput, Packet loss & CPU load 1 Jul 05

MTU 1500 bytes. There was no packet re-ordering.

RAL gftp0440 --> 441 30 Jun 05 1,000,000 packets Data taken: 11:45 - 13:00 UK time	cern op71 --> op72 1 Jul 05 (plots say 30Jun but its the correct data) 1,000,000 packets Data taken: 01:00 - 03:15 CERN time (UK+1)

Throughput at line speed for packets >400 bytes	Looks reasonable
Packet Loss


~0.001-0.01% loss for most packet sizes Better than the CERN-->RAL test	Considerable packet loss 0.01 - 1% for all packet sizes and spacings. Small packets >=400 bytes had over 10% loss for spacings less than 10 us.
Packet re-ordering

none - good its level2 on a LAN
% CPU use in kernel mode sending node


% CPU use in kernel mode receiving node


Both CPUs are used for the transfers. RAL systems seem to require less CPU that CERN IA-64 Itanium 2 30 + 20% for 1472 byte packets vs 50 + 30%


	larger values correspond to shorter packet spacing
Packet Loss plot


Plot shows that number reported lost by UDPmon is equal to those discarded by the stack Suggests Insufficient CPU power - but dont see saturation in the kernel CPU plots.	Plot shows that number reported lost by UDPmon is equal to those discarded by the stack - Suggests Insufficient CPU power.