µ
µ
Verbs Consumer Work Request Send Work Queue WQE WQE WQE Receive Work Queue WQE WQE WQE Queue Pair To the fabric Work Completion Completion Queue CQE CQE CQE HCA Hardware
Transactions and Operations Transactions and Operations Channel Adapter Channel Adapter Send Receive Send Receive Transport Transport Layer (Network Layer) Transport Packet Link Layer Packet Port Physical Layer Port Physical Link
n n
1..N
8MB P AGE SIZE 2048 4bytes 4096 = 2
0 1... 8 1
1 1... 8 0 1... 8 0 9... 16 1 1 9 2 10
1200 1000 Native PCI-attached Bridged Xen2MX-plain Xen2MX-tuned 800 Latency (us) 600 400 200 0 1 4 16 64 256 1K 4K 16K 64K 256K Message Size (Bytes) µ µ
140 120 Native PCI-attached Bridged Xen2MX-plain Xen2MX-tuned 100 Latency (us) 80 60 40 20 0 1 4 16 64 256 1K 4K 16K Message Size (Bytes) µ
1200 1000 Native PCI-attached Bridged Xen2MX-plain Xen2MX-tuned 800 Bandwidth (MB/s) 600 400 200 0 256 1K 4K 16K 64K 256K 1M 4M Message Size (Bytes)
1GB messagesize
3 2.5 Normalized Time 2 1.5 1 deregister register recv send 0.5 0 TUNED PLAIN TUNED PLAIN TUNED PLAIN TUNED PLAIN TUNED PLAIN TUNED PLAIN 64k 128k 256k 512k 1M 2M
3 2.5 Normalized Time 2 1.5 1 0.5 deregister poke host register recv send 0 TUNED PLAIN TUNED PLAIN TUNED PLAIN TUNED PLAIN TUNED PLAIN TUNED PLAIN 64k 128k 256k 512k 1M 2M
1200 1000 Xen2MX Throughput Bridged Throughput Xen2MX CPU utilization Bridged CPU utilization 100 80 800 Throughput (MB/s) 600 400 60 40 200 20 0 1 2 4 8 Number of VMs per container 0
35 30 Guest Driver Domain 25 Percentage 20 15 10 5 0 XEN2MX BRIDGED XEN2MX BRIDGED XEN2MX BRIDGED XEN2MX BRIDGED 1 2 4 8 25000 20000 domu_softirq domu_irq domu_system domu_user Time (us) 15000 10000 5000 0 BRIDGED XEN2MX BRIDGED XEN2MX BRIDGED XEN2MX BRIDGED 1 2 4 8 XEN2MX
100 1200 80 1000 CPU utilization (%) 60 40 1 VM 2VMs 4VMs 8VMs 1 VM 2 VMs 4 VMs 8 VMs 800 600 Throughput (MB/s) 400 20 200 0 64K 256K 1M 4M 16M Message Size (bytes) 0
< >
100 1200 80 1000 CPU utilization (%) 60 40 Bridged CPU utilization Xen2MX CPU utilization Bridged throughput Xen2MX throughput 800 600 Throughput (MB/s) 400 20 200 0 64K 256K 1M 4M 16M Message Size (bytes) 0 1... 20 0 21... 40 1 1 21 2 22 >