[夏肇毅知識運算網 Mark Hsia's Knowledge Computing Web] 夏肇毅部落格

我們知道一個Socket的 最大連線數是65535

要如何讓伺服器同時連接1200萬個連線呢?

在這篇文章內敘述了他們的做法:

Scaling to 12 Million Concurrent Connections: How ...

伺服器端:

增大 kernel 最大值:

echo 20000500 > /proc/sys/fs/nr_open


增大socket descriptors至2000萬:

ulimit -n 20000500

客戶端:

客戶埠數:

增大客戶臨時埠範圍至65535:

 sysctl -w net.ipv4.ip_local_port_range="500 65535"

 增大socket descriptors至200萬:
echo 3000000 > /proc/sys/fs/nr_open
ulimit -n 2000000


在TIME_WAIT 時 重複使用 socket :
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle


 增大TCP 的memory pages allocated至2303190 (約8.7 GB) :
sysctl -w net.ipv4.tcp_mem="383865   511820   2303190"

Linux Kernel 調整

調整socket buffer sizes:

sysctl -w net.ipv4.tcp_rmem="1024   4096   16384"
sysctl -w net.ipv4.tcp_wmem="1024   4096   16384"
sysctl -w net.ipv4.tcp_moderate_rcvbuf="0"

平衡硬體中斷

看一下網路卡p1p1的中斷:

# grep p1p1 /proc/interrupts

平衡每一tx/rx queue的硬體中斷到不同CPU 核心:

[root@t15 ~]# cat smp_affinity.sh

echo "1" >/proc/irq/77/smp_affinity

echo "2" >/proc/irq/78/smp_affinity

echo "4" >/proc/irq/79/smp_affinity

echo "8" >/proc/irq/80/smp_affinity

echo "10" >/proc/irq/81/smp_affinity

echo "20" >/proc/irq/82/smp_affinity

echo "40" >/proc/irq/83/smp_affinity

echo "80" >/proc/irq/84/smp_affinity

echo "100" >/proc/irq/85/smp_affinity

echo "200" >/proc/irq/86/smp_affinity

echo "400" >/proc/irq/87/smp_affinity

echo "800" >/proc/irq/88/smp_affinity

echo "1000" >/proc/irq/89/smp_affinity

echo "2000" >/proc/irq/90/smp_affinity

echo "4000" >/proc/irq/91/smp_affinity

echo "8000" >/proc/irq/92/smp_affinity

echo "10000" >/proc/irq/93/smp_affinity

echo "20000" >/proc/irq/94/smp_affinity

echo "40000" >/proc/irq/95/smp_affinity

echo "80000" >/proc/irq/96/smp_affinity

echo "100000" >/proc/irq/97/smp_affinity

echo "200000" >/proc/irq/98/smp_affinity

echo "400000" >/proc/irq/99/smp_affinity

echo "800000" >/proc/irq/100/smp_affinity

 看一下中斷是否平衡到 24 CPU 核心內:

# grep p1p1 /proc/interrupts

參考資料:

Scaling to 12 Million Concurrent Connections: How ...