多核心的處理器最早是透過對稱多處理 SMP (Symmetric multiprocessing) 的方式,所有的 CPU 核心對於記憶體的存取是共用的,但是當 CPU 核心數太多時反而是一個限制,當不同的處理器需要交換資料時都是透過系統匯流排將資料儲存在記憶體中,但當核心數多時,交換資料變成常態, CPU 與記憶體之間的速度跟不上 CPU 處理的速度.越多的核心反而讓整體效能降低.
因此有了 Intel 的 NUMA (Non-uniform memory access),他把 CPU 與記憶體區分成不同的結點 Node (不同的 CPU 各自擁有記憶體),彼此的 CPU 節點再透過 QPI (Intel QuickPath Interconnect) 這個介面做溝通.
關於 CPU 的演進可以參考鳥哥網站 http://linux.vbird.org/linux_enterprise/cputune.php
測試環境為 Ubuntu 16.04 64bits
來看一下我系統的下的 NUMA 狀態,可以使用 numactl , numastat 這兩個指令,為非預設安裝,需要透過 apt 來安裝.
root@ubuntu:~# apt-get install numactl Reading package lists... Done Building dependency tree Reading state information... Done The following package was automatically installed and is no longer required: ubuntu-core-launcher Use 'apt autoremove' to remove it. The following NEW packages will be installed: numactl 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 30.2 kB of archives. After this operation, 117 kB of additional disk space will be used. Get:1 http://tw.archive.ubuntu.com/ubuntu xenial/universe amd64 numactl amd64 2.0.11-1ubuntu1 [30.2 kB] Fetched 30.2 kB in 0s (40.9 kB/s) Selecting previously unselected package numactl. (Reading database ... 205238 files and directories currently installed.) Preparing to unpack .../numactl_2.0.11-1ubuntu1_amd64.deb ... Unpacking numactl (2.0.11-1ubuntu1) ... Processing triggers for man-db (2.7.5-1) ... Setting up numactl (2.0.11-1ubuntu1) ...
透過 numastat 可以看到我的系統有兩個 Node (Node0 與 Node1)
root@ubuntu:~# numastat
node0 node1
numa_hit 6018617 4385882
numa_miss 0 0
numa_foreign 0 0
interleave_hit 4800 15116
local_node 6003635 4380820
other_node 14982 5062
上面的數值所代表
- numa_hit
Memory successfully allocated on this node as intended.
記憶體成功配置至此節點 - numa_miss
Memory allocated on this node despite the process preferring some different node. Each numa_miss has a numa_foreign on another node.
原先預定的節點的記憶體不足,而配置至此節點. numa_miss 與另一個節點的 numa_foreign 是相對應的. - numa_foreign
Memory intended for this node, but actually allocated on some different node. Each numa_foreign has a numa_miss on another node.
原先預定至此節點的記憶體但被配置至其他節點上. numa_foreign 與另一個節點的 numa_miss 是相對應的. - interleave_hit
Interleaved memory successfully allocated on this node as intended.
The number of interleave policy allocations that were intended for a specific node and succeeded there. - local_node
Memory allocated on this node while a process was running on it.
該節點上的程序成功配置到該節點的記憶體空間. - other_node
Memory allocated on this node while a process was running on some other node.
該節點上的程序,成功配置到另一個節點的記憶體空間.
numactl 指令參考
–hardware , -H : Show inventory of available node on the system.
root@ubuntu:~# numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 8 9 10 11 node 0 size: 3939 MB node 0 free: 3034 MB node 1 cpus: 4 5 6 7 12 13 14 15 node 1 size: 4029 MB node 1 free: 3468 MB node distances: node 0 1 0: 10 20 1: 20 10
–show, -s : Show NUMA policy setting of the current process.
root@ubuntu:~# numactl --show policy: default preferred node: current physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cpubind: 0 1 nodebind: 0 1 membind: 0 1