[Node] 内存溢出与 old-space 大小调整

2020-08-21 本文已影响0人何幻

1. 内存溢出

V8 为 Node.js 应用，默认只会分配了大概 1400 MB（仅本地测试的结果）的内存空间。
超出了这个限额，就会内存溢出。

我们来写一段程序，制造一个内存溢出错误，

// index.js

const main = n => {
  const is = [];
  while (true) {
    const js = [];
    for (let j = 0; j < n; j++) {
      const ks = [];
      for (let k = 0; k < n; k++) {
        const ls = [];
        for (let l = 0; l < n; l++) {
          ls.push([]);
        }
        ks.push(ls);
      }
      js.push(ks);
    }
    is.push(js);
    showMemoryUsage();
  }
};

const showMemoryUsage = () => {
  const { rss, heapTotal, heapUsed } = process.memoryUsage();
  log(`rss = ${byteToMB(rss)}, heapTotal = ${byteToMB(heapTotal)}, heapUsed = ${byteToMB(heapUsed)}`);
};

const byteToMB = byte => `${(byte / 1024 / 1024).toFixed(2)} MB`;
const log = message => console.log(message);

main(10);

执行一下，

$ node index.js
rss = 22.30 MB, heapTotal = 6.23 MB, heapUsed = 3.88 MB
rss = 22.74 MB, heapTotal = 9.23 MB, heapUsed = 3.83 MB
...
rss = 1450.73 MB, heapTotal = 1426.73 MB, heapUsed = 1400.11 MB
rss = 1450.77 MB, heapTotal = 1426.73 MB, heapUsed = 1400.17 MB

<--- Last few GCs --->

[40615:0x103800000]    19849 ms: Mark-sweep 1398.1 (1424.7) -> 1398.0 (1425.7) MB, 2473.5 / 0.0 ms  (+ 0.0 ms in 20 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2592 ms) (average mu = 0.124, current mu = 0.084) alloca

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x234ab8cdbe3d]
Security context: 0x016d8749e6c1 <JSObject>
    1: main [0x16d0208b7c1] [/Users/.../index.js:~1] [pc=0x234ab8ce5238](this=0x016d5928d461 <JSGlobal Object>,n=10)
    2: /* anonymous */ [0x16d0208b7f9] [/Users/.../index.js:29] [bytecode=0x16df5ad7991 offset=43](this=0x016d0208b929 <Object map = 0x16d3ce02571>,exports=0x016d0208b929 <Object map = 0x16d3ce02571>,require=0x0...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100d68631 node::Abort() (.cold.1) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 2: 0x10003b124 node_module_register [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 3: 0x10003b2e5 node::OnFatalError(char const*, char const*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 4: 0x1001a9097 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 5: 0x1001a9031 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 6: 0x100593592 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 7: 0x100595d33 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 8: 0x100591bda v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 9: 0x10058fac5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
10: 0x10059cc55 v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
11: 0x10059ccbf v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
12: 0x10056cdb3 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
13: 0x1007fddaf v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
14: 0x234ab8cdbe3d
15: 0x234ab8ce5238
16: 0x234ab8c918d5
17: 0x234ab8c918d5
18: 0x234ab8c918d5
19: 0x234ab8c918d5
20: 0x234ab8c918d5
[1]    40615 abort      node index.js

可以看到当内存占用达到 1400 MB 时，就报错了，

rss = 1450.77 MB, heapTotal = 1426.73 MB, heapUsed = 1400.17 MB

2. new-space & old-space

V8 使用了分代垃圾回收机制，将堆（heap）划分成了几个不同的空间。

参考一篇比较老的文章，A tour of V8: Garbage Collection #Heap organization

New-space
Old-pointer-space
Old-data-space
Large-object-space
Code-space
Cell-space, property-cell-space and map-space

文中提到了 new-space 和 old-space，

In the vast majority of programs, objects tend to die young: most objects have a very short lifetime, while a small minority of objects tend to live much longer. To take advantage of this behavior, V8 divides the heap into two generations. Objects are allocated in new-space, which is fairly small (between 1 and 8 MB, depending on behavior heuristics). Allocation in new space is very cheap: we just have an allocation pointer which we increment whenever we want to reserve space for a new object. When the allocation pointer reaches the end of new space, a scavenge (minor garbage collection cycle) is triggered, which quickly removes the dead objects from new space. Objects which have survived two minor garbage collections are promoted to old-space. Old-space is garbage collected during a mark-sweep or mark-compact (major garbage collection cycle), which is much less frequent. A major garbage collection cycle is triggered when we have promoted a certain amount of memory to old space. This threshold shifts over time depending on the size of old space and the behavior of the program.

大致含义是，新对象一般会先分配到 new-space 中（只有 1 - 8 M 大小），
new-space 满了以后会触发一次小型的（minor）垃圾回收，
如果对象经历了两次小型（minor）垃圾回收还活着，就会被移到 old-space 中（这个空间比较大）。
old-space 满了会经历一次大型（major）垃圾回收（采用标记-清除回收算法）。

所以，如果我们一直分配新对象且不释放它，就会最终被放到 old-space 中。
一旦 old-space 占用率超出限额，就会造成内存溢出。

注：
V8 的垃圾回收机制也是与时俱进的，
最近的一些进展，可参考 Trash talk: the Orinoco garbage collector，
上面介绍的内容大同小异。

3. 调整 old-space 大小

有以下两种方式，可以调整 old-space 大小，
在某些情况下，或许能暂时避免 Node.js 内存溢出。

--max-old-space-size=SIZE

$ node --max-old-space-size=2048 index.js

NODE_OPTIONS=options...

$ NODE_OPTIONS='--max-old-space-size=2048' node index.js

我们来看看效果，

$ node --max-old-space-size=2048 index.js
rss = 22.30 MB, heapTotal = 6.23 MB, heapUsed = 3.88 MB
rss = 22.74 MB, heapTotal = 9.23 MB, heapUsed = 3.83 MB
...

rss = 2110.81 MB, heapTotal = 2084.73 MB, heapUsed = 2047.35 MB
rss = 2110.86 MB, heapTotal = 2084.73 MB, heapUsed = 2047.40 MB

<--- Last few GCs --->

[40880:0x103800000]    36415 ms: Mark-sweep 2045.2 (2082.2) -> 2045.1 (2082.7) MB, 3743.6 / 0.0 ms  (+ 0.0 ms in 13 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 3907 ms) (average mu = 0.108, current mu = 0.061) alloca[40880:0x103800000]    36455 ms: Scavenge 2046.4 (2083.7) -> 2046.4 (2084.7) MB, 13.2 / 0.0 ms  (average mu = 0.108, current mu = 0.061) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x37e08555be3d]
Security context: 0x10607f39e6c1 <JSObject>
    1: main [0x1060cbe8c051] [/Users/.../index.js:~1] [pc=0x37e0855f0ab8](this=0x10607f10d461 <JSGlobal Object>,n=10)
    2: /* anonymous */ [0x1060cbe8c089] [/Users/.../index.js:29] [bytecode=0x10602e4d79e1 offset=43](this=0x1060cbe8c1b9 <Object map = 0x1060e9202571>,exports=0x1060cbe8c1b9 <Object map = 0x1060e9202571>,requir...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100d68631 node::Abort() (.cold.1) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 2: 0x10003b124 node_module_register [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 3: 0x10003b2e5 node::OnFatalError(char const*, char const*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 4: 0x1001a9097 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 5: 0x1001a9031 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 6: 0x100593592 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 7: 0x100595d33 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 8: 0x100591bda v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
 9: 0x10058fac5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
10: 0x10059cc55 v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
11: 0x10059ccbf v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
12: 0x10056cdb3 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
13: 0x1007fddaf v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/.../.nvm/versions/node/v10.21.0/bin/node]
14: 0x37e08555be3d
15: 0x37e0855f0ab8
16: 0x37e0855118d5
17: 0x37e0855118d5
18: 0x37e0855118d5
19: 0x37e0855118d5
20: 0x37e0855118d5
[1]    40880 abort      node --max-old-space-size=2048 index.js

我们看到产生内存溢出时，内存占用情况不同了，

# 默认 old-space 大小
rss = 1450.77 MB, heapTotal = 1426.73 MB, heapUsed = 1400.17 MB

# 调整 old-space 大小为 2048 MB
rss = 2110.86 MB, heapTotal = 2084.73 MB, heapUsed = 2047.40 MB

参考

A tour of V8: Garbage Collection
Trash talk: the Orinoco garbage collector
Node.js - Command line options: --max-old-space-size=SIZE
Node.js - Command line options: NODE_OPTIONS=options...

[Node] 内存溢出与 old-space 大小调整

1. 内存溢出

2. new-space & old-space

3. 调整 old-space 大小

参考

猜你喜欢

热点阅读