【mongoDB】mongoDB 从节点进程退出原因分析

2024-01-14 本文已影响0人 Bogon

mongoDB部署架构为1主1从1仲裁，从节点进程突然崩溃退出，日志如下：

2024-01-15T15:41:58.123+0800 E STORAGE  [conn82] WiredTiger error (14) [1705304518:123136][8227:0x7f1239a7d700], file:fileDB/index-300-911693708088245535.wt, WT_CURSOR.search_near: /path/to/fileDB/index-300-911693708088245535.wt: handle-read: pread: failed to read 8192 bytes at offset 434053120: Bad address
2024-01-15T15:41:58.123+0800 I -        [conn82] Invariant failure: ret resulted in status UnknownError: 14: Bad address at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp 840
2024-01-15T15:41:58.123+0800 I -        [conn82] 

***aborting after invariant() failure


2024-01-15T15:41:58.141+0800 F -        [conn82] Got signal: 6 (Aborted).

 0x7f126de1a1f1 0x7f126de19409 0x7f126de198ed 0x7f126b5667e0 0x7f126b1f54f5 0x7f126b1f6cd5 0x7f126d0acce7 0x7f126daf845e 0x7f126d42acb7 0x7f126d42af6f 0x7f126d43b333 0x7f126d419afe 0x7f126d43b333 0x7f126d42f726 0x7f126d43b333 0x7f126d74651a 0x7f126d746e3b 0x7f126d32695b 0x7f126d2fd53f 0x7f126d2fec61 0x7f126d91db1b 0x7f126d51d172 0x7f126d51f176 0x7f126d11dc1d 0x7f126d11e54d 0x7f126dd79031 0x7f126b55eaa1 0x7f126b2abc4d


----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F126C88E000","o":"158C1F1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F126C88E000","o":"158B409"},{"b":"7F126C88E000","o":"158B8ED"},{"b":"7F126B557000","o":"F7E0"},{"b":"7F126B1C3000","o":"324F5","s":"gsignal"},{"b":"7F126B1C3000","o":"33CD5","s":"abort"},{"b":"7F126C88E000","o":"81ECE7","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj"},{"b":"7F126C88E000","o":"126A45E"},{"b":"7F126C88E000","o":"B9CCB7","s":"_ZN5mongo9IndexScan13initIndexScanEv"},{"b":"7F126C88E000","o":"B9CF6F","s":"_ZN5mongo9IndexScan6doWorkEPm"},{"b":"7F126C88E000","o":"BAD333","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"7F126C88E000","o":"B8BAFE","s":"_ZN5mongo10FetchStage6doWorkEPm"},{"b":"7F126C88E000","o":"BAD333","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"7F126C88E000","o":"BA1726","s":"_ZN5mongo10LimitStage6doWorkEPm"},{"b":"7F126C88E000","o":"BAD333","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"7F126C88E000","o":"EB851A","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"7F126C88E000","o":"EB8E3B","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"7F126C88E000","o":"A9895B","s":"_ZN5mongo7FindCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE"},{"b":"7F126C88E000","o":"A6F53F","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"7F126C88E000","o":"A70C61","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"7F126C88E000","o":"108FB1B","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"7F126C88E000","o":"C8F172"},{"b":"7F126C88E000","o":"C91176","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"7F126C88E000","o":"88FC1D","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE"},{"b":"7F126C88E000","o":"89054D"},{"b":"7F126C88E000","o":"14EB031"},{"b":"7F126B557000","o":"7AA1"},{"b":"7F126B1C3000","o":"E8C4D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.17", "gitVersion" : "7c14a47868643bb691a507a92fe25541f998eca4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-431.el6.x86_64", "version" : "#1 SMP Fri Nov 22 03:15:09 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "b" : "7F126C88E000", "elfType" : 3, "buildId" : "DA910B7EE39255ECC5F9EF734C1643A8B6DA72A3" }, { "b" : "7FFF5F4FF000", "elfType" : 3, "buildId" : "81A81BE2E44C93640ADEDB62ADC93A47F4A09DD1" }, { "b" : "7ED6F45FF000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "D256E285C5E11D9A99EB04CA7651003A8F67B64E" }, { "b" : "7ED382A1A000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "1EDB45C205A844A75EBBB4F0075E705803FFB85B" }, { "b" : "7ED38E412000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "552CEC3216281CCFD7FA6432C723D50163255823" }, { "b" : "7ED38E60E000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "2AF795BFFD122309BA3359FEBABB5D0967403D17" }, { "b" : "7ED38D78A000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "4AAEE970B045D8BF946578B9C7F3AB5CDE9AB44A" }, { "b" : "7ED382574000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDC925E58FE28DCA536993EB13179C739F1E6566" }, { "b" : "7ED38E357000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4EA475CD3FD3B69B6C95D9381FA74B36DB4992EF" }, { "b" : "7ED38E3C3000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "3E5ABB69E7969FB2C80A7D3637D62395D6C3F827" }, { "b" : "7ED38FC6B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "97AF4B77212F74CFF72B6C013E6AA2D74A97EF60" }, { "b" : "7ED6F357F000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9A737F8BF10FC99C37CC404D3FC188F6E11FEDD9" }, { "b" : "7ED6F3E98000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "8D3D6E28DF6EB3752642A7031AAC17D39EA4265D" }, { "b" : "7ED6F6894000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "7EC54D6E88BB7D2C1284117C2A483496A01EAAF4" }, { "b" : "7ED6F3268000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "CC89B4C8CDCCD32BA610BC72784DC3B7E9BD9E19" }, { "b" : "7ED38B84E000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "4786A2A5D30B121601958E84D643C70C13C4FBA5" }, { "b" : "7ED38C638000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7ED6F382D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E0C522C589F775C324330BE09CE67DC83950A213" }, { "b" : "7ED6F2E2A000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7ED38B40B000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f126de1a1f1]
 mongod(+0x158B409) [0x7f126de19409]
 mongod(+0x158B8ED) [0x7f126de198ed]
 libpthread.so.0(+0xF7E0) [0x7f126b5667e0]
 libc.so.6(gsignal+0x35) [0x7f126b1f54f5]
 libc.so.6(abort+0x175) [0x7f126b1f6cd5]
 mongod(_ZN5mongo25fassertFailedWithLocationEiPKcj+0x0) [0x7f126d0acce7]
 mongod(+0x126A45E) [0x7f126daf845e]
 mongod(_ZN5mongo9IndexScan13initIndexScanEv+0x2C7) [0x7f126d42acb7]
 mongod(_ZN5mongo9IndexScan6doWorkEPm+0x14F) [0x7f126d42af6f]
 mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x7f126d43b333]
 mongod(_ZN5mongo10FetchStage6doWorkEPm+0x29E) [0x7f126d419afe]
 mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x7f126d43b333]
 mongod(_ZN5mongo10LimitStage6doWorkEPm+0x76) [0x7f126d42f726]
 mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x7f126d43b333]
 mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x19A) [0x7f126d74651a]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x7f126d746e3b]
 mongod(_ZN5mongo7FindCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE+0x106B) [0x7f126d32695b]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x4FF) [0x7f126d2fd53f]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0xF81) [0x7f126d2fec61]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x23B) [0x7f126d91db1b]
 mongod(+0xC8F172) [0x7f126d51d172]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x746) [0x7f126d51f176]
 mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1FD) [0x7f126d11dc1d]
 mongod(+0x89054D) [0x7f126d11e54d]
 mongod(+0x14EB031) [0x7f126dd79031]
 libpthread.so.0(+0x7AA1) [0x7f126b55eaa1]
 libc.so.6(clone+0x6D) [0x7f126b2abc4d]
-----  END BACKTRACE  -----

2024-01-15T15:41:58.123+0800 E STORAGE  [conn82] WiredTiger error (14) [1705304518:123136][8227:0x7f1239a7d700], file:fileDB/index-300-911693708088245535.wt, WT_CURSOR.search_near: /path/to/fileDB/index-300-911693708088245535.wt: handle-read: pread: failed to read 8192 bytes at offset 434053120: Bad address
2024-01-15T15:41:58.123+0800 I -        [conn82] Invariant failure: ret resulted in status UnknownError: 14: Bad address at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp 840
2024-01-15T15:41:58.123+0800 I -        [conn82] 

***aborting after invariant() failure


2024-01-15T15:41:58.141+0800 F -        [conn82] Got signal: 6 (Aborted).

 0x7f126de1a1f1 0x7f126de19409 0x7f126de198ed 0x7f126b5667e0 0x7f126b1f54f5 0x7f126b1f6cd5 0x7f126d0acce7 0x7f126daf845e 0x7f126d42acb7 0x7f126d42af6f 0x7f126d43b333 0x7f126d419afe 0x7f126d43b333 0x7f126d42f726 0x7f126d43b333 0x7f126d74651a 0x7f126d746e3b 0x7f126d32695b 0x7f126d2fd53f 0x7f126d2fec61 0x7f126d91db1b 0x7f126d51d172 0x7f126d51f176 0x7f126d11dc1d 0x7f126d11e54d 0x7f126dd79031 0x7f126b55eaa1 0x7f126b2abc4d

根据给出的日志，可以看到以下关键信息：

错误信息 "WiredTiger error (14)" 指示了一个WiredTiger存储引擎的错误。
错误信息 "Bad address" 表明在读取文件时遇到了地址错误。
错误信息 "Invariant failure" 表明出现了不符合预期的情况。
错误信息 "Got signal: 6 (Aborted)" 表明进程因收到信号6（Abort）而终止。

综合分析，可能的原因是：

文件损坏：日志中提到的文件 "/path/to/fileDB/index-300-911693708088245535.wt" 可能出现了损坏，导致读取错误。
硬件故障：错误信息中的 "Bad address" 可能是由硬件故障（如内存错误）引起的。
数据库引擎问题：WiredTiger存储引擎可能存在某种错误或不兼容性，导致进程崩溃。

为了进一步确定问题的根本原因，可能需要进行以下步骤：

检查文件：确认文件 "/path/to/fileDB/index-300-911693708088245535.wt" 的完整性和一致性，尝试修复或替换该文件。
检查硬件：检查服务器硬件是否存在故障，例如内存错误或其他硬件问题。
更新软件：确保使用的MongoDB版本和WiredTiger存储引擎版本是最新的，有可能存在已知的问题和已修复的错误。
检查其他日志：查看更多相关日志，特别是在崩溃前的日志，以寻找其他可能的线索。

如果问题仍然存在，可能需要向MongoDB的官方支持渠道寻求进一步的帮助和调试。

----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F126C88E000","o":"158C1F1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F126C88E000","o":"158B409"},{"b":"7F126C88E000","o":"158B8ED"},{"b":"7F126B557000","o":"F7E0"},{"b":"7F126B1C3000","o":"324F5","s":"gsignal"},{"b":"7F126B1C3000","o":"33CD5","s":"abort"},{"b":"7F126C88E000","o":"81ECE7","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj"},{"b":"7F126C88E000","o":"126A45E"},{"b":"7F126C88E000","o":"B9CCB7","s":"_ZN5mongo9IndexScan13initIndexScanEv"},{"b":"7F126C88E000","o":"B9CF6F","s":"_ZN5mongo9IndexScan6doWorkEPm"},{"b":"7F126C88E000","o":"BAD333","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"7F126C88E000","o":"B8BAFE","s":"_ZN5mongo10FetchStage6doWorkEPm"},{"b":"7F126C88E000","o":"BAD333","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"7F126C88E000","o":"BA1726","s":"_ZN5mongo10LimitStage6doWorkEPm"},{"b":"7F126C88E000","o":"BAD333","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"7F126C88E000","o":"EB851A","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"7F126C88E000","o":"EB8E3B","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"7F126C88E000","o":"A9895B","s":"_ZN5mongo7FindCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE"},{"b":"7F126C88E000","o":"A6F53F","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"7F126C88E000","o":"A70C61","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"7F126C88E000","o":"108FB1B","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"7F126C88E000","o":"C8F172"},{"b":"7F126C88E000","o":"C91176","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"7F126C88E000","o":"88FC1D","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE"},{"b":"7F126C88E000","o":"89054D"},{"b":"7F126C88E000","o":"14EB031"},{"b":"7F126B557000","o":"7AA1"},{"b":"7F126B1C3000","o":"E8C4D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.17", "gitVersion" : "7c14a47868643bb691a507a92fe25541f998eca4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-431.el6.x86_64", "version" : "#1 SMP Fri Nov 22 03:15:09 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "b" : "7F126C88E000", "elfType" : 3, "buildId" : "DA910B7EE39255ECC5F9EF734C1643A8B6DA72A3" }, { "b" : "7FFF5F4FF000", "elfType" : 3, "buildId" : "81A81BE2E44C93640ADEDB62ADC93A47F4A09DD1" }, { "b" : "7ED6F45FF000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "D256E285C5E11D9A99EB04CA7651003A8F67B64E" }, { "b" : "7ED382A1A000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "1EDB45C205A844A75EBBB4F0075E705803FFB85B" }, { "b" : "7ED38E412000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "552CEC3216281CCFD7FA6432C723D50163255823" }, { "b" : "7ED38E60E000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "2AF795BFFD122309BA3359FEBABB5D0967403D17" }, { "b" : "7ED38D78A000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "4AAEE970B045D8BF946578B9C7F3AB5CDE9AB44A" }, { "b" : "7ED382574000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDC925E58FE28DCA536993EB13179C739F1E6566" }, { "b" : "7ED38E357000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4EA475CD3FD3B69B6C95D9381FA74B36DB4992EF" }, { "b" : "7ED38E3C3000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "3E5ABB69E7969FB2C80A7D3637D62395D6C3F827" }, { "b" : "7ED38FC6B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "97AF4B77212F74CFF72B6C013E6AA2D74A97EF60" }, { "b" : "7ED6F357F000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9A737F8BF10FC99C37CC404D3FC188F6E11FEDD9" }, { "b" : "7ED6F3E98000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "8D3D6E28DF6EB3752642A7031AAC17D39EA4265D" }, { "b" : "7ED6F6894000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "7EC54D6E88BB7D2C1284117C2A483496A01EAAF4" }, { "b" : "7ED6F3268000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "CC89B4C8CDCCD32BA610BC72784DC3B7E9BD9E19" }, { "b" : "7ED38B84E000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "4786A2A5D30B121601958E84D643C70C13C4FBA5" }, { "b" : "7ED38C638000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7ED6F382D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E0C522C589F775C324330BE09CE67DC83950A213" }, { "b" : "7ED6F2E2A000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7ED38B40B000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f126de1a1f1]
 mongod(+0x158B409) [0x7f126de19409]
 mongod(+0x158B8ED) [0x7f126de198ed]
 libpthread.so.0(+0xF7E0) [0x7f126b5667e0]
 libc.so.6(gsignal+0x35) [0x7f126b1f54f5]
 libc.so.6(abort+0x175) [0x7f126b1f6cd5]
 mongod(_ZN5mongo25fassertFailedWithLocationEiPKcj+0x0) [0x7f126d0acce7]
 mongod(+0x126A45E) [0x7f126daf845e]
 mongod(_ZN5mongo9IndexScan13initIndexScanEv+0x2C7) [0x7f126d42acb7]
 mongod(_ZN5mongo9IndexScan6doWorkEPm+0x14F) [0x7f126d42af6f]
 mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x7f126d43b333]
 mongod(_ZN5mongo10FetchStage6doWorkEPm+0x29E) [0x7f126d419afe]
 mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x7f126d43b333]
 mongod(_ZN5mongo10LimitStage6doWorkEPm+0x76) [0x7f126d42f726]
 mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x7f126d43b333]
 mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x19A) [0x7f126d74651a]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x7f126d746e3b]
 mongod(_ZN5mongo7FindCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE+0x106B) [0x7f126d32695b]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x4FF) [0x7f126d2fd53f]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0xF81) [0x7f126d2fec61]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x23B) [0x7f126d91db1b]
 mongod(+0xC8F172) [0x7f126d51d172]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x746) [0x7f126d51f176]
 mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1FD) [0x7f126d11dc1d]
 mongod(+0x89054D) [0x7f126d11e54d]
 mongod(+0x14EB031) [0x7f126dd79031]
 libpthread.so.0(+0x7AA1) [0x7f126b55eaa1]
 libc.so.6(clone+0x6D) [0x7f126b2abc4d]
-----  END BACKTRACE  -----

此类日志是 mongod 进程崩溃后生成的错误回溯（backtrace）。
它提供了一些关于崩溃发生时函数调用堆栈的信息，用于帮助定位问题。

如何检测Linux上磁盘是否有问题？

在Linux服务器上检查磁盘是否有坏道或其他问题，可以按照以下步骤进行：

使用SMART工具检测磁盘健康状况：
- 安装smartmontools软件包：运行sudo apt-get install smartmontools（适用于Debian/Ubuntu）或sudo yum install smartmontools（适用于CentOS/RHEL）来安装smartmontools软件包。
- 运行SMART测试：使用sudo smartctl -a /dev/sda命令（假设磁盘设备为/dev/sda）来查看磁盘的SMART属性和健康状态。关注属性中的错误计数、预测故障和健康状况等信息。
运行磁盘检测工具：
- 使用badblocks命令：运行sudo badblocks -v /dev/sda命令（假设磁盘设备为/dev/sda）来扫描磁盘并检测坏道。此命令可能需要较长时间才能完成扫描。
- 使用fsck命令：运行sudo fsck /dev/sda命令（假设磁盘设备为/dev/sda）来检查文件系统的完整性并修复任何错误。请注意，这将在磁盘上进行读写操作，因此请确保在操作之前卸载该磁盘分区。
监视磁盘性能和状态：
- 使用iotop命令：运行sudo iotop命令来监视磁盘的读写速度和进程的磁盘活动情况。
- 使用df命令：运行df -h命令来查看磁盘使用情况和可用空间。
- 使用dmesg命令：运行dmesg | grep -i "error\|fail"命令来检查内核日志中是否存在与磁盘相关的错误或失败信息。

如果要检测基于/dev/sdb创建的逻辑卷/dev/mapper/vg_data-lv_data，可以按照以下步骤进行：

使用SMART工具检测磁盘健康状况：
- 安装smartmontools软件包：运行sudo apt-get install smartmontools（适用于Debian/Ubuntu）或sudo yum install smartmontools（适用于CentOS/RHEL）来安装smartmontools软件包。
- 运行SMART测试：使用sudo smartctl -a /dev/sdb命令来查看/dev/sdb磁盘的SMART属性和健康状态。关注属性中的错误计数、预测故障和健康状况等信息。
运行磁盘检测工具：
- 使用badblocks命令：运行sudo badblocks -v /dev/sdb命令来扫描/dev/sdb磁盘并检测坏道。此命令可能需要较长时间才能完成扫描。
检查逻辑卷的完整性：
- 使用fsck命令：运行sudo fsck /dev/mapper/vg_data-lv_data命令来检查逻辑卷的文件系统完整性并修复任何错误。
监视磁盘性能和状态：
- 使用iotop命令：运行sudo iotop命令来监视磁盘的读写速度和进程的磁盘活动情况。
- 使用df命令：运行df -h命令来查看逻辑卷的使用情况和可用空间。

请注意，以上步骤中的命令可能需要root权限才能运行。
在执行任何磁盘检测或修复操作之前，务必备份重要数据。
如果您对这些步骤不确定，建议咨询系统管理员或专业的Linux技术支持人员。

【mongoDB】mongoDB 从节点进程退出原因分析

猜你喜欢

热点阅读