个人极限测试，看看各种情况达到极限时会出现怎样的情况，然后研究并实验处理方法。
 会抽时间去测试并分享出来，测试内容包括但不限于：把内存占用光、写满磁盘空间、网速达到带宽上限、超出TCP最大连接数、重启系统、删除数据库数据、docker环境下的测试等等。

数据库相关

测试磁盘空间写满

 操作：运行程序断向数据库写入数据，同时使用linux命令不断地创建一些占大量空间的文件，直到磁盘空间被写满。测试数据库是mongodb，日志文件可在/etc/mongod.conf里看到，一般是/var/log/mongodb/mongod.log。
 结果：数据库停止运行，但应用程序还在正常运行，只是由于数据库已经停止运行所以写入数据失败。
 解决办法：做好服务器的监控，当磁盘达到警告阀值时通知管理员做处理。

1
2
3
4
5
# 查看磁盘空间
df
# 快速创建出一个占10G空间的文件
#Linux & and some filesystems (ext4, xfs, btrfs and ocfs2)
fallocate -l 10G 10Gigfile

 从日志文件里很容易看到原因：写入失败，
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
2019-02-22T06:10:43.005+0000 W FTDC     [ftdc] Uncaught exception in 'FileStreamFailed: Failed to write to interim file buffer for full-time diagnostic data capture: /var/lib/mongo/diagnostic.data/metrics.interim.temp' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
2019-02-22T06:11:32.906+0000 E STORAGE  [WTCheckpointThread] WiredTiger error (28) [1550815892:906948][3886:0x7f33b4418700], file:index-8--6709502618593790047.wt, WT_SESSION.checkpoint: __posix_file_write, 579: /var/lib/mongo/index-8--6709502618593790047.wt: handle-write: pwrite: failed to write 4096 bytes at offset 32768: No space left on device Raw: [1550815892:906948][3886:0x7f33b4418700], file:index-8--6709502618593790047.wt, WT_SESSION.checkpoint: __posix_file_write, 579: /var/lib/mongo/index-8--6709502618593790047.wt: handle-write: pwrite: failed to write 4096 bytes at offset 32768: No space left on device
2019-02-22T06:11:32.907+0000 E STORAGE  [WTCheckpointThread] WiredTiger error (28) [1550815892:907029][3886:0x7f33b4418700], file:index-8--6709502618593790047.wt, WT_SESSION.checkpoint: __ckpt_process, 724: index-8--6709502618593790047.wt: fatal checkpoint failure: No space left on device Raw: [1550815892:907029][3886:0x7f33b4418700], file:index-8--6709502618593790047.wt, WT_SESSION.checkpoint: __ckpt_process, 724: index-8--6709502618593790047.wt: fatal checkpoint failure: No space left on device
2019-02-22T06:11:32.907+0000 E STORAGE  [WTCheckpointThread] WiredTiger error (-31804) [1550815892:907044][3886:0x7f33b4418700], file:index-8--6709502618593790047.wt, WT_SESSION.checkpoint: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1550815892:907044][3886:0x7f33b4418700], file:index-8--6709502618593790047.wt, WT_SESSION.checkpoint: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-02-22T06:11:32.907+0000 F -        [WTCheckpointThread] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 409
2019-02-22T06:11:32.907+0000 F -        [WTCheckpointThread]

***aborting after fassert() failure


2019-02-22T06:11:32.932+0000 F -        [WTCheckpointThread] Got signal: 6 (Aborted).
 0x5618ad4df281 0x5618ad4de499 0x5618ad4de97d 0x7f33bdb3c5d0 0x7f33bd796207 0x7f33bd7978f8 0x5618abaefba7 0x5618abbe7fd6 0x5618abc5a501 0x5618aba79cdb 0x5618aba7a0fb 0x5618abd2c051 0x5618abc900ea 0x5618abc36ba6 0x5618abc3c567 0x5618abca7e5c 0x5618abc6b43a 0x5618abc6d82f 0x5618abc6e8ab 0x5618abc5378a 0x5618abbc8817 0x5618acf34ad1 0x5618ad5eef30 0x7f33bdb34dd5 0x7f33bd85dead
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5618AB0D8000","o":"2407281","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5618AB0D8000","o":"2406499"},{"b":"5618AB0D8000","o":"240697D"},{"b":"7F33BDB2D000","o":"F5D0"},{"b":"7F33BD760000","o":"36207","s":"gsignal"},{"b":"7F33BD760000","o":"378F8","s":"abort"},{"b":"5618AB0D8000","o":"A17BA7","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5618AB0D8000","o":"B0FFD6"},{"b":"5618AB0D8000","o":"B82501"},{"b":"5618AB0D8000","o":"9A1CDB","s":"__wt_err_func"},{"b":"5618AB0D8000","o":"9A20FB","s":"__wt_panic"},{"b":"5618AB0D8000","o":"C54051","s":"__wt_block_checkpoint"},{"b":"5618AB0D8000","o":"BB80EA","s":"__wt_bt_write"},{"b":"5618AB0D8000","o":"B5EBA6"},{"b":"5618AB0D8000","o":"B64567","s":"__wt_reconcile"},{"b":"5618AB0D8000","o":"BCFE5C","s":"__wt_cache_op"},{"b":"5618AB0D800

附录

Quickly create a large file on a Linux system - Stack Overflow