このチュートリアルでは、Sysdig システム可視化ツールを Ubuntu 22.04 にインストールする方法について説明します。 シスディグ は、システムを詳細に可視化するシンプルな可視化ツールです。 sysdig のマニュアルページによると;
- sysdig は、システムのトラブルシューティング、分析、調査のためのツールです。
- システムコールやその他の OS イベントのキャプチャ、フィルタリング、デコードに使用できます。
- 稼働中のシステムを検査するため、または後の段階で分析できるトレース ファイルを生成するために使用できます。
- sysdig には強力なフィルタリング言語が含まれており、カスタマイズ可能な出力があり、chisel と呼ばれる Lua スクリプトによって拡張できます。
Ubuntu 22.04 に Sysdig システム可視化ツールをインストールする
このチュートリアルでは、Sysdig を Ubuntu 22.04 システムにインストールします。
Sysdig は、デフォルトの Ubuntu Universe リポジトリーによって提供されます。 ただし、利用可能なバージョンは少し古いです。
apt-cache policy sysdig
sysdig: Installed: (none) Candidate: 0.27.1-0.3build1 Version table: 0.27.1-0.3build1 500 500 http://ke.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
の 現在のリリース バージョン この記事の執筆時点での Sysdig は v0.29.3.
したがって、現在のバージョンの Sysdig を Ubuntu 22.04 にインストールするには 2 つの方法があります。
Ubuntu に Sysdig を自動的にインストールする
Ubuntu に Sysdig を自動的にインストールするには、次のようにインストール スクリプトをダウンロードして実行するだけです。 このスクリプトは、基本的にシステムに Sysdig Draios APT リポジトリをインストールし、そのリポジトリを介して Sysdig およびその他の必要なパッケージをインストールします。
curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash
インストール出力の例。
* Detecting operating system * Installing Sysdig public key Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). OK * Installing Sysdig repository W: https://download.sysdig.com/stable/deb/stable-amd64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. * Installing kernel headers * Installing Sysdig Extracting templates from packages: 100% Preconfiguring packages ... Selecting previously unselected package gcc-11-base:amd64. (Reading database ... 73190 files and directories currently installed.) Preparing to unpack .../gcc-11-base_11.2.0-19ubuntu1_amd64.deb ... Unpacking gcc-11-base:amd64 (11.2.0-19ubuntu1) ... Preparing to unpack .../libc6_2.35-0ubuntu3.1_amd64.deb ... Unpacking libc6:amd64 (2.35-0ubuntu3.1) over (2.35-0ubuntu3) ... Setting up libc6:amd64 (2.35-0ubuntu3.1) ... Selecting previously unselected package libisl23:amd64. (Reading database ... 73195 files and directories currently installed.) Preparing to unpack .../00-libisl23_0.24-2build1_amd64.deb ... Unpacking libisl23:amd64 (0.24-2build1) ... Selecting previously unselected package libmpc3:amd64. Preparing to unpack .../01-libmpc3_1.2.1-2build1_amd64.deb ... Unpacking libmpc3:amd64 (1.2.1-2build1) ... Selecting previously unselected package cpp-11. Preparing to unpack .../02-cpp-11_11.2.0-19ubuntu1_amd64.deb ... Unpacking cpp-11 (11.2.0-19ubuntu1) ... Selecting previously unselected package cpp. Preparing to unpack .../03-cpp_4%3a11.2.0-1ubuntu1_amd64.deb ... Unpacking cpp (4:11.2.0-1ubuntu1) ... Selecting previously unselected package libcc1-0:amd64. Preparing to unpack .../04-libcc1-0_12-20220319-1ubuntu1_amd64.deb ... Unpacking libcc1-0:amd64 (12-20220319-1ubuntu1) ... Selecting previously unselected package libgomp1:amd64. Preparing to unpack .../05-libgomp1_12-20220319-1ubuntu1_amd64.deb ... Unpacking libgomp1:amd64 (12-20220319-1ubuntu1) ... Selecting previously unselected package libitm1:amd64. Preparing to unpack .../06-libitm1_12-20220319-1ubuntu1_amd64.deb ... Unpacking libitm1:amd64 (12-20220319-1ubuntu1) ... Selecting previously unselected package libatomic1:amd64. Preparing to unpack .../07-libatomic1_12-20220319-1ubuntu1_amd64.deb ... Unpacking libatomic1:amd64 (12-20220319-1ubuntu1) ... Selecting previously unselected package libasan6:amd64. Preparing to unpack .../08-libasan6_11.2.0-19ubuntu1_amd64.deb ... Unpacking libasan6:amd64 (11.2.0-19ubuntu1) ... Selecting previously unselected package liblsan0:amd64. Preparing to unpack .../09-liblsan0_12-20220319-1ubuntu1_amd64.deb ... Unpacking liblsan0:amd64 (12-20220319-1ubuntu1) ... Selecting previously unselected package libtsan0:amd64. Preparing to unpack .../10-libtsan0_11.2.0-19ubuntu1_amd64.deb ... Unpacking libtsan0:amd64 (11.2.0-19ubuntu1) ... Selecting previously unselected package libubsan1:amd64. Preparing to unpack .../11-libubsan1_12-20220319-1ubuntu1_amd64.deb ... Unpacking libubsan1:amd64 (12-20220319-1ubuntu1) ... Selecting previously unselected package libquadmath0:amd64. Preparing to unpack .../12-libquadmath0_12-20220319-1ubuntu1_amd64.deb ... Unpacking libquadmath0:amd64 (12-20220319-1ubuntu1) ... Selecting previously unselected package libgcc-11-dev:amd64. Preparing to unpack .../13-libgcc-11-dev_11.2.0-19ubuntu1_amd64.deb ... Unpacking libgcc-11-dev:amd64 (11.2.0-19ubuntu1) ... Selecting previously unselected package gcc-11. Preparing to unpack .../14-gcc-11_11.2.0-19ubuntu1_amd64.deb ... Unpacking gcc-11 (11.2.0-19ubuntu1) ... Selecting previously unselected package gcc. Preparing to unpack .../15-gcc_4%3a11.2.0-1ubuntu1_amd64.deb ... Unpacking gcc (4:11.2.0-1ubuntu1) ... Selecting previously unselected package libdpkg-perl. Preparing to unpack .../16-libdpkg-perl_1.21.1ubuntu2.1_all.deb ... Unpacking libdpkg-perl (1.21.1ubuntu2.1) ... Selecting previously unselected package bzip2. Preparing to unpack .../17-bzip2_1.0.8-5build1_amd64.deb ... Unpacking bzip2 (1.0.8-5build1) ... Selecting previously unselected package make. Preparing to unpack .../18-make_4.3-4.1build1_amd64.deb ... Unpacking make (4.3-4.1build1) ... Selecting previously unselected package lto-disabled-list. Preparing to unpack .../19-lto-disabled-list_24_all.deb ... Unpacking lto-disabled-list (24) ... Selecting previously unselected package dpkg-dev. Preparing to unpack .../20-dpkg-dev_1.21.1ubuntu2.1_all.deb ... Unpacking dpkg-dev (1.21.1ubuntu2.1) ... Selecting previously unselected package libc-dev-bin. Preparing to unpack .../21-libc-dev-bin_2.35-0ubuntu3.1_amd64.deb ... Unpacking libc-dev-bin (2.35-0ubuntu3.1) ... Selecting previously unselected package linux-libc-dev:amd64. Preparing to unpack .../22-linux-libc-dev_5.15.0-43.46_amd64.deb ... Unpacking linux-libc-dev:amd64 (5.15.0-43.46) ... Selecting previously unselected package libcrypt-dev:amd64. Preparing to unpack .../23-libcrypt-dev_1%3a4.4.27-1_amd64.deb ... Unpacking libcrypt-dev:amd64 (1:4.4.27-1) ... Selecting previously unselected package rpcsvc-proto. Preparing to unpack .../24-rpcsvc-proto_1.4.2-0ubuntu6_amd64.deb ... Unpacking rpcsvc-proto (1.4.2-0ubuntu6) ... Preparing to unpack .../25-libtirpc-common_1.3.2-2ubuntu0.1_all.deb ... Unpacking libtirpc-common (1.3.2-2ubuntu0.1) over (1.3.2-2build1) ... Setting up libtirpc-common (1.3.2-2ubuntu0.1) ... (Reading database ... 75034 files and directories currently installed.) Preparing to unpack .../libtirpc3_1.3.2-2ubuntu0.1_amd64.deb ... Unpacking libtirpc3:amd64 (1.3.2-2ubuntu0.1) over (1.3.2-2build1) ... Setting up libtirpc3:amd64 (1.3.2-2ubuntu0.1) ... Selecting previously unselected package libtirpc-dev:amd64. (Reading database ... 75034 files and directories currently installed.) Preparing to unpack .../00-libtirpc-dev_1.3.2-2ubuntu0.1_amd64.deb ... Unpacking libtirpc-dev:amd64 (1.3.2-2ubuntu0.1) ... Selecting previously unselected package libnsl-dev:amd64. Preparing to unpack .../01-libnsl-dev_1.3.0-2build2_amd64.deb ... Unpacking libnsl-dev:amd64 (1.3.0-2build2) ... Selecting previously unselected package libc6-dev:amd64. Preparing to unpack .../02-libc6-dev_2.35-0ubuntu3.1_amd64.deb ... Unpacking libc6-dev:amd64 (2.35-0ubuntu3.1) ... Selecting previously unselected package libstdc++-11-dev:amd64. Preparing to unpack .../03-libstdc++-11-dev_11.2.0-19ubuntu1_amd64.deb ... Unpacking libstdc++-11-dev:amd64 (11.2.0-19ubuntu1) ... Selecting previously unselected package g++-11. Preparing to unpack .../04-g++-11_11.2.0-19ubuntu1_amd64.deb ... Unpacking g++-11 (11.2.0-19ubuntu1) ... Selecting previously unselected package g++. Preparing to unpack .../05-g++_4%3a11.2.0-1ubuntu1_amd64.deb ... Unpacking g++ (4:11.2.0-1ubuntu1) ... Selecting previously unselected package build-essential. Preparing to unpack .../06-build-essential_12.9ubuntu3_amd64.deb ... Unpacking build-essential (12.9ubuntu3) ... Selecting previously unselected package dctrl-tools. Preparing to unpack .../07-dctrl-tools_2.24-3build2_amd64.deb ... Unpacking dctrl-tools (2.24-3build2) ... Selecting previously unselected package dkms. Preparing to unpack .../08-dkms_2.8.7-2ubuntu2_all.deb ... Unpacking dkms (2.8.7-2ubuntu2) ... Selecting previously unselected package libfakeroot:amd64. Preparing to unpack .../09-libfakeroot_1.28-1ubuntu1_amd64.deb ... Unpacking libfakeroot:amd64 (1.28-1ubuntu1) ... Selecting previously unselected package fakeroot. Preparing to unpack .../10-fakeroot_1.28-1ubuntu1_amd64.deb ... Unpacking fakeroot (1.28-1ubuntu1) ... Selecting previously unselected package fonts-dejavu-core. Preparing to unpack .../11-fonts-dejavu-core_2.37-2build1_all.deb ... Unpacking fonts-dejavu-core (2.37-2build1) ... Selecting previously unselected package fontconfig-config. Preparing to unpack .../12-fontconfig-config_2.13.1-4.2ubuntu5_all.deb ... Unpacking fontconfig-config (2.13.1-4.2ubuntu5) ... Selecting previously unselected package libalgorithm-diff-perl. Preparing to unpack .../13-libalgorithm-diff-perl_1.201-1_all.deb ... Unpacking libalgorithm-diff-perl (1.201-1) ... Selecting previously unselected package libalgorithm-diff-xs-perl. Preparing to unpack .../14-libalgorithm-diff-xs-perl_0.04-6build3_amd64.deb ... Unpacking libalgorithm-diff-xs-perl (0.04-6build3) ... Selecting previously unselected package libalgorithm-merge-perl. Preparing to unpack .../15-libalgorithm-merge-perl_0.08-3_all.deb ... Unpacking libalgorithm-merge-perl (0.08-3) ... Selecting previously unselected package libfontconfig1:amd64. Preparing to unpack .../16-libfontconfig1_2.13.1-4.2ubuntu5_amd64.deb ... Unpacking libfontconfig1:amd64 (2.13.1-4.2ubuntu5) ... Selecting previously unselected package libjpeg-turbo8:amd64. Preparing to unpack .../17-libjpeg-turbo8_2.1.2-0ubuntu1_amd64.deb ... Unpacking libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ... Selecting previously unselected package libjpeg8:amd64. Preparing to unpack .../18-libjpeg8_8c-2ubuntu10_amd64.deb ... Unpacking libjpeg8:amd64 (8c-2ubuntu10) ... Selecting previously unselected package libdeflate0:amd64. Preparing to unpack .../19-libdeflate0_1.10-2_amd64.deb ... Unpacking libdeflate0:amd64 (1.10-2) ... Selecting previously unselected package libjbig0:amd64. Preparing to unpack .../20-libjbig0_2.1-3.1build3_amd64.deb ... Unpacking libjbig0:amd64 (2.1-3.1build3) ... Selecting previously unselected package libwebp7:amd64. Preparing to unpack .../21-libwebp7_1.2.2-2_amd64.deb ... Unpacking libwebp7:amd64 (1.2.2-2) ... Selecting previously unselected package libtiff5:amd64. Preparing to unpack .../22-libtiff5_4.3.0-6_amd64.deb ... Unpacking libtiff5:amd64 (4.3.0-6) ... Selecting previously unselected package libxpm4:amd64. Preparing to unpack .../23-libxpm4_1%3a3.5.12-1build2_amd64.deb ... Unpacking libxpm4:amd64 (1:3.5.12-1build2) ... Selecting previously unselected package libgd3:amd64. Preparing to unpack .../24-libgd3_2.3.0-2ubuntu2_amd64.deb ... Unpacking libgd3:amd64 (2.3.0-2ubuntu2) ... Selecting previously unselected package libc-devtools. Preparing to unpack .../25-libc-devtools_2.35-0ubuntu3.1_amd64.deb ... Unpacking libc-devtools (2.35-0ubuntu3.1) ... Selecting previously unselected package libfile-fcntllock-perl. Preparing to unpack .../26-libfile-fcntllock-perl_0.22-3build7_amd64.deb ... Unpacking libfile-fcntllock-perl (0.22-3build7) ... Selecting previously unselected package manpages-dev. Preparing to unpack .../27-manpages-dev_5.10-1ubuntu1_all.deb ... Unpacking manpages-dev (5.10-1ubuntu1) ... Selecting previously unselected package sysdig. Preparing to unpack .../28-sysdig_0.29.3_amd64.deb ... Unpacking sysdig (0.29.3) ... Setting up gcc-11-base:amd64 (11.2.0-19ubuntu1) ... Setting up manpages-dev (5.10-1ubuntu1) ... Setting up lto-disabled-list (24) ... Setting up libxpm4:amd64 (1:3.5.12-1build2) ... Setting up libfile-fcntllock-perl (0.22-3build7) ... Setting up libalgorithm-diff-perl (1.201-1) ... Setting up libdeflate0:amd64 (1.10-2) ... Setting up linux-libc-dev:amd64 (5.15.0-43.46) ... Setting up libgomp1:amd64 (12-20220319-1ubuntu1) ... Setting up bzip2 (1.0.8-5build1) ... Setting up libjbig0:amd64 (2.1-3.1build3) ... Setting up libfakeroot:amd64 (1.28-1ubuntu1) ... Setting up libasan6:amd64 (11.2.0-19ubuntu1) ... Setting up fakeroot (1.28-1ubuntu1) ... update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode Setting up libtirpc-dev:amd64 (1.3.2-2ubuntu0.1) ... Setting up rpcsvc-proto (1.4.2-0ubuntu6) ... Setting up make (4.3-4.1build1) ... Setting up libquadmath0:amd64 (12-20220319-1ubuntu1) ... Setting up libmpc3:amd64 (1.2.1-2build1) ... Setting up libatomic1:amd64 (12-20220319-1ubuntu1) ... Setting up fonts-dejavu-core (2.37-2build1) ... Setting up libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ... Setting up libdpkg-perl (1.21.1ubuntu2.1) ... Setting up libwebp7:amd64 (1.2.2-2) ... Setting up libubsan1:amd64 (12-20220319-1ubuntu1) ... Setting up libnsl-dev:amd64 (1.3.0-2build2) ... Setting up libcrypt-dev:amd64 (1:4.4.27-1) ... Setting up libisl23:amd64 (0.24-2build1) ... Setting up libc-dev-bin (2.35-0ubuntu3.1) ... Setting up libalgorithm-diff-xs-perl (0.04-6build3) ... Setting up libcc1-0:amd64 (12-20220319-1ubuntu1) ... Setting up liblsan0:amd64 (12-20220319-1ubuntu1) ... Setting up dctrl-tools (2.24-3build2) ... Setting up libitm1:amd64 (12-20220319-1ubuntu1) ... Setting up libalgorithm-merge-perl (0.08-3) ... Setting up libtsan0:amd64 (11.2.0-19ubuntu1) ... Setting up libjpeg8:amd64 (8c-2ubuntu10) ... Setting up cpp-11 (11.2.0-19ubuntu1) ... Setting up fontconfig-config (2.13.1-4.2ubuntu5) ... Setting up dpkg-dev (1.21.1ubuntu2.1) ... Setting up libgcc-11-dev:amd64 (11.2.0-19ubuntu1) ... Setting up gcc-11 (11.2.0-19ubuntu1) ... Setting up cpp (4:11.2.0-1ubuntu1) ... Setting up libc6-dev:amd64 (2.35-0ubuntu3.1) ... Setting up libtiff5:amd64 (4.3.0-6) ... Setting up libfontconfig1:amd64 (2.13.1-4.2ubuntu5) ... Setting up gcc (4:11.2.0-1ubuntu1) ... Setting up dkms (2.8.7-2ubuntu2) ... Setting up libgd3:amd64 (2.3.0-2ubuntu2) ... Setting up sysdig (0.29.3) ... Loading new scap-e5c53d648f3c4694385bbe488e7d47eaa36c229a DKMS files... Building for 5.15.0-27-generic Building initial module for 5.15.0-27-generic Done. scap.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/5.15.0-27-generic/updates/dkms/ depmod..... Setting up libstdc++-11-dev:amd64 (11.2.0-19ubuntu1) ... Setting up libc-devtools (2.35-0ubuntu3.1) ... Setting up g++-11 (11.2.0-19ubuntu1) ... Setting up g++ (4:11.2.0-1ubuntu1) ... update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode Setting up build-essential (12.9ubuntu3) ... Processing triggers for man-db (2.10.2-1) ... Processing triggers for libc-bin (2.35-0ubuntu3) ... NEEDRESTART-VER: 3.5 NEEDRESTART-KCUR: 5.15.0-27-generic NEEDRESTART-KEXP: 5.15.0-27-generic NEEDRESTART-KSTA: 1 NEEDRESTART-SVC: cron.service NEEDRESTART-SVC: dbus.service NEEDRESTART-SVC: irqbalance.service NEEDRESTART-SVC: ModemManager.service NEEDRESTART-SVC: multipathd.service NEEDRESTART-SVC: networkd-dispatcher.service NEEDRESTART-SVC: packagekit.service NEEDRESTART-SVC: polkit.service NEEDRESTART-SVC: rsyslog.service NEEDRESTART-SVC: snapd.service NEEDRESTART-SVC: ssh.service NEEDRESTART-SVC: systemd-journald.service NEEDRESTART-SVC: systemd-logind.service NEEDRESTART-SVC: systemd-networkd.service NEEDRESTART-SVC: systemd-resolved.service NEEDRESTART-SVC: systemd-timesyncd.service NEEDRESTART-SVC: systemd-udevd.service NEEDRESTART-SVC: udisks2.service NEEDRESTART-SVC: unattended-upgrades.service NEEDRESTART-SVC: [email protected]
Ubuntu 22.04 に Sysdig を手動でインストールする
Ubuntu 22.04 に Sysdig を手動でインストールするには;
- Sysdig Draios APT リポジトリのインストール
curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public
| gpg --dearmor > /etc/apt/trusted.gpg.d/sysdig.gpg
curl -s https://s3.amazonaws.com/download.draios.com/stable/deb/draios.list
-o /etc/apt/sources.list.d/sysdig-draios.list
- システム パッケージ キャッシュの更新を実行する
apt update
apt install linux-headers-$(uname -r)
- Ubuntu 22.04 に Sysdig システム可視化ツールをインストールする
Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: build-essential bzip2 cpp cpp-11 dctrl-tools dkms dpkg-dev fakeroot fontconfig-config fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1 libc-dev-bin libc-devtools libc6 libc6-dev libcc1-0 libcrypt-dev libdeflate0 libdpkg-perl libfakeroot libfile-fcntllock-perl libfontconfig1 libgcc-11-dev libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 liblsan0 libmpc3 libnsl-dev libquadmath0 libstdc++-11-dev libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp7 libxpm4 linux-libc-dev lto-disabled-list make manpages-dev rpcsvc-proto Suggested packages: bzip2-doc cpp-doc gcc-11-locales debtags menu debian-keyring g++-multilib g++-11-multilib gcc-11-doc gcc-multilib autoconf automake libtool flex bison gdb gcc-doc gcc-11-multilib glibc-doc bzr libgd-tools libstdc++-11-doc make-doc Recommended packages: libnss-nis libnss-nisplus The following NEW packages will be installed: build-essential bzip2 cpp cpp-11 dctrl-tools dkms dpkg-dev fakeroot fontconfig-config fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdeflate0 libdpkg-perl libfakeroot libfile-fcntllock-perl libfontconfig1 libgcc-11-dev libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 liblsan0 libmpc3 libnsl-dev libquadmath0 libstdc++-11-dev libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp7 libxpm4 linux-libc-dev lto-disabled-list make manpages-dev rpcsvc-proto sysdig The following packages will be upgraded: libc6 1 upgraded, 55 newly installed, 0 to remove and 37 not upgraded. Need to get 81.6 MB of archives. After this operation, 246 MB of additional disk space will be used. Do you want to continue? [Y/n] y
プロセス中にカーネル ヘッダーがアップグレードされる場合があります。 したがって、システムを再起動する必要があります。
systemctl reboot -i
インストールされている Sysdig のバージョンを確認する
インストールされているバージョンの確認;
sysdig --version
サンプル出力;
sysdig version 0.29.3
Sysdig システム可視化ツールの実行
Sysdig は、実行するだけでコマンド ラインから実行できます。
sysdig
コマンド ライン引数なしで実行すると、Sysdig はキャプチャされた各システム イベントの情報を次の形式で 1 行に出力します。
*%evt.num %evt.time %evt.cpu %proc.name (%thread.tid) %evt.dir %evt.type %evt.info
どこ;
- イベント番号 増分イベント番号です
- イベント時間 イベントのタイムスタンプです
- evt.cpu イベントがキャプチャされた CPU 番号です。
- proc.name イベントを生成したプロセスの名前です
- スレッド.tid id イベントを生成した TID。これは、シングル スレッド プロセスの PID に対応します。
- evt.dir はイベントの方向です。> は開始イベント、< は終了イベントです。
- evt.type ‘open’ や ‘read’ などのイベントの名前です。
- evt.args イベント引数のリストです。
サンプル出力;
180 13:24:24.825700257 0 sudo (1417.1417) > write fd=8(/dev/tty) size=2 181 13:24:24.825701386 0 sudo (1417.1417) < write res=2 data=.. 182 13:24:24.825701876 0 sudo (1417.1417) > switch next=4681 pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 183 13:24:24.825703162 1 sshd (1407.1407) > getpid 184 13:24:24.825703262 0 ( .4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 185 13:24:24.825704070 1 sshd (1407.1407) < getpid 186 13:24:24.825704104 0 sudo (1417.1417) > rt_sigaction 187 13:24:24.825704316 0 sudo (1417.1417) < rt_sigaction 188 13:24:24.825704837 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 18446744073709551615:?4 8:f1 9:f1 timeout=none sigmask= 189 13:24:24.825706544 0 sudo (1417.1417) > switch next=4685(sysdig) pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 191 13:24:24.825714684 1 sshd (1407.1407) > rt_sigprocmask 193 13:24:24.825715005 1 sshd (1407.1407) < rt_sigprocmask 194 13:24:24.825716551 1 sshd (1407.1407) > ppoll fds=4:41 4:44 10:f1 7:f0 timeout=none sigmask= 195 13:24:24.825718500 1 sshd (1407.1407) < ppoll res=2 fds=4:44 10:f1 196 13:24:24.825719227 1 sshd (1407.1407) > rt_sigprocmask 197 13:24:24.825719449 1 sshd (1407.1407) < rt_sigprocmask 198 13:24:24.825720103 1 sshd (1407.1407) > read fd=10( /dev/ptmx) size=32768 200 13:24:24.825721199 1 sshd (1407.1407) < read res=2 data=.. 201 13:24:24.825722309 1 sshd (1407.1407) > write fd=4(<4t>192.168.56.1:48792->192.168.56.143:22) size=356 202 13:24:24.825724281 0 sysdig (4685.4685) > switch next=4681 pgft_maj=2 pgft_min=831 vm_size=60092 vm_rss=15628 vm_swap=0 203 13:24:24.825727253 0 ( .4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 204 13:24:24.825729171 0 sudo (1417.1417) < ppoll res=1 fds=9:f1 205 13:24:24.825731162 0 sudo (1417.1417) > rt_sigaction 206 13:24:24.825731625 0 sudo (1417.1417) < rt_sigaction 207 13:24:24.825732245 0 sudo (1417.1417) > read fd=9( /dev/ptmx) size=65536 208 13:24:24.825733171 0 sudo (1417.1417) < read res=109 data=23 13:24:24.825264580 0 .[01;32msnapd.[00m (.[01;36m1559.[00m.1579) < .[01;34mfu 209 13:24:24.825733991 0 sudo (1417.1417) > rt_sigaction 210 13:24:24.825734162 0 sudo (1417.1417) < rt_sigaction 211 13:24:24.825734643 0 sudo (1417.1417) > rt_sigprocmask 212 13:24:24.825734870 0 sudo (1417.1417) < rt_sigprocmask 213 13:24:24.825735287 0 sudo (1417.1417) > rt_sigprocmask 214 13:24:24.825735442 0 sudo (1417.1417) < rt_sigprocmask 215 13:24:24.825736119 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 8:f4 8:f1 9:f1 timeout=none sigmask= 216 13:24:24.825738008 0 sudo (1417.1417) < ppoll res=1 fds=8:f4 217 13:24:24.825739012 0 sudo (1417.1417) > rt_sigaction 218 13:24:24.825739197 0 sudo (1417.1417) < rt_sigaction 219 13:24:24.825739623 0 sudo (1417.1417) > write fd=8( /dev/tty) size=109 220 13:24:24.825740912 0 sudo (1417.1417) < write res=109 data=23 13:24:24.825264580 0 .[01;32msnapd.[00m (.[01;36m1559.[00m.1579) < .[01;34mfu 221 13:24:24.825741696 0 sudo (1417.1417) > switch next=4681 pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 222 13:24:24.825743109 0 ( .4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 223 13:24:24.825743972 0 sudo (1417.1417) > rt_sigaction 224 13:24:24.825744338 0 sudo (1417.1417) < rt_sigaction 225 13:24:24.825744876 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 18446744073709551615:?4 8:f1 9:f1 timeout=none sigmask= 226 13:24:24.825746787 0 sudo (1417.1417) > switch next=4685(sysdig) pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 228 13:24:24.825749220 0 sysdig (4685.4685) > switch next=4681 pgft_maj=2 pgft_min=831 vm_size=60092 vm_rss=15628 vm_swap=0 229 13:24:24.825750789 0 ( .4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 230 13:24:24.825752221 0 sudo (1417.1417) < ppoll res=1 fds=9:f1 231 13:24:24.825753358 0 sudo (1417.1417) > rt_sigaction 232 13:24:24.825753560 0 sudo (1417.1417) < rt_sigaction 233 13:24:24.825754261 0 sudo (1417.1417) > read fd=9( /dev/ptmx) size=65536 234 13:24:24.825754931 0 sudo (1417.1417) < read res=2 data=.. 235 13:24:24.825755384 0 sudo (1417.1417) > rt_sigaction 236 13:24:24.825755548 0 sudo (1417.1417) < rt_sigaction 237 13:24:24.825756138 0 sudo (1417.1417) > rt_sigprocmask 238 13:24:24.825756347 0 sudo (1417.1417) < rt_sigprocmask 239 13:24:24.825756827 0 sudo (1417.1417) > rt_sigprocmask 240 13:24:24.825756982 0 sudo (1417.1417) < rt_sigprocmask 241 13:24:24.825757601 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 8:f4 8:f1 9:f1 timeout=none sigmask= 242 13:24:24.825758860 0 sudo (1417.1417) < ppoll res=1 fds=8:f4 243 13:24:24.825759579 0 sudo (1417.1417) > rt_sigaction 244 13:24:24.825759872 0 sudo (1417.1417) < rt_sigaction 245 13:24:24.825760299 0 sudo (1417.1417) > write fd=8( /dev/tty) size=2 246 13:24:24.825761578 0 sudo (1417.1417) < write res=2 data=.. 247 13:24:24.825762138 0 sudo (1417.1417) > switch next=4681 pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 248 13:24:24.825763510 0 ( .4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 249 13:24:24.825764361 0 sudo (1417.1417) > rt_sigaction 250 13:24:24.825764540 0 sudo (1417.1417) < rt_sigaction 251 13:24:24.825765051 0 sudo (1417.1417) > ppoll fds=11:u1 3:p1 18446744073709551615:?4 8:f1 9:f1 timeout=none sigmask= 252 13:24:24.825766787 0 sudo (1417.1417) > switch next=4685(sysdig) pgft_maj=0 pgft_min=477 vm_size=11664 vm_rss=5868 vm_swap=0 255 13:24:24.825776112 1 sshd (1407.1407) < write res=356 data=^.......3.p9...cI.../.eu.....f..I.0D..50.$..kGL...K..S`...A..$...........J..[.. 256 13:24:24.825778159 1 sshd (1407.1407) > getpid 257 13:24:24.825778500 1 sshd (1407.1407) < getpid 259 13:24:24.825781130 1 sshd (1407.1407) > rt_sigprocmask 260 13:24:24.825781448 1 sshd (1407.1407) < rt_sigprocmask 261 13:24:24.825782499 1 sshd (1407.1407) > ppoll fds=4:41 4:44 10:f1 7:f0 timeout=none sigmask= 262 13:24:24.825784482 0 sysdig (4685.4685) > switch next=4681 pgft_maj=2 pgft_min=831 vm_size=60092 vm_rss=15628 vm_swap=0 263 13:24:24.825784780 1 sshd (1407.1407) < ppoll res=2 fds=4:44 10:f1 264 13:24:24.825785407 1 sshd (1407.1407) > rt_sigprocmask 265 13:24:24.825785602 1 sshd (1407.1407) < rt_sigprocmask 266 13:24:24.825786280 1 sshd (1407.1407) > read fd=10( /dev/ptmx) size=32768 267 13:24:24.825786972 0 ( .4681) > switch next=1417(sudo) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0 268 13:24:24.825787376 1 sshd (1407.1407) < read res=111 data=23 13:24:24.825264580 0 .[01;32msnapd.[00m (.[01;36m1559.[00m.1579) < .[01;34mfu 269 13:24:24.825788199 1 sshd (1407.1407) > write fd=4(<4t>192.168.56.1:48792->192.168.56.143:22) size=36 270 13:24:24.825788929 0 sudo (1417.1417) < ppoll res=1 fds=9:f1 271 13:24:24.825789978 1 sshd (1407.1407) < write res=36 data=#.Z.,.}[email protected]~+...>.z.e^8.>.1.+.....
Sysdig コマンド出力のフィルタリング
sysdig の出力は、さまざまなイベント フィールドを使用してフィルタリングできます。
コマンドを使用して、使用可能な sysdig イベント フィールドを確認できます。
sysdig -l
サンプル出力;
------------------------------- Field Class: evt (All event types) Description: These fields can be used for all event types evt.num event number. evt.time event timestamp as a time string that includes the nanosecond part. evt.time.s event timestamp as a time string with no nanoseconds. evt.time.iso8601 event timestamp in ISO 8601 format, including nanoseconds and time zone offset (in UTC). evt.datetime event timestamp as a time string that includes the date. evt.datetime.s event timestamp as a datetime string with no nanoseconds. evt.rawtime absolute event timestamp, i.e. nanoseconds from epoch. evt.rawtime.s integer part of the event timestamp (e.g. seconds since epoch). evt.rawtime.ns fractional part of the absolute event timestamp. evt.reltime number of nanoseconds from the beginning of the capture. evt.reltime.s number of seconds from the beginning of the capture. evt.reltime.ns fractional part (in ns) of the time from the beginning of the capture. evt.pluginname if the event comes from a plugin, the name of the plugin that generated it. The plugin must be currently loaded. evt.plugininfo if the event comes from a plugin, a summary of the event as formatted by the plugin. The plugin must be currently loaded. ------------------------------- Field Class: evt (Syscall events only) Description: Event fields applicable to syscall events. Note that for most events you can access the individual arguments/parameters of each syscall via evt.arg, e.g. evt.arg.filename. evt.latency delta between an exit event and the correspondent enter event, in nanoseconds. evt.latency.s integer part of the event latency delta. evt.latency.ns fractional part of the event latency delta. evt.latency.human delta between an exit event and the correspondent enter event, as a human readable string (e.g. 10.3ms). evt.deltatime delta between this event and the previous event, in nanoseconds. evt.deltatime.s integer part of the delta between this event and the previous event. evt.deltatime.ns fractional part of the delta between this event and the previous event. evt.dir event direction can be either '>' for enter events or '<' for exit events. evt.type The name of the event (e.g. 'open'). evt.type.is allows one to specify an event type, and returns 1 for events that are of that type. For example, evt.type.is.open returns 1 for open events, 0 for any other event. syscall.type For system call events, the name of the system call (e.g. 'open'). Unset for other events (e.g. switch or internal events). Use this field instead of evt.type if you need to make sure that the filtered/printed value is actually a system call. evt.category The event category. Example values are 'file' (for file operations like open and close), 'net' (for network operations like socket and bind), memory (for things like brk or mmap), and so on. evt.cpu number of the CPU where this event happened. evt.args all the event arguments, aggregated into a single string. evt.arg one of the event arguments specified by name or by number. Some events (e.g. return codes or FDs) will be converted into a text representation when possible. E.g. 'evt.arg.fd' or 'evt.arg[0]'. evt.rawarg one of the event arguments specified by name. E.g. 'evt.rawarg.fd'. evt.info for most events, this field returns the same value as evt.args. However, for some events (like writes to /dev/log) it provides higher level information coming from decoding the arguments. evt.buffer the binary data buffer for events that have one, like read(), recvfrom(), etc. Use this field in filters with 'contains' to search into I/O data buffers. evt.buflen the length of the binary data buffer for events that have one, like read(), recvfrom(), etc. evt.res event return value, as a string. If the event failed, the result is an error code string (e.g. 'ENOENT'), otherwise the result is the string 'SUCCESS'. evt.rawres event return value, as a number (e.g. -2). Useful for range comparisons. evt.failed 'true' for events that returned an error status. evt.is_io 'true' for events that read or write to FDs, like read(), send, recvfrom(), etc. evt.is_io_read 'true' for events that read from FDs, like read(), recv(), recvfrom(), etc. evt.is_io_write 'true' for events that write to FDs, like write(), send(), etc. evt.io_dir 'r' for events that read from FDs, like read(); 'w' for events that write to FDs, like write(). evt.is_wait 'true' for events that make the thread wait, e.g. sleep(), select(), poll(). evt.wait_latency for events that make the thread wait (e.g. sleep(), select(), poll()), this is the time spent waiting for the event to return, in nanoseconds. evt.is_syslog 'true' for events that are writes to /dev/log. evt.count This filter field always returns 1 and can be used to count events from inside chisels. evt.count.error This filter field returns 1 for events that returned with an error, and can be used to count event failures from inside chisels. evt.count.error.file This filter field returns 1 for events that returned with an error and are related to file I/O, and can be used to count event failures from inside chisels. evt.count.error.net This filter field returns 1 for events that returned with an error and are related to network I/O, and can be used to count event failures from inside chisels. evt.count.error.memory This filter field returns 1 for events that returned with an error and are related to memory allocation, and can be used to count event failures from inside chisels. evt.count.error.other This filter field returns 1 for events that returned with an error and are related to none of the previous categories, and can be used to count event failures from inside chisels. evt.count.exit This filter field returns 1 for exit events, and can be used to count single events from inside chisels. evt.around (FILTER ONLY) Accepts the event if it's around the specified time interval. The syntax is evt.around[T]=D, where T is the value returned by %evt.rawtime for the event and D is a delta in milliseconds. For example, evt.around[1404996934793590564]=1000 will return the events with timestamp with one second before the timestamp and one second after it, for a total of two seconds of capture. evt.abspath Absolute path calculated from dirfd and name during syscalls like renameat and symlinkat. Use 'evt.abspath.src' or 'evt.abspath.dst' for syscalls that support multiple paths. evt.is_open_read 'true' for open/openat/openat2 events where the path was opened for reading evt.is_open_write 'true' for open/openat/openat2 events where the path was opened for writing evt.is_open_exec 'true' for open/openat/openat2 or creat events where a file is created with execute permissions ------------------------------- Field Class: process Description: Additional information about the process and thread executing the syscall event. proc.pid the id of the process generating the event. proc.exe the first command line argument (usually the executable name or a custom one). proc.name the name (excluding the path) of the executable generating the event. proc.args the arguments passed on the command line when starting the process generating the event. proc.env the environment variables of the process generating the event. proc.cmdline full process command line, i.e. proc.name + proc.args. proc.exeline full process command line, with exe as first argument, i.e. proc.exe + proc.args. proc.cwd the current working directory of the event. proc.nthreads the number of threads that the process generating the event currently has, including the main process thread. proc.nchilds the number of child threads that the process generating the event currently has. This excludes the main process thread. proc.ppid the pid of the parent of the process generating the event. proc.pname the name (excluding the path) of the parent of the process generating the event. proc.pcmdline the full command line (proc.name + proc.args) of the parent of the process generating the event. proc.apid the pid of one of the process ancestors. E.g. proc.apid[1] returns the parent pid, proc.apid[2] returns the grandparent pid, and so on. proc.apid[0] is the pid of the current process. proc.apid without arguments can be used in filters only and matches any of the process ancestors, e.g. proc.apid=1234. proc.aname the name (excluding the path) of one of the process ancestors. E.g. proc.aname[1] returns the parent name, proc.aname[2] returns the grandparent name, and so on. proc.aname[0] is the name of the current process. proc.aname without arguments can be used in filters only and matches any of the process ancestors, e.g. proc.aname=bash. proc.loginshellid the pid of the oldest shell among the ancestors of the current process, if there is one. This field can be used to separate different user sessions, and is useful in conjunction with chisels like spy_user. proc.duration number of nanoseconds since the process started. proc.fdopencount number of open FDs for the process proc.fdlimit maximum number of FDs the process can open. proc.fdusage the ratio between open FDs and maximum available FDs for the process. proc.vmsize total virtual memory for the process (as kb). proc.vmrss resident non-swapped memory for the process (as kb). proc.vmswap swapped memory for the process (as kb). thread.pfmajor number of major page faults since thread start. thread.pfminor number of minor page faults since thread start. thread.tid the id of the thread generating the event. thread.ismain 'true' if the thread generating the event is the main one in the process. thread.exectime CPU time spent by the last scheduled thread, in nanoseconds. Exported by switch events only. thread.totexectime Total CPU time, in nanoseconds since the beginning of the capture, for the current thread. Exported by switch events only. thread.cgroups all the cgroups the thread belongs to, aggregated into a single string. thread.cgroup the cgroup the thread belongs to, for a specific subsystem. E.g. thread.cgroup.cpuacct. thread.vtid the id of the thread generating the event as seen from its current PID namespace. proc.vpid the id of the process generating the event as seen from its current PID namespace. thread.cpu the CPU consumed by the thread in the last second. thread.cpu.user the user CPU consumed by the thread in the last second. thread.cpu.system the system CPU consumed by the thread in the last second. thread.vmsize For the process main thread, this is the total virtual memory for the process (as kb). For the other threads, this field is zero. thread.vmrss For the process main thread, this is the resident non-swapped memory for the process (as kb). For the other threads, this field is zero. proc.sid the session id of the process generating the event. proc.sname the name of the current process's session leader. This is either the process with pid=proc.sid or the eldest ancestor that has the same sid as the current process. proc.tty The controlling terminal of the process. 0 for processes without a terminal. proc.exepath The full executable path of the process. proc.vpgid the process group id of the process generating the event, as seen from its current PID namespace. proc.is_container_healthcheck true if this process is running as a part of the container's health check. proc.is_container_liveness_probe true if this process is running as a part of the container's liveness probe. proc.is_container_readiness_probe true if this process is running as a part of the container's readiness probe. proc.is_exe_writable true if this process' executable file is writable by the same user that spawned the process. ------------------------------- Field Class: user Description: Information about the user executing the specific event. user.uid user ID. user.name user name. user.homedir home directory of the user. user.shell user's shell. user.loginuid audit user id (auid). user.loginname audit user name (auid). ------------------------------- Field Class: group Description: Information about the user group. group.gid group ID. group.name group name. ------------------------------- Field Class: container Description: Container information. If the event is not happening inside a container, both id and name will be set to 'host'. container.id the container id. container.name the container name. container.image the container image name (e.g. falcosecurity/falco:latest for docker). container.image.id the container image id (e.g. 6f7e2741b66b). container.type the container type, eg: docker or rkt container.privileged true for containers running as privileged, false otherwise container.mounts A space-separated list of mount information. Each item in the list has the format
たとえば、SSH 関連のイベントのみを出力するには、次のようにします。
sysdig proc.name=sshd
そしてENTERを押します。
比較演算子を使用することもできます。 =、!=、<、<=、>、>=、contains、icontains、in、exists. 例えば;
sysdig fd.name contains etc
プロセスが存在する必要があります。
sysdig proc.name exists
ブール演算子を使用して、複数のイベントをフィルタリングすることもできます。 そして、または、そうではありません;
sysdig "not (fd.name contains /proc or fd.name contains /dev)"
Sysdig ノミ
Sysdig には、チゼルを呼び出す lua スクリプトもバンドルされています。 これらのスクリプトは、sysdig イベント ストリームを分析して、有用なアクションを実行できます。 たとえば、ユーザー アクティビティの監視、特定の IP アドレスの監視などです。
Sysdig チゼルを一覧表示するには;
sysdig -cl
Category: Application --------------------- httplog HTTP requests log httptop Top HTTP requests memcachelog memcached requests log Category: CPU Usage ------------------- spectrogram Visualize OS latency in real time. subsecoffset Visualize subsecond offset execution time. topcontainers_cpu Top containers by CPU usage topprocs_cpu Top processes by CPU usage Category: Errors ---------------- topcontainers_error Top containers by number of errors topfiles_errors Top files by number of errors topprocs_errors top processes by number of errors Category: I/O ------------- echo_fds Print the data read and written by processes. fdbytes_by I/O bytes, aggregated by an arbitrary filter field fdcount_by FD count, aggregated by an arbitrary filter field fdtime_by FD time group by iobytes Sum of I/O bytes on any type of FD iobytes_file Sum of file I/O bytes spy_file Echo any read/write made by any process to all files. Optionall y, you can provide the name of one file to only intercept reads /writes to that file. stderr Print stderr of processes stdin Print stdin of processes stdout Print stdout of processes topcontainers_file Top containers by R+W disk bytes topfiles_bytes Top files by R+W bytes topfiles_time Top files by time topprocs_file Top processes by R+W disk bytes udp_extract extract data from UDP streams to files. Category: Logs -------------- spy_logs Echo any write made by any process to a log file. Optionally, e xport the events around each log message to file. spy_syslog Print every message written to syslog. Optionally, export the e vents around each syslog message to file. Category: Misc -------------- around Export to file the events around the time range where the given filter matches. Category: Net ------------- iobytes_net Show total network I/O bytes spy_ip Show the data exchanged with the given IP address spy_port Show the data exchanged using the given IP port number topconns Top network connections by total bytes topcontainers_net Top containers by network I/O topports_server Top TCP/UDP server ports by R+W bytes topprocs_net Top processes by network I/O Category: Performance --------------------- bottlenecks Slowest system calls fileslower Trace slow file I/O netlower Trace slow network I/0 proc_exec_time Show process execution time scallslower Trace slow syscalls topscalls Top system calls by number of calls topscalls_time Top system calls by time Category: Security ------------------ list_login_shells List the login shell IDs shellshock_detect print shellshock attacks spy_users Display interactive user activity Category: System State ---------------------- lscontainers List the running containers lsof List (and optionally filter) the open file descriptors. netstat List (and optionally filter) network connections. ps List (and optionally filter) the machine processes. Category: Tracers ----------------- tracers_2_statsd Export spans duration as statds metrics. Use the -i flag to get detailed information about a specific chisel
たとえば、実行中のプロセスを一覧表示するには、次のようにします。
sysdig -c ps
CPU 使用率で上位のプロセスを取得するには;
sysdig -c topprocs_cpu
指定された IP アドレスと交換されたデータを表示します。
sysdig -c spy_ip 192.168.100.1
インタラクティブなユーザー アクティビティを表示します。
sysdig -c spy_users
などなど。
Csysdig: 直感的な sysdig UI ツール
Sysdig には、Csysdig と呼ばれる直感的な UI ツールが付属しています。 top/htop コマンドと同様の方法で機能します。
コマンドラインから Csysdig を起動するだけです。
csysdig
Csysdig にはさまざまなビューがあります。 プレス Fn+F12.
矢印キーを使用してビューを上下にスクロールし、特定のビューを選択できます。 Enter キーを押してビューを表示します。
spy_users ビューのサンプル。
すごいですね。
これで、Ubuntu 22.04 に Sysdig システム可視化ツールをインストールする方法に関するチュートリアルは終了です。
参考文献
その他のチュートリアル
Rocky Linux 8 に ntopng をインストールする
vimで貼り付け行をコピーする方法
Linux での ps コマンドの使用例
The post Ubuntu 22.04 に Sysdig システム可視化ツールをインストールする appeared first on Gamingsym Japan.