AnsibleでUbuntuの初期設定を行う

 
カテゴリー Ansible Linux   タグ

AnsibleでUbuntuの設定を行う

プレイブックの実行

1
2
3
ansible-playbook unattended-upgrades.yml -i hosts
ansible-playbook install-ubuntu-packages.yml -i hosts
ansible-playbook ufw.yml -i hosts

インベントリファイル「hosts」

初期設定を行うinitグループを設定する。

  • ansible_user/ansible_passwordは対象Ubuntuのログインユーザ、パスワード
  • ansible_become_passwordはsudo時のパスワード
1
2
3
4
5
6
7
8
[ubuntu]
target

[ubuntu:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_port=※カスタマイズしたポート※
ansible_user=※ansible用ユーザ名※
ansible_ssh_private_key_file=~/.ssh/id_rsa

プレイブック「unattended-upgrades.yml」

セキュリティパッチや推奨パッチを自動インストールするUnattended-upgradesを設定する。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
---
- hosts: ubuntu

vars:
gather_facts: no
become: true

tasks:
- name: Configure debconf
debconf: name=unattended-upgrades question=unattended-upgrades/enable_auto_updates vtype=boolean value='true'

- name: Install unattended-upgrades
apt: name=unattended-upgrades

- name: Configure unattended-upgrades
command:
cmd: dpkg-reconfigure -f noninteractive unattended-upgrades
creates: /etc/apt/apt.conf.d/20auto-upgrades

- name: Enable periodic for download and clean
lineinfile:
dest: /etc/apt/apt.conf.d/20auto-upgrades
line: "{{ item }}"
with_items:
- 'APT::Periodic::Download-Upgradeable-Packages "1";'
- 'APT::Periodic::AutocleanInterval "7";'

- name: Enable upgrade recommend updates
lineinfile:
dest: /etc/apt/apt.conf.d/50unattended-upgrades
regexp: '^//\t+\"\${distro_id}\:\${distro_codename}-updates\";'
line: "\t\"${distro_id}:${distro_codename}-updates\";"

- name: Enable automatic reboot
lineinfile:
dest: /etc/apt/apt.conf.d/50unattended-upgrades
regexp: '^//Unattended-Upgrade::Automatic-Reboot \"false\";'
line: "Unattended-Upgrade::Automatic-Reboot \"true\";"

- name: Set automatic reboot time
lineinfile:
dest: /etc/apt/apt.conf.d/50unattended-upgrades
regexp: '^//Unattended-Upgrade::Automatic-Reboot-Time \"02:00\";'
line: "Unattended-Upgrade::Automatic-Reboot-Time \"04:00\";"

- name: Enable remove unused kernel packages
lineinfile:
dest: /etc/apt/apt.conf.d/50unattended-upgrades
regexp: '^//Unattended-Upgrade::Remove-Unused-Kernel-Packages \"true\";'
line: "Unattended-Upgrade::Remove-Unused-Kernel-Packages \"true\";"

- name: Enable remove unused dependencies
lineinfile:
dest: /etc/apt/apt.conf.d/50unattended-upgrades
regexp: '^//Unattended-Upgrade::Remove-Unused-Dependencies \"false\";'
line: "Unattended-Upgrade::Remove-Unused-Dependencies \"true\";"

- name: Reboot the machine (Wait for 5 min)
reboot:
reboot_timeout: 300

プレイブック「install-ubuntu-packages.yml」

パッケージを追加する。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
---
- hosts: ubuntu

vars:
gather_facts: no
become: true

tasks:
- name: Install net-tools
apt:
update_cache: yes
name:
- net-tools

- name: Install curl
apt:
update_cache: yes
name:
- curl

- name: Install tree
apt:
update_cache: yes
name:
- tree

- name: Install htop
apt:
update_cache: yes
name:
- htop

プレイブック「ufw.yml」

ファイアウォールを有効化する。
デフォルトDENYでポート番号を変更したSSHポートのみ許可する

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
---
- hosts: ubuntu

vars:
- sshd_port: "{{ lookup('env', 'SSHD_PORT') }}"
gather_facts: no
become: true

tasks:
- name: Deny all access from any
ufw:
default: deny
direction: incoming

- name: Allow all access to port {{ sshd_port }} (SSH)
ufw:
rule: allow
proto: tcp
port: "{{ sshd_port }}"

- name: Enable firewall
ufw:
state: enabled

コメント・シェア

AnsibleでUbuntuに接続可能にする

プレイブックの実行

ansible-playbook add-user-ssh.yml -i hosts

インベントリファイル「hosts」

初期設定を行うinitグループを設定する。

  • ansible_user/ansible_passwordは対象Ubuntuのログインユーザ、パスワード
  • ansible_become_passwordはsudo時のパスワード
1
2
3
4
5
6
7
8
9
[init]
target

[init:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_port=22
ansible_user=※ユーザ名※
ansible_password=※パスワード※
ansible_become_password=※パスワード※

プレイブック「add-user-ssh.yml」

Ansible用のユーザ追加とSSHDの初期設定を行う。

  • lookup(‘env’,’XXXXX’)はAnsible実行ホストの環境変数を参照する
  • sudo実行可能なユーザとしてANSIBLE_USERNAMEで指定したユーザを追加する
  • 追加したユーザにSSH用の公開鍵を登録する
  • SSHDのポート番号変更やルートログイン禁止の設定を行いsshdをリスタートする
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
---
- hosts: init

vars:
- ansible_user_name: "{{ lookup('env', 'ANSIBLE_USERNAME') }}"
- ansible_user_password: "{{ lookup('env', 'ANSIBLE_PASSWORD') }}"
- sshd_port: "{{ lookup('env', 'SSHD_PORT') }}"
gather_facts: no
become: true

tasks:
- name: Add a new user named {{ ansible_user_name }}
user:
name: "{{ ansible_user_name }}"
shell: /bin/bash
password: "{{ ansible_user_password }}"

- name: Add ansible user to the sudoers
copy:
dest: "/etc/sudoers.d/{{ ansible_user_name }}"
content: "{{ ansible_user_name }} ALL=(ALL) NOPASSWD: ALL"

- name: Deploy SSH Key
authorized_key:
user: "{{ ansible_user_name }}"
key: "{{ lookup('file', '/root/.ssh/id_rsa.pub') }}"
state: present

- name: Change port to {{ sshd_port }}
lineinfile:
dest: /etc/ssh/sshd_config
regexp: "^Port"
insertafter: "^#Port"
line: "Port {{ sshd_port }}"

- name: Disable Root Login
lineinfile:
dest: /etc/ssh/sshd_config
regexp: "^PermitRootLogin"
line: "PermitRootLogin no"
state: present
backup: yes
notify:
- restart sshd

handlers:
- name: restart sshd
service:
name: sshd
state: restarted

発生したエラーと対策

エラー「sudo: a password is required」

sudoのパスワードが指定されていない。

1
fatal: [target]: FAILED! => {"changed": false, "module_stderr": "Shared connection to target closed.\r\n", "module_stdout": "sudo: a password is required\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

ansible_become_passwordでsudoパスワードを指定する

エラー「/usr/bin/python: not found」

pythonが見つからない。

ansible_python_interpreter=/usr/bin/python3でpythonのパスを明示する

1
fatal: [target]: FAILED! => {"changed": false, "module_stderr": "Shared connection to target closed.\r\n", "module_stdout": "\r\n/bin/sh: 1: /usr/bin/python: not found\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 127}

コメント・シェア

Ansibleをdocker-composeで動かす

 
カテゴリー Ansible Container   タグ

dockerコンテナをansible実行ホストとして構成する

docker-composeで環境をカプセル化してGitでPlaybookとともに管理したい。

最終的な構成

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
version: '3.5'

services:
tutorial:
image: python:3-slim
tty: true
working_dir: /work
entrypoint: ./entrypoint.sh
command: /bin/bash
volumes:
- .:/work
environment:
ANSIBLE_CONFIG: .
ANSIBLE_USERNAME: ${ANSIBLE_USERNAME}
ANSIBLE_PASSWORD: ${ANSIBLE_PASSWORD}
secrets:
- secret_key
- public_key

secrets:
secret_key:
file: ./.ssh/ubuntu.key
public_key:
file: ./.ssh/ubuntu.key.pub

entrypoint.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#!/bin/bash

#
# SSHキー
#
mkdir -p ~/.ssh
chown -R root:root ~/.ssh
chmod -R 0700 ~/.ssh
cp -ip /run/secrets/secret_key ~/.ssh/id_rsa
cp -ip /run/secrets/public_key ~/.ssh/id_rsa.pub
chmod -R 0600 ~/.ssh

#
# Ansibleインストール
#
apt-get install sshpass -y
pip install ansible

exec "$@"

ansible.cfg

1
2
[defaults]
host_key_checking = False

発生したエラーと対策

SecretsをつかってSSH用の公開鍵、秘密鍵を共有する

Secretsは**/run/secrets/**から参照できる
entrypoint.shでrootアカウントのキーとしてコピーし、パーミッションを設定する。

エラー「you must install the sshpass program」

ログイン時にパスワードを使用したログインを行う場合は、sshpassをインストールする必要がある。

1
fatal: [target]: FAILED! => {"msg": "to use the 'ssh' connection type with passwords, you must install the sshpass program"}

エラー「Please add this host’s fingerprint to your known_hosts file to manage this host.」

初回ログイン時などfingerprintの対話操作が含まれるのでエラーになってしまう。

1
fatal: [target]: FAILED! => {"msg": "Using a SSH password instead of a key is not possible because Host Key checking is enabled and sshpass does not support this.  Please add this host's fingerprint to your known_hosts file to manage this host."}

ansible.cfgをカレントディレクトリに作成しhost_key_checking = Falseを設定する。

1
2
[defaults]
host_key_checking = False

ansible.cfgが読み込まれない

docker-composeを実行するのがWindowsの場合、パーミッションが777になり読み込まれない。
export ANSIBLE_CONFIG=.のように環境変数を定義して明示的にansible.cfgの場所を明示することで読み込ませる。

Python3環境でaptでインストールしたAnsibleが動かない

aptで以下のようにインストールでき、ansibleコマンドも実行可能だが、妙なエラーが出る。

1
2
3
4
apt-get update
apt-get install -y software-properties-common
apt-add-repository --yes --update ppa:ansible/ansible
apt-get install -y ansible
1
fatal: [xxxx.xxxxxx.xxxx]: FAILED! => {"changed": false, "module_stderr": "Shared connection to xxxx.xxxxxx.xxxx closed.\r\n", "module_stdout": "\r\n/home/ansible/.ansible/tmp/ansible-tmp-1621777634.977247-71421390121516/AnsiballZ_user.py:17: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\r\n  import imp\r\nTraceback (most recent call last):\r\n  File \"/tmp/ansible_user_payload_frbeg21u/ansible_user_payload.zip/ansible/module_utils/basic.py\", line 279, in get_distribution\r\nAttributeError: module 'platform' has no attribute '_supported_dists'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1621777634.977247-71421390121516/AnsiballZ_user.py\", line 113, in <module>\r\n    _ansiballz_main()\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1621777634.977247-71421390121516/AnsiballZ_user.py\", line 105, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1621777634.977247-71421390121516/AnsiballZ_user.py\", line 48, in invoke_module\r\n    imp.load_module('__main__', mod, module, MOD_DESC)\r\n  File \"/usr/lib/python3.8/imp.py\", line 234, in load_module\r\n    return load_source(name, filename, file)\r\n  File \"/usr/lib/python3.8/imp.py\", line 169, in load_source\r\n    module = _exec(spec, sys.modules[name])\r\n  File \"<frozen importlib._bootstrap>\", line 604, in _exec\r\n  File \"<frozen importlib._bootstrap_external>\", line 783, in exec_module\r\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\r\n  File \"/tmp/ansible_user_payload_frbeg21u/__main__.py\", line 2611, in <module>\r\n  File \"/tmp/ansible_user_payload_frbeg21u/__main__.py\", line 2516, in main\r\n  File \"/tmp/ansible_user_payload_frbeg21u/__main__.py\", line 403, in __new__\r\n  File \"/tmp/ansible_user_payload_frbeg21u/ansible_user_payload.zip/ansible/module_utils/basic.py\", line 337, in load_platform_subclass\r\n  File \"/tmp/ansible_user_payload_frbeg21u/ansible_user_payload.zip/ansible/module_utils/basic.py\", line 289, in get_distribution\r\nAttributeError: module 'platform' has no attribute 'dist'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

pipでインストールすればOK。

1
pip install ansible

コメント・シェア

FluentdとFluent-Bit

Fluentd Fluent Bit
Scope Containers / Servers Embedded Linux / Containers / Servers
Language C & Ruby C
Memory ~40MB ~650KB
Performance High Performance High Performance
Dependencies Built as a Ruby Gem, it requires a certain number of gems. Zero dependencies, unless some special plugin requires them.
Plugins More than 1000 plugins available Around 70 plugins available
License Apache License v2.0 Apache License v2.0

Both Fluentd and Fluent Bit can work as Aggregators or Forwarders, they both can complement each other or use them as standalone solutions.

どちらも単独でログアグリゲーター&フォワーダーとして機能する。

コンセプト

Event or Record

Internally, an Event always has two components (in an array form):

[TIMESTAMP, MESSAGE]

ログ行はイベントとして扱われ、イベントはタイムスタンプとメッセージのタプルで構成されている。

Filtering

In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering.

There are many use cases when Filtering is required like:

  • Append specific information to the Event like an IP address or metadata.
  • Select a specific piece of the Event content.
  • Drop Events that matches certain pattern.

フィルタリングによってイベントに大して、追加、変更、削除を行うことができる。

Tag

Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from.

INPUTでタグ付けを行い、FILTERやOUTPUTで利用する。

Timestamp

SECONDS.NANOSECONDS

タイムスタンプはSECONDSで示すUnixエポックタイムとNANOSECONDSで示されるナノ秒の組で表現される。

Match

Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule.

MATCHによって該当するタグを持つログを複数の宛先にルーティングするなどのルールを記述できる。

Structured Messages

{“project”: “Fluent Bit”, “created”: 1398289291}

Consider MessagePack as a binary version of JSON on steroids.

メッセージは構造化されたJSONデータ(JSONのバイナリ版であるMessagePack)を使う。

コメント・シェア

FluentBitの最小限の構成

INPUT FORWARD

INPUT/FORWARDにIPで待ち受けて、標準出力に出力する例がある。
OUTPUT/STDOUTで標準出力に出力できる。

1
2
3
4
5
6
7
8
9
10
[INPUT]
Name forward
Listen 0.0.0.0
Port 24224
Buffer_Chunk_Size 1M
Buffer_Max_Size 6M

[OUTPUT]
Name stdout
Match *

PortはTCP port to listen for incoming connections.だ。

FILTER STDOUT

標準出力への出力はFILTERでも可能。
FILTER/STDOUTにFILTERを使った標準出力が示されている。

1
2
3
4
5
6
7
8
9
10
11
[INPUT]
Name cpu
Tag cpu.local

[FILTER]
Name stdout
Match *

[OUTPUT]
Name null
Match *

OUTPUT/NULLは単純にイベントを破棄する。

Logging pipeline locally

Running a Logging Pipeline LocallyではDocker Composeの設定例がある。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
version: "3.7"

services:
fluent-bit:
image: fluent/fluent-bit
volumes:
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
depends_on:
- elasticsearch
elasticsearch:
image: elasticsearch:7.6.2
ports:
- "9200:9200"
environment:
- discovery.type=single-node

Dockerのログを集約するサイドカー構成

docker-composeの設定例

以下がfluent-bitサイドカーとして設定した内容。
fluent-bitは24224ポートで待ち受け、web1、web2コンテナはLoggingDriverを使って待ち受けポートへログを送る。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
version: '3.5'
services:
web1:
image: nginx:latest
ports:
- "8888:80"
depends_on:
- fluent-bit
command: [nginx-debug, '-g', 'daemon off;']
logging:
driver: fluentd
options:
fluentd-address: "localhost:24224"
fluentd-async-connect: "false"
web2:
image: nginx:latest
ports:
- "8889:80"
depends_on:
- fluent-bit
command: [nginx-debug, '-g', 'daemon off;']
logging:
driver: fluentd
options:
fluentd-address: "localhost:24224"
fluentd-async-connect: "false"
fluent-bit:
image: fluent/fluent-bit
volumes:
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
ports:
- "24224:24224"

fluent-bit.confは以下の通り。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[SERVICE]
Log_Level info

[INPUT]
Name forward
Listen 0.0.0.0
Port 24224

[FILTER]
Name stdout
Match *

[OUTPUT]
Name null
Match *

実行例

ログを見ると、web1、web2で出力しているログはfluent-bitへ送られ、fluent-bitのログとして出力されている。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
PS > docker-compose up
Docker Compose is now in the Docker CLI, try `docker compose up`

WARNING: Found orphan containers (stdout_web_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating stdout_fluent-bit_1 ... done
Creating stdout_web2_1 ... done
Creating stdout_web1_1 ... done
Attaching to stdout_fluent-bit_1, stdout_web2_1, stdout_web1_1
fluent-bit_1 | Fluent Bit v1.7.4
fluent-bit_1 | * Copyright (C) 2019-2021 The Fluent Bit Authors
fluent-bit_1 | * Copyright (C) 2015-2018 Treasure Data
fluent-bit_1 | * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
fluent-bit_1 | * https://fluentbit.io
fluent-bit_1 |
fluent-bit_1 | [2021/05/21 06:22:11] [Warning] [config] I cannot open /fluent-bit/etc/parser.conf file
fluent-bit_1 | [2021/05/21 06:22:11] [ info] [engine] started (pid=1)
fluent-bit_1 | [2021/05/21 06:22:11] [ info] [storage] version=1.1.1, initializing...
fluent-bit_1 | [2021/05/21 06:22:11] [ info] [storage] in-memory
fluent-bit_1 | [2021/05/21 06:22:11] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
fluent-bit_1 | [2021/05/21 06:22:11] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
fluent-bit_1 | [2021/05/21 06:22:11] [error] [sp] could not initialize stream processor
fluent-bit_1 | [2021/05/21 06:22:11] [error] [engine] could not initialize stream processor
fluent-bit_1 | [2021/05/21 06:22:11] [Warning] [config] I cannot open /fluent-bit/etc/stream_processor.conf file
web1_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web1_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web1_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web1_1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web2_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web2_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web2_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web2_1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web2_1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web2_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web2_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
web2_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
web1_1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web1_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web1_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
fluent-bit_1 | [0] 4fb66927922a: [1621578134.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578134.000000000, {"log"=>"/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578134.000000000, {"container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578134.000000000, {"source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578134.000000000, {"container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578134.000000000, {"source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578134.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578134.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Configuration complete; ready for start up"}]
fluent-bit_1 | [0] 71d75b79d476: [1621578135.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"}]
fluent-bit_1 | [0] 71d75b79d476: [1621578135.000000000, {"log"=>"/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout"}]
fluent-bit_1 | [0] 71d75b79d476: [1621578135.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh"}]
fluent-bit_1 | [0] 71d75b79d476: [1621578135.000000000, {"container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06"}]
fluent-bit_1 | [0] 71d75b79d476: [1621578135.000000000, {"container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06"}]
fluent-bit_1 | [0] 71d75b79d476: [1621578135.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh"}]
web1_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
web2_1 | 172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 2021/05/21 06:22:33 [error] 31#31: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.24.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "localhost:8889", referrer: "http://localhost:8889/"
web2_1 | 172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET /favicon.ico HTTP/1.1" 404 556 "http://localhost:8889/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.24.0.1 - - [21/May/2021:06:22:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 172.24.0.1 - - [21/May/2021:06:22:41 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.24.0.1 - - [21/May/2021:06:22:43 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.24.0.1 - - [21/May/2021:06:22:54 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1 | [0] 71d75b79d476: [1621578135.000000000, {"source"=>"stdout", "log"=>"/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1"}]
fluent-bit_1 | [0] 71d75b79d476: [1621578135.000000000, {"source"=>"stdout", "log"=>"/docker-entrypoint.sh: Configuration complete; ready for start up", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578153.000000000, {"source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578153.000000000, {"container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:33 +0000] "GET /favicon.ico HTTP/1.1" 404 556 "http://localhost:8889/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578153.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stderr", "log"=>"2021/05/21 06:22:33 [error] 31#31: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.24.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "localhost:8889", referrer: "http://localhost:8889/""}]
fluent-bit_1 | [0] 71d75b79d476: [1621578158.000000000, {"container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578161.000000000, {"source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:41 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1"}]
fluent-bit_1 | [0] 71d75b79d476: [1621578163.000000000, {"source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:43 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"", "container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1"}]
fluent-bit_1 | [0] 4fb66927922a: [1621578165.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1 | [0] 71d75b79d476: [1621578168.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1 | [0] 4fb66927922a: [1621578171.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
Gracefully stopping... (press Ctrl+C again to force)
Stopping stdout_web1_1 ... done
Stopping stdout_web2_1 ... done
Stopping stdout_fluent-bit_1 ... done

コメント・シェア

EFK Compose

Docker Logging Efk Composeで、docker-composeを使ったELKスタックの例がある。

docker-compsoe.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
version: '2'
services:
web:
image: httpd
ports:
- "8080:80"
links:
- fluentd
logging:
driver: "fluentd"
options:
fluentd-address: "localhost:24224"
fluentd-async-connect: "false"
tag: httpd.access
depends_on:
- fluentd

fluentd:
build: ./fluentd
volumes:
- ./fluentd/conf:/fluentd/etc
links:
- "elasticsearch"
ports:
- "24224:24224"
- "24224:24224/udp"

elasticsearch:
image: elasticsearch:7.12.0
expose:
- 9200
ports:
- "9200:9200"

kibana:
image: kibana:7.12.0
links:
- "elasticsearch"
ports:
- "5601:5601"
depends_on:
- elasticsearch

fluentd/conf/fluent.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# fluentd/conf/fluent.conf
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match *.**>
@type copy
<store>
@type elasticsearch
host elasticsearch
port 9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s
</store>
<store>
@type stdout
</store>
</match>

fluentd/Dockerfile

1
2
3
# fluentd/Dockerfile
FROM fluent/fluentd:latest
RUN gem install fluent-plugin-elasticsearch --no-rdoc --no-ri

ElasticSeearchがcode78でエラーになる!?

EKF Composeを起動すると、ElasticSearchがcode 78エラーで停止してしまう。
least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configuredというメッセージが出ている

discoveryの設定が必要

このパラメーターはElastickSearchクラスタに関する設定
ElasticSearch7.X系以降はこの設定が必須になっているようだ。

パラメーター 意味
discovery.seed_hosts 設定ベースのseed_hosts
discovery.seed_providers ファイルベースのseed_hosts
cluster.initial_master_nodes クラスターのマスターノード
1
2
3
4
5
6
7
8
9
10
11
12
13
14
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:00:46,290Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "initialized" }
elasticsearch_1 | {"type": "server", "timestamp": "2021-08-15T01:00:46,290Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "starting ..." }
elasticsearch_1 | {"type": "server", "timestamp": "2021-08-15T01:00:46,309Z", "level": "INFO", "component": "o.e.x.s.c.PersistentCache", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "persistent cache index loaded" }
elasticsearch_1 | {"type": "server", "timestamp": "2021-08-15T01:00:46,406Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "publish_address {192.168.32.2:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1 | {"type": "server", "timestamp": "2021-08-15T01:00:46,524Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
elasticsearch_1 | ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
elasticsearch_1 | bootstrap check failure [1] of [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
elasticsearch_1 | ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log
elasticsearch_1 | {"type": "server", "timestamp": "2021-08-15T01:00:46,535Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "stopping ..." }
elasticsearch_1 | {"type": "server", "timestamp": "2021-08-15T01:00:46,556Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "stopped" }
elasticsearch_1 | {"type": "server", "timestamp": "2021-08-15T01:00:46,557Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "closing ..." }
elasticsearch_1 | {"type": "server", "timestamp": "2021-08-15T01:00:46,569Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "closed" }
elasticsearch_1 | {"type": "server", "timestamp": "2021-08-15T01:00:46,572Z", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "docker-cluster", "node.name": "93ed1cfad299", "message": "Native controller process has stopped - no new native processes can be started" }
ekf-stack_elasticsearch_1 exited with code 78

マスターノードのみ設定する

複数台構成ではないので、マスターノードのみ設定する。

1
2
3
4
5
6
7
8
elasticsearch:
image: elasticsearch:7.12.0
expose:
- 9200
ports:
- "9200:9200"
environment:
- cluster.initial_master_nodes=elasticsearch

code 78はでなくなったがmaster not discovered yet,というメッセージが…

1
elasticsearch_1  | {"type": "server", "timestamp": "2021-08-15T01:56:08,340Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "64a558eac1e4", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch] to bootstrap a cluster: have discovered [{64a558eac1e4}{2UB6bFD4QUSAu9rXFwZFhw}{q72ZliAYQOq5r_Zq7-vFxw}{172.19.0.2}{172.19.0.2:9300}{cdfhilmrstw}{ml.machine_memory=8348790784, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4177526784}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305] from hosts providers and [{64a558eac1e4}{2UB6bFD4QUSAu9rXFwZFhw}{q72ZliAYQOq5r_Zq7-vFxw}{172.19.0.2}{172.19.0.2:9300}{cdfhilmrstw}{ml.machine_memory=8348790784, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4177526784}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

ElasticSearch自体の応答はある。

ElasticSearch width=640

標準のノード名がランダムな値となっているので、ノード名とクラスター名を明示的に設定する。

1
2
3
4
5
6
7
8
9
10
elasticsearch:
image: elasticsearch:7.12.0
expose:
- 9200
ports:
- "9200:9200"
environment:
- node.name=es-node
- cluster.name=es-cluster
- cluster.initial_master_nodes=es-node

設定後に各ポートにアクセスすると

ElasticSearch

ElasticSearch(localhost:9200)にアクセスすると、設定したノード名やクラスター名が表示されている。

ElasticSearch width=640

Kibana

Kibana(localhost:5601)へアクセスするとKibanaのUIが表示される。

Kibana width=640

Kibana width=640

Fluent-bitからがElasticSearchにせつぞくできていない・・・Faraday::Error::ConnectionFailed

正常にログが転送されると、KibanaのIndexManagementにfluentd-YYYYMMDDの形式でインデックスが登録される。
しかし、インデックスが存在しない状態…ログを確認すると、以下のエラーが出ている。

1
2
3
fluentd_1        | 2021-08-15 03:12:14 +0000 [warn]: #0 failed to flush the buffer. retry_time=13 next_retry_seconds=2021-08-15 04:17:15 +0000 chunk="5c98f797a0fdcfe5f8155507b4d141b4" error_class=NameError error="uninitialized constant Faraday::Error::ConnectionFailed"
fluentd_1 | 2021-08-15 03:12:14 +0000 [warn]: #0 suppressed same stacktrace
fluentd_1 | 2021-08-15 03:12:14.571214000 +0000 fluent.warn: {"retry_time":13,"next_retry_seconds":"2021-08-15 04:17:15 +0000","chunk":"5c98f797a0fdcfe5f8155507b4d141b4","error":"#<NameError: uninitialized constant Faraday::Error::ConnectionFailed>","message":"failed to flush the buffer. retry_time=13 next_retry_seconds=2021-08-15 04:17:15 +0000 chunk=\"5c98f797a0fdcfe5f8155507b4d141b4\" error_class=NameError error=\"uninitialized constant Faraday::Error::ConnectionFailed\""}

fluent.confではコンテナ名を指定する

いろいろ思考錯誤した結果、fluent.confのhostではコンテナ名を明示する必要がある。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# fluentd/conf/fluent.conf
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match *.**>
@type copy
<store>
@type elasticsearch
host es-node
port 9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s
</store>
<store>
@type stdout
</store>
</match>

Kibanaでインデックスるが作成されている

ログインすると前とは異なりいきなりログイン後画面に。

Kibana width=640

左上のメニューからStack Managemetを選択。

Kibana width=640

Index Managementを選択。fluentd-YYYYMMDDの形式でインデックスが登録されている。

Kibana width=640

コメント・シェア

Parserを使用していない場合

出力されたログは適切にParseされていない。

1
2
3
fluent-bit_1  | [0] 4fb66927922a: [1621578165.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1 | [0] 71d75b79d476: [1621578168.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1 | [0] 4fb66927922a: [1621578171.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]

既定のnginxのParserを使う

ParserでFILTERでParserを指定する方法が記載されている。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[SERVICE]
Parsers_File /path/to/parsers.conf

[INPUT]
Name dummy
Tag dummy.data
Dummy {"data":"100 0.5 true This is example"}

[FILTER]
Name parser
Match dummy.*
Key_Name data
Parser dummy_test

[OUTPUT]
Name stdout
Match *

これを参考にFILTERを利用する設定を定義する。
parsers.confでは様々なParserが定義されており、今回はnginxを使用する。

1
2
3
4
5
6
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z

Parsers_File parsers.confを設定することで、この既定のParserが利用可能になる。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[SERVICE]
Log_Level info
Parsers_File parsers.conf

[INPUT]
Name forward
Listen 0.0.0.0
Port 24224

[FILTER]
Name parser
Match *
Key_Name log
Parser nginx

[OUTPUT]
Name stdout
Match *

Name parserのFILTER定義を作成する。
対象となるのは現在logで出力されている部分だ。

1
"log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""

Parseされたenginxアクセスログ

nginxのログ部分はParseされて出力された。

1
2
3
4
5
6
7
8
9
10
11
12
web1_1        | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:35:35 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 172.23.0.1 - - [21/May/2021:07:35:36 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1 | [0] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [1] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [2] 9dfdc1a8b440: [1621582535.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [0] 5b4d36337166: [1621582536.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [1] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [2] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]

Parse対象以外の項目を残す

しかし,元々出力されえていたcontainer_idなどが消えてしまっている。
そこでReserve_Data Onを使用して元の項目を残す。ただし、Prase対象のlogは必要ないので、Preserve_Key Offで残さないようにする。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[SERVICE]
Log_Level info
Parsers_File parsers.conf

[INPUT]
Name forward
Listen 0.0.0.0
Port 24224

[FILTER]
Name parser
Match *
Key_Name log
Parser nginx
Preserve_Key Off
Reserve_Data On

[OUTPUT]
Name stdout
Match *

結果は以下。期待通りの項目が出力されている。

1
2
3
4
5
6
7
8
web1_1        | 172.23.0.1 - - [21/May/2021:07:47:03 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:47:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1 | [0] 9dfdc1a8b440: [1621583223.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]
fluent-bit_1 | [1] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]
fluent-bit_1 | [2] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "source"=>"stdout", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1"}]
fluent-bit_1 | [3] 9dfdc1a8b440: [1621583225.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]

コメント・シェア

バッチファイルでファイルリストのforル―プ

 
カテゴリー ShellScript   タグ

ファイルリストをforループで出力する

1
2
3
4
5
@echo off

for %%f in (.\*) do (
echo %%~nxf
)

%~nx変数でファイルパスを加工する

指定されたファイルパスをファイル名に加工する。
実はfor文の変数は加工する機能がある。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
%* バッチ スクリプト内では、すべての引数 (%1、%2、%3、%4、
%5 など) を参照します。

バッチ パラメーター (%n) の置換は拡張されました。次のオプション構文
を使うことができます:

%~1 - すべての引用句 (") を削除して、%1 を展開します。
%~f1 - %1 を完全修飾パス名に展開します。
%~d1 - %1 をドライブ文字だけに展開します。
%~p1 - %1 をパスだけに展開します。
%~n1 - %1 をファイル名だけに展開します。
%~x1 - %1 をファイル拡張子だけに展開します。
%~s1 - 展開されたパスは、短い名前だけを含みます。
%~a1 - %1 をファイル属性に展開します。
%~t1 - %1 をファイルの日付/時刻に展開します。
%~z1 - %1 をファイルのサイズに展開します。
%~$PATH:1 - PATH 環境変数に指定されているディレクトリを検索し、
最初に見つかった完全修飾名に %1 を展開します。
環境変数名が定義されていない場合、または
検索してもファイルが見つからなかった場合は、
この修飾子を指定すると空の文字列に展開されます。

修飾子を組み合わせて、複合結果を得ることもできます:

%~dp1 - %1 をドライブ文字とパスだけに展開します。
%~nx1 - %1 をファイル名と拡張子だけに展開します。
%~dp$PATH:1 - PATH 環境変数に指定されているディレクトリを
検索して %1 を探し、最初に見つかったファイル
のドライブ文字とパスだけに展開します。
%~ftza1 - %1 を DIR の出力行のように展開します。

上の例の %1 と PATH は、他の有効な値で置き換えることができ
ます。%~ 構文は有効な引数の数によって区切られます。%~ 修飾子
は %* と同時には使用できません。

コメント・シェア

タブインデントを増やすとき

対象のコードブロックを選択して、TAB

例えば、以下のコードで<meta><title>の2行を選択し、

1
2
3
4
<head>
<meta charset="utf-8">
<title>TEST PAGE</title>
</head>

TABをおすと

1
2
3
4
<head>
<meta charset="utf-8">
<title>TEST PAGE</title>
</head>

タブインデントを戻すときは

対象コードブロックを選択して、Shift + TAB

同様に、<meta><title>の2行を選択し、

1
2
3
4
<head>
<meta charset="utf-8">
<title>TEST PAGE</title>
</head>

Shift + TABをおすと

1
2
3
4
<head>
<meta charset="utf-8">
<title>TEST PAGE</title>
</head>

コメント・シェア

Git for Windowsの文字コード変換

 
カテゴリー Git   タグ

Bashスクリプトの文字コードがおかしくなる

開発はWindows上で行っているが、実行はDockerを使用している。
この環境でgit pullするとBashスクリプトの改行コードがCRLFにかわり、実行エラーになってしまう事象が発生。

改行コードをかえているもの

gitのcore.autocrlf=trueが変換している。

core.autocrlfの挙動

ローカルリポジトリからのcheckout時とcommit時に変換している

Value checkout(IN) commit(OUT)
true LFからCRLF CRLFからLF
input CRLFからLF
false

どうするか?

リポジトリはLF統一にしたいので、inputを選択。

設定

1
2
3
4
5
6
7
8
9
PS > git config -l
...略...
core.autocrlf=true
...略...
PS > git config --global core.autocrlf input
PS > git config -l
...略...
core.autocrlf=input
...略...

コメント・シェア

nullpo

めも


募集中


Japan