Parserを使用していない場合

出力されたログは適切にParseされていない。

1
2
3
fluent-bit_1  | [0] 4fb66927922a: [1621578165.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1 | [0] 71d75b79d476: [1621578168.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]
fluent-bit_1 | [0] 4fb66927922a: [1621578171.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]

既定のnginxのParserを使う

ParserでFILTERでParserを指定する方法が記載されている。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[SERVICE]
Parsers_File /path/to/parsers.conf

[INPUT]
Name dummy
Tag dummy.data
Dummy {"data":"100 0.5 true This is example"}

[FILTER]
Name parser
Match dummy.*
Key_Name data
Parser dummy_test

[OUTPUT]
Name stdout
Match *

これを参考にFILTERを利用する設定を定義する。
parsers.confでは様々なParserが定義されており、今回はnginxを使用する。

1
2
3
4
5
6
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z

Parsers_File parsers.confを設定することで、この既定のParserが利用可能になる。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[SERVICE]
Log_Level info
Parsers_File parsers.conf

[INPUT]
Name forward
Listen 0.0.0.0
Port 24224

[FILTER]
Name parser
Match *
Key_Name log
Parser nginx

[OUTPUT]
Name stdout
Match *

Name parserのFILTER定義を作成する。
対象となるのは現在logで出力されている部分だ。

1
"log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""

Parseされたenginxアクセスログ

nginxのログ部分はParseされて出力された。

1
2
3
4
5
6
7
8
9
10
11
12
web1_1        | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:35:35 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 172.23.0.1 - - [21/May/2021:07:35:36 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web2_1 | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1 | [0] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [1] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [2] 9dfdc1a8b440: [1621582535.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [0] 5b4d36337166: [1621582536.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [1] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]
fluent-bit_1 | [2] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]

Parse対象以外の項目を残す

しかし,元々出力されえていたcontainer_idなどが消えてしまっている。
そこでReserve_Data Onを使用して元の項目を残す。ただし、Prase対象のlogは必要ないので、Preserve_Key Offで残さないようにする。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[SERVICE]
Log_Level info
Parsers_File parsers.conf

[INPUT]
Name forward
Listen 0.0.0.0
Port 24224

[FILTER]
Name parser
Match *
Key_Name log
Parser nginx
Preserve_Key Off
Reserve_Data On

[OUTPUT]
Name stdout
Match *

結果は以下。期待通りの項目が出力されている。

1
2
3
4
5
6
7
8
web1_1        | 172.23.0.1 - - [21/May/2021:07:47:03 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
web1_1 | 172.23.0.1 - - [21/May/2021:07:47:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
fluent-bit_1 | [0] 9dfdc1a8b440: [1621583223.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]
fluent-bit_1 | [1] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]
fluent-bit_1 | [2] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "source"=>"stdout", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1"}]
fluent-bit_1 | [3] 9dfdc1a8b440: [1621583225.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]