Parserを使用していない場合出力されたログは適切にParseされていない。 123fluent-bit_1 | [0] 4fb66927922a: [1621578165.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:45 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]fluent-bit_1 | [0] 71d75b79d476: [1621578168.000000000, {"container_id"=>"71d75b79d476723bb0cece0516e50ed36ddb99b36db21573eaa11baabeb67c06", "container_name"=>"/stdout_web1_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}]fluent-bit_1 | [0] 4fb66927922a: [1621578171.000000000, {"container_id"=>"4fb66927922a06fd696ed9ee5cc2c5c287592ab13786b9fc9e5704ac3b8077ea", "container_name"=>"/stdout_web2_1", "source"=>"stdout", "log"=>"172.24.0.1 - - [21/May/2021:06:22:51 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-""}] 既定のnginxのParserを使うParserでFILTERでParserを指定する方法が記載されている。 1234567891011121314151617[SERVICE] Parsers_File /path/to/parsers.conf[INPUT] Name dummy Tag dummy.data Dummy {"data":"100 0.5 true This is example"}[FILTER] Name parser Match dummy.* Key_Name data Parser dummy_test[OUTPUT] Name stdout Match * これを参考にFILTERを利用する設定を定義する。parsers.confでは様々なParserが定義されており、今回はnginxを使用する。 123456[PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)") Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z Parsers_File parsers.confを設定することで、この既定のParserが利用可能になる。 123456789101112131415161718[SERVICE] Log_Level info Parsers_File parsers.conf[INPUT] Name forward Listen 0.0.0.0 Port 24224[FILTER] Name parser Match * Key_Name log Parser nginx[OUTPUT] Name stdout Match * Name parserのFILTER定義を作成する。対象となるのは現在logで出力されている部分だ。 1"log"=>"172.24.0.1 - - [21/May/2021:06:22:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"" Parseされたenginxアクセスログnginxのログ部分はParseされて出力された。 123456789101112web1_1 | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"web1_1 | 172.23.0.1 - - [21/May/2021:07:35:34 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"web1_1 | 172.23.0.1 - - [21/May/2021:07:35:35 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"web2_1 | 172.23.0.1 - - [21/May/2021:07:35:36 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"web2_1 | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"web2_1 | 172.23.0.1 - - [21/May/2021:07:35:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"fluent-bit_1 | [0] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]fluent-bit_1 | [1] 9dfdc1a8b440: [1621582534.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]fluent-bit_1 | [2] 9dfdc1a8b440: [1621582535.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]fluent-bit_1 | [0] 5b4d36337166: [1621582536.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]fluent-bit_1 | [1] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}]fluent-bit_1 | [2] 5b4d36337166: [1621582537.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"}] Parse対象以外の項目を残すしかし,元々出力されえていたcontainer_idなどが消えてしまっている。そこでReserve_Data Onを使用して元の項目を残す。ただし、Prase対象のlogは必要ないので、Preserve_Key Offで残さないようにする。 1234567891011121314151617181920[SERVICE] Log_Level info Parsers_File parsers.conf[INPUT] Name forward Listen 0.0.0.0 Port 24224[FILTER] Name parser Match * Key_Name log Parser nginx Preserve_Key Off Reserve_Data On[OUTPUT] Name stdout Match * 結果は以下。期待通りの項目が出力されている。 12345678web1_1 | 172.23.0.1 - - [21/May/2021:07:47:03 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"web1_1 | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"web1_1 | 172.23.0.1 - - [21/May/2021:07:47:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"web1_1 | 172.23.0.1 - - [21/May/2021:07:47:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"fluent-bit_1 | [0] 9dfdc1a8b440: [1621583223.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]fluent-bit_1 | [1] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]fluent-bit_1 | [2] 9dfdc1a8b440: [1621583224.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "source"=>"stdout", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1"}]fluent-bit_1 | [3] 9dfdc1a8b440: [1621583225.000000000, {"remote"=>"172.23.0.1", "host"=>"-", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"304", "size"=>"0", "referer"=>"-", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36", "container_id"=>"9dfdc1a8b440382b3185db05516656af156ce2f220a293857cf6039a32af102c", "container_name"=>"/parser_web1_1", "source"=>"stdout"}]