logstash 설정하기

OS가 centos에서 ubuntu로 변경되어 ubuntu에서도 logstash 설정하는 내용을 남깁니다.

 

설명을 위해서 fortigate log를 파싱하기 위한 logstash 설정 입니다.

내용 확인 후 활용하시면 좋을 것 같아요.

 

logstash 설정 파일 생성하기

logmgmt@syslogserver:~$ sudo cat /etc/logstash/conf.d/logstash.conf
input {
        file {
                path => "/var/log/rsyslog/192.168.000.000/*.log"
                start_position => "beginning"
                tags => ["fortigate"]
        }
}

filter {
        if "fortigate" in [tags] {
                grok {
                        patterns_dir => ["/etc/logstash/pattern.d"]
                                match => { "message" => [ "%{FORTILOG} %{GREEDYDATA:sub_message}" ] }
                }
                kv {
                        source => "sub_message"
                        value_split => "="
                }
                mutate { remove_field => [ "message" ] }
                mutate { remove_field => [ "sub_message" ] }
                if "wan" in [srcintfrole] {
                        geoip {
                                source => "srcip"
                                target => "geoip_src"
                        }
                }
                if [sentbyte] != "" and [rcvdbyte] != "" {
                        bytes {
                                source => "rcvdbyte"
                                target => "receivedbyte"
                        }
                        bytes {
                                source => "sentbyte"
                                target => "sentedbyte"
                        }
                }
                mutate {
                        convert => {
                                "rcvdpkt" => "integer"
                                "sentpkt" => "integer"
                                "proto" => "integer"
                                "srcserver" => "integer"
                                "sessionid" => "integer"
                                "duration" => "integer"
                                "policyid" => "integer"
                                "HOUR" => "integer"
                                "MINUTE" => "integer"
                                "SECOND" => "integer"
                        }
                }
        }
}

output {
        if "fortigate" in [tags] {
                if "traffic" in [type] {
                        elasticsearch {
                                hosts => "http://192.168.000.000:9200"
                                index => "logstash-fortigate-traffic-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "event" in [type] {
                        elasticsearch {
                                hosts => "http://192.168.000.000:9200"
                                index => "logstash-fortigate-event-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "utm" in [type] {
                        elasticsearch {
                                hosts => "http://192.168.000.000:9200"
                                index => "logstash-fortigate-utm-index-%{+YYYY.MM.dd}"
                        }
                }
        }
}
logmgmt@syslogserver:~$

 

 

fortigate 패턴 파일 생성하기

logmgmt@syslogserver:~$ sudo cat /etc/logstash/pattern.d/fortigate.pattern
####################################
###Fortinet Syslog Pattern Types:###
####################################

FORTILOG (?<timestamp>^\w+\s+\d+\s+\d+\:\d+\:\d+)\s(?<log_send_dev>(?:%{IP}|\_gateway))
logmgmt@syslogserver:~$

 

logstash-filter-bytes 필터 설치하기

logmgmt@syslogserver:~$ sudo /usr/share/logstash/bin/logstash-plugin install logstash-filter-bytes
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-filter-bytes
Resolving mixin dependencies
Installing logstash-filter-bytes
Installation successful
logmgmt@syslogserver:~$