OS가 centos에서 ubuntu로 변경되어 ubuntu에서도 logstash 설정하는 내용을 남깁니다.

 

설명을 위해서 fortigate log를 파싱하기 위한 logstash 설정 입니다.

내용 확인 후 활용하시면 좋을 것 같아요.

 

logstash 설정 파일 생성하기

logmgmt@syslogserver:~$ sudo cat /etc/logstash/conf.d/logstash.conf
input {
        file {
                path => "/var/log/rsyslog/192.168.000.000/*.log"
                start_position => "beginning"
                tags => ["fortigate"]
        }
}

filter {
        if "fortigate" in [tags] {
                grok {
                        patterns_dir => ["/etc/logstash/pattern.d"]
                                match => { "message" => [ "%{FORTILOG} %{GREEDYDATA:sub_message}" ] }
                }
                kv {
                        source => "sub_message"
                        value_split => "="
                }
                mutate { remove_field => [ "message" ] }
                mutate { remove_field => [ "sub_message" ] }
                if "wan" in [srcintfrole] {
                        geoip {
                                source => "srcip"
                                target => "geoip_src"
                        }
                }
                if [sentbyte] != "" and [rcvdbyte] != "" {
                        bytes {
                                source => "rcvdbyte"
                                target => "receivedbyte"
                        }
                        bytes {
                                source => "sentbyte"
                                target => "sentedbyte"
                        }
                }
                mutate {
                        convert => {
                                "rcvdpkt" => "integer"
                                "sentpkt" => "integer"
                                "proto" => "integer"
                                "srcserver" => "integer"
                                "sessionid" => "integer"
                                "duration" => "integer"
                                "policyid" => "integer"
                                "HOUR" => "integer"
                                "MINUTE" => "integer"
                                "SECOND" => "integer"
                        }
                }
        }
}

output {
        if "fortigate" in [tags] {
                if "traffic" in [type] {
                        elasticsearch {
                                hosts => "http://192.168.000.000:9200"
                                index => "logstash-fortigate-traffic-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "event" in [type] {
                        elasticsearch {
                                hosts => "http://192.168.000.000:9200"
                                index => "logstash-fortigate-event-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "utm" in [type] {
                        elasticsearch {
                                hosts => "http://192.168.000.000:9200"
                                index => "logstash-fortigate-utm-index-%{+YYYY.MM.dd}"
                        }
                }
        }
}
logmgmt@syslogserver:~$

 

 

fortigate 패턴 파일 생성하기

logmgmt@syslogserver:~$ sudo cat /etc/logstash/pattern.d/fortigate.pattern
####################################
###Fortinet Syslog Pattern Types:###
####################################

FORTILOG (?<timestamp>^\w+\s+\d+\s+\d+\:\d+\:\d+)\s(?<log_send_dev>(?:%{IP}|\_gateway))
logmgmt@syslogserver:~$

 

logstash-filter-bytes 필터 설치하기

logmgmt@syslogserver:~$ sudo /usr/share/logstash/bin/logstash-plugin install logstash-filter-bytes
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-filter-bytes
Resolving mixin dependencies
Installing logstash-filter-bytes
Installation successful
logmgmt@syslogserver:~$
저작자표시 비영리 변경금지 (새창열림)

'기술 노트 > ubuntu' 카테고리의 다른 글

Prometheus 설치  (1) 2023.12.21
kibana 설정하기  (1) 2023.12.05
elasticsearch 설정하기  (0) 2023.12.04
rsyslog 설정하기  (1) 2023.12.04
우분투에서 IP 주소 확인하기  (1) 2023.12.04

fortigate 6.X 와 7.X대 syslog가 변화가 있었네요.

7.X대 logstash 설정은 코드가 좀 더 업그레이드 되었습니다.

[root@tmplogsvr 192.168.x.x]# vi /etc/logstash/conf.d/logstash.conf

input {
        file {
                path => "/var/log/rsyslog/192.168.x.1/*.log"
                start_position => "beginning"
                tags => ["fortigate"]
        }
        file {
                path => "/var/log/rsyslog/192.168.x.2/*.log"
                start_position => "beginning"
                tags => ["fortigate"]
        }
}

filter {
        if "fortigate" in [tags] {
                grok {
                        patterns_dir => ["/etc/logstash/pattern.d"]
                        match => { "message" => [ "%{FORTILOG} %{GREEDYDATA:sub_message}" ] }
                        overwrite => [ "message" ]
                }
                kv {
                        value_split => "="
                }

                if "wan" in [srcintfrole] {
                        geoip {
                                source => "srcip"
                                target => "geoip_src"
                        }
                }

                if [sentbyte] != "" and [rcvdbyte] != "" {
                        bytes {
                                source => "rcvdbyte"
                                target => "rcvdbyte"
                        }
                        bytes {
                                source => "sentbyte"
                                target => "sentbyte"
                        }
                }
                mutate {
                        convert => {
                                "transport" => "integer"
                                "duration" => "integer"
                                "sentpkt" => "integer"
                                "rcvdpkt" => "integer"
                                "srcserver" => "integer"
                                "proto" => "integer"
                                "sessionid" => "integer"
                                "policyid" => "integer"
                                "crscore" => "integer"
                                "craction" => "integer"
                                "srcport" => "integer"
                                "dstport" => "integer"
                                "cpu" => "integer"
                                "mem" => "integer"
                                "totalsession" => "integer"
                                "disk" => "integer"
                                "setuprate" => "integer"
                                "disklograte" => "integer"
                                "fazlograte" => "integer"
                                "freediskstorage" => "integer"
                                "sysuptime" => "integer"
                        }
                }
        }
}

output {
        if "fortigate" in [tags] {
                elasticsearch {
                        hosts => "http://192.168.x.x:9200"
                        index => "logstash-fortigate-index-%{+YYYY.MM.dd}"
                        data_stream => "false"
                }
        }
}
[root@tmplogsvr pattern.d]# vi /etc/logstash/pattern.d/fortigate.pattern
FORTILOG (?<timestamp>^\w+\s+\d+\s+\d+\:\d+\:\d+)\s(?<log_send_dev>(?:%{IP}|\_gateway))
저작자표시 비영리 변경금지 (새창열림)

'기술 노트 > fortigate' 카테고리의 다른 글

LDAP 서버 연결 설정이 안될 때 고려사항  (0) 2023.07.17
AD를 LDAP 서버로 연결 하기  (0) 2023.07.07
syslog 전달 시 _gateway를 IP로 변경하는 방법  (0) 2023.06.15
웹 페이지 접근 보호  (1) 2023.05.23
로그인 알람 설정  (0) 2023.04.06

fortigate log 파싱 부분이 추가되었습니다.

fortigate 로그 파싱할때에는 logstash-filter-bytes를 사용하였습니다.

이유는 logstash에서 로그를 파싱하면 모든 자료는 기본적으로 type이 text입니다.

마찬가지로 숫자도 text로 인식합니다.

그래서 데이터 전송량도 type에 맞게 bytes라는 필터를 추가로 설치하였습니다.

logstash-filter-bytes의 설치는 아래의 작성 글을 참고하시기 바랍니다.

https://dirt-spoon.tistory.com/80

 

input {
        file {
                path => "/var/log/rsyslog/192.168.10.2/*.log"
                start_position => "beginning"
                tags => ["ap1"]
        }
        file {
                path => "/var/log/rsyslog/192.168.10.3/*.log"
                start_position => "beginning"
                tags => ["ap2"]
        }
        file {
                path => "/var/log/rsyslog/192.168.10.4/*.log"
                start_position => "beginning"
                tags => ["ap3"]
        }
        file {
                path => "/var/log/rsyslog/192.168.10.5/*.log"
                start_position => "beginning"
                tags => ["ap4"]
        }
        file {
                path => "/var/log/rsyslog/192.168.0.14/*.log"
                start_position => "beginning"
                tags => ["fortigate"]
        }
}

filter {
        if "ap1" in [tags] or "ap2" in [tags] or "ap3" in [tags] or "ap4" in [tags]  {
                grok {
                        patterns_dir => ["/etc/logstash/pattern.d"]
                        match => {
                                "message" => [
                                        "%{SYSLOGTIMESTAMP:access_time} %{IPORHOST:ip_or_host} %{IPORHOST:process}\[%{BASE10NUM:process_id}\]\: %{GREEDYDATA:sub_message}",
                                        "%{SYSLOGTIMESTAMP:access_time} %{IPORHOST:ip_or_host} \[%{BASE10NUM:process_id}\]\: %{GREEDYDATA:sub_message}",
                                        "%{SYSLOGTIMESTAMP:access_time} %{IPORHOST:ip_or_host} %{WORD:process}\: User\: %{GREEDYDATA:user} last logged %{GREEDYDATA:access_result} in %{GREEDYDATA:access_day}\#%{NUMBER:deauthentication_reason_code}, to %{IP:destination_ip}, from %{IP:source_ip} using %{WORD:access_method}",
                                        "%{SYSLOGTIMESTAMP:access_time} %{IPORHOST:ip_or_host} %{WORD:process}\: User\: %{GREEDYDATA:reason} in %{GREEDYDATA:access_day}\#%{NUMBER:deauthentication_reason_code}, to %{IP:destination_ip}, from %{IP:source_ip} using %{WORD:access_method}"
                                ]
                        }
                }
                mutate { remove_field => [ "message" ] }
        }
        else if "fortigate" in [tags] {
                grok {
                        patterns_dir => ["/etc/logstash/pattern.d"]
                        match => { "message" => [ "%{FORTILOG} %{GREEDYDATA:sub_message}" ] }
                }
                kv {
                        source => "sub_message"
                        value_split => "="
                }
                mutate { remove_field => [ "message" ] }
                mutate { remove_field => [ "sub_message" ] }
                if "wan" in [srcintfrole] {
                        geoip {
                                source => "srcip"
                                target => "geoip_src"
                        }
                }
                if [sentbyte] != "" and [rcvdbyte] != "" {
                        bytes {
                                source => "rcvdbyte"
                                target => "receivedbyte"
                        }
                        bytes {
                                source => "sentbyte"
                                target => "sentedbyte"
                        }
                }
                mutate {
                        convert => {
                                "rcvdpkt" => "integer"
                                "sentpkt" => "integer"
                                "proto" => "integer"
                                "srcserver" => "integer"
                                "sessionid" => "integer"
                                "duration" => "integer"
                                "policyid" => "integer"
                                "HOUR" => "integer"
                                "MINUTE" => "integer"
                                "SECOND" => "integer"
                        }
                }
        }
}

output {
        if "ap1" in [tags] {
                elasticsearch {
                        hosts => "http://192.168.0.17:9200"
                        index => "logstash-ap1-index-%{+YYYY.MM.dd}"
                }
        }
        else if "ap2" in [tags] {
                elasticsearch {
                        hosts => "http://192.168.0.17:9200"
                        index => "logstash-ap2-index-%{+YYYY.MM.dd}"
                }
        }
        else if "ap3" in [tags] {
                elasticsearch {
                        hosts => "http://192.168.0.17:9200"
                        index => "logstash-ap3-index-%{+YYYY.MM.dd}"
                }
        }
        else if "ap4" in [tags] {
                elasticsearch {
                        hosts => "http://192.168.0.17:9200"
                        index => "logstash-ap4-index-%{+YYYY.MM.dd}"
                }
        }
        else if "fortigate" in [tags] {
                if "traffic" in [LOG_TYPE] {
                        elasticsearch {
                                hosts => "http://192.168.0.17:9200"
                                index => "logstash-fortigate-traffic-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "event" in [LOG_TYPE] {
                        elasticsearch {
                                hosts => "http://192.168.0.17:9200"
                                index => "logstash-fortigate-event-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "utm" in [LOG_TYPE] {
                        elasticsearch {
                                hosts => "http://192.168.0.17:9200"
                                index => "logstash-fortigate-utm-index-%{+YYYY.MM.dd}"
                        }
                }
        }
}
저작자표시 비영리 변경금지 (새창열림)

'기술 노트 > logstash' 카테고리의 다른 글

font 깨질 때  (0) 2025.03.24
logstash.conf_230306  (0) 2023.03.06
logstash.conf_23.02.20  (0) 2023.02.20
logstash 파일 파싱하기  (0) 2023.02.20
logstash 설정  (0) 2023.02.20

logstash plugin을 이용하여 logstash-filter-bytes 설치

[root@tmplogsvr bin]# /usr/share/logstash/bin/logstash-plugin install logstash-filter-bytes
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-filter-bytes
Resolving mixin dependencies
Installing logstash-filter-bytes
Installation successful
[root@tmplogsvr bin]#

 

fortigate를 위한 logstash 패턴

####################################
###Fortinet Syslog Pattern Types:###
####################################

FORTILOG (?<TIMESTAMP>^\w+\s+\d+\s+\d+\:\d+\:\d+)\s(?<Client>\d+.\d+.\d+.\d+)\sdate=(?<DAY>\w+\-\w+\-\w+)\stime=(?<HOUR>\d+)\:(?<MINUTE>\d+)\:(?<SECOND>\d+)\sdevname="(?<Device_Name>.*)"\sdevid="(?<DEV_ID>\w+)"\slogid="(?<LOG_ID>\d+)"\stype="(?<LOG_TYPE>\w+)"\ssubtype="(?<SUB_LOG_TYPE>\w+)"

※ 기본 grok.pattern의 설정과 연계되어 있어 grok.pattern도 함께 작성 필요

 

fortigate를 위한 logstash 설정

[root@tmplogsvr bin]# cat /etc/logstash/conf.d/logstash.conf
input {
        file {
                path => "/var/log/rsyslog/192.168.10.14/*.log"
                start_position => "beginning"
                tags => ["fortigate"]
        }
}

filter {
        if "fortigate" in [tags] {
                grok {
                        patterns_dir => ["/etc/logstash/pattern.d"]
                        match => { "message" => [ "%{FORTILOG} %{GREEDYDATA:sub_message}" ] }
                }
                kv {
                        source => "sub_message"
                        value_split => "="
                }
                mutate { remove_field => [ "sub_message" ] }
                if "wan" in [srcintfrole] {
                        geoip {
                                source => "srcip"
                                target => "geoip_src"
                        }
                }
                if [sentbyte] != "" and [rcvdbyte] != "" {
                        bytes {
                                source => "rcvdbyte"
                                target => "receivedbyte"
                        }
                        bytes {
                                source => "sentbyte"
                                target => "sentedbyte"
                        }
                }
                mutate {
                        convert => {
                                "rcvdpkt" => "integer"
                                "sentpkt" => "integer"
                                "proto" => "integer"
                                "srcserver" => "integer"
                                "sessionid" => "integer"
                                "duration" => "integer"
                                "policyid" => "integer"
                                "HOUR" => "integer"
                                "MINUTE" => "integer"
                                "SECOND" => "integer"
                        }
                }
        }
}

output {
        if "fortigate" in [tags] {
                if "traffic" in [LOG_TYPE] {
                        elasticsearch {
                                hosts => ["http://192.168.0.17:9200"]
                                index => "logstash-fortigate-traffic-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "event" in [LOG_TYPE] {
                        elasticsearch {
                                hosts => ["http://192.168.0.17:9200"]
                                index => "logstash-fortigate-event-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "utm" in [LOG_TYPE] {
                        elasticsearch {
                                hosts => ["http://192.168.0.17:9200"]
                                index => "logstash-fortigate-utm-index-%{+YYYY.MM.dd}"
                        }
                }
        }
}
저작자표시 비영리 변경금지 (새창열림)

'기술 노트 > fortigate' 카테고리의 다른 글

웹 페이지 접근 보호  (1) 2023.05.23
로그인 알람 설정  (0) 2023.04.06
interface https http ssh 접속 허용 cli  (0) 2023.03.24
interface status cli  (0) 2023.03.24
system shutdown cli  (0) 2023.03.24

+ Recent posts

티스토리툴바