흙수저의 엔지니어링

/etc/kibana/kibana.yml

logmgmt@syslogserver:~$ sudo cat /etc/kibana/kibana.yml
# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html

# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
server.publicBaseUrl: "http://192.168.000.000:5601"

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name. This is used for display purposes.
server.name: "logserver"

# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://192.168.000.000:9200"]

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Kibana can also authenticate to Elasticsearch via "service account tokens".
# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
# Use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# The maximum number of sockets that can be used for communications with elasticsearch.
# Defaults to `Infinity`.
#elasticsearch.maxSockets: 1024

# Specifies whether Kibana should use compression for communications with elasticsearch
# Defaults to `false`.
#elasticsearch.compression: false

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# =================== System: Elasticsearch (Optional) ===================
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# =================== System: Logging ===================
# Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'info'
#logging.root.level: debug

# Enables you to specify a file where Kibana stores log output.
logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
#  layout:
#    type: json

# Logs queries sent to Elasticsearch.
#logging.loggers:
#  - name: elasticsearch.query
#    level: debug

# Logs http responses.
#logging.loggers:
#  - name: http.server.response
#    level: debug

# Logs system usage information.
#logging.loggers:
#  - name: metrics.ops
#    level: debug

# =================== System: Other ===================
# The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data
#path.data: data

# Specifies the path where Kibana creates the process ID file.
pid.file: /run/kibana/kibana.pid

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000ms.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English (default) "en", Chinese "zh-CN", Japanese "ja-JP", French "fr-FR".
#i18n.locale: "en"

# =================== Frequently used (Optional)===================

# =================== Saved Objects: Migrations ===================
# Saved object migrations run at startup. If you run into migration-related issues, you might need to adjust these settings.

# The number of documents migrated at a time.
# If Kibana can't start up or upgrade due to an Elasticsearch `circuit_breaking_exception`,
# use a smaller batchSize value to reduce the memory pressure. Defaults to 1000 objects per batch.
#migrations.batchSize: 1000

# The maximum payload size for indexing batches of upgraded saved objects.
# To avoid migrations failing due to a 413 Request Entity Too Large response from Elasticsearch.
# This value should be lower than or equal to your Elasticsearch cluster’s `http.max_content_length`
# configuration option. Default: 100mb
#migrations.maxBatchSizeBytes: 100mb

# The number of times to retry temporary migration failures. Increase the setting
# if migrations fail frequently with a message such as `Unable to complete the [...] step after
# 15 attempts, terminating`. Defaults to 15
#migrations.retryAttempts: 15

# =================== Search Autocomplete ===================
# Time in milliseconds to wait for autocomplete suggestions from Elasticsearch.
# This value must be a whole number greater than zero. Defaults to 1000ms
#unifiedSearch.autocomplete.valueSuggestions.timeout: 1000

# Maximum number of documents loaded by each shard to generate autocomplete suggestions.
# This value must be a whole number greater than zero. Defaults to 100_000
#unifiedSearch.autocomplete.valueSuggestions.terminateAfter: 100000

OS가 centos에서 ubuntu로 변경되어 ubuntu에서도 logstash 설정하는 내용을 남깁니다.

설명을 위해서 fortigate log를 파싱하기 위한 logstash 설정 입니다.

내용 확인 후 활용하시면 좋을 것 같아요.

logstash 설정 파일 생성하기

logmgmt@syslogserver:~$ sudo cat /etc/logstash/conf.d/logstash.conf
input {
        file {
                path => "/var/log/rsyslog/192.168.000.000/*.log"
                start_position => "beginning"
                tags => ["fortigate"]
        }
}

filter {
        if "fortigate" in [tags] {
                grok {
                        patterns_dir => ["/etc/logstash/pattern.d"]
                                match => { "message" => [ "%{FORTILOG} %{GREEDYDATA:sub_message}" ] }
                }
                kv {
                        source => "sub_message"
                        value_split => "="
                }
                mutate { remove_field => [ "message" ] }
                mutate { remove_field => [ "sub_message" ] }
                if "wan" in [srcintfrole] {
                        geoip {
                                source => "srcip"
                                target => "geoip_src"
                        }
                }
                if [sentbyte] != "" and [rcvdbyte] != "" {
                        bytes {
                                source => "rcvdbyte"
                                target => "receivedbyte"
                        }
                        bytes {
                                source => "sentbyte"
                                target => "sentedbyte"
                        }
                }
                mutate {
                        convert => {
                                "rcvdpkt" => "integer"
                                "sentpkt" => "integer"
                                "proto" => "integer"
                                "srcserver" => "integer"
                                "sessionid" => "integer"
                                "duration" => "integer"
                                "policyid" => "integer"
                                "HOUR" => "integer"
                                "MINUTE" => "integer"
                                "SECOND" => "integer"
                        }
                }
        }
}

output {
        if "fortigate" in [tags] {
                if "traffic" in [type] {
                        elasticsearch {
                                hosts => "http://192.168.000.000:9200"
                                index => "logstash-fortigate-traffic-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "event" in [type] {
                        elasticsearch {
                                hosts => "http://192.168.000.000:9200"
                                index => "logstash-fortigate-event-index-%{+YYYY.MM.dd}"
                        }
                }
                else if "utm" in [type] {
                        elasticsearch {
                                hosts => "http://192.168.000.000:9200"
                                index => "logstash-fortigate-utm-index-%{+YYYY.MM.dd}"
                        }
                }
        }
}
logmgmt@syslogserver:~$

fortigate 패턴 파일 생성하기

logmgmt@syslogserver:~$ sudo cat /etc/logstash/pattern.d/fortigate.pattern
####################################
###Fortinet Syslog Pattern Types:###
####################################

FORTILOG (?<timestamp>^\w+\s+\d+\s+\d+\:\d+\:\d+)\s(?<log_send_dev>(?:%{IP}|\_gateway))
logmgmt@syslogserver:~$

logstash-filter-bytes 필터 설치하기

logmgmt@syslogserver:~$ sudo /usr/share/logstash/bin/logstash-plugin install logstash-filter-bytes
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-filter-bytes
Resolving mixin dependencies
Installing logstash-filter-bytes
Installation successful
logmgmt@syslogserver:~$

/etc/elasticsearch/elasticsearch.yml

logmgmt@syslogserver:~$ sudo cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: es-cluster-1
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: logserver-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["127.0.0.1", "[::1]"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["logserver-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 01-12-2023 03:44:21
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: false

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["localhost"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

우분투에 기본적으로 설치되어 있는지는 모르겠지만, 저는 설치가 되어 있어서 설정 부분만 안내합니다.

개인적으로는 syslog는 UDP Port 514로 로그를 수집하는 상황으로 설정합니다.

/etc/rsyslog.conf 설정하기

• module(load="imudp") 라인의 주석 제거
• input(type="imudp" port="514") 라인의 주석 제거

# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf


#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
module(load="imudp") # 주석 제거
input(type="imudp" port="514") # 주석 제거

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

/etc/rsyslog.d/ 폴더에 원하는 conf 파일 생성

• 저는 원격으로 syslog를 수집하기 때문에 10-remote.conf로 생성하였습니다.

logmgmt@syslogserver:/etc/rsyslog.d$ ls -al
total 28
drwxr-xr-x   2 root root  4096 12월  1 18:48 .
drwxr-xr-x 131 root root 12288 12월  1 18:46 ..
-rw-r--r--   1 root root   130 12월  1 18:48 10-remote.conf
-rw-r--r--   1 root root   314  9월 19  2021 20-ufw.conf
-rw-r--r--   1 root root  1122 12월  1 18:47 50-default.conf
logmgmt@syslogserver:/etc/rsyslog.d$

/etc/rsyslog.d/10-remote.conf

logmgmt@syslogserver:cat /etc/rsyslog.d$ cat 10-remote.conf
# syslog 수신 시 저장할 위치 및 파일 설정 템플릿
$template Remote,"/var/log/rsyslog/%fromhost-ip%/%$YEAR%-%$MONTH%-%$DAY%.log"

# syslog 수신 시 fromhost-ip가 127.0.0.1이 아닌 경우 템플릿 Remote 정책을 적용
:fromhost-ip, !isequal, "127.0.0.1" ?Remote

# 템플릿 Remote로 수신한 경우 로그 처리 중지
# 중지하지 않을 경우, OS에서 수집하는 syslog나 message에도 로그가 수집 됨 (중복 수집)
& stop
logmgmt@syslogserver:/etc/rsyslog.d$

우분투에는 ifconfig 명령어가 안 먹히네요.

우분투에서 IP Address 확인하는 방법과 ifconfig 설치 후 IP Address 확인하는 방법을 안내합니다.

우분투에서 IP Address 확인하기 (ip addr)

logmgmt@syslogserver:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d4:ae:52:00:00:00 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
    inet 192.168.000.000/24 brd 192.168.000.255 scope global noprefixroute eno1
       valid_lft forever preferred_lft forever
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether d4:ae:52:00:00:00 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
logmgmt@syslogserver:~$

우분투에서 IP Address 확인하기 (ifconfig)

logmgmt@syslogserver:~$ sudo apt install net-tools
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다... 완료
상태 정보를 읽는 중입니다... 완료
다음 새 패키지를 설치할 것입니다:
  net-tools
0개 업그레이드, 1개 새로 설치, 0개 제거 및 8개 업그레이드 안 함.
204 k바이트 아카이브를 받아야 합니다.
이 작업 후 819 k바이트의 디스크 공간을 더 사용하게 됩니다.
받기:1 http://kr.archive.ubuntu.com/ubuntu jammy/main amd64 net-tools amd64 1.60+git20181103.0eebece-1ubuntu5 [204 kB]
내려받기 204 k바이트, 소요시간 2초 (100.0 k바이트/초)
Selecting previously unselected package net-tools.
(데이터베이스 읽는중 ...현재 283267개의 파일과 디렉터리가 설치되어 있습니다.)
Preparing to unpack .../net-tools_1.60+git20181103.0eebece-1ubuntu5_amd64.deb ...
Unpacking net-tools (1.60+git20181103.0eebece-1ubuntu5) ...
net-tools (1.60+git20181103.0eebece-1ubuntu5) 설정하는 중입니다 ...
Processing triggers for man-db (2.10.2-1) ...
logmgmt@syslogserver:~$ ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.000.000  netmask 255.255.255.0  broadcast 192.168.000.255
        ether d4:ae:52:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 2757378  bytes 3668986690 (3.6 GB)
        RX errors 0  dropped 16327  overruns 0  frame 0
        TX packets 159042  bytes 12797747 (12.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether d4:ae:52:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1684  bytes 162952 (162.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1684  bytes 162952 (162.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

logmgmt@syslogserver:~$

상황 설명

서버에 disk가 2개 연결되어 있습니다.

이 중 /dev/sdb 디스크가 사용되지 않고 있어서, /var/log/rsyslog 폴더에 mount를 할 계획입니다.

디스크 확인하기

• /dev/sda
• /dev/sdb

logmgmt@syslogserver:~$ sudo parted -l
Model: SEAGATE ST1000NM0023 (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type      File system     Flags
 1      1049kB  32.8GB  32.8GB  primary   linux-swap(v1)
 2      32.8GB  33.8GB  1023MB  extended
 5      32.8GB  33.8GB  1023MB  logical   ext4
 3      33.8GB  34.3GB  512MB   primary   fat32           boot, esp
 4      34.3GB  1000GB  966GB   primary   ext4


Model: ATA TOSHIBA MG03ACA2 (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  2000GB  2000GB  primary  ntfs


logmgmt@syslogserver:~$

디스크 ubuntu에서 사용할 수 있도록 변환하기

• /dev/sdb

logmgmt@syslogserver:~$ sudo parted /dev/sdb
GNU Parted 3.4
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) print
Model: ATA TOSHIBA MG03ACA2 (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start  End  Size  File system  Name  Flags

(parted)

 mount 할 디스크의 partition 및 용량 설정하기

• /dev/sdb 용량 전체로 할당 (디스크 용량을 나눠 2개 이상으로 설정할 경우 partition은 2개 이상이 될 수 있음)
• /dev/sdb1 으로 partition 설정 (디스크로 사용할 수 있는 구역, 예 - C:\, D:\)

(parted) mkpart primary ext4 0% 100%
(parted) print
Model: ATA TOSHIBA MG03ACA2 (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  2000GB  2000GB  ext4         primary

(parted) quit
Information: You may need to update /etc/fstab.

logmgmt@syslogserver:~$

mount할 디스크의 device 경로 확인하기

• /dev/sdb1 (partition이 늘어날 경우 /dev/sdb2 등으로 늘어날 수 있음)

logmgmt@syslogserver:~$ fdisk -l /dev/sdb
fdisk: cannot open /dev/sdb: Permission denied
logmgmt@syslogserver:~$ sudo fdisk -l /dev/sdb
Disk /dev/sdb: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: TOSHIBA MG03ACA2
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C016A0C2-47FB-4524-8662-62F857982457

Device     Start        End    Sectors  Size Type
/dev/sdb1   2048 3907028991 3907026944  1.8T Linux filesystem

mount할 디스크 포멧하기

logmgmt@syslogserver:~$ sudo mkfs -t ext4 /dev/sdb1
mke2fs 1.46.5 (30-Dec-2021)
/dev/sdb1 contains a ntfs file system labelled '새 볼륨'
Proceed anyway? (y,N) y
Creating filesystem with 488378368 4k blocks and 122101760 inodes
Filesystem UUID: fe303f5e-5ba7-49e4-8b14-f9a5347f2e42
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): # 엔터
done
Writing superblocks and filesystem accounting information: #엔터
done

logmgmt@syslogserver:~$

디스크 mount 하여 리눅스에서 사용하기

logmgmt@syslogserver:~$ sudo mount -t auto /dev/sdb1 /var/log/rsyslog
logmgmt@syslogserver:~$ df -m
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs               1596     2      1594   1% /run
/dev/sda4         905596 12700    846822   2% /
tmpfs               7977     0      7977   0% /dev/shm
tmpfs                  5     0         5   0% /run/lock
/dev/sda5            943   191       688  22% /boot
/dev/sda3            488     7       481   2% /boot/efi
tmpfs               1596     1      1596   1% /run/user/128
tmpfs               1596     1      1596   1% /run/user/1000
/dev/sdb1        1876685     1   1781282   1% /var/log/rsyslog #mount 완료
logmgmt@syslogserver:~$

자동으로 mount 하기

• /etc/fstab에 등록하기 위해서는 disk의 UUID 정보가 필요

logmgmt@syslogserver:~$ sudo blkid | grep /dev/sdb1
/dev/sdb1: UUID="fe303f5e-5ba7-49e4-8b14-f9a5347f2e42" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="39587ea9-0582-4082-b3d3-6b077fc63fc9"
logmgmt@syslogserver:~$

자동 mount 등록 설정하기 (/etc/fstab)

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda4 during installation
UUID=c64bd3a1-9b2a-4d9a-bfbf-1fd93994a1dc /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda5 during installation
UUID=f3d05dfd-77c8-4a66-b9ba-4ca013060550 /boot           ext4    defaults        0       2
# /boot/efi was on /dev/sda3 during installation
UUID=1785-5B0F  /boot/efi       vfat    umask=0077      0       1
# swap was on /dev/sda1 during installation
UUID=7d8047cd-6cb8-41ec-a532-c0d6d8e4de1f none            swap    sw              0       0
# /var/log/rsyslog was on /dev/sdb1 during installation
UUID=fe303f5e-5ba7-49e4-8b14-f9a5347f2e42 /var/log/rsyslog        ext4    defaults 0       1 # 등록

자동 mount 설정 오류 확인하기

• 메시지가 아무것도 없으면 정상 작동

logmgmt@syslogserver:~$ sudo mount -a
logmgmt@syslogserver:~$

CentOS 에서 ubuntu로 ELK를 재설치하여 기록을 남깁니다.

참고로 저는 최신 버전이 아닌 8.11.0 으로 설치합니다.

elastalert plugin이 8.11.0 까지 지원하네요.

https://github.com/Karql/elastalert-kibana-plugin/releases

아래의 3개는 elasticsearch나 logstash에서 설치하면 따로 하지 않아도 됩니다.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg

sudo apt-get install apt-transport-https

echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list

여기서부터 따라하시면 됩니다.

logmgmt@syslogserver:~$ apt list | grep kibana

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

kibana/stable 8.11.1 amd64
logmgmt@syslogserver:~$ sudo apt install kibana=8.11.0
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다... 완료
상태 정보를 읽는 중입니다... 완료
다음 새 패키지를 설치할 것입니다:
  kibana
0개 업그레이드, 1개 새로 설치, 0개 제거 및 7개 업그레이드 안 함.
314 M바이트 아카이브를 받아야 합니다.
이 작업 후 907 M바이트의 디스크 공간을 더 사용하게 됩니다.
받기:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 kibana amd64 8.11.0 [314 MB]
내려받기 314 M바이트, 소요시간 21초 (15.3 M바이트/초)
Selecting previously unselected package kibana.
(데이터베이스 읽는중 ...현재 196269개의 파일과 디렉터리가 설치되어 있습니다.)
Preparing to unpack .../kibana_8.11.0_amd64.deb ...
Unpacking kibana (8.11.0) ...
kibana (8.11.0) 설정하는 중입니다 ...
Creating kibana group... OK
Creating kibana user... OK
Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/8.11/production.html#openssl-legacy-provider
Created Kibana keystore in /etc/kibana/kibana.keystore
logmgmt@syslogserver:~$

CentOS 에서 ubuntu로 ELK를 재설치하여 기록을 남깁니다.

참고로 저는 최신 버전이 아닌 8.11.0 으로 설치합니다.

elastalert plugin이 8.11.0 까지 지원하네요.

https://github.com/Karql/elastalert-kibana-plugin/releases

아래의 3개는 elasticsearch나 kibana에서 설치하면 따로 하지 않아도 됩니다.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg

sudo apt-get install apt-transport-https

echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list

여기서부터 따라하시면 됩니다.

logmgmt@syslogserver:~$ apt list | grep logstash

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

golang-github-bshuster-repo-logrus-logstash-hook-dev/jammy,jammy 0.4.1-2 all
logstash/stable 1:8.11.1-1 amd64
mcollective-plugins-logstash/jammy,jammy 0.0.0~git20120507.df2fa81-0ubuntu2 all
logmgmt@syslogserver:~$ sudo apt install logstash=1:8.11.0-1
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다... 완료
상태 정보를 읽는 중입니다... 완료
다음 새 패키지를 설치할 것입니다:
  logstash
0개 업그레이드, 1개 새로 설치, 0개 제거 및 6개 업그레이드 안 함.
350 M바이트 아카이브를 받아야 합니다.
이 작업 후 607 M바이트의 디스크 공간을 더 사용하게 됩니다.
받기:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 logstash amd64 1:8.11.0-1 [350 MB]
내려받기 350 M바이트, 소요시간 23초 (15.4 M바이트/초)
Selecting previously unselected package logstash.
(데이터베이스 읽는중 ...현재 181487개의 파일과 디렉터리가 설치되어 있습니다.)
Preparing to unpack .../logstash_1%3a8.11.0-1_amd64.deb ...
Unpacking logstash (1:8.11.0-1) ...
logstash (1:8.11.0-1) 설정하는 중입니다 ...
logmgmt@syslogserver:~$

logmgmt@syslogserver:~$ sudo systemctl daemon-reload
logmgmt@syslogserver:~$ sudo systemctl enable logstash.service
Created symlink /etc/systemd/system/multi-user.target.wants/logstash.service → /lib/systemd/system/logstash.service.
logmgmt@syslogserver:~$

CentOS 에서 ubuntu로 ELK를 재설치하여 기록을 남깁니다.

아래 사이트를 보고 따라하시면 됩니다.

https://www.elastic.co/guide/en/elasticsearch/reference/8.11/deb.html#deb-repo

참고로 저는 최신 버전이 아닌 8.11.0 으로 설치합니다.

elastalert plugin이 8.11.0 까지 지원하네요.

https://github.com/Karql/elastalert-kibana-plugin/releases

여기서부터 설치 기록 입니다.

logmgmt@syslogserver:~$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

logmgmt@syslogserver:~$ sudo apt install apt-transport-https
[sudo] password for logmgmt:
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다... 완료
상태 정보를 읽는 중입니다... 완료
다음 새 패키지를 설치할 것입니다:
  apt-transport-https
0개 업그레이드, 1개 새로 설치, 0개 제거 및 3개 업그레이드 안 함.
1,510 바이트 아카이브를 받아야 합니다.
이 작업 후 170 k바이트의 디스크 공간을 더 사용하게 됩니다.
받기:1 http://kr.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 apt-transport-https all 2.4.11 [1,510 B]
내려받기 1,510 바이트, 소요시간 1초 (1,636 바이트/초)
Selecting previously unselected package apt-transport-https.
(데이터베이스 읽는중 ...현재 180154개의 파일과 디렉터리가 설치되어 있습니다.)
Preparing to unpack .../apt-transport-https_2.4.11_all.deb ...
Unpacking apt-transport-https (2.4.11) ...
apt-transport-https (2.4.11) 설정하는 중입니다 ...
logmgmt@syslogserver:~$

logmgmt@syslogserver:~$ echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main
logmgmt@syslogserver:~$

logmgmt@syslogserver:~$ apt list | grep elasticsearch

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

elasticsearch/stable 8.11.1 amd64
logmgmt@syslogserver:~$ sudo apt update && sudo apt install elasticsearch=8.11.0
기존:1 https://artifacts.elastic.co/packages/8.x/apt stable InRelease
기존:2 http://kr.archive.ubuntu.com/ubuntu jammy InRelease
기존:3 http://security.ubuntu.com/ubuntu jammy-security InRelease
기존:4 http://kr.archive.ubuntu.com/ubuntu jammy-updates InRelease
기존:5 http://kr.archive.ubuntu.com/ubuntu jammy-backports InRelease
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다... 완료
상태 정보를 읽는 중입니다... 완료
5 packages can be upgraded. Run 'apt list --upgradable' to see them.
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다... 완료
상태 정보를 읽는 중입니다... 완료
다음 새 패키지를 설치할 것입니다:
  elasticsearch
0개 업그레이드, 1개 새로 설치, 0개 제거 및 5개 업그레이드 안 함.
630 M바이트 아카이브를 받아야 합니다.
이 작업 후 1,316 M바이트의 디스크 공간을 더 사용하게 됩니다.
받기:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 elasticsearch amd64 8.11.0 [630 MB]
내려받기 630 M바이트, 소요시간 2분 29초 (4,221 k바이트/초)
Selecting previously unselected package elasticsearch.
(데이터베이스 읽는중 ...현재 180158개의 파일과 디렉터리가 설치되어 있습니다.)
Preparing to unpack .../elasticsearch_8.11.0_amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (8.11.0) ...
elasticsearch (8.11.0) 설정하는 중입니다 ...
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : JDDAw8w6wb-PPQYIxLgV

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
logmgmt@syslogserver:~$

logmgmt@syslogserver:~$ sudo systemctl daemon-reload
logmgmt@syslogserver:~$ sudo systemctl enable elasticsearch.service
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service → /lib/systemd/system/elasticsearch.service.
logmgmt@syslogserver:~$

윈도우즈 서버 설치 후에는 반드시 해야 하는 작업이죠~

실행 창에 gpedit.msc 실행

아래 위치로 이동

• Local Computer Policy → Computer Configuration → Administrative Templates → Windows Components → Windows Update → Configure Automatic Updates
• (로컬그룹정책 → 컴퓨터 구성 → 관리 템플릿 → 윈도우즈 구성요소 → 윈도우즈 업데이트 → 자동 업데이트 구성)

• 구성되지 않음 → 사용으로 변경 (Not Configured → Enabled)
• 2 - 다운로드 및 자동 설치 알림 (2 - Notify for download and auto install