흙수저의 엔지니어링

Prometheus 설치하기

logmgmt@syslogserver:~$ sudo apt install prometheus
패키지 목록을 읽는 중입니다... 완료
의존성 트리를 만드는 중입니다... 완료
상태 정보를 읽는 중입니다... 완료
The following additional packages will be installed:
  fonts-glyphicons-halflings javascript-common libio-pty-perl libipc-run-perl libjs-bootstrap libjs-bootstrap4 libjs-d3 libjs-eonasdan-bootstrap-datetimepicker libjs-jquery libjs-jquery-hotkeys
  libjs-moment libjs-moment-timezone libjs-mustache libjs-popper.js libjs-rickshaw libjs-sizzle libtime-duration-perl moreutils node-jquery prometheus-node-exporter
  prometheus-node-exporter-collectors smartmontools
제안하는 패키지:
  apache2 | lighttpd | httpd gsmartcontrol smart-notifier mailx | mailutils
다음 새 패키지를 설치할 것입니다:
  fonts-glyphicons-halflings javascript-common libio-pty-perl libipc-run-perl libjs-bootstrap libjs-bootstrap4 libjs-d3 libjs-eonasdan-bootstrap-datetimepicker libjs-jquery libjs-jquery-hotkeys
  libjs-moment libjs-moment-timezone libjs-mustache libjs-popper.js libjs-rickshaw libjs-sizzle libtime-duration-perl moreutils node-jquery prometheus prometheus-node-exporter
  prometheus-node-exporter-collectors smartmontools
0개 업그레이드, 23개 새로 설치, 0개 제거 및 15개 업그레이드 안 함.
26.1 M바이트 아카이브를 받아야 합니다.
이 작업 후 280 M바이트의 디스크 공간을 더 사용하게 됩니다.
계속 하시겠습니까? [Y/n] y
받기:1 http://kr.archive.ubuntu.com/ubuntu jammy/main amd64 libjs-jquery all 3.6.0+dfsg+~3.5.13-1 [321 kB]
받기:2 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-jquery-hotkeys all 0~20130707+git2d51e3a9+dfsg-2ubuntu1 [11.4 kB]
받기:3 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 fonts-glyphicons-halflings all 1.009~3.4.1+dfsg-2 [117 kB]
받기:4 http://kr.archive.ubuntu.com/ubuntu jammy/main amd64 javascript-common all 11+nmu1 [5,936 B]
받기:5 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-popper.js all 1.16.1+ds-5 [53.8 kB]
받기:6 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-bootstrap4 all 4.6.0+dfsg1-4 [534 kB]
받기:7 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-bootstrap all 3.4.1+dfsg-2 [124 kB]
받기:8 http://kr.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libjs-moment all 2.29.1+ds-3ubuntu0.2 [146 kB]
받기:9 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-eonasdan-bootstrap-datetimepicker all 4.17.47-5 [28.4 kB]
받기:10 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-moment-timezone all 0.5.34+dfsg-1+2021e [759 kB]
받기:11 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-mustache all 2.3.2-2 [20.6 kB]
받기:12 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-d3 all 3.5.17-4 [132 kB]
받기:13 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-rickshaw all 1.5.1.dfsg-5 [646 kB]
받기:14 http://kr.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 prometheus amd64 2.31.2+ds1-1ubuntu1.22.04.2 [18.3 MB]
받기:15 http://kr.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 prometheus-node-exporter amd64 1.3.1-1ubuntu0.22.04.2 [3,887 kB]
받기:16 http://kr.archive.ubuntu.com/ubuntu jammy/main amd64 libio-pty-perl amd64 1:1.15-2build2 [35.1 kB]
받기:17 http://kr.archive.ubuntu.com/ubuntu jammy/main amd64 libipc-run-perl all 20200505.0-1 [89.8 kB]
받기:18 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-sizzle all 2.3.6+ds+~2.3.3-1 [32.3 kB]
받기:19 http://kr.archive.ubuntu.com/ubuntu jammy/main amd64 libtime-duration-perl all 1.21-1 [13.1 kB]
받기:20 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 moreutils amd64 0.66-1 [63.3 kB]
받기:21 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 node-jquery all 3.6.0+dfsg+~3.5.13-1 [160 kB]
받기:22 http://kr.archive.ubuntu.com/ubuntu jammy/universe amd64 prometheus-node-exporter-collectors all 0+git20211024.8eeeffb-1 [27.1 kB]
받기:23 http://kr.archive.ubuntu.com/ubuntu jammy/main amd64 smartmontools amd64 7.2-1build2 [583 kB]
내려받기 26.1 M바이트, 소요시간 11초 (2,298 k바이트/초)
Selecting previously unselected package libjs-jquery.
(데이터베이스 읽는중 ...현재 283647개의 파일과 디렉터리가 설치되어 있습니다.)
Preparing to unpack .../00-libjs-jquery_3.6.0+dfsg+~3.5.13-1_all.deb ...
Unpacking libjs-jquery (3.6.0+dfsg+~3.5.13-1) ...
Selecting previously unselected package libjs-jquery-hotkeys.
Preparing to unpack .../01-libjs-jquery-hotkeys_0~20130707+git2d51e3a9+dfsg-2ubuntu1_all.deb ...
Unpacking libjs-jquery-hotkeys (0~20130707+git2d51e3a9+dfsg-2ubuntu1) ...
Selecting previously unselected package fonts-glyphicons-halflings.
Preparing to unpack .../02-fonts-glyphicons-halflings_1.009~3.4.1+dfsg-2_all.deb ...
Unpacking fonts-glyphicons-halflings (1.009~3.4.1+dfsg-2) ...
Selecting previously unselected package javascript-common.
Preparing to unpack .../03-javascript-common_11+nmu1_all.deb ...
Unpacking javascript-common (11+nmu1) ...
Selecting previously unselected package libjs-popper.js.
Preparing to unpack .../04-libjs-popper.js_1.16.1+ds-5_all.deb ...
Unpacking libjs-popper.js (1.16.1+ds-5) ...
Selecting previously unselected package libjs-bootstrap4.
Preparing to unpack .../05-libjs-bootstrap4_4.6.0+dfsg1-4_all.deb ...
Unpacking libjs-bootstrap4 (4.6.0+dfsg1-4) ...
Selecting previously unselected package libjs-bootstrap.
Preparing to unpack .../06-libjs-bootstrap_3.4.1+dfsg-2_all.deb ...
Unpacking libjs-bootstrap (3.4.1+dfsg-2) ...
Selecting previously unselected package libjs-moment.
Preparing to unpack .../07-libjs-moment_2.29.1+ds-3ubuntu0.2_all.deb ...
Unpacking libjs-moment (2.29.1+ds-3ubuntu0.2) ...
Selecting previously unselected package libjs-eonasdan-bootstrap-datetimepicker.
Preparing to unpack .../08-libjs-eonasdan-bootstrap-datetimepicker_4.17.47-5_all.deb ...
Unpacking libjs-eonasdan-bootstrap-datetimepicker (4.17.47-5) ...
Selecting previously unselected package libjs-moment-timezone.
Preparing to unpack .../09-libjs-moment-timezone_0.5.34+dfsg-1+2021e_all.deb ...
Unpacking libjs-moment-timezone (0.5.34+dfsg-1+2021e) ...
Selecting previously unselected package libjs-mustache.
Preparing to unpack .../10-libjs-mustache_2.3.2-2_all.deb ...
Unpacking libjs-mustache (2.3.2-2) ...
Selecting previously unselected package libjs-d3.
Preparing to unpack .../11-libjs-d3_3.5.17-4_all.deb ...
Unpacking libjs-d3 (3.5.17-4) ...
Selecting previously unselected package libjs-rickshaw.
Preparing to unpack .../12-libjs-rickshaw_1.5.1.dfsg-5_all.deb ...
Unpacking libjs-rickshaw (1.5.1.dfsg-5) ...
Selecting previously unselected package prometheus.
Preparing to unpack .../13-prometheus_2.31.2+ds1-1ubuntu1.22.04.2_amd64.deb ...
Unpacking prometheus (2.31.2+ds1-1ubuntu1.22.04.2) ...
Selecting previously unselected package prometheus-node-exporter.
Preparing to unpack .../14-prometheus-node-exporter_1.3.1-1ubuntu0.22.04.2_amd64.deb ...
Unpacking prometheus-node-exporter (1.3.1-1ubuntu0.22.04.2) ...
Selecting previously unselected package libio-pty-perl.
Preparing to unpack .../15-libio-pty-perl_1%3a1.15-2build2_amd64.deb ...
Unpacking libio-pty-perl (1:1.15-2build2) ...
Selecting previously unselected package libipc-run-perl.
Preparing to unpack .../16-libipc-run-perl_20200505.0-1_all.deb ...
Unpacking libipc-run-perl (20200505.0-1) ...
Selecting previously unselected package libjs-sizzle.
Preparing to unpack .../17-libjs-sizzle_2.3.6+ds+~2.3.3-1_all.deb ...
Unpacking libjs-sizzle (2.3.6+ds+~2.3.3-1) ...
Selecting previously unselected package libtime-duration-perl.
Preparing to unpack .../18-libtime-duration-perl_1.21-1_all.deb ...
Unpacking libtime-duration-perl (1.21-1) ...
Selecting previously unselected package moreutils.
Preparing to unpack .../19-moreutils_0.66-1_amd64.deb ...
Unpacking moreutils (0.66-1) ...
Selecting previously unselected package node-jquery.
Preparing to unpack .../20-node-jquery_3.6.0+dfsg+~3.5.13-1_all.deb ...
Unpacking node-jquery (3.6.0+dfsg+~3.5.13-1) ...
Selecting previously unselected package prometheus-node-exporter-collectors.
Preparing to unpack .../21-prometheus-node-exporter-collectors_0+git20211024.8eeeffb-1_all.deb ...
Unpacking prometheus-node-exporter-collectors (0+git20211024.8eeeffb-1) ...
Selecting previously unselected package smartmontools.
Preparing to unpack .../22-smartmontools_7.2-1build2_amd64.deb ...
Unpacking smartmontools (7.2-1build2) ...
smartmontools (7.2-1build2) 설정하는 중입니다 ...
Created symlink /etc/systemd/system/smartd.service → /lib/systemd/system/smartmontools.service.
Created symlink /etc/systemd/system/multi-user.target.wants/smartmontools.service → /lib/systemd/system/smartmontools.service.
javascript-common (11+nmu1) 설정하는 중입니다 ...
libjs-popper.js (1.16.1+ds-5) 설정하는 중입니다 ...
libio-pty-perl (1:1.15-2build2) 설정하는 중입니다 ...
fonts-glyphicons-halflings (1.009~3.4.1+dfsg-2) 설정하는 중입니다 ...
libjs-moment (2.29.1+ds-3ubuntu0.2) 설정하는 중입니다 ...
libjs-sizzle (2.3.6+ds+~2.3.3-1) 설정하는 중입니다 ...
prometheus-node-exporter (1.3.1-1ubuntu0.22.04.2) 설정하는 중입니다 ...
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus-node-exporter.service → /lib/systemd/system/prometheus-node-exporter.service.
libjs-d3 (3.5.17-4) 설정하는 중입니다 ...
libjs-moment-timezone (0.5.34+dfsg-1+2021e) 설정하는 중입니다 ...
libipc-run-perl (20200505.0-1) 설정하는 중입니다 ...
libtime-duration-perl (1.21-1) 설정하는 중입니다 ...
libjs-bootstrap4 (4.6.0+dfsg1-4) 설정하는 중입니다 ...
libjs-jquery (3.6.0+dfsg+~3.5.13-1) 설정하는 중입니다 ...
libjs-mustache (2.3.2-2) 설정하는 중입니다 ...
libjs-jquery-hotkeys (0~20130707+git2d51e3a9+dfsg-2ubuntu1) 설정하는 중입니다 ...
node-jquery (3.6.0+dfsg+~3.5.13-1) 설정하는 중입니다 ...
libjs-bootstrap (3.4.1+dfsg-2) 설정하는 중입니다 ...
libjs-eonasdan-bootstrap-datetimepicker (4.17.47-5) 설정하는 중입니다 ...
moreutils (0.66-1) 설정하는 중입니다 ...
libjs-rickshaw (1.5.1.dfsg-5) 설정하는 중입니다 ...
prometheus-node-exporter-collectors (0+git20211024.8eeeffb-1) 설정하는 중입니다 ...
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-apt.timer → /lib/systemd/system/prometheus-node-exporter-apt.timer.
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-ipmitool-sensor.timer → /lib/systemd/system/prometheus-node-exporter-ipmitool-sensor.timer.
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-mellanox-hca-temp.timer → /lib/systemd/system/prometheus-node-exporter-mellanox-hca-temp.timer.
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-nvme.timer → /lib/systemd/system/prometheus-node-exporter-nvme.timer.
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-smartmon.timer → /lib/systemd/system/prometheus-node-exporter-smartmon.timer.
prometheus (2.31.2+ds1-1ubuntu1.22.04.2) 설정하는 중입니다 ...
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /lib/systemd/system/prometheus.service.
Processing triggers for fontconfig (2.13.1-4.2ubuntu5) ...
Processing triggers for man-db (2.10.2-1) ...
logmgmt@syslogserver:~$

Prometheus 웹 접속하기

http://{서버IP}:9090 

기본 설정에 필요한 부분만 수정하였습니다.

각자 설치 시 재 설정 파일과 비교하시면 좋을 것아요.

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    scrape_interval: 5m
    scrape_timeout: 1m

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: 'snmp'
    static_configs:
      - targets: ['192.168.10.2']  # SNMP device.
      - targets: ['192.168.10.3']  # SNMP device.
      - targets: ['192.168.10.4']  # SNMP device.
      - targets: ['192.168.10.5']  # SNMP device.
    metrics_path: /snmp
    params:
      module: [if_mib]
      community: [<각자 설정한 commuity  값>]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.0.17:9116  # The SNMP exporter's real hostname:port.
  - job_name: 'grafana_metrics'
    scrape_interval: 15s
    scrape_timeout: 5s

    static_configs:
      - targets: ['192.168.0.17:3000']

프로메테우스 다운로드

https://prometheus.io/download/

프로메테우스 설치

아래 매뉴얼을 따라하시면 별도로 다운로드를 하지 않으셔도 됩니다.

[root@tmplogsvr ~]# groupadd --system prometheus
[root@tmplogsvr ~]# useradd -s /sbin/nologin --system -g prometheus prometheus
[root@tmplogsvr ~]# mkdir /var/lib/prometheus
[root@tmplogsvr ~]# for i in rules rules.d files_sd; do mkdir -p /etc/prometheus/${i}; done
[root@tmplogsvr ~]# curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest   | grep browser_download_url   | grep linux-amd64   | cut -d '"' -f 4   | wget -qi -
[root@tmplogsvr ~]# tar xvf prometheus-2.42.0.linux-amd64.tar.gz
prometheus-2.42.0.linux-amd64/
prometheus-2.42.0.linux-amd64/NOTICE
prometheus-2.42.0.linux-amd64/consoles/
prometheus-2.42.0.linux-amd64/consoles/index.html.example
prometheus-2.42.0.linux-amd64/consoles/node.html
prometheus-2.42.0.linux-amd64/consoles/prometheus-overview.html
prometheus-2.42.0.linux-amd64/consoles/node-disk.html
prometheus-2.42.0.linux-amd64/consoles/prometheus.html
prometheus-2.42.0.linux-amd64/consoles/node-overview.html
prometheus-2.42.0.linux-amd64/consoles/node-cpu.html
prometheus-2.42.0.linux-amd64/console_libraries/
prometheus-2.42.0.linux-amd64/console_libraries/menu.lib
prometheus-2.42.0.linux-amd64/console_libraries/prom.lib
prometheus-2.42.0.linux-amd64/prometheus.yml
prometheus-2.42.0.linux-amd64/LICENSE
prometheus-2.42.0.linux-amd64/promtool
prometheus-2.42.0.linux-amd64/prometheus
[root@tmplogsvr ~]# cd prometheus-2.42.0.linux-amd64/
[root@tmplogsvr prometheus-2.42.0.linux-amd64]# cp ./prometheus promtool /usr/local/bin
[root@tmplogsvr prometheus-2.42.0.linux-amd64]# cp -r prometheus.yml consoles/ console_libraries/ /etc/prometheus/
[root@tmplogsvr prometheus-2.42.0.linux-amd64]# cd
[root@tmplogsvr ~]# echo '
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target' > /etc/systemd/system/prometheus.service
[root@tmplogsvr ~]# chown -R prometheus:prometheus /etc/prometheus
[root@tmplogsvr ~]# chmod -R 775 /etc/prometheus/
[root@tmplogsvr ~]# chown -R prometheus:prometheus /var/lib/prometheus/
[root@tmplogsvr ~]# systemctl daemon-reload
[root@tmplogsvr ~]# systemctl enable prometheus
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /etc/systemd/system/prometheus.service.
[root@tmplogsvr ~]#

프로메테우스 실행

[root@tmplogsvr ~]# systemctl start prometheus.service
[root@tmplogsvr ~]# systemctl status prometheus.service
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-03-22 10:04:33 KST; 4s ago
     Docs: https://prometheus.io/docs/introduction/overview/
 Main PID: 233870 (prometheus)
    Tasks: 13 (limit: 99645)
   Memory: 20.5M
   CGroup: /system.slice/prometheus.service
           └─233870 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.lib>

 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.564Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3
 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.565Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3
 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.565Z caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3
 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.565Z caller=head.go:722 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.402µs wal_replay_dur>
 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.566Z caller=main.go:1014 level=info fs_type=EXT4_SUPER_MAGIC
 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.566Z caller=main.go:1017 level=info msg="TSDB started"
 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.566Z caller=main.go:1197 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.568Z caller=main.go:1234 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml total>
 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.568Z caller=main.go:978 level=info msg="Server is ready to receive web requests."
 3월 22 10:04:33 tmplogsvr prometheus[233870]: ts=2023-03-22T01:04:33.568Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..."
lines 1-20/20 (END)

웹페이지 접근

http://{서버IP}:9090