Added basic Monitoring

2025-12-06 07:48:26 +01:00 · 2025-03-07 18:28:57 +01:00
parent e782848efd
commit bf19c05148
9 changed files with 486 additions and 0 deletions
--- a/script/monitor/README.md
+++ b/script/monitor/README.md
@@ -0,0 +1,126 @@
+
+How to setup a monitoring dashboard
+===================================
+
+Inspired by: https://pimylifeup.com/ubuntu-chromium-kiosk/
+
+Steps
+-----
+
+
+
+### 1.) Install Ubuntu Server (no desktop) on your computer than set hostname and timezone.
+
+```sh
+hostnamectl set-hostname check.local
+timedatectl set-timezone Europe/Berlin
+```
+
+
+
+### 2.) Install minimal GUI and Tools.
+
+```sh
+apt install ubuntu-desktop-minimal
+apt install language-pack-gnome-de
+apt install xdotool
+apt install dbus-x11
+```
+
+
+
+### 3.) Create a kiosk user with home-directory.
+
+```sh
+useradd -m kiosk
+```
+
+and disable Welocme-Screen
+```sh
+echo "yes" > /home/kiosk/.config/gnome-initial-setup-done
+```
+
+
+
+### 4.) Edit following file `nano /etc/gdm3/custom.conf` to turn of wayland and turn on autologin for user 'kiosk'.
+
+```
+[daemon]
+# Uncomment the line below to force the login screen to use Xorg
+#WaylandEnable=false
+
+WaylandEnable=false
+
+# Enabling automatic login
+#  AutomaticLoginEnable = true
+#  AutomaticLogin = user1
+
+AutomaticLoginEnable = true
+AutomaticLogin = kiosk
+```
+
+
+
+### 5.) Configure GUI of user kiosk to prevent monitor from sleeping
+
+```sh
+#gsettings list-recursively
+
+# Does not work
+#sudo -u kiosk gsettings set org.gnome.desktop.session idle-delay 0
+
+# Set idle-delay from "uint32 300" to "uint32 0", needs 'apt install dbus-x11'
+# You can check the value in "GUI-Session of kiosk -> Settings -> Power"
+sudo -u kiosk dbus-launch dconf write /org/gnome/desktop/session/idle-delay "uint32 0"
+```
+
+
+
+### 6.) Create custom service to start firefox loading the page.
+
+Therefore create a file `/etc/systemd/system/kiosk.service` with this content:
+
+```
+[Unit]
+Description=Firefox Kiosk
+Wants=graphical.target
+After=graphical.target
+
+[Service]
+Environment=DISPLAY=:0
+# Set firefox language, needs 'apt install language-pack-gnome-de' 
+Environment=LANG=de_DE.UTF-8
+Type=simple
+# Always a fresh firefox ('-' allow error if common does not exist)
+ExecStartPre=-/usr/bin/rm -r /home/kiosk/snap/firefox/common
+# Move Mouse (should also work on small screens), needs 'apt install dbus-x11'
+ExecStartPre=/usr/bin/xdotool mousemove 4096 2160
+# See: https://wiki.mozilla.org/Firefox/CommandLineOptions (just -kiosk URL => Start-Assistant, so use -url too)
+ExecStart=/usr/bin/firefox -fullscreen -kiosk -url http://monitor.example.net/check.html
+Restart=always
+RestartSec=30
+User=kiosk
+Group=kiosk
+
+[Install]
+WantedBy=graphical.target
+```
+
+
+
+### 7.) Enable the service and reboot
+
+```sh
+systemctl enable kiosk
+reboot
+```
+
+
+
+Troubleshouting
+---------------
+
+```
+systemctl disable pd-mapper.service
+apt purge cloud-init -y && apt autoremove --purge -y
+```
--- a/script/monitor/check.html
+++ b/script/monitor/check.html
@@ -0,0 +1,204 @@
+<html>
+<head>
+<meta charset="utf-8">
+<title>Monitoring Dashboard</title>
+
+<style>
+    html, body {
+        --background-theme-color: #001EA0;
+        --cell-space: 20px;
+        --logo-height: 50px;
+
+        background-color: #cccccc;
+        font-family: Verdana;
+        font-size: 14pt;
+        color: #ffffff;
+        height: 100%;
+        margin: 0;
+    }
+    @media screen and (orientation: portrait) {
+        body {
+            zoom: 200%
+        }
+    }
+    #header {
+        background-color: var(--background-theme-color); 
+        position: sticky;
+        top: 0;
+        height: calc(var(--logo-height) + (2 * var(--cell-space)));
+        width: 100%;
+    }
+    #header img {
+        height: var(--logo-height);
+        margin: var(--cell-space);
+        vertical-align: middle;
+    }
+    #header h1 {
+        display: inline;
+        font-weight: normal;
+        vertical-align: middle;
+    }
+    #content {
+        min-height: 100%;
+    }
+    #footer {
+        background-color: var(--background-theme-color);
+        position: sticky;
+        bottom: 0px;
+        padding: var(--cell-space);
+        text-align: center;
+        vertical-align: middle;
+        font-size: 22pt;
+    }
+
+    #checks {
+        display: grid;
+        grid-template-columns: repeat(auto-fit, minmax(350px, 1fr));
+        padding: var(--cell-space);
+        grid-gap: var(--cell-space);
+    }
+    #checks > div {
+        border: 1px solid black;
+        border-radius: 10px;
+        padding: 10px;
+        text-align: center;
+
+        box-shadow: 0 2px 5px 0 rgba(0, 0, 0, 0.3), 0 2px 10px 0 rgba(0, 0, 0, 0.2);
+    }
+    #checks > div.ok {
+        background-color: #66aa22;
+        color: #222222;	
+    }
+    #checks > div.info {
+        background-color: #88cc44;
+        color: #222222;	
+    }
+    #checks > div.warn {
+        background-color: #ffdd00;
+        color: #222222;	
+    }
+    #checks > div.fail {
+        background-color: #ff0000;
+    }
+    #checks > div.timeout {
+        background-color: var(--background-theme-color);
+    }
+
+</style>
+</head>
+<body>
+    <div id="header">
+        <img src="logo.png"></img>
+        <h1>Monitoring</h1>
+    </div>
+    <div id="content">
+        <div id="checks">
+            <div class="check warn">Loading...</div>
+        </div>
+    </div>
+    <div id="footer">
+        Köln, <span id="datetime"></span>
+    </div>
+
+<script>
+    var connectionAlive = true;
+
+    function downloadCheckFile(callback) {
+        var xmlHttp = new XMLHttpRequest();
+        xmlHttp.open('GET', "check.txt", false);
+        xmlHttp.onreadystatechange=function() {
+            if(xmlHttp.readyState==4) {
+                callback(xmlHttp.responseText);
+            }
+        }
+        try {
+            xmlHttp.send(null);
+            if (xmlHttp.status >= 200 && xmlHttp.status < 304) {
+                return true;
+            } else {
+                return false;
+            }
+        } catch (e) {
+            return false;
+        }
+    }
+    function convertToHtml(checkText) {
+        if (!connectionAlive) {
+            return '<div class="fail">CONNECTION FAILED</div>';
+        }
+
+        var html = "";
+        var lines = checkText.split(/\n/);
+
+        for(var lineNo = 2; lineNo < lines.length; lineNo++) {
+            var line = lines[lineNo];
+
+            var parts = line.split('?');
+            if (parts.length > 1) {
+
+                var name = parts[0].trim().split("_").join(" ");
+                var resultParts = parts[1].trim().split('#');
+                var result = resultParts[0];
+                var message = resultParts[1];
+
+                if(name == 'MISSED') {
+                    var fileTimeParts = message.split('-');
+                    var fileTime = new Date();
+                    fileTime.setHours(parseInt(fileTimeParts[0]));
+                    fileTime.setMinutes(parseInt(fileTimeParts[1]));
+                    fileTime.setSeconds(parseInt(fileTimeParts[2]));
+
+                    var scriptTime = new Date();
+                    scriptTime.setMinutes(scriptTime.getMinutes() - 2);
+                    if (scriptTime.getTime() < fileTime.getTime()){
+                        if (result == "0") {
+                            html += '<div class="ok">EVERYTHING OK<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                        } else {
+                            html += '<div class="fail">FAILED: ' + result + '<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                        }
+                    } else {
+                        html += '<div class="check fail">CHECKS TOO OLD<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                    }
+                } else {
+                    if(result.indexOf('OK') >= 0) {
+                        html += '<div class="ok">'+ name;
+                    } else if(result.indexOf('INFO') >= 0) {
+                        html += '<div class="info">'+ name;
+                    } else if(result.indexOf('TIMEOUT') >= 0) {
+                        html += '<div class="timeout">'+ name;
+                    } else if(result.indexOf('WARN') >= 0) {
+                        html += '<div class="warn">'+ name;
+                    } else {
+                        html += '<div class="fail">' + name;
+                    }
+                    if(message) {
+                        html += '<br/>' + message.trim();
+                    }
+                    html += '</div>';
+                }
+            }
+        }
+        return html;
+    }
+    function exchangeChecks(text) {
+        document.getElementById("checks").innerHTML = convertToHtml(text);
+    }
+    function refreshTime() {
+        document.getElementById("datetime").innerHTML = new Date().toLocaleString("de-DE", {timeZone: "Europe/Berlin"});
+    }
+    function refreshChecks() {
+        connectionAlive = downloadCheckFile(exchangeChecks);
+    }
+    function reloadIfAlive() {
+        if (connectionAlive) {
+            location.reload();
+        }
+    }
+    setInterval(refreshTime, 1000);
+    setInterval(refreshChecks, 5000);
+    setInterval(reloadIfAlive, 300000);
+    refreshTime();
+    refreshChecks();
+</script>
+</body>
+</html>
--- a/script/monitor/check.sh
+++ b/script/monitor/check.sh
@@ -0,0 +1,86 @@
+#!/bin/bash
+
+
+
+function doChecks(){
+    local readonly _TMPDIR="${1:?"doChecks(): Missing parameter TMPDIR:"}"
+    local readonly _COLOR="${2:-"monocrom"}"
+
+    local _DATETIME=$(date +%H-%M-%S)
+
+    mkdir -p ${_TMPDIR}
+    rm ${_TMPDIR}/* > /dev/null 2>&1
+
+    for check in /monitoring/checks/*.on
+    do
+        local _CHECK_FILENAME="${check##*/}"
+        echo -n "${_CHECK_FILENAME%%.on}?" > "${_TMPDIR}/${_CHECK_FILENAME}"
+        timeout -k 10s 20s bash ${check} >> "${_TMPDIR}/${_CHECK_FILENAME}" 2> /dev/null || echo "TIMEOUT#Timeout" >> "${_TMPDIR}/${_CHECK_FILENAME}" &
+    done
+    wait
+
+    local _FAILED=0
+    echo "CHECK?RESULT[#MESSAGE]:"
+    echo "-----------------------"
+    for resultFile in ${_TMPDIR}/*
+    do
+        cat "${resultFile}"
+        grep -q "FAIL" ${resultFile} && _FAILED=$(expr ${_FAILED} + 1)
+    done
+
+    if [ "${_COLOR}" == "color" ]; then
+        #color is for console-output
+        echo "-----------------------"
+        if [ ${_FAILED} -ne 0 ]; then
+            echo "MISSED?${_FAILED}#${_DATETIME}"
+        else
+            echo "MISSED?${_FAILED}#${_DATETIME}"
+        fi
+    else
+        echo "MISSED?${_FAILED}#${_DATETIME}"
+    fi
+
+    rm -r ${_TMPDIR} > /dev/null 2>&1
+    return 0
+}
+
+function usage(){
+    printf "\nUsage: /monitoring/check.sh <command> <options>"
+    echo
+    echo "possible commands:"
+    echo
+    echo "- all"
+    echo "    Executes all checks."
+    echo "- auto <out_file>"
+    echo "    Executes quiet all checks and saves the result in the given out_file."
+    return 0
+}
+
+main(){
+    case "${1:-""}" in
+        all)
+            printf "Checks werden ausgeführt..." \
+                && doChecks "/tmp/checks" color \
+                && printf "Success" \
+                && return 0
+            ;;
+        auto)
+            # If just a filename is given it is created in /tmp, because of 'cd /tmp' 
+            cd /tmp \
+                && doChecks "/tmp/checks$(date +%N)" > "$2.new" \
+                && mv -f "$2.new" "$2" \
+                && return 0
+            return 1
+            ;;
+		*)
+            [ "${1:+isset}" == "isset" ] \
+                && printf "Parameter '${1}' ist kein gültiger Befehl.\n"
+            usage
+            return 0
+            ;;
+    esac
+
+    return 1
+}
+
+main "$@" || exit 1
--- a/script/monitor/checks/GENERIC_NGINX_CHECK.sh
+++ b/script/monitor/checks/GENERIC_NGINX_CHECK.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+_SERVER="${1:?"FQDN of server missing"}"
+_PORT="${2:-"22"}"
+_USER="monitoring"
+
+#grep:
+# -F                         Use fixed text, no regexp which has to be interpreted
+
+#cut:
+# -d                         Delimiter, marker where to cut (here ;)
+# -f                         Index of column to show (One based, so there is no -f0)
+_RESULT="$(ssh -p "${_PORT}" "${_USER}"@"${_SERVER}" 'systemctl status nginx.service' | grep -F Active: | grep -F running | cut -d';' -f2)"
+! [ -z "${_RESULT}" ] && echo "OK#UPTIME:${_RESULT}" || echo "FAIL"
--- a/script/monitor/checks/GENERIC_PING_CHECK.sh
+++ b/script/monitor/checks/GENERIC_PING_CHECK.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+_SERVER="${1:?"FQDN of server missing"}"
+
+# -4                         Use IPv4
+# -W            SECONDS      Wait seconds for an answer
+# -c            COUNT_VALUE  Count of pings being executed
+_RESULT="$(ping -4 -W 1 -c 1 "${_SERVER}" | grep "time=" | cut -d'=' -f4)"
+! [ -z "${_RESULT}" ] && echo "OK#RTT: ${_RESULT}" || echo "FAIL#PLEASE USE FALLBACK!"
--- a/script/monitor/checks/GENERIC_POOL_SIZE_CHECK.sh
+++ b/script/monitor/checks/GENERIC_POOL_SIZE_CHECK.sh
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+SERVER="${1:?"FQDN of server missing"}"
+FILE="pool-size.txt"
+
+# --connect-timeout SECONDS  Maximum time allowed for connection
+# -k                         Allow connections to SSL sites without certs (H)
+# -L                         Follow redirects (H)
+# --max-time        SECONDS  Maximum time allowed for the transfer
+# -s                         Silent mode. Don't output anything
+# -f                         Fail fast with no output on HTTP errors (otherwise no exit-code > 0 on 404)
+RESULT="$(curl --connect-timeout 10 --max-time 10 -k -s -f https://$SERVER/monitoring/$FILE || echo WARN#404 on $FILE check HTTPS)"
+echo $RESULT
--- a/script/monitor/checks/GENERIC_REMOTECHECK
+++ b/script/monitor/checks/GENERIC_REMOTECHECK
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+# --connect-timeout SECONDS  Maximum time allowed for connection
+# -k                         Allow connections to SSL sites without certs (H)
+# -L                         Follow redirects (H)
+# --max-time        SECONDS  Maximum time allowed for the transfer
+# -s                         Silent mode. Don't output anything
+URL="${1:?"URL missing"}"
+RESULTS="$(curl  --connect-timeout 10 --max-time 10 -k -s "$URL" 2>/dev/null)"
+CURTIME="$[ $(date +%s) - 10 * 60 ]"
+TIME="$(echo "$RESULTS" | tail -n 1)"
+if (echo $TIME | grep -E "[^0-9"] > /dev/null); then echo "FAIL"; exit; fi
+RES="$(([ "$CURTIME" -gt "$TIME" ] && echo "TIMEOUT") || (echo "$RESULTS" | head -n 1))"
+echo $RES
+echo "$RESULTS" | tail -n +2 | head -n -1
+
+
--- a/script/monitor/checks/GENERIC_URL_CHECK.sh
+++ b/script/monitor/checks/GENERIC_URL_CHECK.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+_URL="${1:?"URL of site missing"}"
+
+#curl:
+# --connect-timeout SECONDS  Maximum time allowed for connection
+# -k                         Allow connections to SSL sites without certs (H)
+# -L                         Follow redirects (H)
+# --max-time        SECONDS  Maximum time allowed for the transfer
+# -s                         Silent mode. Don't output anything
+# --head                     Show head information only
+# --no-progress-meter        Clean output for grep
+
+#grep:
+# -q                         Quite, no output just status codes
+# -F                         Interpret search term as plain text
+((curl --connect-timeout 10 --max-time 10 -k -s --head --no-progress-meter "${_URL}" | grep -qF '200 OK') && echo OK) || echo FAIL
--- a/script/monitor/logo.png
+++ b/script/monitor/logo.png