checkOrStartSSHMaster() improved

.gitignore

@@ -0,0 +1,10 @@
+# Ignore the file '/overrideOwnDomain' because this is per host individually.
+/overrideOwnDomain
+
+# Ignore the subfolders only, because their content are other git repositories.
+# But 'definitions and 'states' should be prepared by cloning this repository.
+/definitions/*/
+/states/*/
+
+# Ignore environment files
+.env

README.md

@@ -1,35 +1,109 @@
-Infrastructure System (ISS)
+Core Infrastructure System (CIS)
-===========================
+================================
 
-Setup a new host
+The main idea is to use git to keep scripts, definitions and state in sync across all hosts.  
-----------------
+Currently an operating instance uses one repository for this core functionality and scripts,  
+another to distibute the definitions and a third one to share the state.
 
-### Preconditions
+If a script or a definition has to be changed an independent working copy is needed to push the adaptions.  
-To deploy the system you have to clone this repository to the host as root user.
+States can be changed by a host itself. Then we need a mechanism that informs all hosts to execute a `git pull`.
-Therefore you have to register the SSH public key of that root user as deploy key to allow readonly access to this repository.
+
-We use the modern ed25519 keys, so the public key of root is stored at this location:
+We use a Git server as syncronisation point and use a web hook to send the notification.  
+Because the should not be an agent to be installed on each host, we use jenkins to execute an update script via ssh.
+
+This allows us to use standard software without having to program something that may contain a security problem.
+
+
+
+Setup the first or a new host
+-----------------------------
+
+1. Update the host and ensure git is installed
+2. Set the long hostname (fqdn)
+3. Create ssh keys for user root (ssh key type ed25519)
+
+You can use this script to do so: [prepareThisHostBeforeCloning.sh](./prepareThisHostBeforeCloning.sh)
+
+
+
+### Ensure the existence of the repositories for your definitions and the state
+
+This should be necessary just if you set up the first host.  
+You can use the following scripts to assist the process:
+
+- [prepareDefinitionsRepository.sh](./prepareDefinitionsRepository.sh)
+- [prepareStatesRepository.sh](./prepareStatesRepository.sh)
+
+
+
+### Register the public ssh key of user root
+
+This is an example for `example.net` as domain of the host.
+
+1. __Scripts:__  
+   The public ssh key of the root user must be registered as a deploy key for the this repository,  
+   which grants __readonly access__.
+
+   A root user of a host should only be able to update the local cloned repository (`cis`) to a new version via `git pull`.
+
+2. __Definitions:__  
+   The public ssh key of the root user must be registered as a deploy key for the definitions repository,  
+   which grants __readonly access__.  
+
+   User root should only be able to update the local cloned repository (`cis-definition-example.net`) to a new version via `git pull`.
+
+3. __States:__  
+   The public ssh key of the root user must be registered as a deploy key for the states repository,  
+   which grants __write access__.  
+
+   User root should be able to push new state to the cloned repository (`cis-state-example.net`) via `git push`.
+
+
+
+### Clone the Infrastructure System (cis) repository and complete the setup
+After you registered the printed root's public key of this host you can clone the repository and execute the setup script:
+```sh
+# Note the tailing '/cis', because we want to clone the repository to that folder
+git clone ssh://git@git.example.dev:22448/cis.git /cis
+
+# Execute the setup script
+/cis/setupCoreOntoThisHost.sh
+```
+
+<br>
+<br>
+<br>
+
+
+
+Setup a new host step by step manually
+--------------------------------------
+
+To deploy cis you have to clone this repository to the host as root user.
+Therefore you have to set the correct long hostname (fqdn) create a pair of ssh keys (key type ed25519) for user root  
+and register the SSH public key of root as __deploy key__ to allow readonly access to this repository:
 
 1. First become root:
    ```sh
    sudo -i
    ```
 
-2. Set the long hostname:
+2. Update Ubuntu:
-   ```sh
-   hostnamectl set-hostname "the-new-unique-long-hostname (fqdn, eg.: host1.example.net)"
-   ```
-
-3. Update Ubuntu:
    ```sh
    # DO NOT SKIP THIS STEP
    apt update; apt upgrade -y
    ```
 
-4. Install git if needed:
+3. Install git if needed:
    ```sh
    git --version > /dev/null || apt install git
    ```
 
+4. Set the long hostname:
+   ```sh
+   hostnamectl set-hostname "the-new-unique-long-hostname (fqdn, eg.: host1.example.net)"
+   ```
+
 5. If not exist generate the ssh key pair and print the public key of the user root:
    ```sh
    # -t    type of the key pair
@@ -41,40 +115,17 @@ We use the modern ed25519 keys, so the public key of root is stored at this loca
        || (ssh-keygen \
            -t ed25519 \
            -f "/root/.ssh/id_ed25519" -q -N "" \
-           -C "$(date +%Y%m%d):root@$(hostname -b)" \
+           -C "$(date +%Y%m%d)-root@$(hostname -b)" \
        && cat "/root/.ssh/id_ed25519.pub")
    ```
 
-   This key has to be registerd via gitea web ui as deploy key into the repositories as documented in chapter "Register public host key".
+   This key has to be registerd via gitea web ui as deploy key into this repository.
 
 
 
-### Register public host key
-This is an example for `example.net` as domain of the host owner.
-
-1. Repository `iss`, allow __readonly__ access only.
-2. Repository `iss-definition-example.net`, allow __readonly__ access only.
-3. Repository `iss-state-example.net`, allow __writable__ access.
-
-
-
-### Clone the Infrastructure System (iss) repository
-After you registered the printed root's public key of this host you can clone the repository and execute the setup script:
-```sh
-# Note the tailing '/iss', because we want to clone the repository to that folder
-git clone ssh://git@git.example.dev:22448/iss.git /iss
-
-# Execute the setup script
-/iss/setupCoreOntoThisHost.sh
-```
-
-<br>
-<br>
-<br>
-
 How it works
 ------------
-We add a webhook to each gitea repository that belongs to ISS:
+We add a webhook to each gitea repository that belongs to CIS:
  - __Taget URL:__   https://YOUR.JENKINS.DOMAIN/generic-webhook-trigger/invoke?token=YOUR_TOKEN
  - __HTTP-Method:__ POST
  - __Trigger On:__  Push Events
@@ -94,11 +145,11 @@ cat "${JENKINS_HOME}/.ssh/id_ed25519.pub" \
     || (ssh-keygen \
         -t ed25519 \
         -f "${JENKINS_HOME}/.ssh/id_ed25519" -q -N "" \
-        -C "$(date +%Y%m%d):$(whoami)@$(echo ${JENKINS_URL} | cut -d/ -f3)" \
+        -C "$(date +%Y%m%d)-$(whoami)@$(echo ${JENKINS_URL} | cut -d/ -f3)" \
     && cat "${JENKINS_HOME}/.ssh/id_ed25519.pub")
 
 # add your host here, note the tailing '&' to run it in parallel
-ssh -o StrictHostKeyChecking=no jenkins@192.168.X.Y /iss/update_repositories.sh ( --scripts | --definitions | --states ) &
+ssh -o StrictHostKeyChecking=no jenkins@192.168.X.Y /cis/updateRepositories.sh ( --scripts | --definitions | --states ) &
 
 #wait for all background processes to complete
 wait

core/addAndCheckGitRepository.sh

@@ -6,10 +6,10 @@
 
 
 function checkPermissions(){
-    local _FOLDER _REPOSITORY
+    local _FOLDER _RIGHTS
     _FOLDER="${1:?"Missing first parameter FOLDER"}"
     _RIGHTS="${2:?"Missing second parameter RIGHTS"}"
-    readonly _FOLDER _REPOSITORY
+    readonly _FOLDER _RIGHTS
 
     [ "${_RIGHTS}" == "readonly" ] \
         && [ -d "${_FOLDER}/.git" ] \
@@ -21,30 +21,9 @@ function checkPermissions(){
         && git -C "${_FOLDER}" push --dry-run &> /dev/null \
         && return 0
 
-    echo "FAIL: The rights of the repository are incorrect:  ("$(readlink -f ${0})")"
+    echo "FAIL: The rights of the repository are incorrect:  ("$(readlink -f ${0})")" >&2
-    echo "  - '${_FOLDER}' is not '${_RIGHTS}'"
+    echo "  - '${_FOLDER}' is not '${_RIGHTS}'" >&2
-    echo "  - check the settings of gitea."
+    echo "  - check the settings of gitea." >&2
-    return 1
-}
-
-function checkRemoteRepository() {
-    local _FOLDER _REPOSITORY
-    _FOLDER="${1:?"Missing first parameter FOLDER"}"
-    _REPOSITORY="${2:?"Missing second parameter REPOSITORY"}"
-    readonly _FOLDER _REPOSITORY
-
-    #Should exist after successful clone only, therefore the remote repository exists and was accessible.
-    [ -d "${_FOLDER}/.git" ] \
-        && return 0
-
-    #Checks if repository exists and is accessible.
-    ! [ -d "${_FOLDER}/.git" ] \
-        && git ls-remote "${_REPOSITORY}" \
-        && return 0
-
-    echo "FAIL: The remote repository is not accessible:     ("$(readlink -f ${0})")"
-    echo "  - '${_REPOSITORY}'"
-    echo "  - check the settings of gitea."
     return 1
 }
 
@@ -54,42 +33,67 @@ function cloneOrPull {
     _REPOSITORY="${2:?"Missing second parameter REPOSITORY"}"
     readonly _FOLDER _REPOSITORY
 
-    ! [ -d "${_FOLDER}/.git" ] \
-        && git clone "${_REPOSITORY}" "${_FOLDER}" &> /dev/null \
-        && return 0
-
     [ -d "${_FOLDER}/.git" ] \
         && git -C "${_FOLDER}" pull &> /dev/null \
         && return 0
 
-    echo "FAIL: The local repository is not updatable:       ("$(readlink -f ${0})")"
+    ! [ -d "${_FOLDER}/.git" ] \
-    echo "  - '${_FOLDER}'"
+        && git clone "${_REPOSITORY}" "${_FOLDER}" &> /dev/null \
-    echo "  - check your network and the permissions in gitea."
+        && return 0
+
+    echo "FAIL: The local repository is not updatable:       ("$(readlink -f ${0})")" >&2
+    echo "  - '${_FOLDER}'" >&2
+    echo "  - check your network and the permissions in gitea." >&2
+    return 1
+}
+
+function printRepository(){
+    local _FOLDER _CONFIGURED_REPOSITORY _SUGGESTED_REPOSITORY
+    _FOLDER="${1:?"Missing first parameter FOLDER"}"
+    _CONFIGURED_REPOSITORY="$(git -C "${_FOLDER:?"Missing FOLDER"}" config --get remote.origin.url 2> /dev/null)"
+    _SUGGESTED_REPOSITORY="${2}"
+    readonly _FOLDER _CONFIGURED_REPOSITORY _SUGGESTED_REPOSITORY
+
+    ! [ -z "${_CONFIGURED_REPOSITORY}" ] \
+        && echo "${_CONFIGURED_REPOSITORY}" \
+        && return 0
+
+    while true; do
+        read -e -p "Enter ssh URL to clone Repository: " -i "${_SUGGESTED_REPOSITORY}" _REPOSITORY
+        echo "${_REPOSITORY}" | grep -F 'git@' &> /dev/null \
+            && git ls-remote "${_REPOSITORY}" &> /dev/null \
+            && echo "${_REPOSITORY:?"Missing REPOSITORY: e.g. ssh://git@your.domain.com/cis.git"}" \
+            && return 0
+    done
+
+    echo "FAIL: The remote repository is not accessible:     ("$(readlink -f ${0})")" >&2
+    echo "  - '${_REPOSITORY}'" >&2
+    echo "  - check the settings of gitea." >&2
     return 1
 }
 
 # Note that an unprivileged user can use this script successfully,
 # if no user has to be added to the host because it already exists.
 function addAndCheckGitRepository() {
-    local _FOLDER _REPOSITORY
+    local _FOLDER _REPOSITORY _RIGHTS
     _FOLDER="${1:?"Missing first parameter FOLDER"}"
-    _REPOSITORY="${2:?"Missing second parameter REPOSITORY: e.g. ssh://git@your.domain.com/iss.git "}"
+    _RIGHTS="${2:?"Missing second parameter RIGHTS: (readonly, writable) "}"
-    _RIGHTS="${3:?"Missing third parameter RIGHTS: (readonly, writable) "}"
+    _REPOSITORY="$(printRepository "${_FOLDER}" "${3}")"
-    readonly _FOLDER _REPOSITORY
+    readonly _FOLDER _REPOSITORY _RIGHTS
 
-    checkRemoteRepository "${_FOLDER}" "${_REPOSITORY}" \
+    echo \
-        && cloneOrPull "${_FOLDER}" "${_REPOSITORY}" \
+        && cloneOrPull "${_FOLDER}" "${_REPOSITORY:?"Missing REPOSITORY: e.g. ssh://git@your.domain.com/cis.git"}" \
         && checkPermissions "${_FOLDER}" "${_RIGHTS}" \
         && echo "SUCCESS: The git repository is usable.             ("$(readlink -f ${0})")" \
         && echo "  - remote repository: '${_REPOSITORY}'" \
         && echo "  - local repository:  '${_FOLDER}' (${_RIGHTS})" \
         && return 0
 
-    echo "FAIL: The repository is not functional:            ("$(readlink -f ${0})")"
+    echo "FAIL: The repository is not functional:            ("$(readlink -f ${0})")" >&2
-    echo "  - remote repository: '${_REPOSITORY}'"
+    echo "  - remote repository: '${_REPOSITORY}'" >&2
-    echo "  - local repository:  '${_FOLDER}'"
+    echo "  - local repository:  '${_FOLDER}'" >&2
-    echo "  - due to an error or insufficient rights or"
+    echo "  - due to an error or insufficient rights or" >&2
-    echo "  - one check failed."
+    echo "  - one check failed." >&2
     return 1
 }
 
@@ -98,4 +102,6 @@ addAndCheckGitRepository \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${3} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/addNormalUser.sh

@@ -18,6 +18,8 @@ function addNormalUser() {
         && echo "  - '${_USER}'" \
         && return 0
 
+    # useradd is a low level utility ... use adduser(8) instead.
+    # See: https://askubuntu.com/questions/345974/what-is-the-difference-between-adduser-and-useradd
     [ "$(id -u)" == "0" ] \
         && adduser --gecos 'Normal user' --disabled-password "${_USER}" \
         && chown -R "${_USER}:${_USER}" "/home/${_USER}" \
@@ -27,13 +29,14 @@ function addNormalUser() {
         && echo "  - existing home directories were taken over" \
         && return 0
 
-    echo "FAIL: The user could not be created:               ("$(readlink -f ${0})")"
+    echo "FAIL: The user could not be created:               ("$(readlink -f ${0})")" >&2
-    echo "  - '${_USER}'"
+    echo "  - '${_USER}'" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
 # sanitizes all parameters
-addNormalUser \
+addNormalUser "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
+    && exit 0
-    && exit 0 || exit 1
+
+exit 1

core/addToCrontabEveryHour.sh

@@ -5,14 +5,17 @@
 
 
 
+# Folders always ends with an tailing '/'
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/core/*}/"               #Removes longest  matching pattern '/core/*' from the end
+
 # Note that an unprivileged user can use this script successfully,
 # if no user has to be added to the host because it already exists.
 function addToCrontabEveryHour() {
-    local _ROOT _MINUTE_VALUE _STRING
+    local _MINUTE_VALUE _STRING
-    _ROOT="${0%%/core/*}/"                              #Removes longest  matching pattern '/core/*' from the end
     ! [ -z "${2##*[!0-9]*}" ] && _MINUTE_VALUE=$((${2}%60)) # if second parameter is integer then (minute-value % 60) as safe guard
     _STRING="${_MINUTE_VALUE:?"Missing MINUTE_VALUE"} * * * * ${1:?"Missing first parameter COMMAND"} > /dev/null 2>&1"
-    readonly _ROOT _MINUTE_VALUE _STRING
+    readonly _MINUTE_VALUE _STRING
 
     [ "$(id -u)" == "0" ] \
         && crontab -l | grep -qF "${_STRING:?"Missing CRON_STRING"}" \
@@ -21,11 +24,11 @@ function addToCrontabEveryHour() {
         && return 0
 
     [ "$(id -u)" == "0" ] \
-        && echo "${_ROOT:?"Missing ROOT"}" | grep "home" &> /dev/null \
+        && echo "${_CIS_ROOT:?"Missing CIS_ROOT"}" | grep -F 'home' &> /dev/null \
         && echo "SUCCESS: Although the entry will be skipped:       ("$(readlink -f ${0})")" \
         && echo "  - '${_STRING}'" \
         && echo "  that is because the current environment is:" \
-        && echo "    - ${_ROOT}" \
+        && echo "    - ${_CIS_ROOT}" \
         && return 0
 
     [ "$(id -u)" == "0" ] \
@@ -37,9 +40,9 @@ function addToCrontabEveryHour() {
         && echo "  - '${_STRING}'" \
         && return 0
 
-    echo "FAIL: Entry could not be registered to crontab:    ("$(readlink -f ${0})")"
+    echo "FAIL: Entry could not be registered to crontab:    ("$(readlink -f ${0})")" >&2
-    echo "  - '${_STRING:?"Missing CRON_STRING"}'"
+    echo "  - '${_STRING:?"Missing CRON_STRING"}'" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
@@ -47,4 +50,6 @@ function addToCrontabEveryHour() {
 addToCrontabEveryHour \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/default/etc/sudoers.d/allow-jenkins-updateRepositories

@@ -0,0 +1,6 @@
+Cmnd_Alias C_JENKINS = \
+  /cis/updateRepositories.sh --core, \
+  /cis/updateRepositories.sh --scripts, \
+  /cis/updateRepositories.sh --definitions, \
+  /cis/updateRepositories.sh --states
+jenkins ALL = (root) NOPASSWD: C_JENKINS

core/defineAuthorizedKeysOfUser.sh

@@ -44,24 +44,24 @@ function prepareFolder() {
         && echo "  - '${_SSH_FOLDER}'" \
         && return 0
 
-    echo "FAIL: The ssh folder could not be prepared:        ("$(readlink -f ${0})")"
+    echo "FAIL: The ssh folder could not be prepared:        ("$(readlink -f ${0})")" >&2
-    echo "  - '${_SSH_FOLDER}'"
+    echo "  - '${_SSH_FOLDER}'" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
 function defineAuthorizedKeysOfUser() {
-    local _ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
+    local _CIS_ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
     _DEFINITIONS="$(realpath -s "${1:?"Missing first parameter DEFINITIONS: 'ROOT/definitions/DOMAIN'"}")"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    _CIS_ROOT="${_DEFINITIONS%%/definitions/*}/"  #Removes longest  matching pattern '/definitions/*' from the end
     _DOMAIN="${_DEFINITIONS##*/definitions/}"     #Removes longest  matching pattern '*/definitions/' from the begin
     _DOMAIN="${_DOMAIN%/}"                        #Removes shortest matching pattern '/'              from the end
     #Build from components for safety
-    _DEFINITIONS="${_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
+    _DEFINITIONS="${_CIS_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
 
     _USER="${2:?"Missing second parameter USER"}"
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
+    _CORE_SCRIPTS="${_CIS_ROOT:?"Missing ROOT"}core/"
-    readonly _ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
+    readonly _CIS_ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
 
     case "${_USER:?"Missing USER"}" in
         root)
@@ -83,4 +83,6 @@ function defineAuthorizedKeysOfUser() {
 defineAuthorizedKeysOfUser \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/ensureUsageOfDefinitions.sh

@@ -12,46 +12,76 @@ function printIfEqual() {
 }
 
 function isCoreDefinition() {
-    echo "${1:?"Missing first parameter FILE"}" | grep "/root/.ssh/authorized_keys" &> /dev/null \
+    echo "${1:?"Missing first parameter FILE"}" | grep -F '/root/.ssh/authorized_keys' &> /dev/null \
         && return 0
 
-    echo "${1:?"Missing first parameter FILE"}" | grep "/home/jenkins/.ssh/authorized_keys" &> /dev/null \
+    echo "${1:?"Missing first parameter FILE"}" | grep -F '/home/jenkins/.ssh/authorized_keys' &> /dev/null \
         && return 0
 
-    echo "${1:?"Missing first parameter FILE"}" | grep "/etc/sudoers.d/allow-jenkins-updateRepositories" &> /dev/null \
+    echo "${1:?"Missing first parameter FILE"}" | grep -F '/etc/sudoers.d/allow-jenkins-updateRepositories' &> /dev/null \
         && return 0
 
     return 1
 }
 
+function filterInvalidAuthorizedKeysFilesOfRoot() {
+    local _FILE_DEFINED
+    _FILE_DEFINED="${1:?"Missing DEFINITION FILE"}"
+    readonly _FILE_DEFINED
+
+    #If the full filename contains 'root/.ssh/authorized_keys' then check the content.
+    #Skip lines starting with '#' and if at least one remaining line contains 'ssh' and '@' then print the filename. 
+    echo "${_FILE_DEFINED}" | grep -F 'root/.ssh/authorized_keys' &> /dev/null \
+        && grep -vE '^[[:blank:]]*#' "${_FILE_DEFINED}" | grep -F 'ssh' | grep -F '@' &> /dev/null \
+        && echo "${_FILE_DEFINED}" \
+        && return 0
+
+    #If the full filename contains 'root/.ssh/authorized_keys' print nothing because the file has to be invalid.
+    echo "${_FILE_DEFINED}" | grep -F 'root/.ssh/authorized_keys' &> /dev/null \
+        && echo \
+        && return 0
+
+    #Print the full filename because it does not contain 'root/.ssh/authorized_keys'
+    echo "${_FILE_DEFINED}"
+    return 0
+}
+
 function printSelectedDefinition() {
-    local _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
+    local _DEFINITIONS  _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
-    _CORE_FILE_DEFINED_ALL_HOSTS="${1:?"Missing DEFINITIONS"}/core/all${2:?"Missing CURRENT_FULLFILE"}"
+    _DEFINITIONS="${1:?"Missing CIS_ROOT"}definitions/${2:?"Missing DOMAIN"}/"
-    _CORE_FILE_DEFINED_THIS_HOST="${1:?"Missing DEFINITIONS"}/core/$(hostname -s)${2:?"Missing CURRENT_FULLFILE"}"
+    _CORE_DEFAULT_ALL_HOSTS="${1:?"Missing CIS_ROOT"}core/default${3:?"Missing CURRENT_FULLFILE"}"
-    _FILE_DEFINED_ALL_HOSTS="${1:?"Missing DEFINITIONS"}/hosts/all${2:?"Missing CURRENT_FULLFILE"}"
+    _CORE_FILE_DEFINED_ALL_HOSTS="${_DEFINITIONS:?"Missing DEFINITIONS"}core/all${3:?"Missing CURRENT_FULLFILE"}"
-    _FILE_DEFINED_THIS_HOST="${1:?"Missing DEFINITIONS"}/hosts/$(hostname -s)${2:?"Missing CURRENT_FULLFILE"}"
+    _CORE_FILE_DEFINED_THIS_HOST="${_DEFINITIONS:?"Missing DEFINITIONS"}core/$(hostname -s)${3:?"Missing CURRENT_FULLFILE"}"
-    readonly _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
+    _FILE_DEFINED_ALL_HOSTS="${_DEFINITIONS:?"Missing DEFINITIONS"}hosts/all${3:?"Missing CURRENT_FULLFILE"}"
+    _FILE_DEFINED_THIS_HOST="${_DEFINITIONS:?"Missing DEFINITIONS"}hosts/$(hostname -s)${3:?"Missing CURRENT_FULLFILE"}"
+    readonly _DEFINITIONS _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
 
     #The following are special definitions that affect the core functionality.
     #Try this host first because it should be priorized.
-    isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_CORE_FILE_DEFINED_THIS_HOST}" ] \
-        && echo "${_CORE_FILE_DEFINED_THIS_HOST}" \
+        && filterInvalidAuthorizedKeysFilesOfRoot "${_CORE_FILE_DEFINED_THIS_HOST}" \
         && return 0
 
     #The following are special definitions that affect the core functionality.
-    isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_CORE_FILE_DEFINED_ALL_HOSTS}" ] \
-        && echo "${_CORE_FILE_DEFINED_ALL_HOSTS}" \
+        && filterInvalidAuthorizedKeysFilesOfRoot "${_CORE_FILE_DEFINED_ALL_HOSTS}" \
+        && return 0
+
+    #The following are special definitions that affect the core functionality.
+    isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
+        && [ -s "${_CORE_DEFAULT_ALL_HOSTS}" ] \
+        && filterInvalidAuthorizedKeysFilesOfRoot "${_CORE_DEFAULT_ALL_HOSTS}" \
         && return 0
 
     #Try this host first because it should be priorized.
-    ! isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    ! isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_FILE_DEFINED_THIS_HOST}" ] \
         && echo "${_FILE_DEFINED_THIS_HOST}" \
         && return 0
 
-    ! isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    ! isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_FILE_DEFINED_ALL_HOSTS}" ] \
         && echo "${_FILE_DEFINED_ALL_HOSTS}" \
         && return 0
@@ -71,11 +101,6 @@ function createSymlinkToDefinition() {
         && [ "$(sha256sum "${_DEFINED_FULLFILE}" | cut -d' ' -f1)" == "$(sha256sum "${_CURRENT_FULLFILE}" | cut -d' ' -f1)" ] \
         && echo "The content of the current file already matches the definition, but it will be replaced by a symlink..."
 
-    [ -f "${_CURRENT_FULLFILE}" ] \
-        && [ "$(sha256sum "${_DEFINED_FULLFILE}" | cut -d' ' -f1)" == "$(sha256sum "${_CURRENT_FULLFILE}" | cut -d' ' -f1)" ] \
-        && echo "The content of the current file already matches the definition, but it will be replaced by a symlink..."
-
-
     [ -f "${_CURRENT_FULLFILE}" ] \
         && mv "${_CURRENT_FULLFILE:?"Missing CURRENT_FULLFILE"}" "${_SAVED_FULLFILE:?"Missing SAVED_FULLFILE"}" \
         && echo "Current file has been backed up to: '${_SAVED_FULLFILE}'"
@@ -92,17 +117,17 @@ function createSymlinkToDefinition() {
 }
 
 function ensureUsageOfDefinitions() {
-    local _ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
+    local _CIS_ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
-    _DEFINITIONS="$(realpath -s "${1:?"Missing first parameter DEFINITIONS: 'ROOT/definitions/DOMAIN'"}")"
+    _DEFINITIONS="$(realpath -s "${1:?"Missing first parameter DEFINITIONS: 'ROOT/definitions/DOMAIN'"}")/"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    _CIS_ROOT="${_DEFINITIONS%%/definitions/*}/"  #Removes longest  matching pattern '/definitions/*' from the end
     _DOMAIN="${_DEFINITIONS##*/definitions/}"     #Removes longest  matching pattern '*/definitions/' from the begin
     _DOMAIN="${_DOMAIN%/}"                        #Removes shortest matching pattern '/'              from the end
     #Build from components for safety
-    _DEFINITIONS="$(printIfEqual "${_DEFINITIONS}" "${_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}")"
+    _DEFINITIONS="$(printIfEqual "${_DEFINITIONS}" "${_CIS_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/")"
 
 
-    _CURRENT_FOLDER="$(dirname "${2:?"Missing second parameter CURRENT_FULLFILE"}")"
+    _CURRENT_FULLFILE="${2:?"Missing second parameter CURRENT_FULLFILE"}"
-    _CURRENT_FOLDER="${_CURRENT_FOLDER%/}/"  #Removes shortest matching pattern '/' from the end
+    _CURRENT_FOLDER="${_CURRENT_FULLFILE%/*}/"  #Removes shortest matching pattern '/*' from the end
     ! [ -d "${_CURRENT_FOLDER}" ] \
         && echo "FAIL: The folder cannot be read:                   ("$(readlink -f ${0})")" \
         && echo "  - '${_CURRENT_FOLDER}'" \
@@ -118,10 +143,16 @@ function ensureUsageOfDefinitions() {
     _CURRENT_FULLFILE="${_CURRENT_FOLDER:?"Missing CURRENT_FOLDER"}${_CURRENT_FILE:?"Missing CURRENT_FILE"}"
 
 
-    _DEFINED_FULLFILE="$(printSelectedDefinition "${_DEFINITIONS}" "${_CURRENT_FULLFILE}")"
+    _DEFINED_FULLFILE="$(printSelectedDefinition "${_CIS_ROOT}" "${_DOMAIN}" "${_CURRENT_FULLFILE}")"
     _NOW="$(date +%Y%m%d_%H%M)"
-    _SAVED_FULLFILE="${_CURRENT_FULLFILE}-backup@${_NOW:?"Missing NOW"}"
+    _SAVED_FULLFILE="${_CURRENT_FULLFILE}.backup@${_NOW:?"Missing NOW"}"
-    readonly _ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
+    readonly _CIS_ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
+
+    [ -z "${_DEFINED_FULLFILE}" ] \
+        && echo \
+        && echo "URGENT WARNING: If an 'authorized_keys' file of root is replaced by an invalid version," \
+        && echo "                you may lose access to this host!" \
+        && echo
 
     ! [ -f "${_DEFINED_FULLFILE}" ] \
         && echo "FAIL: No definition available for this file:       ("$(readlink -f ${0})")" \
@@ -138,11 +169,11 @@ function ensureUsageOfDefinitions() {
         && echo "  - '${_DEFINED_FULLFILE}'" \
         && return 0
 
-    echo "${_ROOT:?"Missing ROOT"}" | grep "home" &> /dev/null \
+    echo "${_CIS_ROOT:?"Missing CIS_ROOT"}" | grep -F 'home' &> /dev/null \
         && echo "SUCCESS: Although this definition will be skipped: ("$(readlink -f ${0})")" \
         && echo "  - '${_DEFINED_FULLFILE}'" \
         && echo "  that is because the current environment is:" \
-        && echo "    - ${_ROOT}" \
+        && echo "    - ${_CIS_ROOT}" \
         && echo "  following file is in use:" \
         && echo "    - $(readlink -f "${_CURRENT_FULLFILE}")" \
         && return 0
@@ -165,8 +196,8 @@ function ensureUsageOfDefinitions() {
         && echo "- '${_DEFINED_FULLFILE}'" \
         && return 0
 
-    echo "FAIL: The definition could not be ensured:         ("$(readlink -f ${0})")"
+    echo "FAIL: The definition could not be ensured:         ("$(readlink -f ${0})")" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
@@ -174,4 +205,6 @@ function ensureUsageOfDefinitions() {
 ensureUsageOfDefinitions \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/printCisRoot.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/core/*}/"               #Removes longest  matching pattern '/core/*' from the end
+
+[ -d "${_CIS_ROOT}" ] \
+    && [ -d "${_CIS_ROOT}definitions/" ] \
+    && [ -d "${_CIS_ROOT}states/" ] \
+    && echo "${_CIS_ROOT}" \
+    && exit 0
+
+echo "FAIL: Unable to detect CIS_ROOT" >&2
+exit 1

core/printOwnDomain.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+
+#WARNING: Used for core functionality in setup.sh
+#         DO NOT rename the script and test changes well!
+
+
+
+# Folders always ends with an tailing '/'
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/core/*}/"               #Removes longest  matching pattern '/core/*' from the end
+_OVERRIDE_DOMAIN_FILE="${_CIS_ROOT:?"Missing CIS_ROOT"}overrideOwnDomain"
+
+# There has to be one dot at least.
+_BOOT_DOMAIN="$(hostname -b | grep -F '.' | cut -d. -f2-)"
+
+# Take OVERRIDING_DOMAIN_FILE without empty lines and comments, then take the first line without leading spaces
+_OVERRIDE_DOMAIN="$(grep -vE '^[[:space:]]*$|^[[:space:]]*#' "${_OVERRIDE_DOMAIN_FILE}" 2> /dev/null | head -n 1 | xargs)"
+
+! [ -z "${_OVERRIDE_DOMAIN}" ] \
+    && [ "${_OVERRIDE_DOMAIN}" != "${_BOOT_DOMAIN}" ] \
+    && echo "WARNING: Domain has been overridden by: ${_OVERRIDE_DOMAIN_FILE}" >&2 \
+    && echo "${_OVERRIDE_DOMAIN}" \
+    && exit 0
+
+! [ -z "${_BOOT_DOMAIN}" ] \
+    && echo "${_BOOT_DOMAIN}" \
+    && exit 0
+
+echo "It was impossible to find out the domain of this host, please prepare this host first." >&2
+exit 1

manual/README.md

@@ -0,0 +1,30 @@
+Manual
+======
+
+Each script `${FULL_SCRIPTNAME}.sh` has it's corresponding manual here:
+ - `manuals/${FULL_SCRIPTNAME}.sh.md`
+
+According to the rule above, this directory `manuals/core/*`  
+has the same structure as `../core/*` and contains all manuals explaining the core functionality.
+
+And this directory `manuals/script/*`  
+has the same structure as `../script/*` and contains all manuals explaining the purpose of each script.
+
+
+
+Core functionality
+------------------
+
+This set of scripts is the absolute minimum to provide the core functionality:
+ - `setupCoreOntoThisHost.sh` bootstraps an empty new host, after the scripts repository was cloned  
+   [Read this for further details](./setupCoreOntoThisHost.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/addAndCheckGitRepository.sh`](./core/addAndCheckGitRepository.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/addNormalUser.sh`](./core/addNormalUser.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/addToCrontabEveryHour.sh`](./core/addToCrontabEveryHour.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/defineAuthorizedKeysOfUser.sh`](./core/defineAuthorizedKeysOfUser.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/ensureUsageOfDefinitions.sh`](./core/ensureUsageOfDefinitions.sh.md)
+
+
+
+The scripts
+-----------

manual/core/addNormalUser.sh.md

@@ -0,0 +1,8 @@
+[core/addNormalUser.sh](..//core/addNormalUser.sh)
+==================================================
+
+This script adds a normal user to the host on which it was called.
+
+__Parameters:__
+  1. USER (mandantory)
+          Name of the user who will be added to the host

manual/core/defineAuthorizedKeysOfUser.sh.md

@@ -0,0 +1,8 @@
+[core/defineAuthorizedKeysOfUser.sh](../core/defineAuthorizedKeysOfUser.sh)
+===========================================================================
+
+This script defines the `authorized_keys` file of the given user.
+It will create a link pointing to the file in definitions to distribute the same settings across all hosts.
+
+__Parameters:__
+  1. USER (mandantory) Name of the user who will receive the ssh settings.

manual/script/host/README.md

@@ -0,0 +1,9 @@
+script/host
+===========
+
+This folder contains scripts managing the host.  
+For example:
+  - certificates
+  - git
+  - ssh
+  - users

manual/script/host/user/README.md

@@ -0,0 +1,4 @@
+script/host/user
+================
+
+This folder contains scripts managing users of a host.

prepareDefinitionsRepository.sh

@@ -0,0 +1,69 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && echo "This script prepares the user 'root' of this host and the host itself," \
+    && echo "so this script is allowed to be executed if you are root only." \
+    && exit 1
+
+# There has to be one dot at least.
+_BOOT_DOMAIN="$(hostname -b | grep -F '.' | cut -d. -f2-)"
+
+[ -z "${_BOOT_DOMAIN}" ] \
+    && echo "It was impossible to find out the domain of this host, please prepare this host first." \
+    && exit 1
+
+_REOPSITORY_NAME="cis-definition-${_BOOT_DOMAIN}"
+
+
+
+#Generate file 'README.md'
+mkdir -p /tmp/skeleton/definition
+cat << EOF > /tmp/skeleton/definition/README.md
+This repository contains the definitions of the domain “$_BOOT_DOMAIN” by the Core Infrastructure System.
+EOF
+
+
+
+#Use current file 'authorized_keys' of root as definition
+mkdir -p /tmp/skeleton/definition/core/all/root/.ssh
+cp /root/.ssh/authorized_keys /tmp/skeleton/definition/core/all/root/.ssh/authorized_keys
+
+
+
+#Generate file 'authorized_keys' for user jenkins
+mkdir -p /tmp/skeleton/definition/core/all/home/jenkins/.ssh
+cat << EOF > /tmp/skeleton/definition/core/all/home/jenkins/.ssh/authorized_keys
+#------------------------------------------------------
+# Enter the public ssh key of your jenkins server here.
+#------------------------------------------------------
+EOF
+
+
+
+cat << EOF
+
+The first content for your repository for the definitions of the '$_BOOT_DOMAIN' domain has been created.
+
+Please create a definition repository.
+To follow the naming convention name it '$_REOPSITORY_NAME'
+
+Please DO NOT use the SSH key of root for this.
+Maybe you can use https and user password for pushing the first commit.
+
+Go to folder '/tmp/skeleton/definition' and check the content of all 'authorized_keys' files,
+correct them if required to prevent losing access to your hosts.
+
+The public ssh key of your jenkins server has to be added.
+
+Only now follow the instructions as our git server shows.
+For example:
+
+  cd /tmp/skeleton/definition
+  git init
+  git checkout -b main
+  git add .
+  git commit -m "first core definitions"
+  git remote add origin https://git.example.dev/[SOME_PATH/]$_REOPSITORY_NAME.git
+  git push -u origin main
+
+EOF

prepareStatesRepository.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && echo "This script prepares the user 'root' of this host and the host itself," \
+    && echo "so this script is allowed to be executed if you are root only." \
+    && exit 1
+
+# There has to be one dot at least.
+_BOOT_DOMAIN="$(hostname -b | grep -F '.' | cut -d. -f2-)"
+
+[ -z "${_BOOT_DOMAIN}" ] \
+    && echo "It was impossible to find out the domain of this host, please prepare this host first." \
+    && exit 1
+
+_REOPSITORY_NAME="cis-state-${_BOOT_DOMAIN}"
+
+
+
+#Generate README.md
+mkdir -p /tmp/skeleton/state
+cat << EOF > /tmp/skeleton/state/README.md
+This repository contains the states of the domain “$_BOOT_DOMAIN” by the Core Infrastructure System.
+EOF
+
+
+
+cat << EOF
+
+The first content for your repository for the state of the '$_BOOT_DOMAIN' domain has been created.
+
+Please create a states repository.
+To follow the naming convention name it '$_REOPSITORY_NAME'
+
+Please DO NOT use the SSH key of root for this.
+Maybe you can use https and user password for pushing the first commit.
+
+Then go to folder '/tmp/skeleton/state' and follow the instructions as your git server shows.
+For example:
+
+  cd /tmp/skeleton/state
+  git init
+  git checkout -b main
+  git add .
+  git commit -m "first state"
+  git remote add origin https://git.example.dev/[SOME_PATH/]$_REOPSITORY_NAME.git
+  git push -u origin main
+
+EOF

prepareThisHostBeforeCloning.sh

@@ -8,46 +8,66 @@
 function setNeededHostnameOrExit() {
     _FQDN="${1:?"Missing unique long hostname (fqdn, eg.: host1.example.net) for this host as first parameter."}"
 
-    echo "${_FQDN}" | grep '\.' &> /dev/null \
+    echo "${_FQDN}" | grep -F '.' &> /dev/null \
         && hostnamectl set-hostname "${_FQDN}" \
         && return 0
 
-    echo "FAILED: setting full qualified domain name, given value was:"
+    echo "FAILED: setting full qualified domain name does not contain a domain,"
-    echo "  - ${_FQDN}"
+    echo "        given value was: ${_FQDN}"
     exit 1
 }
 
-function prepare() {
+function prepareThisHost() {
     git --version > /dev/null || (apt update; apt upgrade -y; apt install git)
 
     echo
     echo "Public SSH-Key for root@$(hostname -b):"
+    cat "/root/.ssh/id_ed25519.pub" \
+        && return 0
+
     # -t    type of the key pair
     # -f    defines the filenames (we use the standard for the selected type here)
     # -q    quiet, no output or interaction
     # -N "" means the private key will not be secured by a passphrase
     # -C    defines a comment
-    cat "/root/.ssh/id_ed25519.pub" \
+    ssh-keygen \
-        || (ssh-keygen \
         -t ed25519 \
         -f "/root/.ssh/id_ed25519" -q -N "" \
-            -C "$(date +%Y%m%d)-root@$(hostname -b)" \
+        -C "$(date +%Y%m%d)-root@$(hostname -b)"
-        && cat "/root/.ssh/id_ed25519.pub")
+
+    cat "/root/.ssh/id_ed25519.pub" \
+        && return 0
 
     echo
-    echo "Now you have to register the public ssh-key from above into your git-server to grant these access rights:"
+    echo "FAILED: somthing went wrong during the generation the ssh keys."
+    echo "        These keys are mandantory. You can try to restart this script."
+    echo
+}
+
+function showFurtherSteps() {}
+    echo
+    echo "IMPORTANT: It is assumed that repositories for definitions and states already exist"
+    echo "           and comply with the naming convention."
+    echo "  Otherwise, these repositories must be created first!"
+    echo
+    echo "To grant the correct access rights, you have to register the above-mentioned ssh key,"
+    echo "as deploy key in these repositories of the Git server:"
     echo "  - scripts repository     (allow readonly access only),"
     echo "  - definitions repository (allow readonly access only),"
     echo "  - states repository      (allow writable access)."
     echo
-    echo "After all access rights are granted you can clone the Infrastructure System:"
+    echo "After all access rights are granted you can clone the Core Infrastructure System:"
-    echo "  e.g.: git clone ssh://git@git.example.dev:22448/iss.git /iss"
+    echo "  e.g.: git clone ssh://git@git.example.dev:22448/cis.git /cis"
     echo
-    echo "Finally call 'setupCoreOntoThisHost.sh' from the root directory of the repository:"
+    echo "Finally call 'setupCoreOntoThisHost.sh' from the root directory:"
-    echo "  e.g.: /iss/setupCoreOntoThisHost.sh"
+    echo "  e.g.: /cis/setupCoreOntoThisHost.sh"
     echo
 }
 
 # sanitizes all parameters
 setNeededHostnameOrExit "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && prepare
+    && prepareThisHost \
+    && showFurtherSteps \
+    && exit 0
+
+exit 1

script/check/README.md

@@ -0,0 +1,4 @@
+runAllChecks.sh
+===============
+
+This script processes all checks of a host to verify the right configuration.

script/check/host/all/app_docker-compose_is_installed.check.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+_CURRENT_APP='docker compose version'
+
+${_CURRENT_APP} > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/app_docker_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+docker --version > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/app_nginx_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+nginx -v > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/app_nginx_starts_reliable.check.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# Fail because of unnecessary custom config
+grep "Wants=network-online.target" /lib/systemd/system/nginx.service > /dev/null 2>&1 \
+    && [ -f "/etc/systemd/system/nginx.service" ] \
+    && exit 1
+
+# Success if system config is ok
+grep "Wants=network-online.target" /lib/systemd/system/nginx.service > /dev/null 2>&1 \
+    && exit 0
+
+# Success if custom config fixes system config
+grep "Wants=network-online.target" /etc/systemd/system/nginx.service > /dev/null 2>&1 \
+    && exit 0
+
+exit 1

script/check/host/all/core_cron_starts_setup_as_fallback.check.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+crontab -l | grep -E "[0-9]{1,2}[ \*]{8}[[:blank:]]*\/cis\/setupCoreOntoThisHost.sh" > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/core_git_app_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+git --version > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/core_hostname_is_long.check.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/hostname'
+
+#The file must be readable, then
+#the number of lines containing a '.' must be zero.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "$(grep -cF '.' "${_CURRENT_FILE}")" -gt 0 ] \
+    && exit 0
+exit 1

script/check/host/all/core_ssh_app_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+ssh -V > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/core_ssh_authorized_keys_of_jenkins_points_to_definitions.check.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+_CURRENT_FILE='/home/jenkins/.ssh/authorized_keys'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#File has to be readable, then
+#search for '/definitions/' in the path of current file, after readlink expanded a potential symlink.
+[ -r "${_CURRENT_FILE}" ] \
+    && readlink -f "${_CURRENT_FILE}" | grep -q "/definitions/" \
+    && exit 0
+
+exit 1

script/check/host/all/core_ssh_authorized_keys_of_root_is_empty_or_points_to_definitions.check.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+_CURRENT_FILE='/root/.ssh/authorized_keys'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#No file is ok
+[ ! -e "${_CURRENT_FILE}" ] \
+    && exit 0
+
+#The file must be readable, then
+#all comments and all blank lines are removed, after which the number of remaining lines must be zero.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "0" == "$(cat "${_CURRENT_FILE}" | sed 's/[[:blank:]]*#.*//' | sed '/^$/d' | grep -c .)" ] \
+    && exit 0
+
+#File has to be readable, then
+#search for '/definitions/' in the path of current file, after readlink expanded a potential symlink.
+[ -r "${_CURRENT_FILE}" ] \
+    && readlink -f "${_CURRENT_FILE}" | grep -q "/definitions/" \
+    && exit 0
+
+exit 1

script/check/host/all/core_ssh_key_of_root_exists_as_expected.check.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_CURRENT_FILE='/root/.ssh/id_ed25519'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#File has to be readable and no passphrase should be needed.
+ssh-keygen -y -P "" -f "${_CURRENT_FILE}" &> /dev/null \
+    && exit 0
+
+exit 1

script/check/host/all/core_sudoers_file_of_jenkins_points_to_definitions.check.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/sudoers.d/allow-jenkins-updateRepositories'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#File has to be readable, then
+#search for '/definitions/' in the path of current file, after readlink expanded a potential symlink.
+[ -r "${_CURRENT_FILE}" ] \
+    && readlink -f "${_CURRENT_FILE}" | grep -q "/definitions/" \
+    && exit 0
+
+exit 1

script/check/host/all/core_user_jenkins_exists.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_USER='jenkins'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+id -u "${_CURRENT_USER}" > /dev/null 2>&1 \
+    && exit 0
+
+exit 1

script/check/host/all/system_localtime_contains_cet_and_cest.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/localtime'
+
+#The file must be readable, then
+#the number of lines containing "CET" must be greater than zero, and
+#the number of lines containing "CEST" must also be greater than zero.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "$(zdump -v "${_CURRENT_FILE}" | head -n 10 | grep 'CET' | grep -c .)" -gt "0" ] \
+    && [ "$(zdump -v "${_CURRENT_FILE}" | head -n 10 | grep 'CEST' | grep -c .)" -gt "0" ] \
+    && exit 0
+exit 1

script/check/host/all/system_timezone_is_berlin.check.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/timezone'
+
+#The file must be readable, then
+#the number of lines containing "Europe/Berlin" must be one.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "1" == "$(cat "${_CURRENT_FILE}" | grep 'Europe/Berlin' | grep -c .)" ] \
+    && exit 0
+exit 1

script/check/host/all/system_unattended_upgrades_are_disabled.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+! systemctl is-enabled unattended-upgrades.service > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_app_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+zfs --version > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_atime_of_rootfs_zpool1.check.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_CURRENT_POOL='zpool1'
+
+#Check if the tool 'zfs' is available, then
+#retrieve the property 'atime' from 'zpool1', without header and compare the result with 'off'
+#because this the feature 'atime' logs each access, there are many avoidable writes.
+
+#Set with: 'zfs set atime=off zpool1'
+zfs version &> /dev/null  \
+    && [ "$(zfs get atime -Ho value ${_CURRENT_POOL} 2> /dev/null)" == "off" ] \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_compression_of_rootfs_zpool1.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_POOL='zpool1'
+
+#Check if the tool 'zfs' is available, then
+#retrieve the property 'compression' from 'zpool1', without header and compare the result with 'lz4'
+
+#Set with: 'zfs set compression=lz4 zpool1'
+zfs version &> /dev/null  \
+    && [ "$(zfs get compression -Ho value ${_CURRENT_POOL} 2> /dev/null)" == "lz4" ] \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_mountpoint_of_rootfs_zpool1.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_ZFS='zpool1'
+
+#Check if the tool 'zfs' is available, then
+#retrieve the property 'mountpoint' from 'zpool1', without header and compare the result with '/zpool1'
+
+#Set with: 'zfs set mountpount=default'
+zfs version &> /dev/null \
+    && [ "$(zfs get mountpoint -Ho value ${_CURRENT_ZFS} 2> /dev/null)" == "/${_CURRENT_ZFS}" ] \
+    && exit 0
+exit 1

script/check/host/all/system_zpool_alignment_of_pool.check.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+_CURRENT_POOL='zpool1'
+
+#Check if the tool 'zpool' is available, then
+#retrieve the property 'ashift' from 'zpool1', without header and compare the result with '12'
+zpool version &> /dev/null \
+    && [ "$(zpool get ashift -Ho value ${_CURRENT_POOL} 2> /dev/null)" == "12" ] \
+    && exit 0
+exit 1

script/check/runAllChecks.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SCRIPT%%/script/check/*}/"               #Removes longest  matching pattern '/script/check/*' from the end
+_SCRIPT_PATH="${_CIS_ROOT:?"Missing CIS_ROOT"}script/"
+_OWN_DOMAIN="$(${_CIS_ROOT}core/printOwnDomain.sh)"
+_OWN_DEFINITIONS="${_CIS_ROOT}definitions/${_OWN_DOMAIN:?"Missing OWN_DOMAIN"}/"
+
+
+
+function run_as_root() {
+    [ "0" == "$(id -u)" ] \
+        && echo OK \
+        && return 0
+
+    echo FAIL
+    return 1
+}
+
+function scripts_are_updateable_by_git() {
+        git -C "${_SCRIPT_PATH:?"Missing SCRIPT_PATH"}" pull > /dev/null 2>&1 \
+        && echo OK \
+        && return 0
+
+    echo FAIL
+    return 1
+}
+
+function allChecks() {
+    local _CHECK_PATH _MODE_PATH
+    _CHECK_PATH="${1:?"allChecks(): Missing first parameter CHECK_PATH"}check/"
+    _MODE_PATH="${2:-all}/"
+    readonly _CHECK_PATH _MODE_PATH
+
+    echo "  - ${_CHECK_PATH}host/${_MODE_PATH}*.check.sh"
+    [ "$(ls -1 ${_CHECK_PATH}host/${_MODE_PATH}*.check.sh 2> /dev/null | grep -cE '.*')" == "0"  ] \
+       && echo "  nothing to do" \
+       && return 0
+
+    for _CURRENT_CHECK in ${_CHECK_PATH}host/${_MODE_PATH}*.check.sh; do
+        _NAME="$(basename ${_CURRENT_CHECK} | cut -d'.' -f1)"
+        _CONTEXT="$(echo ${_NAME} | cut -d'_' -f1)"
+        _CHECK="$(echo ${_NAME} | cut -d'_' -f2- | tr '_' ' ')"
+        _RESULT="$("${_CURRENT_CHECK}" && echo OK || echo FAIL)"
+        echo "  ${_CONTEXT^^} ${_CHECK}: ${_RESULT}"
+    done
+}
+
+echo "PRECONDITION run as root: $(run_as_root)"
+echo "PRECONDITION scripts are updateable by git: $(scripts_are_updateable_by_git)"
+echo
+echo "Check all (common):"
+allChecks "${_SCRIPT_PATH}"
+echo "Check all (own):"
+allChecks "${_OWN_DEFINITIONS}"
+echo "Check this host:"
+allChecks "${_OWN_DEFINITIONS}" "$(hostname -s)"

script/docker/printAllBridgedNetworksWithRange.sh

@@ -0,0 +1,4 @@
+#/bin/bash
+
+docker network inspect $(docker network ls | grep -F 'bridge' | cut -d' ' -f1) \
+    | jq -r '.[] | .Name + " " + .IPAM.Config[0].Subnet' -

script/docker/printContainerNamesOfComposition.sh

@@ -0,0 +1,23 @@
+#/bin/bash
+
+_COMPOSITION_FILE="${1:-./docker-compose.yml}"
+
+[ -d "${_COMPOSITION_FILE}" ] \
+    && echo "A valid composition file ('docker-compose.yml') is needed. Given parameter was: ${_COMPOSITION_FILE}" >&2 \
+    && exit 1
+
+_DOCKER_COMPOSE_CMD=""
+
+[ "${_DOCKER_COMPOSE_CMD}" = "" ] \
+    && docker compose version 2> /dev/null | grep -q version \
+    && _DOCKER_COMPOSE_CMD="docker compose"
+
+[ "${_DOCKER_COMPOSE_CMD}" = "" ] \
+    && docker-compose version 2> /dev/null | grep -q version \
+    && _DOCKER_COMPOSE_CMD="docker-compose"
+
+[ "${_DOCKER_COMPOSE_CMD}" = "" ] \
+    && echo "Command 'docker compose' not found" >&2 \
+    && exit 1
+
+${_DOCKER_COMPOSE_CMD} -f "${_COMPOSITION_FILE}" images | tail -n +2 | cut -d' ' -f1

script/host/net/printAllShortManagedHostnamesFromHostsFile.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# Select just lines containing 'managedHost'.
+# 1.) Remove everything after a '#' (including the #).
+# 2.) Remove every indenting.
+# 3.) Remove blanks (spaces or tabs) at the end of lines.
+# 4.) Replace blanks (spaces or tabs) with one ';' between the values.
+# 5.) Delete empty lines.
+# Then cut the second field
+# Then cut the first field to get the short hostname
+grep 'managedHost' /etc/hosts \
+    | sed -e 's/#.*//' \
+        -e 's/^[[:blank:]]*//' \
+        -e 's/[[:blank:]]*$//' \
+        -e 's/\s\+/;/g' \
+        -e '/^$/d' \
+    | cut -d';' -f2 \
+    | cut -d'.' -f1
+

script/host/net/printOwnDefaultMACAdress.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+cat /sys/class/net/e*/address \
+    | head -n 1

script/host/net/printOwnIPv4Adress.sh

@@ -0,0 +1,108 @@
+#!/bin/bash
+
+#grep -E '(:|^(127|169\.254|10|172\.(1(6|7|8|9)|2[0-9]|30|31)|192\.168|(22(4|5|6|7|8|9)|23(0|1|2|3|4|5|6|7|8|9))).*)' findet:
+#  loopback:  127.0.0.0/8
+#  linklocal: 169.254.0.0/16
+#  private:   10.0.0.0/8,
+#             172.16.0.0/12,   (172.16… bis 172.31…)
+#             192.168.0.0/16
+# multicast:  224.0.0.0/4      (224… bis 239…)
+
+
+function all() {
+    # Select just lines containing 'inet'.
+    # 1.) Remove every indenting.
+    # 2.) Remove 'inet '.
+    # 3.) Remove everything after a '/' (including the /).
+    ip -4 addr \
+        | grep 'inet' \
+        | sed -e 's/^[[:blank:]]*//' \
+            -e 's/inet //' \
+            -e 's/\/.*//'
+}
+
+function routed() {
+    local _DEVICE
+    _DEVICE="$(ip -4 route show default | xargs -n 1 | grep -A1 -i dev | tail -n 1)"
+    readonly _DEVICE
+
+    ip -4 addr show dev "${_DEVICE:?"Missing DEVICE"}" scope global \
+        | grep 'inet' | xargs -n 1 \
+        | grep -A1 'inet' \
+        | tail -n 1 \
+        | cut -d/ -f1
+}
+
+function public() {
+    hostname -I | xargs -n 1 \
+        | grep -vE '(:|^(127|169\.254|10|172\.(1(6|7|8|9)|2[0-9]|30|31)|192\.168|(22(4|5|6|7|8|9)|23(0|1|2|3|4|5|6|7|8|9))).*)'
+}
+
+# Maybe use "resolvectl status" to get DNS Server and specify 'nslookup'
+function published() {
+    local _BOOT_HOSTNAME
+    _BOOT_HOSTNAME="$(hostname -b)"
+    readonly _BOOT_HOSTNAME
+
+    nslookup -type=A "${_BOOT_HOSTNAME:?"Missing BOOT_HOSTNAME"}" | xargs -n 1 \
+        | grep -A2 -i "${_BOOT_HOSTNAME}" \
+        | grep -A1 -i 'address' \
+        | tail -n1
+}
+
+function verified() {
+    local _PUBLISHED_IP
+    _PUBLISHED_IP="$(published)"
+    readonly _PUBLISHED_IP
+
+    [ -z "${_PUBLISHED_IP}" ] \
+        && return 0
+
+    all | grep "${_PUBLISHED_IP}"
+}
+
+function usage() {
+    echo "Use one of the following options:"
+    echo "  --all       : prints all IPv4 addresses"
+    echo "  --routed    : prints the IPv4 address used to send traffic to the default gateway"
+    echo "  --public    : prints all IPv4 addresses direct accessable from the internet"
+    echo "  --published : prints the IPv4 address provided by DNS using this host's name"
+    echo "  --verified  : prints the IPv4 included in 'all' und respended by 'published'"
+}
+
+
+
+function main(){
+
+    case "${1}" in
+        --all)
+            all
+            return 0
+            ;;
+        --routed)
+            routed
+            return 0
+            ;;
+        --public)
+            public
+            return 0
+            ;;
+        --published)
+            published
+            return 0
+            ;;
+        --verified)
+            verified
+            return 0
+            ;;
+        *)
+            usage
+            return 1
+            ;;
+    esac
+
+    return 1
+
+}
+
+ main "$@" && exit 0 || exit 1

script/host/net/printOwnIPv6Adress.sh

@@ -0,0 +1,109 @@
+#!/bin/bash
+
+#grep -E '(^::1|(^fc.*|^fd.*)|^fe80::.*|^ff.*)' findet:
+#  loopback: ::1/128
+#  uniquelocal: fc00::/7   (fc00… bis fdff…)
+#  linklocal:   fe80::/64
+#  multicast:   ff00::/8   (ff…)
+
+
+
+function all() {
+    # Select just lines containing 'inet6'.
+    # 1.) Remove every indenting.
+    # 2.) Remove 'inet6 '.
+    # 3.) Remove everything after a '/' (including the /).
+    ip -6 addr \
+        | grep 'inet6' \
+        | sed -e 's/^[[:blank:]]*//' \
+            -e 's/inet6 //' \
+            -e 's/\/.*//'
+}
+
+function routed() {
+    local _DEVICE
+    _DEVICE="$(ip -6 route show default | xargs -n 1 | grep -A1 -i dev | tail -n 1)"
+    readonly _DEVICE
+
+    ip -6 addr show dev "${_DEVICE:?"Missing DEVICE"}" scope global \
+        | grep 'inet6' \
+        | xargs -n 1 \
+        | grep -A1 'inet6' \
+        | grep ':' \
+        | cut -d/ -f1
+}
+
+function public() {
+    hostname -I | xargs -n 1 \
+        | grep ':' \
+        | grep -vE '(^::1|(^fc.*|^fd.*)|^fe80::.*|^ff.*)'
+}
+
+# Maybe use "resolvectl status" to get DNS Server and specify 'nslookup'
+function published() {
+    local _BOOT_HOSTNAME
+    _BOOT_HOSTNAME="$(hostname -b)"
+    readonly _BOOT_HOSTNAME
+
+    nslookup -type=AAAA "${_BOOT_HOSTNAME:?"Missing BOOT_HOSTNAME"}" | xargs -n 1 \
+        | grep -A2 -i "${_BOOT_HOSTNAME}" \
+        | grep -A1 -i address \
+        | tail -n1
+}
+
+function verified() {
+    local _PUBLISHED_IP
+    _PUBLISHED_IP="$(published)"
+    readonly _PUBLISHED_IP
+
+    [ -z "${_PUBLISHED_IP}" ] \
+        && return 0
+
+    all | grep "${_PUBLISHED_IP}"
+}
+
+function usage() {
+    echo "Use one of the following options:"
+    echo "  --all       : prints all IPv6 addresses"
+    echo "  --routed    : prints the IPv6 address used to send traffic to the default gateway"
+    echo "  --public    : prints all IPv6 addresses direct accessable from the internet"
+    echo "  --published : prints the IPv6 address provided by DNS using this host's name"
+    echo "  --verified  : prints the IPv6 included in 'all' und respended by 'published'"
+}
+
+
+
+function main(){
+
+    case "${1}" in
+        --all)
+            all
+            return 0
+            ;;
+        --routed)
+            routed
+            return 0
+            ;;
+        --public)
+            public
+            return 0
+            ;;
+        --published)
+            published
+            return 0
+            ;;
+        --verified)
+            verified
+            return 0
+            ;;
+        *)
+            usage
+            return 1
+            ;;
+    esac
+
+    return 1
+
+}
+
+ main "$@" && exit 0 || exit 1

script/host/net/printOwnMACAdresses.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+cat /sys/class/net/e*/address

script/host/net/printOwnShortHostnameFromHostsFile.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+
+# Select just lines containing 'inet'.
+# 1.) Remove every indenting.
+# 2.) Remove 'inet '.
+# 3.) Remove everything after a '/' (including the /).
+# Search each IP of the IPv4-list in file '/etc/hosts' 
+# Select just lines containing 'managedHost'.
+# 1.) Remove everything after a '#' (including the #).
+# 2.) Remove every indenting.
+# 3.) Remove blanks (spaces or tabs) at the end of lines.
+# 4.) Replace blanks (spaces or tabs) with one ';' between the values.
+# 5.) Delete empty lines.
+# Then cut the second field
+# Then cut the first field to get the short hostname
+ip -4 addr \
+    | grep 'inet' \
+    | sed -e 's/^[[:blank:]]*//' \
+        -e 's/inet //' \
+        -e 's/\/.*//' \
+    | xargs -i grep {} /etc/hosts \
+    | grep 'managedHost' \
+    | sed -e 's/#.*//' \
+        -e 's/^[[:blank:]]*//' \
+        -e 's/[[:blank:]]*$//' \
+        -e 's/\s\+/;/g' \
+        -e '/^$/d' \
+    | cut -d';' -f2 \
+    | cut -d'.' -f1
+

script/host/nginx/restartIfConfigurationIsValid.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+nginx -t &> /dev/null \
+    && systemctl restart nginx.service \
+    && exit 0
+
+exit 1

script/host/nginx/setup.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+function main() {
+    local _SCRIPTPATH _DH_PATH _SELF_SIGNED_PATH
+    _SCRIPTPATH="$(cd -- "$(dirname "$0")" > /dev/null 2>&1; pwd -P)"
+    _DH_PATH="/etc/ssl/private"
+    _SELF_SIGNED_PATH="/etc/ssl/private"
+    readonly _SCRIPTPATH _DH_PATH _SELF_SIGNED_PATH
+
+    ! dpkg -s nginx > /dev/null 2>&1 \
+        && apt-get --yes install nginx-full \
+        && echo "Nginx erfolgreich installiert." \
+        || echo "Nginx ist bereits installiert."
+
+    ! dpkg -s openssl > /dev/null 2>&1 \
+        && apt-get --yes install openssl \
+        && echo "OpenSSL erfolgreich installiert." \
+        || echo "OpenSSL ist bereits installiert."
+
+    ! [ -f "${_DH_PATH}/dhparam4096.pem" ] \
+        && mkdir -p "${_DH_PATH}" \
+        && chmod go-rwx "${_DH_PATH}" \
+        && openssl dhparam -out "${_DH_PATH}/dhparam4096.pem" 4096 \
+        && echo "Diffie-Hellman-Parameters erfolgreich erstellt." \
+        || echo "Diffie-Hellman-Parameters bereits vorhanden."
+
+    ! [ -f "${_SELF_SIGNED_PATH}/selfsigned-private.key" ] \
+        && mkdir -p "${_SELF_SIGNED_PATH}" \
+        && chmod go-rwx "${_SELF_SIGNED_PATH}" \
+        && openssl req -x509 -days 36524 -nodes -newkey rsa:4096 \
+             -keyout "${_SELF_SIGNED_PATH}/selfsigned-private.key" \
+             -out "${_SELF_SIGNED_PATH}/selfsigned-fullchain.crt" \
+        && echo "Selbstsignierte Standardschlüssel erfolgreich erstellt." \
+        || echo "Selbstsignierte Standardschlüssel bereits vorhanden."
+
+#TODO Links erstellen
+#    [ -d "/etc/nginx/" ] \
+#        && cp "${_SCRIPTPATH}/etc_nginx_conf.d/"* "/etc/nginx/conf.d/" \
+#        && mkdir -p /etc/nginx/ssl-trusted \
+#        && cp "${_SCRIPTPATH}/etc_nginx_ssl-trusted/"* "/etc/nginx/ssl-trusted/" \
+#        && mkdir -p /var/www/letsencrypt/.well-known/acme-challenge \
+#        && echo "Basis-Konfiguration erfolgreich erstellt." \
+#        || echo "Basis-Konfiguration bereits vorhanden."
+
+    echo \
+        && echo "Nginx neu starten:" \
+        && nginx -t \
+        && systemctl restart nginx.service \
+        && return 0
+
+    return 1
+}
+
+main "$@" && exit 0 || exit 1

script/host/nginx/update_nginx_conf.sh

@@ -0,0 +1,375 @@
+#!/bin/bash
+NGINX_DIR="/etc/nginx"
+HOSTNAME=$(hostname)
+
+if [ ! -d "$NGINX_DIR" ]; then
+	exit
+fi
+rm $NGINX_DIR/site-*/*
+TEMP_HOST_FILE=`mktemp`
+cp /etc/hosts $TEMP_HOST_FILE
+INDENT=4
+
+function appendProxyServerBlock() {
+	INDENT=$((INDENT+4))
+	local DOMAIN
+	local PORT
+	local INCLUDE_DOMAIN
+	local SSL
+	local FILE
+	DOMAIN=$1
+	PORT=$2
+	INCLUDE_DOMAIN=$3
+	FILE=$4
+	if [[ "$5" == "ssl" ]]; then
+		SSL=" ssl"
+	fi
+
+	if [[ -z "$DOMAIN" || -z "$INCLUDE_DOMAIN" ]]; then
+		return
+	fi
+	echo "$(echo "" | pr -to $INDENT)append proxy server block: '${DOMAIN}'${SSL}"
+	cat >> "$FILE" << EOF
+server {
+	listen ${PORT}${SSL};
+
+	server_name ${DOMAIN};
+EOF
+
+	if [[ ! -z "$SSL" ]]; then
+		cat >> "$FILE" << EOF
+
+	ssl_certificate /etc/nginx/ssl/${DOMAIN}/server.crt;
+	ssl_certificate_key /etc/nginx/ssl/${DOMAIN}/server.key;
+
+	add_header Strict-Transport-Security max-age=15552000;
+EOF
+	else
+		cat >> "$FILE" << EOF
+	server_name www.${DOMAIN};
+EOF
+	fi
+
+	cat >> "$FILE" << EOF
+	root /var/www;
+
+	proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
+	proxy_set_header Host \$http_host;
+	proxy_set_header X-Forwarded-Proto \$scheme;
+	proxy_set_header X-Real-IP \$remote_addr;
+
+	# WebSocket support
+	proxy_http_version 1.1;
+	proxy_set_header Upgrade \$http_upgrade;
+	proxy_set_header Connection \$connection_upgrade;
+
+	proxy_redirect off;
+	location /.well-known/acme-challenge {
+		root /tmp/acme;
+	}
+
+	include /etc/nginx/site-${INCLUDE_DOMAIN}/*;
+}
+EOF
+	INDENT=$((INDENT-4))
+}
+
+function appendProxy() {
+	INDENT=$((INDENT+4))
+	local DOMAIN
+	local INCLUDE_DOMAIN
+	local FILE
+	DOMAIN=$1
+	INCLUDE_DOMAIN=$2
+	FILE="$NGINX_DIR/sites-enabled/${DOMAIN}"
+
+
+	if [[ -z "$TARGET_PUBLIC_PORT" ]]; then
+		TARGET_PUBLIC_PORT=80
+	fi
+
+	if [[ -z "$TARGET_PUBLIC_PORT_SSL" ]]; then
+		TARGET_PUBLIC_PORT_SSL=443
+	fi
+
+	echo "$(echo "" | pr -to $INDENT)creating proxy for domain ${DOMAIN}"
+	#VHost-Datei leeren
+	printf "" > "$FILE"
+
+	#Proxy mit ssl (redirect HTTP -> HTTPS)
+	if [ -f "$NGINX_DIR/ssl/${DOMAIN}/server.crt" ]; then
+
+		if [ -f "$NGINX_DIR/ssl/${DOMAIN}/allow_http" ]; then
+			appendProxyServerBlock "${DOMAIN}" "${TARGET_PUBLIC_PORT}" "${INCLUDE_DOMAIN}" "$FILE"
+		else
+			cat >> "${FILE}" << EOF
+server {
+	listen ${TARGET_PUBLIC_PORT};
+	server_name ${DOMAIN};
+	server_name www.${DOMAIN};
+	location /.well-known/acme-challenge {
+		root /tmp/acme;
+	}
+	root /var/www;
+	${CUSTOM_NGINX_HTTP_CONFIG}
+
+	add_header Strict-Transport-Security max-age=15552000;
+	location / {
+		return 301 https://${DOMAIN}:${TARGET_PUBLIC_PORT_SSL}\$request_uri;
+	}
+}
+EOF
+		fi
+		appendProxyServerBlock "${DOMAIN}" "${TARGET_PUBLIC_PORT_SSL}" "${INCLUDE_DOMAIN}" "$FILE" "ssl"
+
+	# Proxy ohne ssl
+	else
+		appendProxyServerBlock "${DOMAIN}" "${TARGET_PUBLIC_PORT}" "${INCLUDE_DOMAIN}" "$FILE"
+	fi
+	INDENT=$((INDENT-4))
+}
+
+function configureProxyForTargetDomain() {
+	INDENT=$((INDENT+4))
+	local DOMAIN=$1
+
+	echo "$(echo "" | pr -to $INDENT)configure proxy for domain $DOMAIN"
+	appendProxy "${DOMAIN}" "${DOMAIN}"
+
+	# Proxy für Domain mit www Präfix
+	if [ -f "$NGINX_DIR/ssl/www.${DOMAIN}/server.crt" ]; then
+		FILE="$NGINX_DIR/sites-enabled/www.${DOMAIN}"
+		printf "" > "$FILE"
+		appendProxyServerBlock "www.${DOMAIN}" "${TARGET_PUBLIC_PORT_SSL}" "${DOMAIN}" "$FILE" "ssl"
+	fi
+
+	mkdir -p "$NGINX_DIR/site-${DOMAIN}"
+	if [[ -z "$PROXY_CONTEXTS" ]]; then
+		if [[ -z "$CONTAINER_HTTPS_PORT" ]]; then
+			cat >> $NGINX_DIR/site-${DOMAIN}/${TARGET_CONTAINER} << EOF
+				location / {
+					proxy_pass http://${CONTAINER_IP}:${CONTAINER_HTTP_PORT};
+				}
+EOF
+		else
+			cat >> $NGINX_DIR/site-${DOMAIN}/${TARGET_CONTAINER} << EOF
+				location / {
+					proxy_pass https://${CONTAINER_IP}:${CONTAINER_HTTPS_PORT};
+				}
+EOF
+		fi
+	else
+		if [[ ! -z "$ROOT_REDIRECT" ]]; then
+			cat >> $NGINX_DIR/site-${DOMAIN}/${TARGET_CONTAINER} << EOF
+				location = / {
+					return 302 \$scheme://${DOMAIN}/${ROOT_REDIRECT};
+				}
+EOF
+		fi
+		for PROXY_CONTEXT in $PROXY_CONTEXTS; do
+			if [[ -z "$CONTAINER_HTTPS_PORT" ]]; then
+				cat >> $NGINX_DIR/site-${DOMAIN}/${TARGET_CONTAINER} << EOF
+					location /${PROXY_CONTEXT} {
+						proxy_pass http://${CONTAINER_IP}:${CONTAINER_HTTP_PORT}/${PROXY_CONTEXT};
+					}
+EOF
+			else
+				cat >> $NGINX_DIR/site-${DOMAIN}/${TARGET_CONTAINER} << EOF
+					location /${PROXY_CONTEXT} {
+						proxy_pass https://${CONTAINER_IP}:${CONTAINER_HTTPS_PORT}/${PROXY_CONTEXT};
+					}
+EOF
+			fi
+		done;
+	fi
+
+
+	CONTAINER_CONFIG_DIR="/invra/state/$(cat /invra/hostowner)/containers"
+	for FWD in $PROXY_FORWARDS; do
+		SOURCE_PATH="`echo $FWD | cut -d: -f1`"
+		TARGET_URL="`echo $FWD | cut -d: -s -f2-`"
+		CONTINUE=0
+
+		# Prüfen ob Proxy bereits durch neues Schema im invra/state angelegt wurde
+		while read PROXY_FILE; do
+			CUR_HOST_FILE="$(dirname "$(dirname "$PROXY_FILE")")/current-host"
+			CUR_HOST=$(cat $CUR_HOST_FILE)
+			if [[ -f "$CUR_HOST_FILE" && ( "$CUR_HOST" != "$HOSTNAME" || "$DOMAIN" != "$HOSTNAME" ) ]]; then
+				CONTINUE=1
+				break
+			fi
+		#Process Substitution nutzen, damit CONTINUE-Variable die Schleife überlebt
+		done < <(grep -lER "^/?${SOURCE_PATH}/?$" ${CONTAINER_CONFIG_DIR}/*/httpproxy/${DOMAIN} 2> /dev/null)
+
+		if [ $CONTINUE -eq 1 ]; then
+			continue
+		fi
+		echo "$(echo "" | pr -to $INDENT)create proxy for context-path '${SOURCE_PATH}' to URL '${TARGET_URL}'"
+		cat >> $NGINX_DIR/site-${DOMAIN}/${TARGET_CONTAINER} << EOF
+				location /${SOURCE_PATH} {
+					proxy_pass ${TARGET_URL};
+				}
+EOF
+
+	done
+	INDENT=$((INDENT-4))
+}
+
+function getVar() {
+	local _VAR_NAME=$1
+	local _CONF_FILE=$2
+	local _RESULT
+	_RESULT=$(grep -E "^${_VAR_NAME}=" "${_CONF_FILE}" | grep -oE "[^=]+$")
+	_CLEAN_RESULT=$(echo "$_RESULT" | sed -E 's/[()"]//g')
+	GET_VAR_RESULT=$_CLEAN_RESULT
+}
+
+echo "creating proxy forwards..."
+TARGET_PUBLIC_PORT=""
+TARGET_PUBLIC_PORT_SSL=""
+for CONTAINER in /invra/state/$(cat /invra/hostowner)/containers/*; do
+	CONTAINER_HOST="$(cat $CONTAINER/current-host)"
+	TARGET_CONTAINER="$(basename "$CONTAINER")"
+
+	echo "    creating forward proxies for container '${TARGET_CONTAINER}' on Host '${CONTAINER_HOST}'"
+	for DOMAIN_FILE in $CONTAINER/httpproxy/*; do
+		if [ ! -f "${DOMAIN_FILE}" ]; then
+			continue
+		fi
+
+		TARGET_DOMAIN="$(basename "$DOMAIN_FILE")"
+
+		#Proxy für Domain, die direkt auf diesen Host verweisen, überspringen => werden für die gehosteten Containern später angelegt
+		if [[ "$TARGET_DOMAIN" == "$HOSTNAME" && "$CONTAINER_HOST" == "$HOSTNAME" ]]; then
+			echo "    skipping '${TARGET_DOMAIN}'"
+			continue
+		fi
+
+		appendProxy "${TARGET_DOMAIN}" "${TARGET_DOMAIN}"
+
+		cat "$DOMAIN_FILE" | while read PROXY_CONTEXT; do
+
+			echo "            with context path '${PROXY_CONTEXT}'"
+
+			mkdir -p "$NGINX_DIR/site-${TARGET_DOMAIN}"
+			case $PROXY_CONTEXT in
+				/) TARGET_LOCATION="" ;;
+				/*) TARGET_LOCATION="${PROXY_CONTEXT}" ;;
+				*) TARGET_LOCATION="/${PROXY_CONTEXT}"; PROXY_CONTEXT="${TARGET_LOCATION}" ;;
+			esac
+
+			if [[ "$CONTAINER_HOST" == "$HOSTNAME" ]]; then
+				# neuer docker client
+				CONTAINER_IP=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}:{{end}}' $TARGET_CONTAINER | cut -d: -f1)
+				if [ -z "$CONTAINER_IP" ]; then
+					# alter docker client
+					CONTAINER_IP=$(docker inspect -f '{{.NetworkSettings.IPAddress}}' $TARGET_CONTAINER)
+					if [ -z "$CONTAINER_IP" ]; then
+						continue
+					fi
+				fi
+				CONTAINER_CONFIG="/persistent/${TARGET_CONTAINER}/containersettings"
+				getVar "CONTAINER_HTTP_PORT" "${CONTAINER_CONFIG}"
+				CONTAINER_HTTP_PORT=${GET_VAR_RESULT:-8080}
+
+				cat >> "$NGINX_DIR/site-${TARGET_DOMAIN}/${TARGET_CONTAINER}" << EOF
+					location ${PROXY_CONTEXT} {
+						proxy_pass http://${CONTAINER_IP}:${CONTAINER_HTTP_PORT};
+					}
+EOF
+				CONTAINER_CONFIG=""
+				CONTAINER_HTTP_PORT=""
+			else
+				cat >> "$NGINX_DIR/site-${TARGET_DOMAIN}/fw-${TARGET_CONTAINER}" << EOF
+					location ${PROXY_CONTEXT} {
+						proxy_pass https://${CONTAINER_HOST}${TARGET_LOCATION};
+						proxy_set_header Host ${TARGET_DOMAIN};
+					}
+EOF
+			fi
+		done
+	done
+done
+
+echo ""
+echo "creating proxies for local container..."
+for CONTAINER_CONFIG in /persistent/*/containersettings; do
+
+	getVar "TARGET_CONTAINER" "${CONTAINER_CONFIG}"
+	TARGET_CONTAINER=$GET_VAR_RESULT
+	getVar "CONTAINER_HTTP_PORT" "${CONTAINER_CONFIG}"
+	CONTAINER_HTTP_PORT=${GET_VAR_RESULT:-8080}
+	getVar "CONTAINER_HTTPS_PORT" "${CONTAINER_CONFIG}"
+	CONTAINER_HTTPS_PORT=$GET_VAR_RESULT
+	getVar "TARGET_DOMAIN" "${CONTAINER_CONFIG}"
+	TARGET_DOMAIN=$GET_VAR_RESULT
+	getVar "TARGET_PUBLIC_PORT" "${CONTAINER_CONFIG}"
+	TARGET_PUBLIC_PORT=$GET_VAR_RESULT
+	getVar "TARGET_PUBLIC_PORT_SSL" "${CONTAINER_CONFIG}"
+	TARGET_PUBLIC_PORT_SSL=$GET_VAR_RESULT
+	getVar "ADDITIONAL_TARGET_DOMAIN" "${CONTAINER_CONFIG}"
+	ADDITIONAL_TARGET_DOMAIN=$GET_VAR_RESULT
+	getVar "PROXY_CONTEXTS" "${CONTAINER_CONFIG}"
+	PROXY_CONTEXTS=$GET_VAR_RESULT
+	getVar "PROXY_FORWARDS" "${CONTAINER_CONFIG}"
+	PROXY_FORWARDS=$GET_VAR_RESULT
+	getVar "CUSTOM_NGINX_HTTP_CONFIG" "${CONTAINER_CONFIG}"
+	CUSTOM_NGINX_HTTP_CONFIG=""
+	grep -E "CUSTOM_NGINX_HTTP_CONFIG" "${CONTAINER_CONFIG}" > /dev/null 
+	if [[ $? -eq 0 ]]; then
+		echo "CUSTOM_NGINX_HTTP_CONFIG wird in containersettings nicht mehr unterstützt"
+	fi
+	getVar "ROOT_REDIRECT" "${CONTAINER_CONFIG}"
+	ROOT_REDIRECT=$GET_VAR_RESULT
+
+	if [[ "${TARGET_CONTAINER}" == "" ]]; then
+		echo "'${CONTAINER_CONFIG}' enthält keinen TARGET_CONTAINER"
+		continue
+	fi
+
+	CURRENT_HOST_FILE="/invra/state/$(cat /invra/hostowner)/containers/${TARGET_CONTAINER}/current-host"
+	if [ -f "$CURRENT_HOST_FILE" ]; then
+		CURRENT_HOST="$(cat "$CURRENT_HOST_FILE")"
+		if [ ! -z "$CURRENT_HOST" ] && [[ "$CURRENT_HOST" != "$HOSTNAME" ]]; then
+			continue
+		fi
+	fi
+	echo "    configuring container '$TARGET_CONTAINER'"
+	# neuer docker client
+	CONTAINER_IP=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}:{{end}}' $TARGET_CONTAINER | cut -d: -f1)
+	if [ -z "$CONTAINER_IP" ]; then
+		# alter docker client
+		CONTAINER_IP=$(docker inspect -f '{{.NetworkSettings.IPAddress}}' $TARGET_CONTAINER)
+		if [ -z "$CONTAINER_IP" ]; then
+			continue
+		fi
+	fi
+	echo "        updating ip from container $TARGET_CONTAINER"
+	sed -i "s/.*$TARGET_CONTAINER\.cont.*//" $TEMP_HOST_FILE
+	sed -i '/^\s*$/d' "${TEMP_HOST_FILE}" #löscht alle Zeilen, die nur unsichtbare Zeichen enthalten
+	sed -i "/# DOCKER-IPS/a\\$CONTAINER_IP $TARGET_CONTAINER\.cont" $TEMP_HOST_FILE
+
+	if [ -z "$TARGET_DOMAIN" ]; then
+		continue
+	fi
+
+	for DOMAIN_I in ${TARGET_DOMAIN}; do
+		OLDSETTINGSFILE="$NGINX_DIR/site-${DOMAIN_I}/${TARGET_CONTAINER}"
+		if [ -f $OLDSETTINGSFILE ]; then
+			rm $OLDSETTINGSFILE
+		fi
+		configureProxyForTargetDomain "${DOMAIN_I}"
+	done
+
+	if [[ ! -z "${ADDITIONAL_TARGET_DOMAIN}" ]]; then
+		for DOMAIN_I in ${ADDITIONAL_TARGET_DOMAIN}; do
+			appendProxy "${DOMAIN_I}" "$(echo ${TARGET_DOMAIN} | awk '{print $1}')"
+		done
+	fi
+
+done
+cat $TEMP_HOST_FILE > /etc/hosts
+rm $TEMP_HOST_FILE
+service nginx configtest
+service nginx reload
+

script/host/user/addUserToSudoGroup.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+sudo usermod --append --groups sudo "${1:?"Missing first parameter USER"}"

script/host/user/removeUserFromSudoGroup.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+sudo usermod --remove --groups sudo "${1:?"Missing first parameter USER"}"

script/host/zfs/snapshot/snapshot-cleanup.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+MIN_MIN=$(date --date="- 5 minutes" -u "+%Y%m%d%H%M")
+HOUR_MIN=$(date --date="- 1 days" -u "+%Y%m%d%H")
+DAY_MIN=$(date --date="- 7 days" -u "+%Y%m%d")
+MONTH_MIN=$(date --date="- 3 years" -u "+%Y%m")
+ 
+zfs list -Hr -o name -t snapshot -r "zpool1/persistent" | grep -E "^zpool1/persistent/[a-zA-Z0-9_-]+@(SNAPHOURLY|SNAPDAILY|SNAPMONTHLY|SNAPMINUTLY)_[0-9]{6,12}$" | while read SNAPSHOT; do
+        SNAPSHOT_TIME=$(echo "$SNAPSHOT" | grep -oE "[0-9]+$")
+        if [[ ${#SNAPSHOT_TIME} == 12 && "$SNAPSHOT_TIME" < "${MIN_MIN}" ]]; then
+                zfs destroy "${SNAPSHOT}"
+        fi
+        if [[ ${#SNAPSHOT_TIME} == 10 && "$SNAPSHOT_TIME" < "${HOUR_MIN}" ]]; then
+                zfs destroy "${SNAPSHOT}"
+        fi
+        if [[ ${#SNAPSHOT_TIME} == 8 && "${SNAPSHOT_TIME}" < "${DAY_MIN}" ]]; then
+                zfs destroy "${SNAPSHOT}"
+        fi
+        if [[ ${#SNAPSHOT_TIME} == 6 && "${SNAPSHOT_TIME}" < "${MONTH_MIN}" ]]; then
+                zfs destroy "${SNAPSHOT}"
+        fi
+done
+ 
+ 
+ 
+MONTH_MIN_QA=$(date --date="- 1 month" -u "+%Y%m")
+ 
+zfs list -Hr -o name -t snapshot -r "zpool1/persistent" | grep -E "^zpool1/persistent/[a-zA-Z0-9_-]+-qa@SNAPMONTHLY_[0-9]{6}$" | while read SNAPSHOT_QA; do
+        SNAPSHOT_TIME_QA=$(echo "$SNAPSHOT_QA" | grep -oE "[0-9]+$")
+        if [[ "${SNAPSHOT_TIME_QA}" < "${MONTH_MIN_QA}" ]]; then
+                zfs destroy "${SNAPSHOT_QA}"
+        fi
+done
+

script/host/zfs/snapshot/snapshot-minutly.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+
+_TIMESTAMP="$(date -u "+%Y%m%d%H%M")"
+_ZFS_FILESYSTEM="${1:?"Missing first parameter ZFS_FILESYSTEM."}"
+echo "${_ZFS_FILESYSTEM}" | grep -E '\-prod$' &> /dev/null \
+    && zfs snapshot "${_ZFS_FILESYSTEM}@SNAPMINUTLY_${_TIMESTAMP}" \
+    && exit 0
+
+echo "Snapshot konnte nicht angelegt werden:"
+echo "  - ${_ZFS_FILESYSTEM}@SNAPMINUTLY_${_TIMESTAMP}"
+echo "  (Minuten-Snapshots sollen nur auf 'PROD'-Containeren angelegt werden, sodass diese dann syncronisiert werden)"
+exit 1

script/host/zfs/snapshot/snapshot.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+HOUR=$(date -u "+%Y%m%d%H")
+DAY=${HOUR:0:8}
+MONTH=${HOUR:0:6}
+HOSTOWNER=$(cat /invra/hostowner)
+if [ ! -d /tmp/locks ]; then
+	mkdir /tmp/locks
+fi
+
+zfs list -Hr -o name zpool1/persistent | grep -v -- -BACKUP | tail -n +2 | while read DATASET; do
+	CONTAINER=${DATASET#zpool1/persistent/}
+	(
+		flock -n 9 || exit 1
+
+		MODE_FILE="/invra/state/$HOSTOWNER/containers/$CONTAINER/snapshot-mode"
+		HOURLY=1
+		DAILY=1
+		MONTHLY=1
+
+		if [ -f "$MODE_FILE" ]; then
+			grep -i "NONE" "$MODE_FILE" &> /dev/null
+			if [ $? -eq 0 ]; then
+				exit
+			fi
+			grep -i "HOURLY" "$MODE_FILE" &> /dev/null
+			if [ $? -ne 0 ]; then
+				HOURLY=0
+			fi
+			grep -i "DAILY" "$MODE_FILE" &> /dev/null
+			if [ $? -ne 0 ]; then
+				DAILY=0
+			fi
+			grep -i "MONTHLY" "$MODE_FILE" &> /dev/null
+			if [ $? -ne 0 ]; then
+				MONTHLY=0
+			fi
+		fi
+		SNAPSHOT_HOUR="${DATASET}@SNAPHOURLY_${HOUR}"
+		SNAPSHOT_DAY="${DATASET}@SNAPDAILY_${DAY}"
+		SNAPSHOT_MONTH="${DATASET}@SNAPMONTHLY_${MONTH}"
+
+		zfs list -H -t snapshot -o name -r "$DATASET" | grep -E "^${SNAPSHOT_HOUR}$" > /dev/null
+		if [[ $? -ne 0 && $HOURLY -eq 1 ]]; then
+			zfs snapshot "${SNAPSHOT_HOUR}"
+		fi
+
+		zfs list -H -t snapshot -o name -r "$DATASET" | grep -E "^${SNAPSHOT_DAY}$" > /dev/null
+		if [[ $? -ne 0 && $DAILY -eq 1 ]]; then
+			zfs snapshot "${SNAPSHOT_DAY}"
+		fi
+
+		zfs list -H -t snapshot -o name -r "$DATASET" | grep -E "^${SNAPSHOT_MONTH}$" > /dev/null
+		if [[ $? -ne 0 && $MONTHLY -eq 1 ]]; then
+			zfs snapshot "${SNAPSHOT_MONTH}"
+		fi
+	) 9>>/tmp/locks/snapshot.${CONTAINER}.lock
+done

script/host/zfs/sync/synccontainer-all.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+HOSTOWNER=$(cat /invra/hostowner)
+BACKUPHOST=$(hostname)
+STATE_DIR=/invra/state/${HOSTOWNER}/containers/;
+
+screen -ls | grep -oE "[0-9]+\.synccontainer\.[a-zA-Z0-9_-]+" | while read -r SCREEN_SESSION; do
+	CONTAINER=$(echo "$SCREEN_SESSION" | grep -oE "[^.]+$")
+	PID=$(echo "$SCREEN_SESSION" | grep -oE "^[0-9]+")
+	grep -iE "^${BACKUPHOST}$" ${STATE_DIR}/${CONTAINER}/standby-hosts > /dev/null
+	if [ $? -ne 0 ]; then
+		echo "quit screen session ${SCREEN_SESSION}"
+		screen -XS "$PID" quit
+	fi
+done
+
+grep -lrE "^${BACKUPHOST}$" /invra/state/${HOSTOWNER}/containers/*/standby-hosts > /dev/null
+if [ $? -eq 0 ]; then
+	grep -lrE "^${BACKUPHOST}$" /invra/state/${HOSTOWNER}/containers/*/standby-hosts | while read -r STANDBY_FILE; do
+		CONTAINER=$(basename $(dirname ${STANDBY_FILE}))
+		screen -ls | grep -oE "[0-9]+\.synccontainer\.$CONTAINER" > /dev/null
+		if [ $? -ne 0 ]; then
+			echo "starte container sync"
+			screen -dmS "synccontainer.$CONTAINER" /invra/scripts/hosts/zfs/synccontainer.sh "$CONTAINER"
+		fi
+	done
+fi
+

script/host/zfs/sync/synccontainer-check.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+TMP="$(mktemp)"
+(
+	HOSTNAME="$(hostname)"
+	HOSTOWNER="$(cat /invra/hostowner)"
+	MAX_BEHIND=0
+	CURRENT_UNIXTIME=$(date -u +%s)
+	echo "OK#Checks running" 
+	for CONTAINER_PATH in /invra/state/${HOSTOWNER}/containers/*; do 
+		grep -E "^${HOSTNAME}$" "${CONTAINER_PATH}/standby-hosts" &> /dev/null || continue;
+		CONTAINER_NAME="$(basename "$CONTAINER_PATH")";
+		TS=$(zfs list -o name -r -t snapshot "zpool1/persistent/${CONTAINER_NAME}-BACKUP" | grep "@SYNC_${HOSTNAME}" | head -n1 | grep -oP "\\d{4}-\\d{2}-\\d{2}_\\d{2}:\\d{2}:\\d{2}")
+		LAST_SNAPSHOT_TIME="$(echo "${TS}" | sed "s/_/ /g")"
+		LAST_SNAPSHOT_UNIXTIME=$(date -u --date="TZ=\"UTC\" ${LAST_SNAPSHOT_TIME}" +%s)
+		SECONDS_BEHIND=$[ $CURRENT_UNIXTIME - $LAST_SNAPSHOT_UNIXTIME ]
+		if [ "$SECONDS_BEHIND" -gt "$MAX_BEHIND" ]; then
+			MAX_BEHIND="$SECONDS_BEHIND"
+		fi
+		if [ "$SECONDS_BEHIND" -gt 30 ]; then
+			echo "LAGGING_SYNC_${CONTAINER_NAME}_${HOSTNAME}?FAIL#${SECONDS_BEHIND} behind"
+			
+		fi
+	done
+	echo $CURRENT_UNIXTIME
+) > "$TMP"
+chmod 655 "$TMP"
+mkdir -p /var/www/html/monitoring &>/dev/null
+mv "$TMP" /var/www/html/monitoring/synccontainer.check.txt
+
+

script/host/zfs/sync/synccontainer-receiver.sh

@@ -0,0 +1,64 @@
+#!/bin/bash
+CONTAINER=${1:?"CONTAINER missing"}
+CONTAINER=$(echo $1 | sed -E 's|[^a-zA-Z0-9_-]*||g')
+(
+	flock -n 9 || exit 1
+
+	BACKUPHOST=$(hostname)
+	HOSTOWNER=$(cat /invra/hostowner)
+	SOURCEHOST=$(cat /invra/state/${HOSTOWNER}/containers/${CONTAINER}/current-host)
+
+	MOUNTPOINT="none"
+	DATASET="zpool1/persistent/${CONTAINER}-BACKUP"
+	SNAPSHOT_PREFIX="${DATASET}@SYNC_${BACKUPHOST}_"
+
+	LAST_SNAPSHOT_NAME=""
+	RESUME_TOKEN=""
+	zfs list -Hr -o name -s name "${DATASET}" | grep -E "^${DATASET}$" > /dev/null
+	if [ $? -eq 0 ]; then
+		LAST_SNAPSHOT_NAME=$(zfs list -H -o name -S name -t snapshot -r "${DATASET}" | grep -E "^${SNAPSHOT_PREFIX}" | head -n 1)
+		RESUME_TOKEN="$(zfs get -o value -H receive_resume_token "${DATASET}")"
+	fi
+
+	if [[ "x$RESUME_TOKEN" != "x" && "x$RESUME_TOKEN" != "x-" ]]; then
+		echo "Resume token present trying to resume at $RESUME_TOKEN"
+		LAST_SNAPSHOT_NAME="RESUME"
+	fi
+	
+	if [[ "x${LAST_SNAPSHOT_NAME}" != "x" && "${LAST_SNAPSHOT_NAME}" != "RESUME" ]]; then
+		zfs rollback -r "${LAST_SNAPSHOT_NAME}"
+	fi
+
+	# Beiim zfs receive in der nächsten Zeile fehlt noch das "-s" für resumable streams. Der tzrlxsrv kann das aber momentan nicht. Fehlermeldung: cannot receive resume stream: kernel modules must be upgraded to receive this stream.
+	(while sleep 1; do echo; done) | ssh -o ConnectTimeout=20 -C invencom@${SOURCEHOST} "sudo /invra/scripts/hosts/zfs/synccontainer-sender.sh \"${BACKUPHOST}\" \"${CONTAINER}\" \"${LAST_SNAPSHOT_NAME#$SNAPSHOT_PREFIX}\"" \"${RESUME_TOKEN}\" | zfs receive -v "${DATASET}"
+	if [ $? -ne 0 ]; then
+		exit 1
+	fi
+
+	# Dataset gegen Veränderungen sichern
+	zfs set readonly=on "${DATASET}"
+	zfs set "mountpoint=${MOUNTPOINT}" "${DATASET}"
+
+	# Aufsetzpunkte fremder Synchronisierer wegräumen
+	zfs list -t snapshot -o name -r "${DATASET}" | grep -- "${DATASET}@SYNC" | grep -v -i "_${BACKUPHOST}_" | while read SNAP; do
+		echo "Destroying $SNAP"
+		zfs destroy $SNAP
+	done
+
+	# Alte Snapshots wegräumen
+	while read -r ZEILE
+	do
+		if [ "$ZEILE" = "" ]; then
+			break
+		fi
+		if [[ "$ZEILE" > "$LAST_SNAPSHOT_NAME"  ]]; then
+			break
+		fi
+		zfs destroy "$ZEILE"
+	done < <(zfs list -Hr -o name -s name -t snapshot "${DATASET}" | grep -E "^${SNAPSHOT_PREFIX}")
+) 9>>/tmp/synccontainer.${CONTAINER}.lock
+
+if [ $? -ne 0 ]; then
+	exit 1
+fi
+exit 0

script/host/zfs/sync/synccontainer-sender.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+BACKUPHOST=${1:?"BACKUPHOST missing"}
+CONTAINER=${2:?"CONTAINER missing"}
+BACKUPHOST=$(echo $1 | sed -E 's|[^a-zA-Z0-9._-]*||g')
+CONTAINER=$(echo $2 | sed -E 's|[^a-zA-Z0-9_-]*||g')
+LAST_SNAPSHOT=$(echo $3 | sed -E 's|[^a-zA-Z0-9._:-]*||g')
+NEW_SNAPSHOT=$(date -u "+%Y-%m-%d_%H:%M:%S")
+
+if [[ "${LAST_SNAPSHOT}" == "RESUME" ]]; then
+	RESUME_TOKEN=$(echo $4 | sed -E 's|[^a-zA-Z0-9._:-]*||g')
+        zfs send -t "${RESUME_TOKEN}"
+        exit
+fi
+
+DATASET="zpool1/persistent/$CONTAINER"
+SNAPSHOT_PREFIX="${DATASET}@SYNC_${BACKUPHOST}_"
+LAST_SNAPSHOT_NAME="${SNAPSHOT_PREFIX}${LAST_SNAPSHOT}"
+NEW_SNAPSHOT_NAME="${SNAPSHOT_PREFIX}${NEW_SNAPSHOT}"
+SNAPSHOT_FOUND=""
+
+# Existiert der Snapshot?
+while read -r ZEILE
+do
+	if [[ "$ZEILE" == "$LAST_SNAPSHOT_NAME" ]]; then
+		SNAPSHOT_FOUND="1"
+		continue
+	fi
+done < <(zfs list -H -o name -s name -t snapshot "${DATASET}" | grep -E "^${SNAPSHOT_PREFIX}")
+
+# Falls ja, alle anderen Snapshots wegräumen - eine frühere Version des Skripts hat hier nur die Älteren weggeräumt. Das führt allerdings zum Vollmüllen
+# mit neueren Snapshots, wenn der Sync immer wieder fehlschlägt - im Einzelfall bis zur Unbenutzbarkeit des Senders
+if [[ "${SNAPSHOT_FOUND}x" == "1x" ]]; then
+	while read -r ZEILE
+	do
+	        if [[ "$ZEILE" == "$LAST_SNAPSHOT_NAME" ]]; then
+        	        continue
+	        fi
+        	zfs destroy "$ZEILE"
+	done < <(zfs list -H -o name -s name -t snapshot "${DATASET}" | grep -E "^${SNAPSHOT_PREFIX}")
+fi
+
+zfs snapshot "$NEW_SNAPSHOT_NAME"
+
+if [[ "$LAST_SNAPSHOT" != "" ]]; then
+	if [[ "$SNAPSHOT_FOUND" == "" ]]; then
+		echo "Angeforderter Snapshot '${LAST_SNAPSHOT}' nicht vorhanden"
+		exit 1;
+	fi
+	zfs send -I "${LAST_SNAPSHOT_NAME}" "${NEW_SNAPSHOT_NAME}"
+else
+	zfs send "${NEW_SNAPSHOT_NAME}"
+fi

script/host/zfs/sync/synccontainer.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+BACKUPHOST=$(hostname)
+CONTAINER=${1:?"Kein Container angegeben"}
+DATASET="zpool1/persistent/$CONTAINER"
+SNAPSHOT_PREFIX="${DATASET}@SYNC_${BACKUPHOST}_"
+
+while true; do
+
+	/invra/scripts/hosts/zfs/synccontainer-receiver.sh "$CONTAINER"
+	sleep 5
+
+#	LAST_SNAPSHOT_NAME=$(zfs list -Hr -o name -S name -t snapshot "${DATASET}" | grep -E "^${SNAPSHOT_PREFIX}" | head -n 1)
+#	LAST_SNAPSHOT_TIME=${LAST_SNAPSHOT_NAME#${SNAPSHOT_PREFIX}}
+#	LAST_SNAPSHOT_TIME="$(echo "${LAST_SNAPSHOT_TIME}" | sed "s/_/ /g")"
+#	LAST_SNAPSHOT_UNIXTIME=$(date -u --date="TZ=\"UTC\" ${LAST_SNAPSHOT_TIME}" +%s)
+#	CURRENT_UNIXTIME=$(date -u +%s)
+#	SECONDS_BEHIND=$[ $CURRENT_UNIXTIME - $LAST_SNAPSHOT_UNIXTIME ]
+#	mkdir -p /var/www/html/monitoring > /dev/null 2>&1
+#	echo $CURRENT_UNIXTIME > "/var/www/html/monitoring/containersync.${CONTAINER}"
+#	echo "OK: $SECONDS_BEHIND seconds behind" >> "/var/www/html/monitoring/containersync.${CONTAINER}"
+
+done
+

script/monitor/README.md

@@ -0,0 +1,32 @@
+
+Monitoring - How it works
+=========================
+
+Basics
+------
+
+You have to set up the monitoring host first. That host will monitor your other machines.
+Execute `/cis/script/monitor/setupMonitoringHost.sh` to start the process.
+
+As usual you can configure this feature via definitions.
+```
+# Path of this feature's scripts       : '/cis/script                 /monitor'
+# Path of the corresponding definitions: '/cis/definitions/YOUR.DOMAIN/monitor'
+ls -lha '/cis/script/monitor'
+ls -lha '/cis/definitions/YOUR.DOMAIN/monitor'
+```
+
+You can modify the appearance and place your own `check.css` or `logo.png` into the definitions folder:
+
+ - /cis/definitions/YOUR.DOMAIN/monitor/check.css
+ - /cis/definitions/YOUR.DOMAIN/monitor/logo.png
+
+After the change, you must call `/cis/script/monitor/setupMonitoringHost.sh` again,  
+because it creates links in '/var/www/html/' and gives the definitions priority over the script.
+
+
+
+Dashboard
+---------
+
+You can set up an dashboard following this manual [SETUP_DASHBOARD.md](SETUP_DASHBOARD.md)

script/monitor/SETUP_DASHBOARD.md

@@ -0,0 +1,126 @@
+
+How to setup a monitoring dashboard
+===================================
+
+Inspired by: https://pimylifeup.com/ubuntu-chromium-kiosk/
+
+Steps
+-----
+
+
+
+### 1.) Install Ubuntu Server (no desktop) on your computer than set hostname and timezone.
+
+```sh
+hostnamectl set-hostname check.local
+timedatectl set-timezone Europe/Berlin
+```
+
+
+
+### 2.) Install minimal GUI and Tools.
+
+```sh
+apt install ubuntu-desktop-minimal
+apt install language-pack-gnome-de
+apt install xdotool
+apt install dbus-x11
+```
+
+
+
+### 3.) Create a kiosk user with home-directory.
+
+```sh
+useradd -m kiosk
+```
+
+and disable Welocme-Screen
+```sh
+echo "yes" > /home/kiosk/.config/gnome-initial-setup-done
+```
+
+
+
+### 4.) Edit following file `nano /etc/gdm3/custom.conf` to turn of wayland and turn on autologin for user 'kiosk'.
+
+```
+[daemon]
+# Uncomment the line below to force the login screen to use Xorg
+#WaylandEnable=false
+
+WaylandEnable=false
+
+# Enabling automatic login
+#  AutomaticLoginEnable = true
+#  AutomaticLogin = user1
+
+AutomaticLoginEnable = true
+AutomaticLogin = kiosk
+```
+
+
+
+### 5.) Configure GUI of user kiosk to prevent monitor from sleeping
+
+```sh
+#gsettings list-recursively
+
+# Does not work
+#sudo -u kiosk gsettings set org.gnome.desktop.session idle-delay 0
+
+# Set idle-delay from "uint32 300" to "uint32 0", needs 'apt install dbus-x11'
+# You can check the value in "GUI-Session of kiosk -> Settings -> Power"
+sudo -u kiosk dbus-launch dconf write /org/gnome/desktop/session/idle-delay "uint32 0"
+```
+
+
+
+### 6.) Create custom service to start firefox loading the page.
+
+Therefore create a file `/etc/systemd/system/kiosk.service` with this content:
+
+```
+[Unit]
+Description=Firefox Kiosk
+Wants=graphical.target
+After=graphical.target
+
+[Service]
+Environment=DISPLAY=:0
+# Set firefox language, needs 'apt install language-pack-gnome-de' 
+Environment=LANG=de_DE.UTF-8
+Type=simple
+# Always a fresh firefox ('-' allow error if common does not exist)
+ExecStartPre=-/usr/bin/rm -r /home/kiosk/snap/firefox/common
+# Move Mouse (should also work on small screens), needs 'apt install dbus-x11'
+ExecStartPre=/usr/bin/xdotool mousemove 4096 2160
+# See: https://wiki.mozilla.org/Firefox/CommandLineOptions (just -kiosk URL => Start-Assistant, so use -url too)
+ExecStart=/usr/bin/firefox -fullscreen -kiosk -url http://monitor.example.net/check.html
+Restart=always
+RestartSec=30
+User=kiosk
+Group=kiosk
+
+[Install]
+WantedBy=graphical.target
+```
+
+
+
+### 7.) Enable the service and reboot
+
+```sh
+systemctl enable kiosk
+reboot
+```
+
+
+
+Troubleshouting
+---------------
+
+```
+systemctl disable pd-mapper.service
+apt purge cloud-init -y && apt autoremove --purge -y
+```

script/monitor/check.css

@@ -0,0 +1,77 @@
+html, body {
+    --background-theme-color: #001EA0;
+    --cell-space: 20px;
+    --logo-height: 50px;
+
+    background-color: #cccccc;
+    font-family: Verdana;
+    font-size: 14pt;
+    color: #ffffff;
+    height: 100%;
+    margin: 0;
+}
+@media screen and (orientation: portrait) {
+    body {
+        zoom: 200%
+    }
+}
+#header {
+    background-color: var(--background-theme-color); 
+    position: sticky;
+    top: 0;
+    height: calc(var(--logo-height) + (2 * var(--cell-space)));
+    width: 100%;
+}
+#header img {
+    height: var(--logo-height);
+    margin: var(--cell-space);
+    vertical-align: middle;
+}
+#header h1 {
+    display: inline;
+    font-weight: normal;
+    vertical-align: middle;
+}
+#content {
+    min-height: 100%;
+}
+#footer {
+    background-color: var(--background-theme-color);
+    position: sticky;
+    bottom: 0px;
+    padding: var(--cell-space);
+    text-align: center;
+    vertical-align: middle;
+    font-size: 22pt;
+}
+#checks {
+    display: grid;
+    grid-template-columns: repeat(auto-fit, minmax(350px, 1fr));
+    padding: var(--cell-space);
+    grid-gap: var(--cell-space);
+}
+#checks > div {
+    border: 1px solid black;
+    border-radius: 10px;
+    padding: 10px;
+    text-align: center;
+    box-shadow: 0 2px 5px 0 rgba(0, 0, 0, 0.3), 0 2px 10px 0 rgba(0, 0, 0, 0.2);
+}
+#checks > div.ok {
+    background-color: #66aa22;
+    color: #222222;	
+}
+#checks > div.info {
+    background-color: #88cc44;
+    color: #222222;	
+}
+#checks > div.warn {
+    background-color: #ffdd00;
+    color: #222222;	
+}
+#checks > div.fail {
+    background-color: #ff0000;
+}
+#checks > div.timeout {
+    background-color: var(--background-theme-color);
+}

script/monitor/check.html

@@ -0,0 +1,122 @@
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Monitoring Dashboard</title>
+    <link rel="stylesheet" href="check.css">
+</head>
+<body>
+    <div id="header">
+        <img src="logo.png"></img>
+        <h1>Monitoring</h1>
+    </div>
+    <div id="content">
+        <div id="checks">
+            <div class="check warn">Loading...</div>
+        </div>
+    </div>
+    <div id="footer">
+        Köln, <span id="datetime"></span>
+    </div>
+
+    <script>
+        var connectionAlive = true;
+
+        function downloadCheckFile(callback) {
+            var xmlHttp = new XMLHttpRequest();
+            xmlHttp.open('GET', "check.txt", false);
+            xmlHttp.onreadystatechange=function() {
+                if(xmlHttp.readyState==4) {
+                    callback(xmlHttp.responseText);
+                }
+            }
+            try {
+                xmlHttp.send(null);
+                if (xmlHttp.status >= 200 && xmlHttp.status < 304) {
+                    return true;
+                } else {
+                    return false;
+                }
+            } catch (e) {
+                return false;
+            }
+        }
+        function convertToHtml(checkText) {
+            if (!connectionAlive) {
+                return '<div class="fail">CONNECTION FAILED</div>';
+            }
+
+            var html = "";
+            var lines = checkText.split(/\n/);
+
+            for(var lineNo = 2; lineNo < lines.length; lineNo++) {
+                var line = lines[lineNo];
+
+                var parts = line.split('?');
+                if (parts.length > 1) {
+
+                    var name = parts[0].trim().split("_").join(" ");
+                    var resultParts = parts[1].trim().split('#');
+                    var result = resultParts[0];
+                    var message = resultParts[1];
+
+                    if(name == 'MISSED') {
+                        var fileTimeParts = message.split('-');
+                        var fileTime = new Date();
+                        fileTime.setHours(parseInt(fileTimeParts[0]));
+                        fileTime.setMinutes(parseInt(fileTimeParts[1]));
+                        fileTime.setSeconds(parseInt(fileTimeParts[2]));
+
+                        var scriptTime = new Date();
+                        scriptTime.setMinutes(scriptTime.getMinutes() - 2);
+                        if (scriptTime.getTime() < fileTime.getTime()){
+                            if (result == "0") {
+                                html += '<div class="ok">EVERYTHING OK<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                            } else {
+                                html += '<div class="fail">FAILED: ' + result + '<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                            }
+                        } else {
+                            html += '<div class="check fail">CHECKS TOO OLD<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                        }
+                    } else {
+                        if(result.indexOf('OK') >= 0) {
+                            html += '<div class="ok">'+ name;
+                        } else if(result.indexOf('INFO') >= 0) {
+                            html += '<div class="info">'+ name;
+                        } else if(result.indexOf('TIMEOUT') >= 0) {
+                            html += '<div class="timeout">'+ name;
+                        } else if(result.indexOf('WARN') >= 0) {
+                            html += '<div class="warn">'+ name;
+                        } else {
+                            html += '<div class="fail">' + name;
+                        }
+                        if(message) {
+                            html += '<br/>' + message.trim();
+                        }
+                        html += '</div>';
+                    }
+                }
+            }
+            return html;
+        }
+        function exchangeChecks(text) {
+            document.getElementById("checks").innerHTML = convertToHtml(text);
+        }
+        function refreshTime() {
+            document.getElementById("datetime").innerHTML = new Date().toLocaleString("de-DE", {timeZone: "Europe/Berlin"});
+        }
+        function refreshChecks() {
+            connectionAlive = downloadCheckFile(exchangeChecks);
+        }
+        function reloadIfAlive() {
+            if (connectionAlive) {
+                location.reload();
+            }
+        }
+        setInterval(refreshTime, 1000);
+        setInterval(refreshChecks, 5000);
+        setInterval(reloadIfAlive, 300000);
+        refreshTime();
+        refreshChecks();
+    </script>
+</body>
+</html>

script/monitor/check.sh

@@ -0,0 +1,88 @@
+#!/bin/bash
+
+# Folders always ends with an tailing '/'
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/script/monitor/*}/"               #Removes longest  matching pattern '/script/monitor/*' from the end
+_CORE_SCRIPTS="${_CIS_ROOT:?"Missing CIS_ROOT"}core/"
+_CURRENT_DOMAIN="$("${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}printOwnDomain.sh")"
+_DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_CURRENT_DOMAIN:?"Missing CURRENT_DOMAIN"}/"
+
+# Checks for the entire domain
+_DOMAIN_CHECKS="${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/checks/"
+
+
+
+function doChecks(){
+    local readonly _TMPDIR="${1:?"doChecks(): Missing parameter TMPDIR:"}"
+
+    local _DATETIME=$(date +%H-%M-%S)
+
+    mkdir -p ${_TMPDIR}
+    rm ${_TMPDIR}/* > /dev/null 2>&1
+
+    for check in ${_DOMAIN_CHECKS}*.on
+    do
+        local _CHECK_FILENAME="${check##*/}"
+        echo -n "${_CHECK_FILENAME%%.on}?" > "${_TMPDIR}/${_CHECK_FILENAME}"
+        timeout -k 10s 20s bash ${check} >> "${_TMPDIR}/${_CHECK_FILENAME}" 2> /dev/null || echo "TIMEOUT#Timeout" >> "${_TMPDIR}/${_CHECK_FILENAME}" &
+    done
+    wait
+
+    local _FAILED=0
+    echo "CHECK?RESULT[#MESSAGE]:"
+    echo "-----------------------"
+    for resultFile in ${_TMPDIR}/*
+    do
+        cat "${resultFile}"
+        grep -q "FAIL" ${resultFile} && _FAILED=$(expr ${_FAILED} + 1)
+    done
+    echo "MISSED?${_FAILED}#${_DATETIME}"
+
+    rm -r ${_TMPDIR} > /dev/null 2>&1
+    return 0
+}
+
+function usage(){
+    printf "\nUsage: /monitoring/check.sh <command> <options>"
+    echo
+    echo "possible commands:"
+    echo
+    echo "- all"
+    echo "    Executes all checks."
+    echo "- auto <out_file>"
+    echo "    Executes quiet all checks and saves the result in the given out_file."
+    echo "      (e.g. add the following line to crontab: '* * * * * /cis/script/monitor/check.sh auto /var/www/html/check.txt'"
+    echo "       to update the file '/var/www/html/check.txt' every minute as 'check.html' needs it.)"
+    return 0
+}
+
+main(){
+    case "${1:-""}" in
+        all)
+            echo "Checks werden ausgeführt..." \
+                && echo \
+                && doChecks "/tmp/checks" color \
+                && echo \
+                && echo "Success" \
+                && return 0
+            ;;
+        auto)
+            # If just a filename is given it is created in /tmp, because of 'cd /tmp' 
+            cd /tmp \
+                && doChecks "/tmp/checks$(date +%N)" > "$2.new" \
+                && mv -f "$2.new" "$2" \
+                && return 0
+            return 1
+            ;;
+		*)
+            [ "${1:+isset}" == "isset" ] \
+                && echo "Parameter '${1}' ist kein gültiger Befehl."
+            usage
+            return 0
+            ;;
+    esac
+
+    return 1
+}
+
+main "$@" || exit 1

script/monitor/checks/EXAMPLE_CHECK.off

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+_CHECK="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_CHECK%%/definitions/*}/"             #Removes longest  matching pattern '/definitions/*' from the end
+_GENERIC_CHECKS="${_CIS_ROOT:?"Missing CIS_ROOT"}script/monitor/generic/"
+
+${_GENERIC_CHECKS:?"Missing GENERIC_CHECKS"}OVERRIDDEN_DOMAIN_CHECK.sh "your-host.your-domain.net"

script/monitor/generic/NGINX_CHECK.sh

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 3 ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit 0 &> /dev/null \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function checkViaHTTP() {
+    _STATUS="$(curl -I http://${_REMOTE_HOSTNAME_FQDN} 2>/dev/null | head -n 1 | cut -d$' ' -f2)"
+    [ "${_STATUS}" == "200" ] \
+        && echo "OK" \
+        && return 0
+
+    return 1
+}
+
+function checkViaHTTPS() {
+    _STATUS="$(curl -k -I https://${_REMOTE_HOSTNAME_FQDN} 2>/dev/null | head -n 1 | cut -d$' ' -f2)"
+    [ "${_STATUS}" == "200" ] \
+        && echo "OK" \
+        && return 0
+
+    return 1
+}
+
+#grep:
+# -E                         Use regexp, '.*' => any chars between 'Active:' and '(running)', the round brackets are escaped.
+
+#cut:
+# -d                         Delimiter, marker where to cut (here ;)
+# -f                         Index of column to show (One based, so there is no -f0)
+function checkViaSSH() {
+    checkOrStartSSHMaster \
+        || return 1
+
+    _RESULT=$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'systemctl status nginx.service' | grep -E 'Active:.*\(running\)' | cut -d';' -f2)
+    ! [ -z "${_RESULT}" ] && echo "OK#UPTIME:${_RESULT}" || echo "FAIL"
+}
+
+#checkViaHTTP && exit 0
+#checkViaHTTPS && exit 0
+checkViaSSH && exit 0
+
+exit 1

script/monitor/generic/OVERRIDDEN_DOMAIN_CHECK.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 3 ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit 0 &> /dev/null \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function testDomain(){
+    checkOrStartSSHMaster \
+        || return 1
+
+    local _RESULT="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'bash /cis/core/printOwnDomain.sh' 2>&1 1>/dev/null)"
+
+    [ -z "${_RESULT}" ] \
+        && echo "OK" \
+        && return 0
+
+    echo "WARNING#Check hosts '/cis/core/printOwnDomain'"
+    return 0
+}
+
+testDomain && exit 0

script/monitor/generic/PING_CHECK.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+_SERVER="${1:?"FQDN of server missing"}"
+
+# -4                         Use IPv4
+# -W            SECONDS      Wait seconds for an answer
+# -c            COUNT_VALUE  Count of pings being executed
+_RESULT="$(ping -4 -W 1 -c 1 "${_SERVER}" | grep "time=" | cut -d'=' -f4)"
+! [ -z "${_RESULT}" ] && echo "OK#RTT: ${_RESULT}" || echo "FAIL#PLEASE USE FALLBACK!"

script/monitor/generic/URL_CHECK.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+_URL="${1:?"URL of site missing"}"
+
+#curl:
+# --connect-timeout SECONDS  Maximum time allowed for connection
+# -k                         Allow connections to SSL sites without certs (H)
+# -L                         Follow redirects (H)
+# --max-time        SECONDS  Maximum time allowed for the transfer
+# -s                         Silent mode. Don't output anything
+# --head                     Show head information only
+# --no-progress-meter        Clean output for grep
+
+#grep:
+# -q                         Quite, no output just status codes
+# -F                         Interpret search term as plain text
+((curl --connect-timeout 10 --max-time 10 -k -s --head --no-progress-meter "${_URL}" | grep -qF '200 OK') && echo OK) || echo FAIL

script/monitor/generic/ZFS_POOL_CHECK.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_ZFS_POOL="${2:?"Name of zfs pool missing: e.g. zpool1"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 3 ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit 0 &> /dev/null \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function testPool(){
+    checkOrStartSSHMaster \
+        || return 1
+
+    local _RESPONSE="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'zpool status ${_ZFS_POOL} | grep -F scrub')"
+    local _RESULT=$(echo "${_RESPONSE}" | grep -F 'scrub repaired 0B' | grep -F '0 errors')
+    _RESULT="${_RESULT#*on}"  #Removes shortest matching pattern '*on' from the begin
+
+    [ -z "${_RESULT}" ] \
+        && echo "FAIL#CHECK POOL: ${_ZFS_POOL}" \
+        && return 0
+
+    echo "OK#Scrubbed on ${_RESULT}."
+    return 0
+}
+
+testPool && exit 0
+
+exit 1

script/monitor/generic/ZFS_SYNC_CHECK.sh

@@ -0,0 +1,106 @@
+#!/bin/bash
+
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SCRIPT%%/script/monitor/*}/"             #Removes longest  matching pattern '/script/monitor/*' from the end
+_DOMAIN="$("${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh")"
+_COMPOSITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/compositions/"
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+# This is crucial:
+#  - default value for the filter part is extracted from the first parameter (FQDN)
+#  - but you can override this part to to adapt the test during a change of the domain.
+#    (e.g. the short hostname can be an option - or even a better default in the future)
+_ZFS_SNAPSHOT_FILTER="@SYNC_${2:-"${_REMOTE_HOSTNAME_FQDN:?"Missing REMOTE_HOSTNAME_FQDN"}"}"
+
+_MODE="${3:-"normal"}"
+_NOW_UTC_UNIXTIME=$(date -u +%s)
+_DEBUG_PATH="/tmp/monitor/"
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 3 ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit 0 &> /dev/null \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function checkSync() {
+    checkOrStartSSHMaster \
+        || return 1
+
+    [ "${_MODE}" == "debug" ] \
+        && mkdir -p "${_DEBUG_PATH}" > /dev/null \
+        && echo "Now: ${_NOW_UTC_UNIXTIME}" > ${_DEBUG_PATH}SECONDS_BEHIND_${_REMOTE_HOSTNAME_FQDN}.txt
+
+    ! [ -d "${_COMPOSITIONS:?"Missing COMPOSITIONS"}" ] \
+        && echo "WARN#no compositions" \
+        && return 0
+
+    [ "${_MODE}" == "debug" ] \
+        && echo "Snapshot filter: ${_ZFS_SNAPSHOT_FILTER}" >> ${_DEBUG_PATH}SECONDS_BEHIND_${_REMOTE_HOSTNAME_FQDN}.txt
+
+    # This retrieves the list of the interesting snapshots including creation timestamp
+    _SNAPSHOTS="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} zfs list -po creation,name -r -t snapshot zpool1/persistent | grep -F ${_ZFS_SNAPSHOT_FILTER})"
+    [ "${_MODE}" == "debug" ] \
+        && echo "${_SNAPSHOTS}" > ${_DEBUG_PATH}SNAPSHOTS_${_REMOTE_HOSTNAME_FQDN}.txt
+
+    [ -z "${_SNAPSHOTS}" ] \
+        && echo "FAIL#no snapshots" \
+        && return 1
+
+    echo "OK#Checks running"
+
+    for _COMPOSITION_PATH in ${_COMPOSITIONS}*; do
+
+        # If remote host is found than it is responsible for this container-composition, otherwise skip
+        #   (grep -E "^[[:blank:]]*something" means. Line has to start with "something", leading blank chars are ok.)
+        grep -E "^[[:blank:]]*${_REMOTE_HOSTNAME_SHORT}" "${_COMPOSITION_PATH}/zfssync-hosts" &> /dev/null \
+            || continue;
+
+        _COMPOSITION_NAME="${_COMPOSITION_PATH##*/}"   #Removes longest  matching pattern '*/' from the begin
+        _LAST_SNAPSHOT_UNIXTIME="$(echo "${_SNAPSHOTS}" | grep ${_COMPOSITION_NAME} | tail -n 1 | cut -d' ' -f1)"
+        _SECONDS_BEHIND=$[ ${_NOW_UTC_UNIXTIME} - ${_LAST_SNAPSHOT_UNIXTIME} ]
+
+        [ "${_MODE}" == "debug" ] \
+            && echo "${_LAST_SNAPSHOT_UNIXTIME} ${_COMPOSITION_NAME} on ${_REMOTE_HOSTNAME_FQDN} behind: ${_SECONDS_BEHIND}s" >> ${_DEBUG_PATH}SECONDS_BEHIND_${_REMOTE_HOSTNAME_FQDN}.txt
+
+        [ "${_SECONDS_BEHIND}" -lt 40 ] \
+            && continue
+
+        [ "${_SECONDS_BEHIND}" -lt 60 ] \
+            && echo "ZFSSYNC_of_${_REMOTE_HOSTNAME_SHORT}_LAGGING?WARN#${_COMPOSITION_NAME} ${_SECONDS_BEHIND}s" \
+            && continue
+
+        echo "ZFSSYNC_of_${_REMOTE_HOSTNAME_SHORT}_LAGGING?FAIL#${_COMPOSITION_NAME} ${_SECONDS_BEHIND}s"
+    done
+}
+
+
+
+RESULTS="$(checkSync)"
+
+[ "${_MODE}" == "debug" ] \
+    && echo "$RESULTS" > ${_DEBUG_PATH}RESULTS_${_REMOTE_HOSTNAME_FQDN}.txt
+
+echo "$RESULTS"

script/monitor/generic/ZFS_USAGE_CHECK.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 3 ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit 0 &> /dev/null \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function testSpace(){
+    checkOrStartSSHMaster \
+        || return 1
+
+    local _RESULT="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'zpool list -H -o capacity,name')"
+    local _SPACE_USED=$(echo "${_RESULT}" | /usr/bin/tail -n 1 | /usr/bin/cut -f1)
+    local _POOL=$(echo "${_RESULT}" | /usr/bin/tail -n 1 | /usr/bin/cut -f2)
+
+    [ -z "${_SPACE_USED}" ] \
+        && echo "FAIL#NO value" \
+        && return 0
+
+    [ "${1:?"Missing OK_THRESHOLD"}" -ge "${_SPACE_USED%\%*}" ] \
+        && echo "OK#${_SPACE_USED} used ${_POOL}." \
+        && return 0
+
+    [ "${2:?"Missing INFO_THRESHOLD"}" -ge "${_SPACE_USED%\%*}" ] \
+        && echo "INFO#${_SPACE_USED} already used ${_POOL}." \
+        && return 0
+
+    echo "FAIL#${_SPACE_USED} used ${_POOL}!"
+    return 0
+}
+
+testSpace 80 90 && exit 0
+
+exit 1

script/monitor/setupMonitoringHost.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && sudo "${0}" \
+    && exit 0
+
+
+
+_SETUP="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SETUP%%/script/monitor/*}/"             #Removes longest  matching pattern '/script/monitor/*' from the end
+_DOMAIN="$("${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh")"
+_DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/"
+
+
+
+function checkPreconditions() {
+    [ -d "${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/checks" ] \
+        && return 0
+
+    echo "No folder for your defined checks found: ${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/checks"
+    echo "Please create it and add all your custom monitoring checks there, following this convention: 'NAME_OF_THE_CHECK.on'"
+    echo "A check has to be switched 'on' to be executed, so you can rename a check to 'NAME_OF_THE_CHECK.off' and it will be ignored."
+    echo
+    echo "You can copy the file '/cis/script/monitor/checks/EXAMPLE_CHECK.off' to your check definitions folder and modify it."
+    return 1
+}
+
+
+
+function printSelectedDefinition() {
+    local _FILE_DEFINED_DOMAIN _FILE_DEFINED_DEFAULT
+    _FILE_DEFINED_DOMAIN="${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/${1:?"Missing CURRENT_FULLFILE"}"
+    _FILE_DEFINED_DEFAULT="${_CIS_ROOT:?"Missing CIS_ROOT"}script/monitor/${1:?"Missing CURRENT_FULLFILE"}"
+    readonly _FILE_DEFINED_DOMAIN _FILE_DEFINED_DEFAULT
+
+    [ -s "${_FILE_DEFINED_DOMAIN}" ] \
+        && echo "${_FILE_DEFINED_DOMAIN}" \
+        && return 0
+
+    [ -s "${_FILE_DEFINED_DEFAULT}" ] \
+        && echo "${_FILE_DEFINED_DEFAULT}" \
+        && return 0
+
+    return 1
+}
+
+function setupPublicFile() {
+    ! [ -d "/var/www/html" ] \
+        && echo "Missing folder '/var/www/html'. Is a webserver installed?" \
+        && return 1
+
+    [ -L "/var/www/html/${1:?"Missing filename"}" ] \
+        && [ "$(readlink -f /var/www/html/${1:?"Missing filename"})" == "$(printSelectedDefinition ${1:?"Missing filename"})" ] \
+        && echo "Link '/var/www/html/${1:?"Missing filename"}' already exists pointing to the expected file:" \
+        && echo "  - '$(readlink -f /var/www/html/${1:?"Missing filename"})'" \
+        && return 0
+
+    ln -f -s "$(printSelectedDefinition ${1:?"Missing filename"})" "/var/www/html/${1:?"Missing filename"}" \
+        && echo "Link '/var/www/html/${1:?"Missing filename"}' created successfully:" \
+        && echo "  - '$(readlink -f /var/www/html/${1:?"Missing filename"})'" \
+        && return 0
+}
+
+echo "Setup the monitoring host that monitors the others ... " \
+    && checkPreconditions \
+    && setupPublicFile "check.html" \
+    && setupPublicFile "check.css" \
+    && setupPublicFile "logo.png" \
+    && exit 0
+
+exit 1

script/monitor/setupServiceProvidingHost.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && sudo "${0}" \
+    && exit 0
+
+
+
+_SETUP="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SETUP%%/script/monitor/*}/"             #Removes longest  matching pattern '/script/monitor/*' from the end
+_CORE_SCRIPTS="${_CIS_ROOT:?"Missing CIS_ROOT"}core/"
+_DOMAIN="$("${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh")"
+_DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/"
+
+
+
+echo "Setup the user and permission to enable the monitoring this host ... " \
+    && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addNormalUser.sh" monitoring \
+    && echo \
+    && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}defineAuthorizedKeysOfUser.sh" "${_DEFINITIONS}" monitoring \
+    && exit 0
+
+exit 1

setupCoreOntoThisHost.sh

@@ -6,6 +6,12 @@
 
 
 
+# Folders always ends with an tailing '/'
+_SETUP="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SETUP%/setupCoreOntoThisHost.sh}/"             #Removes shortest matching pattern '/setupCoreOntoThisHost.sh' from the end
+_CORE_SCRIPTS="${_CIS_ROOT:?"Missing CIS_ROOT"}core/"
+
+
 
 function checkPathsAreAvaiable() {
     grep --version &> /dev/null \
@@ -31,22 +37,21 @@ function checkGitIsAvailable() {
 }
 
 function checkPreconditions() {
-    local _ROOT _DOMAIN
+    local _DOMAIN
-    _ROOT="${1:?"Missing parameter ROOT"}"
+    _DOMAIN="${1}" # Optional parameter DOMAIN
-    _DOMAIN="${2}" # Optional parameter DOMAIN
+    readonly _DOMAIN
-    readonly _ROOT _DOMAIN
 
     ! [ -z "${_DOMAIN}" ] \
         && [ "$(hostname -d)" != "${_DOMAIN}" ] \
         && echo \
-        && echo "WARNING: system-domain DOES NOT MATCH domainOfHostOwner: '$(hostname -d)' != '${_DOMAIN}'" \
+        && echo "WARNING: system-domain DOES NOT MATCH overrideOwnDomain: '$(hostname -d)' != '${_DOMAIN}'" \
         && echo
 
     # Given domain verfügbar (nicht leer)
     ! [ -z "${_DOMAIN}" ] \
         && checkPathsAreAvaiable \
         && checkGitIsAvailable \
-        && git -C "${_ROOT}" pull &> /dev/null \
+        && git -C "${_CIS_ROOT:?"Missing CIS_ROOT"}" pull &> /dev/null \
         && return 0
 
     echo
@@ -69,63 +74,65 @@ function checkPreconditions() {
 }
 
 function getOrSetDomain() {
-    local _ROOT _DOMAIN_FILE _GIVEN_DOMAIN
+    local _CURRENT_DOMAIN _GIVEN_DOMAIN _OVERRIDE_DOMAIN_FILE
-    _ROOT="${1:?"Missing parameter ROOT"}"
+    _CURRENT_DOMAIN="$("${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}printOwnDomain.sh")"
-    _DOMAIN_FILE="${_ROOT:?"Missing ROOT"}domainOfHostOwner"
+    _GIVEN_DOMAIN="${1}" # Optional parameter DOMAIN
-    _GIVEN_DOMAIN="${2}" # Optional parameter DOMAIN
+    _OVERRIDE_DOMAIN_FILE="${_CIS_ROOT:?"Missing CIS_ROOT"}overrideOwnDomain"
-    readonly _ROOT _DOMAIN_FILE _GIVEN_DOMAIN
+    readonly _CURRENT_DOMAIN _GIVEN_DOMAIN _OVERRIDE_DOMAIN_FILE
 
-    # Wenn DOMAIN_FILE enhält lesbare Daten
+    ! [ -z "${_CURRENT_DOMAIN}" ] \
-    grep '[^[:space:]]' "${_DOMAIN_FILE:?"Missing DOMAIN_FILE"}" &> /dev/null \
+        && [ -z "${_GIVEN_DOMAIN}" ] \
-        && cat "${_DOMAIN_FILE}" \
+        && echo "${_CURRENT_DOMAIN}" \
         && return 0
 
-    # Der boot-hostname muss mindestens einen Punkt enthalten, dann wird die hintere Hälfte als Domain genommen
+    ! [ -z "${_CURRENT_DOMAIN}" ] \
-    hostname -b | grep "\." | cut -d. -f2- > "${_DOMAIN_FILE}"
+        && [ "${_CURRENT_DOMAIN}" == "${_GIVEN_DOMAIN}" ] \
-    grep '[^[:space:]]' "${_DOMAIN_FILE}" &> /dev/null \
+        && echo "${_CURRENT_DOMAIN}" \
-        && cat "${_DOMAIN_FILE}" \
         && return 0
 
-    # Given domain is set (nicht leer)
+    # If there is a given domain it will be set or it will override the current one
-    ! [ -z "${_GIVEN_DOMAIN}" ] \
+    [ -z "${_CURRENT_DOMAIN}" ] \
+        && ! [ -z "${_GIVEN_DOMAIN}" ] \
         && [ "$(id -u)" == "0" ] \
+        && echo "Setting hostname to: $(hostname -s).${_GIVEN_DOMAIN}" >&2 \
         && hostnamectl set-hostname "$(hostname -s).${_GIVEN_DOMAIN}" \
-        && hostname -b | grep "\." | cut -d. -f2- > "${_DOMAIN_FILE}" \
+        && echo "${_GIVEN_DOMAIN}" \
-        && grep '[^[:space:]]' "${_DOMAIN_FILE}" &> /dev/null \
+        && return 0
-        && cat "${_DOMAIN_FILE}" \
+
+    ! [ -z "${_GIVEN_DOMAIN}" ] \
+        && echo "Overwriting domain to: ${_GIVEN_DOMAIN}" >&2 \
+        && echo "${_GIVEN_DOMAIN}" > "${_OVERRIDE_DOMAIN_FILE}" \
+        && echo "${_GIVEN_DOMAIN}" \
         && return 0
 
     return 1
 }
 
 function getRemoteRepositoryPath() {
-    local _ROOT
+    _REPOSITORY="$(git -C "${_CIS_ROOT:?"Missing CIS_ROOT"}" config --get remote.origin.url 2> /dev/null | grep -i 'git@')"
-    _ROOT="${1:?"Missing parameter ROOT"}"
+    _PATH="${_REPOSITORY%/*}"                        #Removes shortest matching pattern '/*' from the end
-    readonly _ROOT
+    ! [ -z "${_PATH}" ] \
-
+        && echo "${_PATH}/" \
-    _RESULT="$(git -C "${_ROOT:?"Missing ROOT"}" remote show origin | grep -i 'fetch' | xargs -n 1 | grep -i 'ssh://')"
-    _RESULT="${_RESULT%/*}"                        #Removes shortest matching pattern '/*' from the end
-    ! [ -z "${_RESULT}" ] \
-        && echo "${_RESULT}" \
         && return 0
 
     return 1
 }
 
 function addDefinition(){
-    local _ROOT _CORE_SCRIPTS _DEFINITIONS _REPOSITORY
+    local _DEFINITIONS _REPOSITORY
-    _DEFINITIONS="${1:?"Missing parameter DEFINITIONS"}"
+    _DEFINITIONS="${1:?"Missing first parameter DEFINITIONS"}"
-    _REPOSITORY="${2:?"Missing parameter REPOSITORY"}"
+    _REPOSITORY="$(getRemoteRepositoryPath)cis-definition-${2:?"Missing second parameter DOMAIN"}.git"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    readonly _DEFINITIONS _REPOSITORY
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
+
-    readonly _ROOT _CORE_SCRIPTS _DEFINITIONS _REPOSITORY
     [ "$(id -u)" == "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" "${_REPOSITORY}" readonly \
+        && echo "Running setup as 'root' trying to add definition repository:" \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" readonly "${_REPOSITORY}" \
         && echo "  - definitions are usable for this host." \
         && return 0
 
     [ "$(id -u)" != "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" "${_REPOSITORY}" writable \
+        && echo "Running setup as 'user' trying to add definition repository:" \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" writable "${_REPOSITORY}" \
         && echo "  - definitions are usable, as working copy." \
         && return 0
 
@@ -133,20 +140,22 @@ function addDefinition(){
 }
 
 function addState() {
-    local _ROOT _CORE_SCRIPTS _STATES _REPOSITORY
+    local _STATES _REPOSITORY
-    _STATES="${1:?"Missing parameter STATES"}"
+    _STATES="${1:?"Missing first parameter STATES"}"
-    _REPOSITORY="${2:?"Missing parameter REPOSITORY"}"
+    _REPOSITORY="$(getRemoteRepositoryPath)cis-state-${2:?"Missing second parameter DOMAIN"}.git"
-    _ROOT="${_STATES%%/states/*}/"             #Removes longest  matching pattern '/states/*' from the end
+    readonly _STATES _REPOSITORY
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
-    readonly _ROOT _CORE_SCRIPTS _STATES _REPOSITORY
 
     [ "$(id -u)" == "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" "${_REPOSITORY}" writable \
+        && echo "Running setup as 'root' trying to add state repository:" \
+        && echo \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" writable "${_REPOSITORY}" \
         && echo "  - states are usable for this host." \
         && return 0
 
     [ "$(id -u)" != "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" "${_REPOSITORY}" writable \
+        && echo "Running setup as 'user' trying to add state repository:" \
+        && echo \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" writable "${_REPOSITORY}" \
         && echo "  - states are usable, as working copy." \
         && return 0
 
@@ -154,13 +163,10 @@ function addState() {
 }
 
 function setupCoreFunctionality() {
-    local _ROOT _CORE_SCRIPTS _DEFINITIONS _MINUTE_FROM_OWN_IP _SETUP
+    local _DEFINITIONS _MINUTE_FROM_OWN_IP
     _DEFINITIONS="${1:?"Missing DEFINITIONS: 'ROOT/definitions/DOMAIN'"}"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    _MINUTE_FROM_OWN_IP="$(hostname -I | xargs -n 1 | grep -F '.' | head -n 1 | cut -d. -f4 || echo 0)" #uses last value from first own ipv4 or 0 as minute value
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
+    readonly _DEFINITIONS _MINUTE_FROM_OWN_IP
-    _MINUTE_FROM_OWN_IP="$(hostname -I | xargs -n 1 | grep -F . | head -n 1 | cut -d. -f4 || echo 0)" #uses last value from first own ipv4 or 0 as minute value
-    _SETUP="${2:?"Missing SETUP"}"
-    readonly _ROOT _CORE_SCRIPTS _DEFINITIONS _MINUTE_FROM_OWN_IP _SETUP
 
     [ "$(id -u)" != "0" ] \
         && echo "Configuration of host skipped because of insufficient rights." \
@@ -176,45 +182,39 @@ function setupCoreFunctionality() {
         && echo \
         && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}ensureUsageOfDefinitions.sh" "${_DEFINITIONS}" /etc/sudoers.d/allow-jenkins-updateRepositories \
         && echo \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addToCrontabEveryHour.sh" "${_SETUP}" "${_MINUTE_FROM_OWN_IP}" \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addToCrontabEveryHour.sh" "${_SETUP:?"Missing SETUP"}" "${_MINUTE_FROM_OWN_IP}" \
         && return 0
 
     return 1
 }
 
 function setup() {
-    local _ROOT _DEFINITIONS _DEFINITIONS_REPOSITORY _DOMAIN _REPOSITORY_PATH _SETUP _STATES _STATES_REPOSITORY
+    local _DEFINITIONS _DOMAIN _STATES
-    _SETUP="$(readlink -f "${0}" 2> /dev/null)"
+    _DOMAIN="$(getOrSetDomain "${1}")"
-    _ROOT="$(dirname ${_SETUP:?"Missing SETUP"} 2> /dev/null || echo "/iss")/"
-    _DOMAIN="$(getOrSetDomain "${_ROOT:?"Missing ROOT"}" "${1}")"
-    _REPOSITORY_PATH="$(getRemoteRepositoryPath "${_ROOT:?"Missing ROOT"}")"
 
-    ! checkPreconditions "${_ROOT:?"Missing ROOT"}" "${_DOMAIN}" \
+    ! checkPreconditions "${_DOMAIN}" \
         && return 1
 
-    _DEFINITIONS="${_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
+    _DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
-    _DEFINITIONS_REPOSITORY="${_REPOSITORY_PATH:?"Missing REPOSITORY_PATH"}/iss-definition-${_DOMAIN:?"Missing DOMAIN"}.git"
+    _STATES="${_CIS_ROOT:?"Missing CIS_ROOT"}states/${_DOMAIN:?"Missing DOMAIN"}"
-    _STATES="${_ROOT:?"Missing ROOT"}states/${_DOMAIN:?"Missing DOMAIN"}"
+    readonly _DEFINITIONS _DOMAIN _STATES
-    _STATES_REPOSITORY="${_REPOSITORY_PATH:?"Missing REPOSITORY_PATH"}/iss-state-${_DOMAIN:?"Missing DOMAIN"}.git"
-    readonly _ROOT _DEFINITIONS _DEFINITIONS_REPOSITORY _DOMAIN _REPOSITORY_PATH _SETUP _STATES _STATES_REPOSITORY
 
     echo \
-        && echo "Running setup using repositories of: '${_REPOSITORY_PATH:?"Missing REPOSITORY_PATH"}' ..." \
+        && addDefinition "${_DEFINITIONS:?"Missing DEFINITIONS"}" "${_DOMAIN:?"Missing DOMAIN"}" \
         && echo \
-        && addDefinition "${_DEFINITIONS:?"Missing DEFINITIONS"}" "${_DEFINITIONS_REPOSITORY:?"Missing DEFINITIONS_REPOSITORY"}" \
+        && addState "${_STATES:?"Missing STATES"}" "${_DOMAIN:?"Missing DOMAIN"}" \
-        && echo \
-        && addState "${_STATES:?"Missing STATES"}" "${_STATES_REPOSITORY:?"Missing STATES_REPOSITORY"}" \
         && echo \
         && echo "Using definitions: '${_DEFINITIONS:?"Missing DEFINITIONS"}' ..." \
-        && setupCoreFunctionality "${_DEFINITIONS:?"Missing DEFINITIONS"}" "${_SETUP:?"Missing SETUP"}" \
+        && setupCoreFunctionality "${_DEFINITIONS:?"Missing DEFINITIONS"}" \
         && return 0
 
-    echo "FAIL: setup is incomplete:                        ("$(readlink -f ${0})")"
+    echo "FAIL: setup is incomplete:                         ("$(readlink -f ${0})")" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
 # sanitizes all parameters
-setup \
+setup "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
+    && exit 0
-    && exit 0 || exit 1
+
+exit 1

updateRepositories.sh

@@ -20,54 +20,61 @@
 
 
 function update_repositories() {
-    local _ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
+    local _CIS_ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
     _UPDATE_REPOSITORIES="$(readlink -f "${0}" 2> /dev/null)"
-    _MODE="${1:-"all"}"
+    _CIS_ROOT="${_UPDATE_REPOSITORIES%/updateRepositories.sh}/"               #Removes shortest matching pattern '/updateRepositories.sh' from the end
-    _ROOT="$(dirname ${_UPDATE_REPOSITORIES:?"Missing UPDATE_REPOSITORIES"} 2> /dev/null || echo "/iss")/"
+    _MODE="${1:-"--core"}"
-    _DOMAIN="$(cat ${_ROOT:?"Missing ROOT"}domainOfHostOwner)"
+    _DOMAIN="$(${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh)"
-    _DEFINITIONS="${_ROOT}definitions/${_DOMAIN:?"Missing DOMAIN from file: ${_ROOT}domainOfHostOwner"}/"
+    _DEFINITIONS="${_CIS_ROOT}definitions/${_DOMAIN:?"Missing DOMAIN from file: ${_CIS_ROOT}domainOfHostOwner"}/"
-    _STATES="${_ROOT}states/${_DOMAIN:?"Missing DOMAIN from file: ${_ROOT}domainOfHostOwner"}/"
+    _STATES="${_CIS_ROOT}states/${_DOMAIN:?"Missing DOMAIN from file: ${_CIS_ROOT}domainOfHostOwner"}/"
-    readonly _ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
+    readonly _CIS_ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
 
     [ "${_MODE}" == "--repair" ] \
-        && (git -C "${_ROOT}" reset --hard origin/master; \
+        && (git -C "${_CIS_ROOT}" reset --hard origin/main; \
-            git -C "${_DEFINITIONS}" reset --hard origin/master; \
+            git -C "${_DEFINITIONS}" reset --hard origin/main; \
-            git -C "${_STATES}" reset --hard origin/master; \
+            git -C "${_STATES}" reset --hard origin/main; \
             echo "Run repairs") \
         && return 0
 
     [ "${_MODE}" == "--test" ] \
-        && git -C "${_ROOT}" pull \
+        && git -C "${_CIS_ROOT}" pull \
         && git -C "${_DEFINITIONS}" pull \
         && git -C "${_STATES}" pull \
         && echo "Run in testMode successfully." \
         && return 0
 
     [ "${_MODE}" == "--scripts" ] \
-        && echo "Host $HOSTNAME updating scripts: ${_ROOT} ..." \
+        && printf "Host $HOSTNAME updating scripts: ${_CIS_ROOT} ... " \
-        && (git -C "${_ROOT}" pull &> /dev/null &) \
+        && (git -C "${_CIS_ROOT}" pull &> /dev/null) \
+        && echo "(done)" \
         && return 0
 
     [ "${_MODE}" == "--definitions" ] \
         && echo "Host ${HOSTNAME} updating definitions: ${_DEFINITIONS} ... " \
-        && (git -C "${_DEFINITIONS}" pull &> /dev/null &) \
+        && (git -C "${_DEFINITIONS}" pull &> /dev/null) \
+        && echo "(done)" \
         && return 0
 
     [ "${_MODE}" == "--states" ] \
         && echo "Host ${HOSTNAME} updating states: ${_STATES} ... " \
-        && (git -C "${_STATES}" pull &> /dev/null &) \
+        && (git -C "${_STATES}" pull &> /dev/null) \
+        && echo "(done)" \
         && return 0
 
-    echo "Host ${HOSTNAME} updating ${_MODE}:" \
+    [ "${_MODE}" == "--core" ] \
-        && echo "  - ${_ROOT}" \
+        && echo "Host ${HOSTNAME} updating core including scripts, definitions and states: ${_STATES} ... " \
-        && echo "  - ${_DEFINITIONS}" \
+        && (git -C "${_CIS_ROOT}" pull &> /dev/null) \
-        && echo "  - ${_STATES}"
+        && (git -C "${_DEFINITIONS}" pull &> /dev/null) \
-    git -C "${_ROOT}" pull &> /dev/null
+        && (git -C "${_STATES}" pull &> /dev/null) \
-    git -C "${_DEFINITIONS}" pull &> /dev/null
+        && echo "(done)" \
-    git -C "${_STATES}" pull &> /dev/null
+        && return 0
+
+    echo "FAILED: an error occurred during an update."
+    return 1
 }
 
 # sanitizes all parameters
-update_repositories \
+update_repositories "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
+    && exit 0
-    && exit 0 || exit 1
+
+exit 1