normalized ssh key comments

.gitignore

@@ -0,0 +1,10 @@
+# Ignore the file '/overrideOwnDomain' because this is per host individually.
+/overrideOwnDomain
+
+# Ignore the subfolders only, because their content are other git repositories.
+# But 'definitions and 'states' should be prepared by cloning this repository.
+/definitions/*/
+/states/*/
+
+# Ignore environment files
+.env

README.md

@@ -1,35 +1,109 @@
-Infrastructure System (ISS)
+Core Infrastructure System (CIS)
-===========================
+================================
 
-Setup a new host
+The main idea is to use git to keep scripts, definitions and state in sync across all hosts.  
-----------------
+Currently an operating instance uses one repository for this core functionality and scripts,  
+another to distibute the definitions and a third one to share the state.
 
-### Preconditions
+If a script or a definition has to be changed an independent working copy is needed to push the adaptions.  
-To deploy the system you have to clone this repository to the host as root user.
+States can be changed by a host itself. Then we need a mechanism that informs all hosts to execute a `git pull`.
-Therefore you have to register the SSH public key of that root user as deploy key to allow readonly access to this repository.
+
-We use the modern ed25519 keys, so the public key of root is stored at this location:
+We use a Git server as syncronisation point and use a web hook to send the notification.  
+Because the should not be an agent to be installed on each host, we use jenkins to execute an update script via ssh.
+
+This allows us to use standard software without having to program something that may contain a security problem.
+
+
+
+Setup the first or a new host
+-----------------------------
+
+1. Update the host and ensure git is installed
+2. Set the long hostname (fqdn)
+3. Create ssh keys for user root (ssh key type ed25519)
+
+You can use this script to do so: [prepareThisHostBeforeCloning.sh](./prepareThisHostBeforeCloning.sh)
+
+
+
+### Ensure the existence of the repositories for your definitions and the state
+
+This should be necessary just if you set up the first host.  
+You can use the following scripts to assist the process:
+
+- [prepareDefinitionsRepository.sh](./prepareDefinitionsRepository.sh)
+- [prepareStatesRepository.sh](./prepareStatesRepository.sh)
+
+
+
+### Register the public ssh key of user root
+
+This is an example for `example.net` as domain of the host.
+
+1. __Scripts:__  
+   The public ssh key of the root user must be registered as a deploy key for the this repository,  
+   which grants __readonly access__.
+
+   A root user of a host should only be able to update the local cloned repository (`cis`) to a new version via `git pull`.
+
+2. __Definitions:__  
+   The public ssh key of the root user must be registered as a deploy key for the definitions repository,  
+   which grants __readonly access__.  
+
+   User root should only be able to update the local cloned repository (`cis-definition-example.net`) to a new version via `git pull`.
+
+3. __States:__  
+   The public ssh key of the root user must be registered as a deploy key for the states repository,  
+   which grants __write access__.  
+
+   User root should be able to push new state to the cloned repository (`cis-state-example.net`) via `git push`.
+
+
+
+### Clone the Infrastructure System (cis) repository and complete the setup
+After you registered the printed root's public key of this host you can clone the repository and execute the setup script:
+```sh
+# Note the tailing '/cis', because we want to clone the repository to that folder
+git clone ssh://git@git.example.dev:22448/cis.git /cis
+
+# Execute the setup script
+/cis/setupCoreOntoThisHost.sh
+```
+
+<br>
+<br>
+<br>
+
+
+
+Setup a new host step by step manually
+--------------------------------------
+
+To deploy cis you have to clone this repository to the host as root user.
+Therefore you have to set the correct long hostname (fqdn) create a pair of ssh keys (key type ed25519) for user root  
+and register the SSH public key of root as __deploy key__ to allow readonly access to this repository:
 
 1. First become root:
    ```sh
    sudo -i
    ```
 
-2. Set the long hostname:
+2. Update Ubuntu:
-   ```sh
-   hostnamectl set-hostname "the-new-unique-long-hostname (fqdn, eg.: host1.example.net)"
-   ```
-
-3. Update Ubuntu:
    ```sh
    # DO NOT SKIP THIS STEP
    apt update; apt upgrade -y
    ```
 
-4. Install git if needed:
+3. Install git if needed:
    ```sh
    git --version > /dev/null || apt install git
    ```
 
+4. Set the long hostname:
+   ```sh
+   hostnamectl set-hostname "the-new-unique-long-hostname (fqdn, eg.: host1.example.net)"
+   ```
+
 5. If not exist generate the ssh key pair and print the public key of the user root:
    ```sh
    # -t    type of the key pair
@@ -41,40 +115,17 @@ We use the modern ed25519 keys, so the public key of root is stored at this loca
        || (ssh-keygen \
            -t ed25519 \
            -f "/root/.ssh/id_ed25519" -q -N "" \
-           -C "$(date +%Y%m%d):root@$(hostname -b)" \
+           -C "$(date +%Y%m%d)-root@$(hostname -b)" \
        && cat "/root/.ssh/id_ed25519.pub")
    ```
 
-   This key has to be registerd via gitea web ui as deploy key into the repositories as documented in chapter "Register public host key".
+   This key has to be registerd via gitea web ui as deploy key into this repository.
 
 
 
-### Register public host key
-This is an example for `example.net` as domain of the host owner.
-
-1. Repository `iss`, allow __readonly__ access only.
-2. Repository `iss-definition-example.net`, allow __readonly__ access only.
-3. Repository `iss-state-example.net`, allow __writable__ access.
-
-
-
-### Clone the Infrastructure System (iss) repository
-After you registered the printed root's public key of this host you can clone the repository and execute the setup script:
-```sh
-# Note the tailing '/iss', because we want to clone the repository to that folder
-git clone ssh://git@git.example.dev:22448/iss.git /iss
-
-# Execute the setup script
-/iss/setupCoreOntoThisHost.sh
-```
-
-<br>
-<br>
-<br>
-
 How it works
 ------------
-We add a webhook to each gitea repository that belongs to ISS:
+We add a webhook to each gitea repository that belongs to CIS:
  - __Taget URL:__   https://YOUR.JENKINS.DOMAIN/generic-webhook-trigger/invoke?token=YOUR_TOKEN
  - __HTTP-Method:__ POST
  - __Trigger On:__  Push Events
@@ -94,11 +145,11 @@ cat "${JENKINS_HOME}/.ssh/id_ed25519.pub" \
     || (ssh-keygen \
         -t ed25519 \
         -f "${JENKINS_HOME}/.ssh/id_ed25519" -q -N "" \
-        -C "$(date +%Y%m%d):$(whoami)@$(echo ${JENKINS_URL} | cut -d/ -f3)" \
+        -C "$(date +%Y%m%d)-$(whoami)@$(echo ${JENKINS_URL} | cut -d/ -f3)" \
     && cat "${JENKINS_HOME}/.ssh/id_ed25519.pub")
 
 # add your host here, note the tailing '&' to run it in parallel
-ssh -o StrictHostKeyChecking=no jenkins@192.168.X.Y /iss/update_repositories.sh ( --scripts | --definitions | --states ) &
+ssh -o StrictHostKeyChecking=no jenkins@192.168.X.Y /cis/updateRepositories.sh ( --scripts | --definitions | --states ) &
 
 #wait for all background processes to complete
 wait

core/addAndCheckGitRepository.sh

@@ -6,10 +6,10 @@
 
 
 function checkPermissions(){
-    local _FOLDER _REPOSITORY
+    local _FOLDER _RIGHTS
     _FOLDER="${1:?"Missing first parameter FOLDER"}"
     _RIGHTS="${2:?"Missing second parameter RIGHTS"}"
-    readonly _FOLDER _REPOSITORY
+    readonly _FOLDER _RIGHTS
 
     [ "${_RIGHTS}" == "readonly" ] \
         && [ -d "${_FOLDER}/.git" ] \
@@ -21,30 +21,9 @@ function checkPermissions(){
         && git -C "${_FOLDER}" push --dry-run &> /dev/null \
         && return 0
 
-    echo "FAIL: The rights of the repository are incorrect:  ("$(readlink -f ${0})")"
+    echo "FAIL: The rights of the repository are incorrect:  ("$(readlink -f ${0})")" >&2
-    echo "  - '${_FOLDER}' is not '${_RIGHTS}'"
+    echo "  - '${_FOLDER}' is not '${_RIGHTS}'" >&2
-    echo "  - check the settings of gitea."
+    echo "  - check the settings of gitea." >&2
-    return 1
-}
-
-function checkRemoteRepository() {
-    local _FOLDER _REPOSITORY
-    _FOLDER="${1:?"Missing first parameter FOLDER"}"
-    _REPOSITORY="${2:?"Missing second parameter REPOSITORY"}"
-    readonly _FOLDER _REPOSITORY
-
-    #Should exist after successful clone only, therefore the remote repository exists and was accessible.
-    [ -d "${_FOLDER}/.git" ] \
-        && return 0
-
-    #Checks if repository exists and is accessible.
-    ! [ -d "${_FOLDER}/.git" ] \
-        && git ls-remote "${_REPOSITORY}" \
-        && return 0
-
-    echo "FAIL: The remote repository is not accessible:     ("$(readlink -f ${0})")"
-    echo "  - '${_REPOSITORY}'"
-    echo "  - check the settings of gitea."
     return 1
 }
 
@@ -54,42 +33,67 @@ function cloneOrPull {
     _REPOSITORY="${2:?"Missing second parameter REPOSITORY"}"
     readonly _FOLDER _REPOSITORY
 
-    ! [ -d "${_FOLDER}/.git" ] \
-        && git clone "${_REPOSITORY}" "${_FOLDER}" &> /dev/null \
-        && return 0
-
     [ -d "${_FOLDER}/.git" ] \
         && git -C "${_FOLDER}" pull &> /dev/null \
         && return 0
 
-    echo "FAIL: The local repository is not updatable:       ("$(readlink -f ${0})")"
+    ! [ -d "${_FOLDER}/.git" ] \
-    echo "  - '${_FOLDER}'"
+        && git clone "${_REPOSITORY}" "${_FOLDER}" &> /dev/null \
-    echo "  - check your network and the permissions in gitea."
+        && return 0
+
+    echo "FAIL: The local repository is not updatable:       ("$(readlink -f ${0})")" >&2
+    echo "  - '${_FOLDER}'" >&2
+    echo "  - check your network and the permissions in gitea." >&2
+    return 1
+}
+
+function printRepository(){
+    local _FOLDER _CONFIGURED_REPOSITORY _SUGGESTED_REPOSITORY
+    _FOLDER="${1:?"Missing first parameter FOLDER"}"
+    _CONFIGURED_REPOSITORY="$(git -C "${_FOLDER:?"Missing FOLDER"}" config --get remote.origin.url 2> /dev/null)"
+    _SUGGESTED_REPOSITORY="${2}"
+    readonly _FOLDER _CONFIGURED_REPOSITORY _SUGGESTED_REPOSITORY
+
+    ! [ -z "${_CONFIGURED_REPOSITORY}" ] \
+        && echo "${_CONFIGURED_REPOSITORY}" \
+        && return 0
+
+    while true; do
+        read -e -p "Enter ssh URL to clone Repository: " -i "${_SUGGESTED_REPOSITORY}" _REPOSITORY
+        echo "${_REPOSITORY}" | grep -F 'git@' &> /dev/null \
+            && git ls-remote "${_REPOSITORY}" &> /dev/null \
+            && echo "${_REPOSITORY:?"Missing REPOSITORY: e.g. ssh://git@your.domain.com/cis.git"}" \
+            && return 0
+    done
+
+    echo "FAIL: The remote repository is not accessible:     ("$(readlink -f ${0})")" >&2
+    echo "  - '${_REPOSITORY}'" >&2
+    echo "  - check the settings of gitea." >&2
     return 1
 }
 
 # Note that an unprivileged user can use this script successfully,
 # if no user has to be added to the host because it already exists.
 function addAndCheckGitRepository() {
-    local _FOLDER _REPOSITORY
+    local _FOLDER _REPOSITORY _RIGHTS
     _FOLDER="${1:?"Missing first parameter FOLDER"}"
-    _REPOSITORY="${2:?"Missing second parameter REPOSITORY: e.g. ssh://git@your.domain.com/iss.git "}"
+    _RIGHTS="${2:?"Missing second parameter RIGHTS: (readonly, writable) "}"
-    _RIGHTS="${3:?"Missing third parameter RIGHTS: (readonly, writable) "}"
+    _REPOSITORY="$(printRepository "${_FOLDER}" "${3}")"
-    readonly _FOLDER _REPOSITORY
+    readonly _FOLDER _REPOSITORY _RIGHTS
 
-    checkRemoteRepository "${_FOLDER}" "${_REPOSITORY}" \
+    echo \
-        && cloneOrPull "${_FOLDER}" "${_REPOSITORY}" \
+        && cloneOrPull "${_FOLDER}" "${_REPOSITORY:?"Missing REPOSITORY: e.g. ssh://git@your.domain.com/cis.git"}" \
         && checkPermissions "${_FOLDER}" "${_RIGHTS}" \
         && echo "SUCCESS: The git repository is usable.             ("$(readlink -f ${0})")" \
         && echo "  - remote repository: '${_REPOSITORY}'" \
         && echo "  - local repository:  '${_FOLDER}' (${_RIGHTS})" \
         && return 0
 
-    echo "FAIL: The repository is not functional:            ("$(readlink -f ${0})")"
+    echo "FAIL: The repository is not functional:            ("$(readlink -f ${0})")" >&2
-    echo "  - remote repository: '${_REPOSITORY}'"
+    echo "  - remote repository: '${_REPOSITORY}'" >&2
-    echo "  - local repository:  '${_FOLDER}'"
+    echo "  - local repository:  '${_FOLDER}'" >&2
-    echo "  - due to an error or insufficient rights or"
+    echo "  - due to an error or insufficient rights or" >&2
-    echo "  - one check failed."
+    echo "  - one check failed." >&2
     return 1
 }
 
@@ -98,4 +102,6 @@ addAndCheckGitRepository \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${3} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/addNormalUser.sh

@@ -18,6 +18,8 @@ function addNormalUser() {
         && echo "  - '${_USER}'" \
         && return 0
 
+    # useradd is a low level utility ... use adduser(8) instead.
+    # See: https://askubuntu.com/questions/345974/what-is-the-difference-between-adduser-and-useradd
     [ "$(id -u)" == "0" ] \
         && adduser --gecos 'Normal user' --disabled-password "${_USER}" \
         && chown -R "${_USER}:${_USER}" "/home/${_USER}" \
@@ -27,13 +29,14 @@ function addNormalUser() {
         && echo "  - existing home directories were taken over" \
         && return 0
 
-    echo "FAIL: The user could not be created:               ("$(readlink -f ${0})")"
+    echo "FAIL: The user could not be created:               ("$(readlink -f ${0})")" >&2
-    echo "  - '${_USER}'"
+    echo "  - '${_USER}'" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
 # sanitizes all parameters
-addNormalUser \
+addNormalUser "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
+    && exit 0
-    && exit 0 || exit 1
+
+exit 1

core/addToCrontabEveryHour.sh

@@ -5,14 +5,17 @@
 
 
 
+# Folders always ends with an tailing '/'
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/core/*}/"               #Removes longest  matching pattern '/core/*' from the end
+
 # Note that an unprivileged user can use this script successfully,
 # if no user has to be added to the host because it already exists.
 function addToCrontabEveryHour() {
-    local _ROOT _MINUTE_VALUE _STRING
+    local _MINUTE_VALUE _STRING
-    _ROOT="${0%%/core/*}/"                              #Removes longest  matching pattern '/core/*' from the end
     ! [ -z "${2##*[!0-9]*}" ] && _MINUTE_VALUE=$((${2}%60)) # if second parameter is integer then (minute-value % 60) as safe guard
     _STRING="${_MINUTE_VALUE:?"Missing MINUTE_VALUE"} * * * * ${1:?"Missing first parameter COMMAND"} > /dev/null 2>&1"
-    readonly _ROOT _MINUTE_VALUE _STRING
+    readonly _MINUTE_VALUE _STRING
 
     [ "$(id -u)" == "0" ] \
         && crontab -l | grep -qF "${_STRING:?"Missing CRON_STRING"}" \
@@ -21,11 +24,11 @@ function addToCrontabEveryHour() {
         && return 0
 
     [ "$(id -u)" == "0" ] \
-        && echo "${_ROOT:?"Missing ROOT"}" | grep "home" &> /dev/null \
+        && echo "${_CIS_ROOT:?"Missing CIS_ROOT"}" | grep -F 'home' &> /dev/null \
         && echo "SUCCESS: Although the entry will be skipped:       ("$(readlink -f ${0})")" \
         && echo "  - '${_STRING}'" \
         && echo "  that is because the current environment is:" \
-        && echo "    - ${_ROOT}" \
+        && echo "    - ${_CIS_ROOT}" \
         && return 0
 
     [ "$(id -u)" == "0" ] \
@@ -37,9 +40,9 @@ function addToCrontabEveryHour() {
         && echo "  - '${_STRING}'" \
         && return 0
 
-    echo "FAIL: Entry could not be registered to crontab:    ("$(readlink -f ${0})")"
+    echo "FAIL: Entry could not be registered to crontab:    ("$(readlink -f ${0})")" >&2
-    echo "  - '${_STRING:?"Missing CRON_STRING"}'"
+    echo "  - '${_STRING:?"Missing CRON_STRING"}'" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
@@ -47,4 +50,6 @@ function addToCrontabEveryHour() {
 addToCrontabEveryHour \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/default/etc/sudoers.d/allow-jenkins-updateRepositories

@@ -0,0 +1,6 @@
+Cmnd_Alias C_JENKINS = \
+  /cis/updateRepositories.sh --core, \
+  /cis/updateRepositories.sh --scripts, \
+  /cis/updateRepositories.sh --definitions, \
+  /cis/updateRepositories.sh --states
+jenkins ALL = (root) NOPASSWD: C_JENKINS

core/defineAuthorizedKeysOfUser.sh

@@ -44,24 +44,24 @@ function prepareFolder() {
         && echo "  - '${_SSH_FOLDER}'" \
         && return 0
 
-    echo "FAIL: The ssh folder could not be prepared:        ("$(readlink -f ${0})")"
+    echo "FAIL: The ssh folder could not be prepared:        ("$(readlink -f ${0})")" >&2
-    echo "  - '${_SSH_FOLDER}'"
+    echo "  - '${_SSH_FOLDER}'" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
 function defineAuthorizedKeysOfUser() {
-    local _ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
+    local _CIS_ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
     _DEFINITIONS="$(realpath -s "${1:?"Missing first parameter DEFINITIONS: 'ROOT/definitions/DOMAIN'"}")"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    _CIS_ROOT="${_DEFINITIONS%%/definitions/*}/"  #Removes longest  matching pattern '/definitions/*' from the end
-    _DOMAIN="${_DEFINITIONS##*/definitions/}"  #Removes longest  matching pattern '*/definitions/' from the begin
+    _DOMAIN="${_DEFINITIONS##*/definitions/}"     #Removes longest  matching pattern '*/definitions/' from the begin
-    _DOMAIN="${_DOMAIN%/}"                     #Removes shortest matching pattern '/'              from the end
+    _DOMAIN="${_DOMAIN%/}"                        #Removes shortest matching pattern '/'              from the end
     #Build from components for safety
-    _DEFINITIONS="${_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
+    _DEFINITIONS="${_CIS_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
 
     _USER="${2:?"Missing second parameter USER"}"
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
+    _CORE_SCRIPTS="${_CIS_ROOT:?"Missing ROOT"}core/"
-    readonly _ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
+    readonly _CIS_ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
 
     case "${_USER:?"Missing USER"}" in
         root)
@@ -83,4 +83,6 @@ function defineAuthorizedKeysOfUser() {
 defineAuthorizedKeysOfUser \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/ensureUsageOfDefinitions.sh

@@ -12,46 +12,76 @@ function printIfEqual() {
 }
 
 function isCoreDefinition() {
-    echo "${1:?"Missing first parameter FILE"}" | grep "/root/.ssh/authorized_keys" &> /dev/null \
+    echo "${1:?"Missing first parameter FILE"}" | grep -F '/root/.ssh/authorized_keys' &> /dev/null \
         && return 0
 
-    echo "${1:?"Missing first parameter FILE"}" | grep "/home/jenkins/.ssh/authorized_keys" &> /dev/null \
+    echo "${1:?"Missing first parameter FILE"}" | grep -F '/home/jenkins/.ssh/authorized_keys' &> /dev/null \
         && return 0
 
-    echo "${1:?"Missing first parameter FILE"}" | grep "/etc/sudoers.d/allow-jenkins-updateRepositories" &> /dev/null \
+    echo "${1:?"Missing first parameter FILE"}" | grep -F '/etc/sudoers.d/allow-jenkins-updateRepositories' &> /dev/null \
         && return 0
 
     return 1
 }
 
+function filterInvalidAuthorizedKeysFilesOfRoot() {
+    local _FILE_DEFINED
+    _FILE_DEFINED="${1:?"Missing DEFINITION FILE"}"
+    readonly _FILE_DEFINED
+
+    #If the full filename contains 'root/.ssh/authorized_keys' then check the content.
+    #Skip lines starting with '#' and if at least one remaining line contains 'ssh' and '@' then print the filename. 
+    echo "${_FILE_DEFINED}" | grep -F 'root/.ssh/authorized_keys' &> /dev/null \
+        && grep -vE '^[[:blank:]]*#' "${_FILE_DEFINED}" | grep -F 'ssh' | grep -F '@' &> /dev/null \
+        && echo "${_FILE_DEFINED}" \
+        && return 0
+
+    #If the full filename contains 'root/.ssh/authorized_keys' print nothing because the file has to be invalid.
+    echo "${_FILE_DEFINED}" | grep -F 'root/.ssh/authorized_keys' &> /dev/null \
+        && echo \
+        && return 0
+
+    #Print the full filename because it does not contain 'root/.ssh/authorized_keys'
+    echo "${_FILE_DEFINED}"
+    return 0
+}
+
 function printSelectedDefinition() {
-    local _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
+    local _DEFINITIONS  _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
-    _CORE_FILE_DEFINED_ALL_HOSTS="${1:?"Missing DEFINITIONS"}/core/all${2:?"Missing CURRENT_FULLFILE"}"
+    _DEFINITIONS="${1:?"Missing CIS_ROOT"}definitions/${2:?"Missing DOMAIN"}/"
-    _CORE_FILE_DEFINED_THIS_HOST="${1:?"Missing DEFINITIONS"}/core/$(hostname -s)${2:?"Missing CURRENT_FULLFILE"}"
+    _CORE_DEFAULT_ALL_HOSTS="${1:?"Missing CIS_ROOT"}core/default${3:?"Missing CURRENT_FULLFILE"}"
-    _FILE_DEFINED_ALL_HOSTS="${1:?"Missing DEFINITIONS"}/hosts/all${2:?"Missing CURRENT_FULLFILE"}"
+    _CORE_FILE_DEFINED_ALL_HOSTS="${_DEFINITIONS:?"Missing DEFINITIONS"}core/all${3:?"Missing CURRENT_FULLFILE"}"
-    _FILE_DEFINED_THIS_HOST="${1:?"Missing DEFINITIONS"}/hosts/$(hostname -s)${2:?"Missing CURRENT_FULLFILE"}"
+    _CORE_FILE_DEFINED_THIS_HOST="${_DEFINITIONS:?"Missing DEFINITIONS"}core/$(hostname -s)${3:?"Missing CURRENT_FULLFILE"}"
-    readonly _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
+    _FILE_DEFINED_ALL_HOSTS="${_DEFINITIONS:?"Missing DEFINITIONS"}hosts/all${3:?"Missing CURRENT_FULLFILE"}"
+    _FILE_DEFINED_THIS_HOST="${_DEFINITIONS:?"Missing DEFINITIONS"}hosts/$(hostname -s)${3:?"Missing CURRENT_FULLFILE"}"
+    readonly _DEFINITIONS _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
 
     #The following are special definitions that affect the core functionality.
     #Try this host first because it should be priorized.
-    isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_CORE_FILE_DEFINED_THIS_HOST}" ] \
-        && echo "${_CORE_FILE_DEFINED_THIS_HOST}" \
+        && filterInvalidAuthorizedKeysFilesOfRoot "${_CORE_FILE_DEFINED_THIS_HOST}" \
         && return 0
 
     #The following are special definitions that affect the core functionality.
-    isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_CORE_FILE_DEFINED_ALL_HOSTS}" ] \
-        && echo "${_CORE_FILE_DEFINED_ALL_HOSTS}" \
+        && filterInvalidAuthorizedKeysFilesOfRoot "${_CORE_FILE_DEFINED_ALL_HOSTS}" \
+        && return 0
+
+    #The following are special definitions that affect the core functionality.
+    isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
+        && [ -s "${_CORE_DEFAULT_ALL_HOSTS}" ] \
+        && filterInvalidAuthorizedKeysFilesOfRoot "${_CORE_DEFAULT_ALL_HOSTS}" \
         && return 0
 
     #Try this host first because it should be priorized.
-    ! isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    ! isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_FILE_DEFINED_THIS_HOST}" ] \
         && echo "${_FILE_DEFINED_THIS_HOST}" \
         && return 0
 
-    ! isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    ! isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_FILE_DEFINED_ALL_HOSTS}" ] \
         && echo "${_FILE_DEFINED_ALL_HOSTS}" \
         && return 0
@@ -71,11 +101,6 @@ function createSymlinkToDefinition() {
         && [ "$(sha256sum "${_DEFINED_FULLFILE}" | cut -d' ' -f1)" == "$(sha256sum "${_CURRENT_FULLFILE}" | cut -d' ' -f1)" ] \
         && echo "The content of the current file already matches the definition, but it will be replaced by a symlink..."
 
-    [ -f "${_CURRENT_FULLFILE}" ] \
-        && [ "$(sha256sum "${_DEFINED_FULLFILE}" | cut -d' ' -f1)" == "$(sha256sum "${_CURRENT_FULLFILE}" | cut -d' ' -f1)" ] \
-        && echo "The content of the current file already matches the definition, but it will be replaced by a symlink..."
-
-
     [ -f "${_CURRENT_FULLFILE}" ] \
         && mv "${_CURRENT_FULLFILE:?"Missing CURRENT_FULLFILE"}" "${_SAVED_FULLFILE:?"Missing SAVED_FULLFILE"}" \
         && echo "Current file has been backed up to: '${_SAVED_FULLFILE}'"
@@ -92,17 +117,17 @@ function createSymlinkToDefinition() {
 }
 
 function ensureUsageOfDefinitions() {
-    local _ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
+    local _CIS_ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
-    _DEFINITIONS="$(realpath -s "${1:?"Missing first parameter DEFINITIONS: 'ROOT/definitions/DOMAIN'"}")"
+    _DEFINITIONS="$(realpath -s "${1:?"Missing first parameter DEFINITIONS: 'ROOT/definitions/DOMAIN'"}")/"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    _CIS_ROOT="${_DEFINITIONS%%/definitions/*}/"  #Removes longest  matching pattern '/definitions/*' from the end
-    _DOMAIN="${_DEFINITIONS##*/definitions/}"  #Removes longest  matching pattern '*/definitions/' from the begin
+    _DOMAIN="${_DEFINITIONS##*/definitions/}"     #Removes longest  matching pattern '*/definitions/' from the begin
-    _DOMAIN="${_DOMAIN%/}"                     #Removes shortest matching pattern '/'              from the end
+    _DOMAIN="${_DOMAIN%/}"                        #Removes shortest matching pattern '/'              from the end
     #Build from components for safety
-    _DEFINITIONS="$(printIfEqual "${_DEFINITIONS}" "${_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}")"
+    _DEFINITIONS="$(printIfEqual "${_DEFINITIONS}" "${_CIS_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/")"
 
 
-    _CURRENT_FOLDER="$(dirname "${2:?"Missing second parameter CURRENT_FULLFILE"}")"
+    _CURRENT_FULLFILE="${2:?"Missing second parameter CURRENT_FULLFILE"}"
-    _CURRENT_FOLDER="${_CURRENT_FOLDER%/}/"  #Removes shortest matching pattern '/' from the end
+    _CURRENT_FOLDER="${_CURRENT_FULLFILE%/*}/"  #Removes shortest matching pattern '/*' from the end
     ! [ -d "${_CURRENT_FOLDER}" ] \
         && echo "FAIL: The folder cannot be read:                   ("$(readlink -f ${0})")" \
         && echo "  - '${_CURRENT_FOLDER}'" \
@@ -118,10 +143,16 @@ function ensureUsageOfDefinitions() {
     _CURRENT_FULLFILE="${_CURRENT_FOLDER:?"Missing CURRENT_FOLDER"}${_CURRENT_FILE:?"Missing CURRENT_FILE"}"
 
 
-    _DEFINED_FULLFILE="$(printSelectedDefinition "${_DEFINITIONS}" "${_CURRENT_FULLFILE}")"
+    _DEFINED_FULLFILE="$(printSelectedDefinition "${_CIS_ROOT}" "${_DOMAIN}" "${_CURRENT_FULLFILE}")"
     _NOW="$(date +%Y%m%d_%H%M)"
-    _SAVED_FULLFILE="${_CURRENT_FULLFILE}-backup@${_NOW:?"Missing NOW"}"
+    _SAVED_FULLFILE="${_CURRENT_FULLFILE}.backup@${_NOW:?"Missing NOW"}"
-    readonly _ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
+    readonly _CIS_ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
+
+    [ -z "${_DEFINED_FULLFILE}" ] \
+        && echo \
+        && echo "URGENT WARNING: If an 'authorized_keys' file of root is replaced by an invalid version," \
+        && echo "                you may lose access to this host!" \
+        && echo
 
     ! [ -f "${_DEFINED_FULLFILE}" ] \
         && echo "FAIL: No definition available for this file:       ("$(readlink -f ${0})")" \
@@ -138,11 +169,11 @@ function ensureUsageOfDefinitions() {
         && echo "  - '${_DEFINED_FULLFILE}'" \
         && return 0
 
-    echo "${_ROOT:?"Missing ROOT"}" | grep "home" &> /dev/null \
+    echo "${_CIS_ROOT:?"Missing CIS_ROOT"}" | grep -F 'home' &> /dev/null \
         && echo "SUCCESS: Although this definition will be skipped: ("$(readlink -f ${0})")" \
         && echo "  - '${_DEFINED_FULLFILE}'" \
         && echo "  that is because the current environment is:" \
-        && echo "    - ${_ROOT}" \
+        && echo "    - ${_CIS_ROOT}" \
         && echo "  following file is in use:" \
         && echo "    - $(readlink -f "${_CURRENT_FULLFILE}")" \
         && return 0
@@ -165,8 +196,8 @@ function ensureUsageOfDefinitions() {
         && echo "- '${_DEFINED_FULLFILE}'" \
         && return 0
 
-    echo "FAIL: The definition could not be ensured:         ("$(readlink -f ${0})")"
+    echo "FAIL: The definition could not be ensured:         ("$(readlink -f ${0})")" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
@@ -174,4 +205,6 @@ function ensureUsageOfDefinitions() {
 ensureUsageOfDefinitions \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/printCisRoot.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/core/*}/"               #Removes longest  matching pattern '/core/*' from the end
+
+[ -d "${_CIS_ROOT}" ] \
+    && [ -d "${_CIS_ROOT}definitions/" ] \
+    && [ -d "${_CIS_ROOT}states/" ] \
+    && echo "${_CIS_ROOT}" \
+    && exit 0
+
+echo "FAIL: Unable to detect CIS_ROOT" >&2
+exit 1

core/printOwnDomain.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+
+#WARNING: Used for core functionality in setup.sh
+#         DO NOT rename the script and test changes well!
+
+
+
+# Folders always ends with an tailing '/'
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/core/*}/"               #Removes longest  matching pattern '/core/*' from the end
+_OVERRIDE_DOMAIN_FILE="${_CIS_ROOT:?"Missing CIS_ROOT"}overrideOwnDomain"
+
+# There has to be one dot at least.
+_BOOT_DOMAIN="$(hostname -b | grep -F '.' | cut -d. -f2-)"
+
+# Take OVERRIDING_DOMAIN_FILE without empty lines and comments, then take the first line without leading spaces
+_OVERRIDE_DOMAIN="$(grep -vE '^[[:space:]]*$|^[[:space:]]*#' "${_OVERRIDE_DOMAIN_FILE}" 2> /dev/null | head -n 1 | xargs)"
+
+! [ -z "${_OVERRIDE_DOMAIN}" ] \
+    && [ "${_OVERRIDE_DOMAIN}" != "${_BOOT_DOMAIN}" ] \
+    && echo "WARNING: Domain has been overridden by: ${_OVERRIDE_DOMAIN_FILE}" >&2 \
+    && echo "${_OVERRIDE_DOMAIN}" \
+    && exit 0
+
+! [ -z "${_BOOT_DOMAIN}" ] \
+    && echo "${_BOOT_DOMAIN}" \
+    && exit 0
+
+echo "It was impossible to find out the domain of this host, please prepare this host first." >&2
+exit 1

manual/README.md

@@ -0,0 +1,30 @@
+Manual
+======
+
+Each script `${FULL_SCRIPTNAME}.sh` has it's corresponding manual here:
+ - `manuals/${FULL_SCRIPTNAME}.sh.md`
+
+According to the rule above, this directory `manuals/core/*`  
+has the same structure as `../core/*` and contains all manuals explaining the core functionality.
+
+And this directory `manuals/script/*`  
+has the same structure as `../script/*` and contains all manuals explaining the purpose of each script.
+
+
+
+Core functionality
+------------------
+
+This set of scripts is the absolute minimum to provide the core functionality:
+ - `setupCoreOntoThisHost.sh` bootstraps an empty new host, after the scripts repository was cloned  
+   [Read this for further details](./setupCoreOntoThisHost.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/addAndCheckGitRepository.sh`](./core/addAndCheckGitRepository.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/addNormalUser.sh`](./core/addNormalUser.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/addToCrontabEveryHour.sh`](./core/addToCrontabEveryHour.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/defineAuthorizedKeysOfUser.sh`](./core/defineAuthorizedKeysOfUser.sh.md)
+ - `setupCoreOntoThisHost.sh` needs [`../core/ensureUsageOfDefinitions.sh`](./core/ensureUsageOfDefinitions.sh.md)
+
+
+
+The scripts
+-----------

manual/core/addNormalUser.sh.md

@@ -0,0 +1,8 @@
+[core/addNormalUser.sh](..//core/addNormalUser.sh)
+==================================================
+
+This script adds a normal user to the host on which it was called.
+
+__Parameters:__
+  1. USER (mandantory)
+          Name of the user who will be added to the host

manual/core/defineAuthorizedKeysOfUser.sh.md

@@ -0,0 +1,8 @@
+[core/defineAuthorizedKeysOfUser.sh](../core/defineAuthorizedKeysOfUser.sh)
+===========================================================================
+
+This script defines the `authorized_keys` file of the given user.
+It will create a link pointing to the file in definitions to distribute the same settings across all hosts.
+
+__Parameters:__
+  1. USER (mandantory) Name of the user who will receive the ssh settings.

manual/script/host/README.md

@@ -0,0 +1,9 @@
+script/host
+===========
+
+This folder contains scripts managing the host.  
+For example:
+  - certificates
+  - git
+  - ssh
+  - users

manual/script/host/user/README.md

@@ -0,0 +1,4 @@
+script/host/user
+================
+
+This folder contains scripts managing users of a host.

prepareDefinitionsRepository.sh

@@ -0,0 +1,69 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && echo "This script prepares the user 'root' of this host and the host itself," \
+    && echo "so this script is allowed to be executed if you are root only." \
+    && exit 1
+
+# There has to be one dot at least.
+_BOOT_DOMAIN="$(hostname -b | grep -F '.' | cut -d. -f2-)"
+
+[ -z "${_BOOT_DOMAIN}" ] \
+    && echo "It was impossible to find out the domain of this host, please prepare this host first." \
+    && exit 1
+
+_REOPSITORY_NAME="cis-definition-${_BOOT_DOMAIN}"
+
+
+
+#Generate file 'README.md'
+mkdir -p /tmp/skeleton/definition
+cat << EOF > /tmp/skeleton/definition/README.md
+This repository contains the definitions of the domain “$_BOOT_DOMAIN” by the Core Infrastructure System.
+EOF
+
+
+
+#Use current file 'authorized_keys' of root as definition
+mkdir -p /tmp/skeleton/definition/core/all/root/.ssh
+cp /root/.ssh/authorized_keys /tmp/skeleton/definition/core/all/root/.ssh/authorized_keys
+
+
+
+#Generate file 'authorized_keys' for user jenkins
+mkdir -p /tmp/skeleton/definition/core/all/home/jenkins/.ssh
+cat << EOF > /tmp/skeleton/definition/core/all/home/jenkins/.ssh/authorized_keys
+#------------------------------------------------------
+# Enter the public ssh key of your jenkins server here.
+#------------------------------------------------------
+EOF
+
+
+
+cat << EOF
+
+The first content for your repository for the definitions of the '$_BOOT_DOMAIN' domain has been created.
+
+Please create a definition repository.
+To follow the naming convention name it '$_REOPSITORY_NAME'
+
+Please DO NOT use the SSH key of root for this.
+Maybe you can use https and user password for pushing the first commit.
+
+Go to folder '/tmp/skeleton/definition' and check the content of all 'authorized_keys' files,
+correct them if required to prevent losing access to your hosts.
+
+The public ssh key of your jenkins server has to be added.
+
+Only now follow the instructions as our git server shows.
+For example:
+
+  cd /tmp/skeleton/definition
+  git init
+  git checkout -b main
+  git add .
+  git commit -m "first core definitions"
+  git remote add origin https://git.example.dev/[SOME_PATH/]$_REOPSITORY_NAME.git
+  git push -u origin main
+
+EOF

prepareStatesRepository.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && echo "This script prepares the user 'root' of this host and the host itself," \
+    && echo "so this script is allowed to be executed if you are root only." \
+    && exit 1
+
+# There has to be one dot at least.
+_BOOT_DOMAIN="$(hostname -b | grep -F '.' | cut -d. -f2-)"
+
+[ -z "${_BOOT_DOMAIN}" ] \
+    && echo "It was impossible to find out the domain of this host, please prepare this host first." \
+    && exit 1
+
+_REOPSITORY_NAME="cis-state-${_BOOT_DOMAIN}"
+
+
+
+#Generate README.md
+mkdir -p /tmp/skeleton/state
+cat << EOF > /tmp/skeleton/state/README.md
+This repository contains the states of the domain “$_BOOT_DOMAIN” by the Core Infrastructure System.
+EOF
+
+
+
+cat << EOF
+
+The first content for your repository for the state of the '$_BOOT_DOMAIN' domain has been created.
+
+Please create a states repository.
+To follow the naming convention name it '$_REOPSITORY_NAME'
+
+Please DO NOT use the SSH key of root for this.
+Maybe you can use https and user password for pushing the first commit.
+
+Then go to folder '/tmp/skeleton/state' and follow the instructions as your git server shows.
+For example:
+
+  cd /tmp/skeleton/state
+  git init
+  git checkout -b main
+  git add .
+  git commit -m "first state"
+  git remote add origin https://git.example.dev/[SOME_PATH/]$_REOPSITORY_NAME.git
+  git push -u origin main
+
+EOF

prepareThisHostBeforeCloning.sh

@@ -8,46 +8,66 @@
 function setNeededHostnameOrExit() {
     _FQDN="${1:?"Missing unique long hostname (fqdn, eg.: host1.example.net) for this host as first parameter."}"
 
-    echo "${_FQDN}" | grep '\.' &> /dev/null \
+    echo "${_FQDN}" | grep -F '.' &> /dev/null \
         && hostnamectl set-hostname "${_FQDN}" \
         && return 0
 
-    echo "FAILED: setting full qualified domain name, given value was:"
+    echo "FAILED: setting full qualified domain name does not contain a domain,"
-    echo "  - ${_FQDN}"
+    echo "        given value was: ${_FQDN}"
     exit 1
 }
 
-function prepare() {
+function prepareThisHost() {
     git --version > /dev/null || (apt update; apt upgrade -y; apt install git)
 
     echo
     echo "Public SSH-Key for root@$(hostname -b):"
+    cat "/root/.ssh/id_ed25519.pub" \
+        && return 0
+
     # -t    type of the key pair
     # -f    defines the filenames (we use the standard for the selected type here)
     # -q    quiet, no output or interaction
     # -N "" means the private key will not be secured by a passphrase
     # -C    defines a comment
+    ssh-keygen \
+        -t ed25519 \
+        -f "/root/.ssh/id_ed25519" -q -N "" \
+        -C "$(date +%Y%m%d)-root@$(hostname -b)"
+
     cat "/root/.ssh/id_ed25519.pub" \
-        || (ssh-keygen \
+        && return 0
-            -t ed25519 \
-            -f "/root/.ssh/id_ed25519" -q -N "" \
-            -C "$(date +%Y%m%d)-root@$(hostname -b)" \
-        && cat "/root/.ssh/id_ed25519.pub")
 
     echo
-    echo "Now you have to register the public ssh-key from above into your git-server to grant these access rights:"
+    echo "FAILED: somthing went wrong during the generation the ssh keys."
+    echo "        These keys are mandantory. You can try to restart this script."
+    echo
+}
+
+function showFurtherSteps() {}
+    echo
+    echo "IMPORTANT: It is assumed that repositories for definitions and states already exist"
+    echo "           and comply with the naming convention."
+    echo "  Otherwise, these repositories must be created first!"
+    echo
+    echo "To grant the correct access rights, you have to register the above-mentioned ssh key,"
+    echo "as deploy key in these repositories of the Git server:"
     echo "  - scripts repository     (allow readonly access only),"
     echo "  - definitions repository (allow readonly access only),"
     echo "  - states repository      (allow writable access)."
     echo
-    echo "After all access rights are granted you can clone the Infrastructure System:"
+    echo "After all access rights are granted you can clone the Core Infrastructure System:"
-    echo "  e.g.: git clone ssh://git@git.example.dev:22448/iss.git /iss"
+    echo "  e.g.: git clone ssh://git@git.example.dev:22448/cis.git /cis"
     echo
-    echo "Finally call 'setupCoreOntoThisHost.sh' from the root directory of the repository:"
+    echo "Finally call 'setupCoreOntoThisHost.sh' from the root directory:"
-    echo "  e.g.: /iss/setupCoreOntoThisHost.sh"
+    echo "  e.g.: /cis/setupCoreOntoThisHost.sh"
     echo
 }
 
 # sanitizes all parameters
 setNeededHostnameOrExit "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && prepare
+    && prepareThisHost \
+    && showFurtherSteps \
+    && exit 0
+
+exit 1

script/check/README.md

@@ -0,0 +1,4 @@
+runAllChecks.sh
+===============
+
+This script processes all checks of a host to verify the right configuration.

script/check/host/all/app_docker-compose_is_installed.check.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+_CURRENT_APP='docker compose version'
+
+${_CURRENT_APP} > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/app_docker_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+docker --version > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/app_nginx_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+nginx -v > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/app_nginx_starts_reliable.check.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# Fail because of unnecessary custom config
+grep "Wants=network-online.target" /lib/systemd/system/nginx.service > /dev/null 2>&1 \
+    && [ -f "/etc/systemd/system/nginx.service" ] \
+    && exit 1
+
+# Success if system config is ok
+grep "Wants=network-online.target" /lib/systemd/system/nginx.service > /dev/null 2>&1 \
+    && exit 0
+
+# Success if custom config fixes system config
+grep "Wants=network-online.target" /etc/systemd/system/nginx.service > /dev/null 2>&1 \
+    && exit 0
+
+exit 1

script/check/host/all/core_cron_starts_setup_as_fallback.check.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+crontab -l | grep -E "[0-9]{1,2}[ \*]{8}[[:blank:]]*\/cis\/setupCoreOntoThisHost.sh" > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/core_git_app_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+git --version > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/core_hostname_is_long.check.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/hostname'
+
+#The file must be readable, then
+#the number of lines containing a '.' must be zero.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "$(grep -cF '.' "${_CURRENT_FILE}")" -gt 0 ] \
+    && exit 0
+exit 1

script/check/host/all/core_ssh_app_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+ssh -V > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/core_ssh_authorized_keys_of_jenkins_points_to_definitions.check.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+_CURRENT_FILE='/home/jenkins/.ssh/authorized_keys'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#File has to be readable, then
+#search for '/definitions/' in the path of current file, after readlink expanded a potential symlink.
+[ -r "${_CURRENT_FILE}" ] \
+    && readlink -f "${_CURRENT_FILE}" | grep -q "/definitions/" \
+    && exit 0
+
+exit 1

script/check/host/all/core_ssh_authorized_keys_of_root_is_empty_or_points_to_definitions.check.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+_CURRENT_FILE='/root/.ssh/authorized_keys'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#No file is ok
+[ ! -e "${_CURRENT_FILE}" ] \
+    && exit 0
+
+#The file must be readable, then
+#all comments and all blank lines are removed, after which the number of remaining lines must be zero.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "0" == "$(cat "${_CURRENT_FILE}" | sed 's/[[:blank:]]*#.*//' | sed '/^$/d' | grep -c .)" ] \
+    && exit 0
+
+#File has to be readable, then
+#search for '/definitions/' in the path of current file, after readlink expanded a potential symlink.
+[ -r "${_CURRENT_FILE}" ] \
+    && readlink -f "${_CURRENT_FILE}" | grep -q "/definitions/" \
+    && exit 0
+
+exit 1

script/check/host/all/core_ssh_key_of_root_exists_as_expected.check.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_CURRENT_FILE='/root/.ssh/id_ed25519'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#File has to be readable and no passphrase should be needed.
+ssh-keygen -y -P "" -f "${_CURRENT_FILE}" &> /dev/null \
+    && exit 0
+
+exit 1

script/check/host/all/core_sudoers_file_of_jenkins_points_to_definitions.check.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/sudoers.d/allow-jenkins-updateRepositories'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#File has to be readable, then
+#search for '/definitions/' in the path of current file, after readlink expanded a potential symlink.
+[ -r "${_CURRENT_FILE}" ] \
+    && readlink -f "${_CURRENT_FILE}" | grep -q "/definitions/" \
+    && exit 0
+
+exit 1

script/check/host/all/core_user_jenkins_exists.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_USER='jenkins'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+id -u "${_CURRENT_USER}" > /dev/null 2>&1 \
+    && exit 0
+
+exit 1

script/check/host/all/system_localtime_contains_cet_and_cest.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/localtime'
+
+#The file must be readable, then
+#the number of lines containing "CET" must be greater than zero, and
+#the number of lines containing "CEST" must also be greater than zero.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "$(zdump -v "${_CURRENT_FILE}" | head -n 10 | grep 'CET' | grep -c .)" -gt "0" ] \
+    && [ "$(zdump -v "${_CURRENT_FILE}" | head -n 10 | grep 'CEST' | grep -c .)" -gt "0" ] \
+    && exit 0
+exit 1

script/check/host/all/system_timezone_is_berlin.check.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/timezone'
+
+#The file must be readable, then
+#the number of lines containing "Europe/Berlin" must be one.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "1" == "$(cat "${_CURRENT_FILE}" | grep 'Europe/Berlin' | grep -c .)" ] \
+    && exit 0
+exit 1

script/check/host/all/system_unattended_upgrades_are_disabled.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+! systemctl is-enabled unattended-upgrades.service > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_app_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+zfs --version > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_atime_of_rootfs_zpool1.check.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_CURRENT_POOL='zpool1'
+
+#Check if the tool 'zfs' is available, then
+#retrieve the property 'atime' from 'zpool1', without header and compare the result with 'off'
+#because this the feature 'atime' logs each access, there are many avoidable writes.
+
+#Set with: 'zfs set atime=off zpool1'
+zfs version &> /dev/null  \
+    && [ "$(zfs get atime -Ho value ${_CURRENT_POOL} 2> /dev/null)" == "off" ] \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_compression_of_rootfs_zpool1.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_POOL='zpool1'
+
+#Check if the tool 'zfs' is available, then
+#retrieve the property 'compression' from 'zpool1', without header and compare the result with 'lz4'
+
+#Set with: 'zfs set compression=lz4 zpool1'
+zfs version &> /dev/null  \
+    && [ "$(zfs get compression -Ho value ${_CURRENT_POOL} 2> /dev/null)" == "lz4" ] \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_mountpoint_of_rootfs_zpool1.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_ZFS='zpool1'
+
+#Check if the tool 'zfs' is available, then
+#retrieve the property 'mountpoint' from 'zpool1', without header and compare the result with '/zpool1'
+
+#Set with: 'zfs set mountpount=default'
+zfs version &> /dev/null \
+    && [ "$(zfs get mountpoint -Ho value ${_CURRENT_ZFS} 2> /dev/null)" == "/${_CURRENT_ZFS}" ] \
+    && exit 0
+exit 1

script/check/host/all/system_zpool_alignment_of_pool.check.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+_CURRENT_POOL='zpool1'
+
+#Check if the tool 'zpool' is available, then
+#retrieve the property 'ashift' from 'zpool1', without header and compare the result with '12'
+zpool version &> /dev/null \
+    && [ "$(zpool get ashift -Ho value ${_CURRENT_POOL} 2> /dev/null)" == "12" ] \
+    && exit 0
+exit 1

script/check/runAllChecks.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SCRIPT%%/script/check/*}/"               #Removes longest  matching pattern '/script/check/*' from the end
+_SCRIPT_PATH="${_CIS_ROOT:?"Missing CIS_ROOT"}script/"
+_OWN_DOMAIN="$(${_CIS_ROOT}core/printOwnDomain.sh)"
+_OWN_DEFINITIONS="${_CIS_ROOT}definitions/${_OWN_DOMAIN:?"Missing OWN_DOMAIN"}/"
+
+
+
+function run_as_root() {
+    [ "0" == "$(id -u)" ] \
+        && echo OK \
+        && return 0
+
+    echo FAIL
+    return 1
+}
+
+function scripts_are_updateable_by_git() {
+        git -C "${_SCRIPT_PATH:?"Missing SCRIPT_PATH"}" pull > /dev/null 2>&1 \
+        && echo OK \
+        && return 0
+
+    echo FAIL
+    return 1
+}
+
+function allChecks() {
+    local _CHECK_PATH _MODE_PATH
+    _CHECK_PATH="${1:?"allChecks(): Missing first parameter CHECK_PATH"}check/"
+    _MODE_PATH="${2:-all}/"
+    readonly _CHECK_PATH _MODE_PATH
+
+    echo "  - ${_CHECK_PATH}host/${_MODE_PATH}*.check.sh"
+    [ "$(ls -1 ${_CHECK_PATH}host/${_MODE_PATH}*.check.sh 2> /dev/null | grep -cE '.*')" == "0"  ] \
+       && echo "  nothing to do" \
+       && return 0
+
+    for _CURRENT_CHECK in ${_CHECK_PATH}host/${_MODE_PATH}*.check.sh; do
+        _NAME="$(basename ${_CURRENT_CHECK} | cut -d'.' -f1)"
+        _CONTEXT="$(echo ${_NAME} | cut -d'_' -f1)"
+        _CHECK="$(echo ${_NAME} | cut -d'_' -f2- | tr '_' ' ')"
+        _RESULT="$("${_CURRENT_CHECK}" && echo OK || echo FAIL)"
+        echo "  ${_CONTEXT^^} ${_CHECK}: ${_RESULT}"
+    done
+}
+
+echo "PRECONDITION run as root: $(run_as_root)"
+echo "PRECONDITION scripts are updateable by git: $(scripts_are_updateable_by_git)"
+echo
+echo "Check all (common):"
+allChecks "${_SCRIPT_PATH}"
+echo "Check all (own):"
+allChecks "${_OWN_DEFINITIONS}"
+echo "Check this host:"
+allChecks "${_OWN_DEFINITIONS}" "$(hostname -s)"

script/monitor/README.md

@@ -0,0 +1,126 @@
+
+How to setup a monitoring dashboard
+===================================
+
+Inspired by: https://pimylifeup.com/ubuntu-chromium-kiosk/
+
+Steps
+-----
+
+
+
+### 1.) Install Ubuntu Server (no desktop) on your computer than set hostname and timezone.
+
+```sh
+hostnamectl set-hostname check.local
+timedatectl set-timezone Europe/Berlin
+```
+
+
+
+### 2.) Install minimal GUI and Tools.
+
+```sh
+apt install ubuntu-desktop-minimal
+apt install language-pack-gnome-de
+apt install xdotool
+apt install dbus-x11
+```
+
+
+
+### 3.) Create a kiosk user with home-directory.
+
+```sh
+useradd -m kiosk
+```
+
+and disable Welocme-Screen
+```sh
+echo "yes" > /home/kiosk/.config/gnome-initial-setup-done
+```
+
+
+
+### 4.) Edit following file `nano /etc/gdm3/custom.conf` to turn of wayland and turn on autologin for user 'kiosk'.
+
+```
+[daemon]
+# Uncomment the line below to force the login screen to use Xorg
+#WaylandEnable=false
+
+WaylandEnable=false
+
+# Enabling automatic login
+#  AutomaticLoginEnable = true
+#  AutomaticLogin = user1
+
+AutomaticLoginEnable = true
+AutomaticLogin = kiosk
+```
+
+
+
+### 5.) Configure GUI of user kiosk to prevent monitor from sleeping
+
+```sh
+#gsettings list-recursively
+
+# Does not work
+#sudo -u kiosk gsettings set org.gnome.desktop.session idle-delay 0
+
+# Set idle-delay from "uint32 300" to "uint32 0", needs 'apt install dbus-x11'
+# You can check the value in "GUI-Session of kiosk -> Settings -> Power"
+sudo -u kiosk dbus-launch dconf write /org/gnome/desktop/session/idle-delay "uint32 0"
+```
+
+
+
+### 6.) Create custom service to start firefox loading the page.
+
+Therefore create a file `/etc/systemd/system/kiosk.service` with this content:
+
+```
+[Unit]
+Description=Firefox Kiosk
+Wants=graphical.target
+After=graphical.target
+
+[Service]
+Environment=DISPLAY=:0
+# Set firefox language, needs 'apt install language-pack-gnome-de' 
+Environment=LANG=de_DE.UTF-8
+Type=simple
+# Always a fresh firefox ('-' allow error if common does not exist)
+ExecStartPre=-/usr/bin/rm -r /home/kiosk/snap/firefox/common
+# Move Mouse (should also work on small screens), needs 'apt install dbus-x11'
+ExecStartPre=/usr/bin/xdotool mousemove 4096 2160
+# See: https://wiki.mozilla.org/Firefox/CommandLineOptions (just -kiosk URL => Start-Assistant, so use -url too)
+ExecStart=/usr/bin/firefox -fullscreen -kiosk -url http://monitor.example.net/check.html
+Restart=always
+RestartSec=30
+User=kiosk
+Group=kiosk
+
+[Install]
+WantedBy=graphical.target
+```
+
+
+
+### 7.) Enable the service and reboot
+
+```sh
+systemctl enable kiosk
+reboot
+```
+
+
+
+Troubleshouting
+---------------
+
+```
+systemctl disable pd-mapper.service
+apt purge cloud-init -y && apt autoremove --purge -y
+```

script/monitor/check.css

@@ -0,0 +1,77 @@
+html, body {
+    --background-theme-color: #001EA0;
+    --cell-space: 20px;
+    --logo-height: 50px;
+
+    background-color: #cccccc;
+    font-family: Verdana;
+    font-size: 14pt;
+    color: #ffffff;
+    height: 100%;
+    margin: 0;
+}
+@media screen and (orientation: portrait) {
+    body {
+        zoom: 200%
+    }
+}
+#header {
+    background-color: var(--background-theme-color); 
+    position: sticky;
+    top: 0;
+    height: calc(var(--logo-height) + (2 * var(--cell-space)));
+    width: 100%;
+}
+#header img {
+    height: var(--logo-height);
+    margin: var(--cell-space);
+    vertical-align: middle;
+}
+#header h1 {
+    display: inline;
+    font-weight: normal;
+    vertical-align: middle;
+}
+#content {
+    min-height: 100%;
+}
+#footer {
+    background-color: var(--background-theme-color);
+    position: sticky;
+    bottom: 0px;
+    padding: var(--cell-space);
+    text-align: center;
+    vertical-align: middle;
+    font-size: 22pt;
+}
+#checks {
+    display: grid;
+    grid-template-columns: repeat(auto-fit, minmax(350px, 1fr));
+    padding: var(--cell-space);
+    grid-gap: var(--cell-space);
+}
+#checks > div {
+    border: 1px solid black;
+    border-radius: 10px;
+    padding: 10px;
+    text-align: center;
+    box-shadow: 0 2px 5px 0 rgba(0, 0, 0, 0.3), 0 2px 10px 0 rgba(0, 0, 0, 0.2);
+}
+#checks > div.ok {
+    background-color: #66aa22;
+    color: #222222;	
+}
+#checks > div.info {
+    background-color: #88cc44;
+    color: #222222;	
+}
+#checks > div.warn {
+    background-color: #ffdd00;
+    color: #222222;	
+}
+#checks > div.fail {
+    background-color: #ff0000;
+}
+#checks > div.timeout {
+    background-color: var(--background-theme-color);
+}

script/monitor/check.html

@@ -0,0 +1,122 @@
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Monitoring Dashboard</title>
+    <link rel="stylesheet" href="check.css">
+</head>
+<body>
+    <div id="header">
+        <img src="logo.png"></img>
+        <h1>Monitoring</h1>
+    </div>
+    <div id="content">
+        <div id="checks">
+            <div class="check warn">Loading...</div>
+        </div>
+    </div>
+    <div id="footer">
+        Köln, <span id="datetime"></span>
+    </div>
+
+    <script>
+        var connectionAlive = true;
+
+        function downloadCheckFile(callback) {
+            var xmlHttp = new XMLHttpRequest();
+            xmlHttp.open('GET', "check.txt", false);
+            xmlHttp.onreadystatechange=function() {
+                if(xmlHttp.readyState==4) {
+                    callback(xmlHttp.responseText);
+                }
+            }
+            try {
+                xmlHttp.send(null);
+                if (xmlHttp.status >= 200 && xmlHttp.status < 304) {
+                    return true;
+                } else {
+                    return false;
+                }
+            } catch (e) {
+                return false;
+            }
+        }
+        function convertToHtml(checkText) {
+            if (!connectionAlive) {
+                return '<div class="fail">CONNECTION FAILED</div>';
+            }
+
+            var html = "";
+            var lines = checkText.split(/\n/);
+
+            for(var lineNo = 2; lineNo < lines.length; lineNo++) {
+                var line = lines[lineNo];
+
+                var parts = line.split('?');
+                if (parts.length > 1) {
+
+                    var name = parts[0].trim().split("_").join(" ");
+                    var resultParts = parts[1].trim().split('#');
+                    var result = resultParts[0];
+                    var message = resultParts[1];
+
+                    if(name == 'MISSED') {
+                        var fileTimeParts = message.split('-');
+                        var fileTime = new Date();
+                        fileTime.setHours(parseInt(fileTimeParts[0]));
+                        fileTime.setMinutes(parseInt(fileTimeParts[1]));
+                        fileTime.setSeconds(parseInt(fileTimeParts[2]));
+
+                        var scriptTime = new Date();
+                        scriptTime.setMinutes(scriptTime.getMinutes() - 2);
+                        if (scriptTime.getTime() < fileTime.getTime()){
+                            if (result == "0") {
+                                html += '<div class="ok">EVERYTHING OK<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                            } else {
+                                html += '<div class="fail">FAILED: ' + result + '<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                            }
+                        } else {
+                            html += '<div class="check fail">CHECKS TOO OLD<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                        }
+                    } else {
+                        if(result.indexOf('OK') >= 0) {
+                            html += '<div class="ok">'+ name;
+                        } else if(result.indexOf('INFO') >= 0) {
+                            html += '<div class="info">'+ name;
+                        } else if(result.indexOf('TIMEOUT') >= 0) {
+                            html += '<div class="timeout">'+ name;
+                        } else if(result.indexOf('WARN') >= 0) {
+                            html += '<div class="warn">'+ name;
+                        } else {
+                            html += '<div class="fail">' + name;
+                        }
+                        if(message) {
+                            html += '<br/>' + message.trim();
+                        }
+                        html += '</div>';
+                    }
+                }
+            }
+            return html;
+        }
+        function exchangeChecks(text) {
+            document.getElementById("checks").innerHTML = convertToHtml(text);
+        }
+        function refreshTime() {
+            document.getElementById("datetime").innerHTML = new Date().toLocaleString("de-DE", {timeZone: "Europe/Berlin"});
+        }
+        function refreshChecks() {
+            connectionAlive = downloadCheckFile(exchangeChecks);
+        }
+        function reloadIfAlive() {
+            if (connectionAlive) {
+                location.reload();
+            }
+        }
+        setInterval(refreshTime, 1000);
+        setInterval(refreshChecks, 5000);
+        setInterval(reloadIfAlive, 300000);
+        refreshTime();
+        refreshChecks();
+    </script>
+</body>
+</html>

script/monitor/check.sh

@@ -0,0 +1,88 @@
+#!/bin/bash
+
+# Folders always ends with an tailing '/'
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/script/monitor/*}/"               #Removes longest  matching pattern '/script/monitor/*' from the end
+_CORE_SCRIPTS="${_CIS_ROOT:?"Missing CIS_ROOT"}core/"
+_CURRENT_DOMAIN="$("${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}printOwnDomain.sh")"
+_DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_CURRENT_DOMAIN:?"Missing CURRENT_DOMAIN"}/"
+
+# Checks for the entire domain
+_DOMAIN_CHECKS="${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/checks/"
+
+
+
+function doChecks(){
+    local readonly _TMPDIR="${1:?"doChecks(): Missing parameter TMPDIR:"}"
+
+    local _DATETIME=$(date +%H-%M-%S)
+
+    mkdir -p ${_TMPDIR}
+    rm ${_TMPDIR}/* > /dev/null 2>&1
+
+    for check in ${_DOMAIN_CHECKS}*.on
+    do
+        local _CHECK_FILENAME="${check##*/}"
+        echo -n "${_CHECK_FILENAME%%.on}?" > "${_TMPDIR}/${_CHECK_FILENAME}"
+        timeout -k 10s 20s bash ${check} >> "${_TMPDIR}/${_CHECK_FILENAME}" 2> /dev/null || echo "TIMEOUT#Timeout" >> "${_TMPDIR}/${_CHECK_FILENAME}" &
+    done
+    wait
+
+    local _FAILED=0
+    echo "CHECK?RESULT[#MESSAGE]:"
+    echo "-----------------------"
+    for resultFile in ${_TMPDIR}/*
+    do
+        cat "${resultFile}"
+        grep -q "FAIL" ${resultFile} && _FAILED=$(expr ${_FAILED} + 1)
+    done
+    echo "MISSED?${_FAILED}#${_DATETIME}"
+
+    rm -r ${_TMPDIR} > /dev/null 2>&1
+    return 0
+}
+
+function usage(){
+    printf "\nUsage: /monitoring/check.sh <command> <options>"
+    echo
+    echo "possible commands:"
+    echo
+    echo "- all"
+    echo "    Executes all checks."
+    echo "- auto <out_file>"
+    echo "    Executes quiet all checks and saves the result in the given out_file."
+    echo "      (e.g. add the following line to crontab: '* * * * * /cis/script/monitor/check.sh auto /var/www/html/check.txt'"
+    echo "       to update the file '/var/www/html/check.txt' every minute as 'check.html' needs it.)"
+    return 0
+}
+
+main(){
+    case "${1:-""}" in
+        all)
+            echo "Checks werden ausgeführt..." \
+                && echo \
+                && doChecks "/tmp/checks" color \
+                && echo \
+                && echo "Success" \
+                && return 0
+            ;;
+        auto)
+            # If just a filename is given it is created in /tmp, because of 'cd /tmp' 
+            cd /tmp \
+                && doChecks "/tmp/checks$(date +%N)" > "$2.new" \
+                && mv -f "$2.new" "$2" \
+                && return 0
+            return 1
+            ;;
+		*)
+            [ "${1:+isset}" == "isset" ] \
+                && echo "Parameter '${1}' ist kein gültiger Befehl."
+            usage
+            return 0
+            ;;
+    esac
+
+    return 1
+}
+
+main "$@" || exit 1

script/monitor/generic/NGINX_CHECK.sh

@@ -0,0 +1,68 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 1 "ssh -O check -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN}" &> /dev/null \
+        && echo "master checked" \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function checkViaHTTP() {
+    _STATUS="$(curl -I http://${_REMOTE_HOSTNAME_FQDN} 2>/dev/null | head -n 1 | cut -d$' ' -f2)"
+    [ "${_STATUS}" == "200" ] \
+        && echo "OK" \
+        && return 0
+
+    return 1
+}
+
+function checkViaHTTPS() {
+    _STATUS="$(curl -k -I https://${_REMOTE_HOSTNAME_FQDN} 2>/dev/null | head -n 1 | cut -d$' ' -f2)"
+    [ "${_STATUS}" == "200" ] \
+        && echo "OK" \
+        && return 0
+
+    return 1
+}
+
+#grep:
+# -E                         Use regexp, '.*' => any chars between 'Active:' and '(running)', the round brackets are escaped.
+
+#cut:
+# -d                         Delimiter, marker where to cut (here ;)
+# -f                         Index of column to show (One based, so there is no -f0)
+function checkViaSSH() {
+    checkOrStartSSHMaster \
+        || return 1
+
+    _RESULT=$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'systemctl status nginx.service' | grep -E 'Active:.*\(running\)' | cut -d';' -f2)
+    ! [ -z "${_RESULT}" ] && echo "OK#UPTIME:${_RESULT}" || echo "FAIL"
+}
+
+#checkViaHTTP && exit 0
+#checkViaHTTPS && exit 0
+checkViaSSH && exit 0
+
+exit 1

script/monitor/generic/PING_CHECK.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+_SERVER="${1:?"FQDN of server missing"}"
+
+# -4                         Use IPv4
+# -W            SECONDS      Wait seconds for an answer
+# -c            COUNT_VALUE  Count of pings being executed
+_RESULT="$(ping -4 -W 1 -c 1 "${_SERVER}" | grep "time=" | cut -d'=' -f4)"
+! [ -z "${_RESULT}" ] && echo "OK#RTT: ${_RESULT}" || echo "FAIL#PLEASE USE FALLBACK!"

script/monitor/generic/URL_CHECK.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+_URL="${1:?"URL of site missing"}"
+
+#curl:
+# --connect-timeout SECONDS  Maximum time allowed for connection
+# -k                         Allow connections to SSL sites without certs (H)
+# -L                         Follow redirects (H)
+# --max-time        SECONDS  Maximum time allowed for the transfer
+# -s                         Silent mode. Don't output anything
+# --head                     Show head information only
+# --no-progress-meter        Clean output for grep
+
+#grep:
+# -q                         Quite, no output just status codes
+# -F                         Interpret search term as plain text
+((curl --connect-timeout 10 --max-time 10 -k -s --head --no-progress-meter "${_URL}" | grep -qF '200 OK') && echo OK) || echo FAIL

script/monitor/generic/ZFS_SYNC_CHECK.sh

@@ -0,0 +1,107 @@
+#!/bin/bash
+
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SCRIPT%%/script/monitor/*}/"             #Removes longest  matching pattern '/script/monitor/*' from the end
+_DOMAIN="$("${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh")"
+_COMPOSITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/compositions/"
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+# This is crucial:
+#  - default value for the filter part is extracted from the first parameter (FQDN)
+#  - but you can override this part to to adapt the test during a change of the domain.
+#    (e.g. the short hostname can be an option - or even a better default in the future)
+_ZFS_SNAPSHOT_FILTER="@SYNC_${2:-"${_REMOTE_HOSTNAME_FQDN:?"Missing REMOTE_HOSTNAME_FQDN"}"}"
+
+_MODE="${3:-"normal"}"
+_NOW_UTC_UNIXTIME=$(date -u +%s)
+_DEBUG_PATH="/tmp/monitor/"
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 1 "ssh -O check -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN}" &> /dev/null \
+        && echo "master checked" \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function checkSync() {
+    checkOrStartSSHMaster \
+        || return 1
+
+    [ "${_MODE}" == "debug" ] \
+        && mkdir -p "${_DEBUG_PATH}" > /dev/null \
+        && echo "Now: ${_NOW_UTC_UNIXTIME}" > ${_DEBUG_PATH}SECONDS_BEHIND_${_REMOTE_HOSTNAME_FQDN}.txt
+
+    ! [ -d "${_COMPOSITIONS:?"Missing COMPOSITIONS"}" ] \
+        && echo "WARN#no compositions" \
+        && return 0
+
+    [ "${_MODE}" == "debug" ] \
+        && echo "Snapshot filter: ${_ZFS_SNAPSHOT_FILTER}" >> ${_DEBUG_PATH}SECONDS_BEHIND_${_REMOTE_HOSTNAME_FQDN}.txt
+
+    # This retrieves the list of the interesting snapshots including creation timestamp
+    _SNAPSHOTS="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} zfs list -po creation,name -r -t snapshot zpool1/persistent | grep -F ${_ZFS_SNAPSHOT_FILTER})"
+    [ "${_MODE}" == "debug" ] \
+        && echo "${_SNAPSHOTS}" > ${_DEBUG_PATH}SNAPSHOTS_${_REMOTE_HOSTNAME_FQDN}.txt
+
+    [ -z "${_SNAPSHOTS}" ] \
+        && echo "FAIL#no snapshots" \
+        && return 1
+
+    echo "OK#Checks running"
+
+    for _COMPOSITION_PATH in ${_COMPOSITIONS}*; do
+
+        # If remote host is found than it is responsible for this container-composition, otherwise skip
+        #   (grep -E "^[[:blank:]]*something" means. Line has to start with "something", leading blank chars are ok.)
+        grep -E "^[[:blank:]]*${_REMOTE_HOSTNAME_SHORT}" "${_COMPOSITION_PATH}/zfssync-hosts" &> /dev/null \
+            || continue;
+
+        _COMPOSITION_NAME="${_COMPOSITION_PATH##*/}"   #Removes longest  matching pattern '*/' from the begin
+        _LAST_SNAPSHOT_UNIXTIME="$(echo "${_SNAPSHOTS}" | grep ${_COMPOSITION_NAME} | tail -n 1 | cut -d' ' -f1)"
+        _SECONDS_BEHIND=$[ ${_NOW_UTC_UNIXTIME} - ${_LAST_SNAPSHOT_UNIXTIME} ]
+
+        [ "${_MODE}" == "debug" ] \
+            && echo "${_LAST_SNAPSHOT_UNIXTIME} ${_COMPOSITION_NAME} on ${_REMOTE_HOSTNAME_FQDN} behind: ${_SECONDS_BEHIND}s" >> ${_DEBUG_PATH}SECONDS_BEHIND_${_REMOTE_HOSTNAME_FQDN}.txt
+
+        [ "${_SECONDS_BEHIND}" -lt 40 ] \
+            && continue
+
+        [ "${_SECONDS_BEHIND}" -lt 60 ] \
+            && echo "ZFSSYNC_of_${_REMOTE_HOSTNAME_SHORT}_LAGGING?WARN#${_COMPOSITION_NAME} ${_SECONDS_BEHIND}s" \
+            && continue
+
+        echo "ZFSSYNC_of_${_REMOTE_HOSTNAME_SHORT}_LAGGING?FAIL#${_COMPOSITION_NAME} ${_SECONDS_BEHIND}s"
+    done
+}
+
+
+
+RESULTS="$(checkSync)"
+
+[ "${_MODE}" == "debug" ] \
+    && echo "$RESULTS" > ${_DEBUG_PATH}RESULTS_${_REMOTE_HOSTNAME_FQDN}.txt
+
+echo "$RESULTS"

script/monitor/generic/ZFS_USAGE_CHECK.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 1 "ssh -O check -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN}" &> /dev/null \
+        && echo "master checked" \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function testSpace(){
+    checkOrStartSSHMaster \
+        || return 1
+
+    local _RESULT="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'zpool list -H -o capacity,name')"
+    local _SPACE_USED=$(echo "${_RESULT}" | /usr/bin/tail -n 1 | /usr/bin/cut -f1)
+    local _POOL=$(echo "${_RESULT}" | /usr/bin/tail -n 1 | /usr/bin/cut -f2)
+
+    [ -z "${_SPACE_USED}" ] \
+        && echo "FAIL#NO value" \
+        && return 0
+
+    [ "${1:?"Missing OK_THRESHOLD"}" -ge "${_SPACE_USED%\%*}" ] \
+        && echo "OK#${_SPACE_USED} used ${_POOL}." \
+        && return 0
+
+    [ "${2:?"Missing INFO_THRESHOLD"}" -ge "${_SPACE_USED%\%*}" ] \
+        && echo "INFO#${_SPACE_USED} already used ${_POOL}." \
+        && return 0
+
+    echo "FAIL#${_SPACE_USED} used ${_POOL}!"
+    return 0
+}
+
+testSpace 80 90 && exit 0
+
+exit 1

script/monitor/setupMonitoringHost.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && sudo "${0}" \
+    && exit 0
+
+
+
+_SETUP="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SETUP%%/script/monitor/*}/"             #Removes longest  matching pattern '/script/monitor/*' from the end
+_DOMAIN="$("${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh")"
+_DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/"
+
+
+
+function printSelectedDefinition() {
+    local _FILE_DEFINED_DOMAIN _FILE_DEFINED_DEFAULT
+    _FILE_DEFINED_DOMAIN="${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/${1:?"Missing CURRENT_FULLFILE"}"
+    _FILE_DEFINED_DEFAULT="${_CIS_ROOT:?"Missing CIS_ROOT"}script/monitor/${1:?"Missing CURRENT_FULLFILE"}"
+    readonly _FILE_DEFINED_DOMAIN _FILE_DEFINED_DEFAULT
+
+    [ -s "${_FILE_DEFINED_DOMAIN}" ] \
+        && echo "${_FILE_DEFINED_DOMAIN}" \
+        && return 0
+
+    [ -s "${_FILE_DEFINED_DEFAULT}" ] \
+        && echo "${_FILE_DEFINED_DEFAULT}" \
+        && return 0
+
+    return 1
+}
+
+function setupPublicFile() {
+    ! [ -d "/var/www/html" ] \
+        && echo "Missing folder '/var/www/html'. Is a webserver installed?" \
+        && return 1
+
+    [ -L "/var/www/html/${1:?"Missing filename"}" ] \
+        && [ "$(readlink -f /var/www/html/${1:?"Missing filename"})" == "$(printSelectedDefinition ${1:?"Missing filename"})" ] \
+        && echo "Link '/var/www/html/${1:?"Missing filename"}' already exists pointing to the expected file:" \
+        && echo "  - '$(readlink -f /var/www/html/${1:?"Missing filename"})'" \
+        && return 0
+
+    ln -f -s "$(printSelectedDefinition ${1:?"Missing filename"})" "/var/www/html/${1:?"Missing filename"}" \
+        && echo "Link '/var/www/html/${1:?"Missing filename"}' created successfully:" \
+        && echo "  - '$(readlink -f /var/www/html/${1:?"Missing filename"})'" \
+        && return 0
+}
+
+echo "Setup the monitoring host that monitors the others ... " \
+    && setupPublicFile "check.html" \
+    && setupPublicFile "check.css" \
+    && setupPublicFile "logo.png" \
+    && exit 0
+
+exit 1

script/monitor/setupServiceProvidingHost.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && sudo "${0}" \
+    && exit 0
+
+
+
+_SETUP="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SETUP%%/script/monitor/*}/"             #Removes longest  matching pattern '/script/monitor/*' from the end
+_CORE_SCRIPTS="${_CIS_ROOT:?"Missing CIS_ROOT"}core/"
+_DOMAIN="$("${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh")"
+_DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/"
+
+
+
+echo "Setup the user and permission to enable the monitoring this host ... " \
+    && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addNormalUser.sh" monitoring \
+    && echo \
+    && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}defineAuthorizedKeysOfUser.sh" "${_DEFINITIONS}" monitoring \
+    && exit 0
+
+exit 1

setupCoreOntoThisHost.sh

@@ -6,6 +6,12 @@
 
 
 
+# Folders always ends with an tailing '/'
+_SETUP="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SETUP%/setupCoreOntoThisHost.sh}/"             #Removes shortest matching pattern '/setupCoreOntoThisHost.sh' from the end
+_CORE_SCRIPTS="${_CIS_ROOT:?"Missing CIS_ROOT"}core/"
+
+
 
 function checkPathsAreAvaiable() {
     grep --version &> /dev/null \
@@ -31,22 +37,21 @@ function checkGitIsAvailable() {
 }
 
 function checkPreconditions() {
-    local _ROOT _DOMAIN
+    local _DOMAIN
-    _ROOT="${1:?"Missing parameter ROOT"}"
+    _DOMAIN="${1}" # Optional parameter DOMAIN
-    _DOMAIN="${2}" # Optional parameter DOMAIN
+    readonly _DOMAIN
-    readonly _ROOT _DOMAIN
 
     ! [ -z "${_DOMAIN}" ] \
         && [ "$(hostname -d)" != "${_DOMAIN}" ] \
         && echo \
-        && echo "WARNING: system-domain DOES NOT MATCH domainOfHostOwner: '$(hostname -d)' != '${_DOMAIN}'" \
+        && echo "WARNING: system-domain DOES NOT MATCH overrideOwnDomain: '$(hostname -d)' != '${_DOMAIN}'" \
         && echo
 
     # Given domain verfügbar (nicht leer)
     ! [ -z "${_DOMAIN}" ] \
         && checkPathsAreAvaiable \
         && checkGitIsAvailable \
-        && git -C "${_ROOT}" pull &> /dev/null \
+        && git -C "${_CIS_ROOT:?"Missing CIS_ROOT"}" pull &> /dev/null \
         && return 0
 
     echo
@@ -69,63 +74,65 @@ function checkPreconditions() {
 }
 
 function getOrSetDomain() {
-    local _ROOT _DOMAIN_FILE _GIVEN_DOMAIN
+    local _CURRENT_DOMAIN _GIVEN_DOMAIN _OVERRIDE_DOMAIN_FILE
-    _ROOT="${1:?"Missing parameter ROOT"}"
+    _CURRENT_DOMAIN="$("${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}printOwnDomain.sh")"
-    _DOMAIN_FILE="${_ROOT:?"Missing ROOT"}domainOfHostOwner"
+    _GIVEN_DOMAIN="${1}" # Optional parameter DOMAIN
-    _GIVEN_DOMAIN="${2}" # Optional parameter DOMAIN
+    _OVERRIDE_DOMAIN_FILE="${_CIS_ROOT:?"Missing CIS_ROOT"}overrideOwnDomain"
-    readonly _ROOT _DOMAIN_FILE _GIVEN_DOMAIN
+    readonly _CURRENT_DOMAIN _GIVEN_DOMAIN _OVERRIDE_DOMAIN_FILE
 
-    # Wenn DOMAIN_FILE enhält lesbare Daten
+    ! [ -z "${_CURRENT_DOMAIN}" ] \
-    grep '[^[:space:]]' "${_DOMAIN_FILE:?"Missing DOMAIN_FILE"}" &> /dev/null \
+        && [ -z "${_GIVEN_DOMAIN}" ] \
-        && cat "${_DOMAIN_FILE}" \
+        && echo "${_CURRENT_DOMAIN}" \
         && return 0
 
-    # Der boot-hostname muss mindestens einen Punkt enthalten, dann wird die hintere Hälfte als Domain genommen
+    ! [ -z "${_CURRENT_DOMAIN}" ] \
-    hostname -b | grep "\." | cut -d. -f2- > "${_DOMAIN_FILE}"
+        && [ "${_CURRENT_DOMAIN}" == "${_GIVEN_DOMAIN}" ] \
-    grep '[^[:space:]]' "${_DOMAIN_FILE}" &> /dev/null \
+        && echo "${_CURRENT_DOMAIN}" \
-        && cat "${_DOMAIN_FILE}" \
         && return 0
 
-    # Given domain is set (nicht leer)
+    # If there is a given domain it will be set or it will override the current one
-    ! [ -z "${_GIVEN_DOMAIN}" ] \
+    [ -z "${_CURRENT_DOMAIN}" ] \
+        && ! [ -z "${_GIVEN_DOMAIN}" ] \
         && [ "$(id -u)" == "0" ] \
+        && echo "Setting hostname to: $(hostname -s).${_GIVEN_DOMAIN}" >&2 \
         && hostnamectl set-hostname "$(hostname -s).${_GIVEN_DOMAIN}" \
-        && hostname -b | grep "\." | cut -d. -f2- > "${_DOMAIN_FILE}" \
+        && echo "${_GIVEN_DOMAIN}" \
-        && grep '[^[:space:]]' "${_DOMAIN_FILE}" &> /dev/null \
+        && return 0
-        && cat "${_DOMAIN_FILE}" \
+
+    ! [ -z "${_GIVEN_DOMAIN}" ] \
+        && echo "Overwriting domain to: ${_GIVEN_DOMAIN}" >&2 \
+        && echo "${_GIVEN_DOMAIN}" > "${_OVERRIDE_DOMAIN_FILE}" \
+        && echo "${_GIVEN_DOMAIN}" \
         && return 0
 
     return 1
 }
 
 function getRemoteRepositoryPath() {
-    local _ROOT
+    _REPOSITORY="$(git -C "${_CIS_ROOT:?"Missing CIS_ROOT"}" config --get remote.origin.url 2> /dev/null | grep -i 'git@')"
-    _ROOT="${1:?"Missing parameter ROOT"}"
+    _PATH="${_REPOSITORY%/*}"                        #Removes shortest matching pattern '/*' from the end
-    readonly _ROOT
+    ! [ -z "${_PATH}" ] \
-
+        && echo "${_PATH}/" \
-    _RESULT="$(git -C "${_ROOT:?"Missing ROOT"}" remote show origin | grep -i 'fetch' | xargs -n 1 | grep -i 'ssh://')"
-    _RESULT="${_RESULT%/*}"                        #Removes shortest matching pattern '/*' from the end
-    ! [ -z "${_RESULT}" ] \
-        && echo "${_RESULT}" \
         && return 0
 
     return 1
 }
 
 function addDefinition(){
-    local _ROOT _CORE_SCRIPTS _DEFINITIONS _REPOSITORY
+    local _DEFINITIONS _REPOSITORY
-    _DEFINITIONS="${1:?"Missing parameter DEFINITIONS"}"
+    _DEFINITIONS="${1:?"Missing first parameter DEFINITIONS"}"
-    _REPOSITORY="${2:?"Missing parameter REPOSITORY"}"
+    _REPOSITORY="$(getRemoteRepositoryPath)cis-definition-${2:?"Missing second parameter DOMAIN"}.git"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    readonly _DEFINITIONS _REPOSITORY
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
+
-    readonly _ROOT _CORE_SCRIPTS _DEFINITIONS _REPOSITORY
     [ "$(id -u)" == "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" "${_REPOSITORY}" readonly \
+        && echo "Running setup as 'root' trying to add definition repository:" \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" readonly "${_REPOSITORY}" \
         && echo "  - definitions are usable for this host." \
         && return 0
 
     [ "$(id -u)" != "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" "${_REPOSITORY}" writable \
+        && echo "Running setup as 'user' trying to add definition repository:" \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" writable "${_REPOSITORY}" \
         && echo "  - definitions are usable, as working copy." \
         && return 0
 
@@ -133,20 +140,22 @@ function addDefinition(){
 }
 
 function addState() {
-    local _ROOT _CORE_SCRIPTS _STATES _REPOSITORY
+    local _STATES _REPOSITORY
-    _STATES="${1:?"Missing parameter STATES"}"
+    _STATES="${1:?"Missing first parameter STATES"}"
-    _REPOSITORY="${2:?"Missing parameter REPOSITORY"}"
+    _REPOSITORY="$(getRemoteRepositoryPath)cis-state-${2:?"Missing second parameter DOMAIN"}.git"
-    _ROOT="${_STATES%%/states/*}/"             #Removes longest  matching pattern '/states/*' from the end
+    readonly _STATES _REPOSITORY
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
-    readonly _ROOT _CORE_SCRIPTS _STATES _REPOSITORY
 
     [ "$(id -u)" == "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" "${_REPOSITORY}" writable \
+        && echo "Running setup as 'root' trying to add state repository:" \
+        && echo \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" writable "${_REPOSITORY}" \
         && echo "  - states are usable for this host." \
         && return 0
 
     [ "$(id -u)" != "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" "${_REPOSITORY}" writable \
+        && echo "Running setup as 'user' trying to add state repository:" \
+        && echo \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" writable "${_REPOSITORY}" \
         && echo "  - states are usable, as working copy." \
         && return 0
 
@@ -154,13 +163,10 @@ function addState() {
 }
 
 function setupCoreFunctionality() {
-    local _ROOT _CORE_SCRIPTS _DEFINITIONS _MINUTE_FROM_OWN_IP _SETUP
+    local _DEFINITIONS _MINUTE_FROM_OWN_IP
     _DEFINITIONS="${1:?"Missing DEFINITIONS: 'ROOT/definitions/DOMAIN'"}"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    _MINUTE_FROM_OWN_IP="$(hostname -I | xargs -n 1 | grep -F '.' | head -n 1 | cut -d. -f4 || echo 0)" #uses last value from first own ipv4 or 0 as minute value
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
+    readonly _DEFINITIONS _MINUTE_FROM_OWN_IP
-    _MINUTE_FROM_OWN_IP="$(hostname -I | xargs -n 1 | grep -F . | head -n 1 | cut -d. -f4 || echo 0)" #uses last value from first own ipv4 or 0 as minute value
-    _SETUP="${2:?"Missing SETUP"}"
-    readonly _ROOT _CORE_SCRIPTS _DEFINITIONS _MINUTE_FROM_OWN_IP _SETUP
 
     [ "$(id -u)" != "0" ] \
         && echo "Configuration of host skipped because of insufficient rights." \
@@ -176,45 +182,39 @@ function setupCoreFunctionality() {
         && echo \
         && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}ensureUsageOfDefinitions.sh" "${_DEFINITIONS}" /etc/sudoers.d/allow-jenkins-updateRepositories \
         && echo \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addToCrontabEveryHour.sh" "${_SETUP}" "${_MINUTE_FROM_OWN_IP}" \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addToCrontabEveryHour.sh" "${_SETUP:?"Missing SETUP"}" "${_MINUTE_FROM_OWN_IP}" \
         && return 0
 
     return 1
 }
 
 function setup() {
-    local _ROOT _DEFINITIONS _DEFINITIONS_REPOSITORY _DOMAIN _REPOSITORY_PATH _SETUP _STATES _STATES_REPOSITORY
+    local _DEFINITIONS _DOMAIN _STATES
-    _SETUP="$(readlink -f "${0}" 2> /dev/null)"
+    _DOMAIN="$(getOrSetDomain "${1}")"
-    _ROOT="$(dirname ${_SETUP:?"Missing SETUP"} 2> /dev/null || echo "/iss")/"
-    _DOMAIN="$(getOrSetDomain "${_ROOT:?"Missing ROOT"}" "${1}")"
-    _REPOSITORY_PATH="$(getRemoteRepositoryPath "${_ROOT:?"Missing ROOT"}")"
 
-    ! checkPreconditions "${_ROOT:?"Missing ROOT"}" "${_DOMAIN}" \
+    ! checkPreconditions "${_DOMAIN}" \
         && return 1
 
-    _DEFINITIONS="${_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
+    _DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
-    _DEFINITIONS_REPOSITORY="${_REPOSITORY_PATH:?"Missing REPOSITORY_PATH"}/iss-definition-${_DOMAIN:?"Missing DOMAIN"}.git"
+    _STATES="${_CIS_ROOT:?"Missing CIS_ROOT"}states/${_DOMAIN:?"Missing DOMAIN"}"
-    _STATES="${_ROOT:?"Missing ROOT"}states/${_DOMAIN:?"Missing DOMAIN"}"
+    readonly _DEFINITIONS _DOMAIN _STATES
-    _STATES_REPOSITORY="${_REPOSITORY_PATH:?"Missing REPOSITORY_PATH"}/iss-state-${_DOMAIN:?"Missing DOMAIN"}.git"
-    readonly _ROOT _DEFINITIONS _DEFINITIONS_REPOSITORY _DOMAIN _REPOSITORY_PATH _SETUP _STATES _STATES_REPOSITORY
 
     echo \
-        && echo "Running setup using repositories of: '${_REPOSITORY_PATH:?"Missing REPOSITORY_PATH"}' ..." \
+        && addDefinition "${_DEFINITIONS:?"Missing DEFINITIONS"}" "${_DOMAIN:?"Missing DOMAIN"}" \
         && echo \
-        && addDefinition "${_DEFINITIONS:?"Missing DEFINITIONS"}" "${_DEFINITIONS_REPOSITORY:?"Missing DEFINITIONS_REPOSITORY"}" \
+        && addState "${_STATES:?"Missing STATES"}" "${_DOMAIN:?"Missing DOMAIN"}" \
-        && echo \
-        && addState "${_STATES:?"Missing STATES"}" "${_STATES_REPOSITORY:?"Missing STATES_REPOSITORY"}" \
         && echo \
         && echo "Using definitions: '${_DEFINITIONS:?"Missing DEFINITIONS"}' ..." \
-        && setupCoreFunctionality "${_DEFINITIONS:?"Missing DEFINITIONS"}" "${_SETUP:?"Missing SETUP"}" \
+        && setupCoreFunctionality "${_DEFINITIONS:?"Missing DEFINITIONS"}" \
         && return 0
 
-    echo "FAIL: setup is incomplete:                        ("$(readlink -f ${0})")"
+    echo "FAIL: setup is incomplete:                         ("$(readlink -f ${0})")" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
 # sanitizes all parameters
-setup \
+setup "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
+    && exit 0
-    && exit 0 || exit 1
+
+exit 1

updateRepositories.sh

@@ -20,54 +20,61 @@
 
 
 function update_repositories() {
-    local _ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
+    local _CIS_ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
     _UPDATE_REPOSITORIES="$(readlink -f "${0}" 2> /dev/null)"
-    _MODE="${1:-"all"}"
+    _CIS_ROOT="${_UPDATE_REPOSITORIES%/updateRepositories.sh}/"               #Removes shortest matching pattern '/updateRepositories.sh' from the end
-    _ROOT="$(dirname ${_UPDATE_REPOSITORIES:?"Missing UPDATE_REPOSITORIES"} 2> /dev/null || echo "/iss")/"
+    _MODE="${1:-"--core"}"
-    _DOMAIN="$(cat ${_ROOT:?"Missing ROOT"}domainOfHostOwner)"
+    _DOMAIN="$(${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh)"
-    _DEFINITIONS="${_ROOT}definitions/${_DOMAIN:?"Missing DOMAIN from file: ${_ROOT}domainOfHostOwner"}/"
+    _DEFINITIONS="${_CIS_ROOT}definitions/${_DOMAIN:?"Missing DOMAIN from file: ${_CIS_ROOT}domainOfHostOwner"}/"
-    _STATES="${_ROOT}states/${_DOMAIN:?"Missing DOMAIN from file: ${_ROOT}domainOfHostOwner"}/"
+    _STATES="${_CIS_ROOT}states/${_DOMAIN:?"Missing DOMAIN from file: ${_CIS_ROOT}domainOfHostOwner"}/"
-    readonly _ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
+    readonly _CIS_ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
 
     [ "${_MODE}" == "--repair" ] \
-        && (git -C "${_ROOT}" reset --hard origin/master; \
+        && (git -C "${_CIS_ROOT}" reset --hard origin/main; \
-            git -C "${_DEFINITIONS}" reset --hard origin/master; \
+            git -C "${_DEFINITIONS}" reset --hard origin/main; \
-            git -C "${_STATES}" reset --hard origin/master; \
+            git -C "${_STATES}" reset --hard origin/main; \
             echo "Run repairs") \
         && return 0
 
     [ "${_MODE}" == "--test" ] \
-        && git -C "${_ROOT}" pull \
+        && git -C "${_CIS_ROOT}" pull \
         && git -C "${_DEFINITIONS}" pull \
         && git -C "${_STATES}" pull \
         && echo "Run in testMode successfully." \
         && return 0
 
     [ "${_MODE}" == "--scripts" ] \
-        && echo "Host $HOSTNAME updating scripts: ${_ROOT} ..." \
+        && printf "Host $HOSTNAME updating scripts: ${_CIS_ROOT} ... " \
-        && (git -C "${_ROOT}" pull &> /dev/null &) \
+        && (git -C "${_CIS_ROOT}" pull &> /dev/null) \
+        && echo "(done)" \
         && return 0
 
     [ "${_MODE}" == "--definitions" ] \
-        && echo "Host ${HOSTNAME} updating definitions: ${_DEFINITIONS} ..." \
+        && echo "Host ${HOSTNAME} updating definitions: ${_DEFINITIONS} ... " \
-        && (git -C "${_DEFINITIONS}" pull &> /dev/null &) \
+        && (git -C "${_DEFINITIONS}" pull &> /dev/null) \
+        && echo "(done)" \
         && return 0
 
     [ "${_MODE}" == "--states" ] \
-        && echo "Host ${HOSTNAME} updating states: ${_STATES} ..." \
+        && echo "Host ${HOSTNAME} updating states: ${_STATES} ... " \
-        && (git -C "${_STATES}" pull &> /dev/null &) \
+        && (git -C "${_STATES}" pull &> /dev/null) \
+        && echo "(done)" \
         && return 0
 
-    echo "Host ${HOSTNAME} updating ${_MODE}:" \
+    [ "${_MODE}" == "--core" ] \
-        && echo "  - ${_ROOT}" \
+        && echo "Host ${HOSTNAME} updating core including scripts, definitions and states: ${_STATES} ... " \
-        && echo "  - ${_DEFINITIONS}" \
+        && (git -C "${_CIS_ROOT}" pull &> /dev/null) \
-        && echo "  - ${_STATES}"
+        && (git -C "${_DEFINITIONS}" pull &> /dev/null) \
-    git -C "${_ROOT}" pull &> /dev/null
+        && (git -C "${_STATES}" pull &> /dev/null) \
-    git -C "${_DEFINITIONS}" pull &> /dev/null
+        && echo "(done)" \
-    git -C "${_STATES}" pull &> /dev/null
+        && return 0
+
+    echo "FAILED: an error occurred during an update."
+    return 1
 }
 
 # sanitizes all parameters
-update_repositories \
+update_repositories "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
+    && exit 0
-    && exit 0 || exit 1
+
+exit 1