Scripts for setting up nginx

.gitignore

@@ -0,0 +1,10 @@
+# Ignore the file '/overrideOwnDomain' because this is per host individually.
+/overrideOwnDomain
+
+# Ignore the subfolders only, because their content are other git repositories.
+# But 'definitions and 'states' should be prepared by cloning this repository.
+/definitions/*/
+/states/*/
+
+# Ignore environment files
+.env

README.md

@@ -1,35 +1,109 @@
-Infrastructure System (ISS)
+Core Infrastructure System (CIS)
-===========================
+================================
 
-Setup a new host
+The main idea is to use git to keep scripts, definitions and state in sync across all hosts.  
-----------------
+Currently an operating instance uses one repository for this core functionality and scripts,  
+another to distibute the definitions and a third one to share the state.
 
-### Preconditions
+If a script or a definition has to be changed an independent working copy is needed to push the adaptions.  
-To deploy the system you have to clone this repository to the host as root user.
+States can be changed by a host itself. Then we need a mechanism that informs all hosts to execute a `git pull`.
-Therefore you have to register the SSH public key of that root user as deploy key to allow readonly access to this repository.
+
-We use the modern ed25519 keys, so the public key of root is stored at this location:
+We use a Git server as syncronisation point and use a web hook to send the notification.  
+Because the should not be an agent to be installed on each host, we use jenkins to execute an update script via ssh.
+
+This allows us to use standard software without having to program something that may contain a security problem.
+
+
+
+Setup the first or a new host
+-----------------------------
+
+1. Update the host and ensure git is installed
+2. Set the long hostname (fqdn)
+3. Create ssh keys for user root (ssh key type ed25519)
+
+You can use this script to do so: [prepareThisHostBeforeCloning.sh](./prepareThisHostBeforeCloning.sh)
+
+
+
+### Ensure the existence of the repositories for your definitions and the state
+
+This should be necessary just if you set up the first host.  
+You can use the following scripts to assist the process:
+
+- [prepareDefinitionsRepository.sh](./prepareDefinitionsRepository.sh)
+- [prepareStatesRepository.sh](./prepareStatesRepository.sh)
+
+
+
+### Register the public ssh key of user root
+
+This is an example for `example.net` as domain of the host.
+
+1. __Scripts:__  
+   The public ssh key of the root user must be registered as a deploy key for the this repository,  
+   which grants __readonly access__.
+
+   A root user of a host should only be able to update the local cloned repository (`cis`) to a new version via `git pull`.
+
+2. __Definitions:__  
+   The public ssh key of the root user must be registered as a deploy key for the definitions repository,  
+   which grants __readonly access__.  
+
+   User root should only be able to update the local cloned repository (`cis-definition-example.net`) to a new version via `git pull`.
+
+3. __States:__  
+   The public ssh key of the root user must be registered as a deploy key for the states repository,  
+   which grants __write access__.  
+
+   User root should be able to push new state to the cloned repository (`cis-state-example.net`) via `git push`.
+
+
+
+### Clone the Infrastructure System (cis) repository and complete the setup
+After you registered the printed root's public key of this host you can clone the repository and execute the setup script:
+```sh
+# Note the tailing '/cis', because we want to clone the repository to that folder
+git clone ssh://git@git.example.dev:22448/cis.git /cis
+
+# Execute the setup script
+/cis/setupCoreOntoThisHost.sh
+```
+
+<br>
+<br>
+<br>
+
+
+
+Setup a new host step by step manually
+--------------------------------------
+
+To deploy cis you have to clone this repository to the host as root user.
+Therefore you have to set the correct long hostname (fqdn) create a pair of ssh keys (key type ed25519) for user root  
+and register the SSH public key of root as __deploy key__ to allow readonly access to this repository:
 
 1. First become root:
    ```sh
    sudo -i
    ```
 
-2. Set the long hostname:
+2. Update Ubuntu:
-   ```sh
-   hostnamectl set-hostname "the-new-unique-long-hostname (fqdn, eg.: host1.example.net)"
-   ```
-
-3. Update Ubuntu:
    ```sh
    # DO NOT SKIP THIS STEP
    apt update; apt upgrade -y
    ```
 
-4. Install git if needed:
+3. Install git if needed:
    ```sh
    git --version > /dev/null || apt install git
    ```
 
+4. Set the long hostname:
+   ```sh
+   hostnamectl set-hostname "the-new-unique-long-hostname (fqdn, eg.: host1.example.net)"
+   ```
+
 5. If not exist generate the ssh key pair and print the public key of the user root:
    ```sh
    # -t    type of the key pair
@@ -41,40 +115,17 @@ We use the modern ed25519 keys, so the public key of root is stored at this loca
        || (ssh-keygen \
            -t ed25519 \
            -f "/root/.ssh/id_ed25519" -q -N "" \
-           -C "$(date +%Y%m%d):root@$(hostname -b)" \
+           -C "$(date +%Y%m%d)-root@$(hostname -b)" \
        && cat "/root/.ssh/id_ed25519.pub")
    ```
 
-   This key has to be registerd via gitea web ui as deploy key into the repositories as documented in chapter "Register public host key".
+   This key has to be registerd via gitea web ui as deploy key into this repository.
 
 
 
-### Register public host key
-This is an example for `example.net` as domain of the host owner.
-
-1. Repository `iss`, allow __readonly__ access only.
-2. Repository `iss-definition-example.net`, allow __readonly__ access only.
-3. Repository `iss-state-example.net`, allow __writable__ access.
-
-
-
-### Clone the Infrastructure System (iss) repository
-After you registered the printed root's public key of this host you can clone the repository and execute the setup script:
-```sh
-# Note the tailing '/iss', because we want to clone the repository to that folder
-git clone ssh://git@git.example.dev:22448/iss.git /iss
-
-# Execute the setup script
-/iss/setupCoreOntoThisHost.sh
-```
-
-<br>
-<br>
-<br>
-
 How it works
 ------------
-We add a webhook to each gitea repository that belongs to ISS:
+We add a webhook to each gitea repository that belongs to CIS:
  - __Taget URL:__   https://YOUR.JENKINS.DOMAIN/generic-webhook-trigger/invoke?token=YOUR_TOKEN
  - __HTTP-Method:__ POST
  - __Trigger On:__  Push Events
@@ -94,11 +145,11 @@ cat "${JENKINS_HOME}/.ssh/id_ed25519.pub" \
     || (ssh-keygen \
         -t ed25519 \
         -f "${JENKINS_HOME}/.ssh/id_ed25519" -q -N "" \
-        -C "$(date +%Y%m%d):$(whoami)@$(echo ${JENKINS_URL} | cut -d/ -f3)" \
+        -C "$(date +%Y%m%d)-$(whoami)@$(echo ${JENKINS_URL} | cut -d/ -f3)" \
     && cat "${JENKINS_HOME}/.ssh/id_ed25519.pub")
 
 # add your host here, note the tailing '&' to run it in parallel
-ssh -o StrictHostKeyChecking=no jenkins@192.168.X.Y /iss/update_repositories.sh ( --scripts | --definitions | --states ) &
+ssh -o StrictHostKeyChecking=no jenkins@192.168.X.Y /cis/updateRepositories.sh ( --scripts | --definitions | --states ) &
 
 #wait for all background processes to complete
 wait

core/addAndCheckGitRepository.sh

@@ -6,10 +6,10 @@
 
 
 function checkPermissions(){
-    local _FOLDER _REPOSITORY
+    local _FOLDER _RIGHTS
     _FOLDER="${1:?"Missing first parameter FOLDER"}"
     _RIGHTS="${2:?"Missing second parameter RIGHTS"}"
-    readonly _FOLDER _REPOSITORY
+    readonly _FOLDER _RIGHTS
 
     [ "${_RIGHTS}" == "readonly" ] \
         && [ -d "${_FOLDER}/.git" ] \
@@ -21,30 +21,9 @@ function checkPermissions(){
         && git -C "${_FOLDER}" push --dry-run &> /dev/null \
         && return 0
 
-    echo "FAIL: The rights of the repository are incorrect:  ("$(readlink -f ${0})")"
+    echo "FAIL: The rights of the repository are incorrect:  ("$(readlink -f ${0})")" >&2
-    echo "  - '${_FOLDER}' is not '${_RIGHTS}'"
+    echo "  - '${_FOLDER}' is not '${_RIGHTS}'" >&2
-    echo "  - check the settings of gitea."
+    echo "  - check the settings of gitea." >&2
-    return 1
-}
-
-function checkRemoteRepository() {
-    local _FOLDER _REPOSITORY
-    _FOLDER="${1:?"Missing first parameter FOLDER"}"
-    _REPOSITORY="${2:?"Missing second parameter REPOSITORY"}"
-    readonly _FOLDER _REPOSITORY
-
-    #Should exist after successful clone only, therefore the remote repository exists and was accessible.
-    [ -d "${_FOLDER}/.git" ] \
-        && return 0
-
-    #Checks if repository exists and is accessible.
-    ! [ -d "${_FOLDER}/.git" ] \
-        && git ls-remote "${_REPOSITORY}" \
-        && return 0
-
-    echo "FAIL: The remote repository is not accessible:     ("$(readlink -f ${0})")"
-    echo "  - '${_REPOSITORY}'"
-    echo "  - check the settings of gitea."
     return 1
 }
 
@@ -54,42 +33,67 @@ function cloneOrPull {
     _REPOSITORY="${2:?"Missing second parameter REPOSITORY"}"
     readonly _FOLDER _REPOSITORY
 
-    ! [ -d "${_FOLDER}/.git" ] \
-        && git clone "${_REPOSITORY}" "${_FOLDER}" &> /dev/null \
-        && return 0
-
     [ -d "${_FOLDER}/.git" ] \
         && git -C "${_FOLDER}" pull &> /dev/null \
         && return 0
 
-    echo "FAIL: The local repository is not updatable:       ("$(readlink -f ${0})")"
+    ! [ -d "${_FOLDER}/.git" ] \
-    echo "  - '${_FOLDER}'"
+        && git clone "${_REPOSITORY}" "${_FOLDER}" &> /dev/null \
-    echo "  - check your network and the permissions in gitea."
+        && return 0
+
+    echo "FAIL: The local repository is not updatable:       ("$(readlink -f ${0})")" >&2
+    echo "  - '${_FOLDER}'" >&2
+    echo "  - check your network and the permissions in gitea." >&2
+    return 1
+}
+
+function printRepository(){
+    local _FOLDER _CONFIGURED_REPOSITORY _SUGGESTED_REPOSITORY
+    _FOLDER="${1:?"Missing first parameter FOLDER"}"
+    _CONFIGURED_REPOSITORY="$(git -C "${_FOLDER:?"Missing FOLDER"}" config --get remote.origin.url 2> /dev/null)"
+    _SUGGESTED_REPOSITORY="${2}"
+    readonly _FOLDER _CONFIGURED_REPOSITORY _SUGGESTED_REPOSITORY
+
+    ! [ -z "${_CONFIGURED_REPOSITORY}" ] \
+        && echo "${_CONFIGURED_REPOSITORY}" \
+        && return 0
+
+    while true; do
+        read -e -p "Enter ssh URL to clone Repository: " -i "${_SUGGESTED_REPOSITORY}" _REPOSITORY
+        echo "${_REPOSITORY}" | grep -F 'git@' &> /dev/null \
+            && git ls-remote "${_REPOSITORY}" &> /dev/null \
+            && echo "${_REPOSITORY:?"Missing REPOSITORY: e.g. ssh://git@your.domain.com/cis.git"}" \
+            && return 0
+    done
+
+    echo "FAIL: The remote repository is not accessible:     ("$(readlink -f ${0})")" >&2
+    echo "  - '${_REPOSITORY}'" >&2
+    echo "  - check the settings of gitea." >&2
     return 1
 }
 
 # Note that an unprivileged user can use this script successfully,
 # if no user has to be added to the host because it already exists.
 function addAndCheckGitRepository() {
-    local _FOLDER _REPOSITORY
+    local _FOLDER _REPOSITORY _RIGHTS
     _FOLDER="${1:?"Missing first parameter FOLDER"}"
-    _REPOSITORY="${2:?"Missing second parameter REPOSITORY: e.g. ssh://git@your.domain.com/iss.git "}"
+    _RIGHTS="${2:?"Missing second parameter RIGHTS: (readonly, writable) "}"
-    _RIGHTS="${3:?"Missing third parameter RIGHTS: (readonly, writable) "}"
+    _REPOSITORY="$(printRepository "${_FOLDER}" "${3}")"
-    readonly _FOLDER _REPOSITORY
+    readonly _FOLDER _REPOSITORY _RIGHTS
 
-    checkRemoteRepository "${_FOLDER}" "${_REPOSITORY}" \
+    echo \
-        && cloneOrPull "${_FOLDER}" "${_REPOSITORY}" \
+        && cloneOrPull "${_FOLDER}" "${_REPOSITORY:?"Missing REPOSITORY: e.g. ssh://git@your.domain.com/cis.git"}" \
         && checkPermissions "${_FOLDER}" "${_RIGHTS}" \
         && echo "SUCCESS: The git repository is usable.             ("$(readlink -f ${0})")" \
         && echo "  - remote repository: '${_REPOSITORY}'" \
         && echo "  - local repository:  '${_FOLDER}' (${_RIGHTS})" \
         && return 0
 
-    echo "FAIL: The repository is not functional:            ("$(readlink -f ${0})")"
+    echo "FAIL: The repository is not functional:            ("$(readlink -f ${0})")" >&2
-    echo "  - remote repository: '${_REPOSITORY}'"
+    echo "  - remote repository: '${_REPOSITORY}'" >&2
-    echo "  - local repository:  '${_FOLDER}'"
+    echo "  - local repository:  '${_FOLDER}'" >&2
-    echo "  - due to an error or insufficient rights or"
+    echo "  - due to an error or insufficient rights or" >&2
-    echo "  - one check failed."
+    echo "  - one check failed." >&2
     return 1
 }
 
@@ -98,4 +102,6 @@ addAndCheckGitRepository \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${3} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/addNormalUser.sh

@@ -18,6 +18,8 @@ function addNormalUser() {
         && echo "  - '${_USER}'" \
         && return 0
 
+    # useradd is a low level utility ... use adduser(8) instead.
+    # See: https://askubuntu.com/questions/345974/what-is-the-difference-between-adduser-and-useradd
     [ "$(id -u)" == "0" ] \
         && adduser --gecos 'Normal user' --disabled-password "${_USER}" \
         && chown -R "${_USER}:${_USER}" "/home/${_USER}" \
@@ -27,13 +29,14 @@ function addNormalUser() {
         && echo "  - existing home directories were taken over" \
         && return 0
 
-    echo "FAIL: The user could not be created:               ("$(readlink -f ${0})")"
+    echo "FAIL: The user could not be created:               ("$(readlink -f ${0})")" >&2
-    echo "  - '${_USER}'"
+    echo "  - '${_USER}'" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
 # sanitizes all parameters
-addNormalUser \
+addNormalUser "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
+    && exit 0
-    && exit 0 || exit 1
+
+exit 1

core/addToCrontabEveryHour.sh

@@ -5,14 +5,17 @@
 
 
 
+# Folders always ends with an tailing '/'
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/core/*}/"               #Removes longest  matching pattern '/core/*' from the end
+
 # Note that an unprivileged user can use this script successfully,
 # if no user has to be added to the host because it already exists.
 function addToCrontabEveryHour() {
-    local _ROOT _MINUTE_VALUE _STRING
+    local _MINUTE_VALUE _STRING
-    _ROOT="${0%%/core/*}/"                              #Removes longest  matching pattern '/core/*' from the end
     ! [ -z "${2##*[!0-9]*}" ] && _MINUTE_VALUE=$((${2}%60)) # if second parameter is integer then (minute-value % 60) as safe guard
     _STRING="${_MINUTE_VALUE:?"Missing MINUTE_VALUE"} * * * * ${1:?"Missing first parameter COMMAND"} > /dev/null 2>&1"
-    readonly _ROOT _MINUTE_VALUE _STRING
+    readonly _MINUTE_VALUE _STRING
 
     [ "$(id -u)" == "0" ] \
         && crontab -l | grep -qF "${_STRING:?"Missing CRON_STRING"}" \
@@ -21,11 +24,11 @@ function addToCrontabEveryHour() {
         && return 0
 
     [ "$(id -u)" == "0" ] \
-        && echo "${_ROOT:?"Missing ROOT"}" | grep "home" &> /dev/null \
+        && echo "${_CIS_ROOT:?"Missing CIS_ROOT"}" | grep -F 'home' &> /dev/null \
         && echo "SUCCESS: Although the entry will be skipped:       ("$(readlink -f ${0})")" \
         && echo "  - '${_STRING}'" \
         && echo "  that is because the current environment is:" \
-        && echo "    - ${_ROOT}" \
+        && echo "    - ${_CIS_ROOT}" \
         && return 0
 
     [ "$(id -u)" == "0" ] \
@@ -37,9 +40,9 @@ function addToCrontabEveryHour() {
         && echo "  - '${_STRING}'" \
         && return 0
 
-    echo "FAIL: Entry could not be registered to crontab:    ("$(readlink -f ${0})")"
+    echo "FAIL: Entry could not be registered to crontab:    ("$(readlink -f ${0})")" >&2
-    echo "  - '${_STRING:?"Missing CRON_STRING"}'"
+    echo "  - '${_STRING:?"Missing CRON_STRING"}'" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
@@ -47,4 +50,6 @@ function addToCrontabEveryHour() {
 addToCrontabEveryHour \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/default/etc/sudoers.d/allow-jenkins-updateRepositories

@@ -0,0 +1,6 @@
+Cmnd_Alias C_JENKINS = \
+  /cis/updateRepositories.sh --core, \
+  /cis/updateRepositories.sh --scripts, \
+  /cis/updateRepositories.sh --definitions, \
+  /cis/updateRepositories.sh --states
+jenkins ALL = (root) NOPASSWD: C_JENKINS

core/defineAuthorizedKeysOfUser.sh

@@ -44,24 +44,24 @@ function prepareFolder() {
         && echo "  - '${_SSH_FOLDER}'" \
         && return 0
 
-    echo "FAIL: The ssh folder could not be prepared:        ("$(readlink -f ${0})")"
+    echo "FAIL: The ssh folder could not be prepared:        ("$(readlink -f ${0})")" >&2
-    echo "  - '${_SSH_FOLDER}'"
+    echo "  - '${_SSH_FOLDER}'" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
 function defineAuthorizedKeysOfUser() {
-    local _ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
+    local _CIS_ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
     _DEFINITIONS="$(realpath -s "${1:?"Missing first parameter DEFINITIONS: 'ROOT/definitions/DOMAIN'"}")"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    _CIS_ROOT="${_DEFINITIONS%%/definitions/*}/"  #Removes longest  matching pattern '/definitions/*' from the end
     _DOMAIN="${_DEFINITIONS##*/definitions/}"     #Removes longest  matching pattern '*/definitions/' from the begin
     _DOMAIN="${_DOMAIN%/}"                        #Removes shortest matching pattern '/'              from the end
     #Build from components for safety
-    _DEFINITIONS="${_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
+    _DEFINITIONS="${_CIS_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
 
     _USER="${2:?"Missing second parameter USER"}"
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
+    _CORE_SCRIPTS="${_CIS_ROOT:?"Missing ROOT"}core/"
-    readonly _ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
+    readonly _CIS_ROOT _CORE_SCRIPTS _DOMAIN _DEFINITIONS _USER
 
     case "${_USER:?"Missing USER"}" in
         root)
@@ -83,4 +83,6 @@ function defineAuthorizedKeysOfUser() {
 defineAuthorizedKeysOfUser \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/ensureUsageOfDefinitions.sh

@@ -12,46 +12,76 @@ function printIfEqual() {
 }
 
 function isCoreDefinition() {
-    echo "${1:?"Missing first parameter FILE"}" | grep "/root/.ssh/authorized_keys" &> /dev/null \
+    echo "${1:?"Missing first parameter FILE"}" | grep -F '/root/.ssh/authorized_keys' &> /dev/null \
         && return 0
 
-    echo "${1:?"Missing first parameter FILE"}" | grep "/home/jenkins/.ssh/authorized_keys" &> /dev/null \
+    echo "${1:?"Missing first parameter FILE"}" | grep -F '/home/jenkins/.ssh/authorized_keys' &> /dev/null \
         && return 0
 
-    echo "${1:?"Missing first parameter FILE"}" | grep "/etc/sudoers.d/allow-jenkins-updateRepositories" &> /dev/null \
+    echo "${1:?"Missing first parameter FILE"}" | grep -F '/etc/sudoers.d/allow-jenkins-updateRepositories' &> /dev/null \
         && return 0
 
     return 1
 }
 
+function filterInvalidAuthorizedKeysFilesOfRoot() {
+    local _FILE_DEFINED
+    _FILE_DEFINED="${1:?"Missing DEFINITION FILE"}"
+    readonly _FILE_DEFINED
+
+    #If the full filename contains 'root/.ssh/authorized_keys' then check the content.
+    #Skip lines starting with '#' and if at least one remaining line contains 'ssh' and '@' then print the filename. 
+    echo "${_FILE_DEFINED}" | grep -F 'root/.ssh/authorized_keys' &> /dev/null \
+        && grep -vE '^[[:blank:]]*#' "${_FILE_DEFINED}" | grep -F 'ssh' | grep -F '@' &> /dev/null \
+        && echo "${_FILE_DEFINED}" \
+        && return 0
+
+    #If the full filename contains 'root/.ssh/authorized_keys' print nothing because the file has to be invalid.
+    echo "${_FILE_DEFINED}" | grep -F 'root/.ssh/authorized_keys' &> /dev/null \
+        && echo \
+        && return 0
+
+    #Print the full filename because it does not contain 'root/.ssh/authorized_keys'
+    echo "${_FILE_DEFINED}"
+    return 0
+}
+
 function printSelectedDefinition() {
-    local _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
+    local _DEFINITIONS  _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
-    _CORE_FILE_DEFINED_ALL_HOSTS="${1:?"Missing DEFINITIONS"}/core/all${2:?"Missing CURRENT_FULLFILE"}"
+    _DEFINITIONS="${1:?"Missing CIS_ROOT"}definitions/${2:?"Missing DOMAIN"}/"
-    _CORE_FILE_DEFINED_THIS_HOST="${1:?"Missing DEFINITIONS"}/core/$(hostname -s)${2:?"Missing CURRENT_FULLFILE"}"
+    _CORE_DEFAULT_ALL_HOSTS="${1:?"Missing CIS_ROOT"}core/default${3:?"Missing CURRENT_FULLFILE"}"
-    _FILE_DEFINED_ALL_HOSTS="${1:?"Missing DEFINITIONS"}/hosts/all${2:?"Missing CURRENT_FULLFILE"}"
+    _CORE_FILE_DEFINED_ALL_HOSTS="${_DEFINITIONS:?"Missing DEFINITIONS"}core/all${3:?"Missing CURRENT_FULLFILE"}"
-    _FILE_DEFINED_THIS_HOST="${1:?"Missing DEFINITIONS"}/hosts/$(hostname -s)${2:?"Missing CURRENT_FULLFILE"}"
+    _CORE_FILE_DEFINED_THIS_HOST="${_DEFINITIONS:?"Missing DEFINITIONS"}core/$(hostname -s)${3:?"Missing CURRENT_FULLFILE"}"
-    readonly _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
+    _FILE_DEFINED_ALL_HOSTS="${_DEFINITIONS:?"Missing DEFINITIONS"}hosts/all${3:?"Missing CURRENT_FULLFILE"}"
+    _FILE_DEFINED_THIS_HOST="${_DEFINITIONS:?"Missing DEFINITIONS"}hosts/$(hostname -s)${3:?"Missing CURRENT_FULLFILE"}"
+    readonly _DEFINITIONS _CORE_FILE_DEFINED_ALL_HOSTS _CORE_FILE_DEFINED_THIS_HOST _FILE_DEFINED_ALL_HOSTS _FILE_DEFINED_THIS_HOST
 
     #The following are special definitions that affect the core functionality.
     #Try this host first because it should be priorized.
-    isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_CORE_FILE_DEFINED_THIS_HOST}" ] \
-        && echo "${_CORE_FILE_DEFINED_THIS_HOST}" \
+        && filterInvalidAuthorizedKeysFilesOfRoot "${_CORE_FILE_DEFINED_THIS_HOST}" \
         && return 0
 
     #The following are special definitions that affect the core functionality.
-    isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_CORE_FILE_DEFINED_ALL_HOSTS}" ] \
-        && echo "${_CORE_FILE_DEFINED_ALL_HOSTS}" \
+        && filterInvalidAuthorizedKeysFilesOfRoot "${_CORE_FILE_DEFINED_ALL_HOSTS}" \
+        && return 0
+
+    #The following are special definitions that affect the core functionality.
+    isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
+        && [ -s "${_CORE_DEFAULT_ALL_HOSTS}" ] \
+        && filterInvalidAuthorizedKeysFilesOfRoot "${_CORE_DEFAULT_ALL_HOSTS}" \
         && return 0
 
     #Try this host first because it should be priorized.
-    ! isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    ! isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_FILE_DEFINED_THIS_HOST}" ] \
         && echo "${_FILE_DEFINED_THIS_HOST}" \
         && return 0
 
-    ! isCoreDefinition "${2:?"Missing CURRENT_FULLFILE"}" \
+    ! isCoreDefinition "${3:?"Missing CURRENT_FULLFILE"}" \
         && [ -s "${_FILE_DEFINED_ALL_HOSTS}" ] \
         && echo "${_FILE_DEFINED_ALL_HOSTS}" \
         && return 0
@@ -71,11 +101,6 @@ function createSymlinkToDefinition() {
         && [ "$(sha256sum "${_DEFINED_FULLFILE}" | cut -d' ' -f1)" == "$(sha256sum "${_CURRENT_FULLFILE}" | cut -d' ' -f1)" ] \
         && echo "The content of the current file already matches the definition, but it will be replaced by a symlink..."
 
-    [ -f "${_CURRENT_FULLFILE}" ] \
-        && [ "$(sha256sum "${_DEFINED_FULLFILE}" | cut -d' ' -f1)" == "$(sha256sum "${_CURRENT_FULLFILE}" | cut -d' ' -f1)" ] \
-        && echo "The content of the current file already matches the definition, but it will be replaced by a symlink..."
-
-
     [ -f "${_CURRENT_FULLFILE}" ] \
         && mv "${_CURRENT_FULLFILE:?"Missing CURRENT_FULLFILE"}" "${_SAVED_FULLFILE:?"Missing SAVED_FULLFILE"}" \
         && echo "Current file has been backed up to: '${_SAVED_FULLFILE}'"
@@ -92,17 +117,17 @@ function createSymlinkToDefinition() {
 }
 
 function ensureUsageOfDefinitions() {
-    local _ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
+    local _CIS_ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
-    _DEFINITIONS="$(realpath -s "${1:?"Missing first parameter DEFINITIONS: 'ROOT/definitions/DOMAIN'"}")"
+    _DEFINITIONS="$(realpath -s "${1:?"Missing first parameter DEFINITIONS: 'ROOT/definitions/DOMAIN'"}")/"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    _CIS_ROOT="${_DEFINITIONS%%/definitions/*}/"  #Removes longest  matching pattern '/definitions/*' from the end
     _DOMAIN="${_DEFINITIONS##*/definitions/}"     #Removes longest  matching pattern '*/definitions/' from the begin
     _DOMAIN="${_DOMAIN%/}"                        #Removes shortest matching pattern '/'              from the end
     #Build from components for safety
-    _DEFINITIONS="$(printIfEqual "${_DEFINITIONS}" "${_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}")"
+    _DEFINITIONS="$(printIfEqual "${_DEFINITIONS}" "${_CIS_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/")"
 
 
-    _CURRENT_FOLDER="$(dirname "${2:?"Missing second parameter CURRENT_FULLFILE"}")"
+    _CURRENT_FULLFILE="${2:?"Missing second parameter CURRENT_FULLFILE"}"
-    _CURRENT_FOLDER="${_CURRENT_FOLDER%/}/"  #Removes shortest matching pattern '/' from the end
+    _CURRENT_FOLDER="${_CURRENT_FULLFILE%/*}/"  #Removes shortest matching pattern '/*' from the end
     ! [ -d "${_CURRENT_FOLDER}" ] \
         && echo "FAIL: The folder cannot be read:                   ("$(readlink -f ${0})")" \
         && echo "  - '${_CURRENT_FOLDER}'" \
@@ -118,10 +143,16 @@ function ensureUsageOfDefinitions() {
     _CURRENT_FULLFILE="${_CURRENT_FOLDER:?"Missing CURRENT_FOLDER"}${_CURRENT_FILE:?"Missing CURRENT_FILE"}"
 
 
-    _DEFINED_FULLFILE="$(printSelectedDefinition "${_DEFINITIONS}" "${_CURRENT_FULLFILE}")"
+    _DEFINED_FULLFILE="$(printSelectedDefinition "${_CIS_ROOT}" "${_DOMAIN}" "${_CURRENT_FULLFILE}")"
     _NOW="$(date +%Y%m%d_%H%M)"
-    _SAVED_FULLFILE="${_CURRENT_FULLFILE}-backup@${_NOW:?"Missing NOW"}"
+    _SAVED_FULLFILE="${_CURRENT_FULLFILE}.backup@${_NOW:?"Missing NOW"}"
-    readonly _ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
+    readonly _CIS_ROOT _CURRENT_FILE _CURRENT_FOLDER _CURRENT_FULLFILE _DEFINITIONS _DOMAIN _DEFINED_FULLFILE _NOW _SAVED_FULLFILE
+
+    [ -z "${_DEFINED_FULLFILE}" ] \
+        && echo \
+        && echo "URGENT WARNING: If an 'authorized_keys' file of root is replaced by an invalid version," \
+        && echo "                you may lose access to this host!" \
+        && echo
 
     ! [ -f "${_DEFINED_FULLFILE}" ] \
         && echo "FAIL: No definition available for this file:       ("$(readlink -f ${0})")" \
@@ -138,11 +169,11 @@ function ensureUsageOfDefinitions() {
         && echo "  - '${_DEFINED_FULLFILE}'" \
         && return 0
 
-    echo "${_ROOT:?"Missing ROOT"}" | grep "home" &> /dev/null \
+    echo "${_CIS_ROOT:?"Missing CIS_ROOT"}" | grep -F 'home' &> /dev/null \
         && echo "SUCCESS: Although this definition will be skipped: ("$(readlink -f ${0})")" \
         && echo "  - '${_DEFINED_FULLFILE}'" \
         && echo "  that is because the current environment is:" \
-        && echo "    - ${_ROOT}" \
+        && echo "    - ${_CIS_ROOT}" \
         && echo "  following file is in use:" \
         && echo "    - $(readlink -f "${_CURRENT_FULLFILE}")" \
         && return 0
@@ -165,8 +196,8 @@ function ensureUsageOfDefinitions() {
         && echo "- '${_DEFINED_FULLFILE}'" \
         && return 0
 
-    echo "FAIL: The definition could not be ensured:         ("$(readlink -f ${0})")"
+    echo "FAIL: The definition could not be ensured:         ("$(readlink -f ${0})")" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
@@ -174,4 +205,6 @@ function ensureUsageOfDefinitions() {
 ensureUsageOfDefinitions \
     "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
     "$(echo ${2} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && exit 0 || exit 1
+    && exit 0
+
+exit 1

core/printCisRoot.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/core/*}/"               #Removes longest  matching pattern '/core/*' from the end
+
+[ -d "${_CIS_ROOT}" ] \
+    && [ -d "${_CIS_ROOT}definitions/" ] \
+    && [ -d "${_CIS_ROOT}states/" ] \
+    && echo "${_CIS_ROOT}" \
+    && exit 0
+
+echo "FAIL: Unable to detect CIS_ROOT" >&2
+exit 1

core/printOwnDomain.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+
+#WARNING: Used for core functionality in setup.sh
+#         DO NOT rename the script and test changes well!
+
+
+
+# Folders always ends with an tailing '/'
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/core/*}/"               #Removes longest  matching pattern '/core/*' from the end
+_OVERRIDE_DOMAIN_FILE="${_CIS_ROOT:?"Missing CIS_ROOT"}overrideOwnDomain"
+
+# There has to be one dot at least.
+_BOOT_DOMAIN="$(hostname -b | grep -F '.' | cut -d. -f2-)"
+
+# Take OVERRIDING_DOMAIN_FILE without empty lines and comments, then take the first line without leading spaces
+_OVERRIDE_DOMAIN="$(grep -vE '^[[:space:]]*$|^[[:space:]]*#' "${_OVERRIDE_DOMAIN_FILE}" 2> /dev/null | head -n 1 | xargs)"
+
+! [ -z "${_OVERRIDE_DOMAIN}" ] \
+    && [ "${_OVERRIDE_DOMAIN}" != "${_BOOT_DOMAIN}" ] \
+    && echo "WARNING: Domain has been overridden by: ${_OVERRIDE_DOMAIN_FILE}" >&2 \
+    && echo "${_OVERRIDE_DOMAIN}" \
+    && exit 0
+
+! [ -z "${_BOOT_DOMAIN}" ] \
+    && echo "${_BOOT_DOMAIN}" \
+    && exit 0
+
+echo "It was impossible to find out the domain of this host, please prepare this host first." >&2
+exit 1

prepareDefinitionsRepository.sh

@@ -0,0 +1,69 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && echo "This script prepares the user 'root' of this host and the host itself," \
+    && echo "so this script is allowed to be executed if you are root only." \
+    && exit 1
+
+# There has to be one dot at least.
+_BOOT_DOMAIN="$(hostname -b | grep -F '.' | cut -d. -f2-)"
+
+[ -z "${_BOOT_DOMAIN}" ] \
+    && echo "It was impossible to find out the domain of this host, please prepare this host first." \
+    && exit 1
+
+_REOPSITORY_NAME="cis-definition-${_BOOT_DOMAIN}"
+
+
+
+#Generate file 'README.md'
+mkdir -p /tmp/skeleton/definition
+cat << EOF > /tmp/skeleton/definition/README.md
+This repository contains the definitions of the domain “$_BOOT_DOMAIN” by the Core Infrastructure System.
+EOF
+
+
+
+#Use current file 'authorized_keys' of root as definition
+mkdir -p /tmp/skeleton/definition/core/all/root/.ssh
+cp /root/.ssh/authorized_keys /tmp/skeleton/definition/core/all/root/.ssh/authorized_keys
+
+
+
+#Generate file 'authorized_keys' for user jenkins
+mkdir -p /tmp/skeleton/definition/core/all/home/jenkins/.ssh
+cat << EOF > /tmp/skeleton/definition/core/all/home/jenkins/.ssh/authorized_keys
+#------------------------------------------------------
+# Enter the public ssh key of your jenkins server here.
+#------------------------------------------------------
+EOF
+
+
+
+cat << EOF
+
+The first content for your repository for the definitions of the '$_BOOT_DOMAIN' domain has been created.
+
+Please create a definition repository.
+To follow the naming convention name it '$_REOPSITORY_NAME'
+
+Please DO NOT use the SSH key of root for this.
+Maybe you can use https and user password for pushing the first commit.
+
+Go to folder '/tmp/skeleton/definition' and check the content of all 'authorized_keys' files,
+correct them if required to prevent losing access to your hosts.
+
+The public ssh key of your jenkins server has to be added.
+
+Only now follow the instructions as our git server shows.
+For example:
+
+  cd /tmp/skeleton/definition
+  git init
+  git checkout -b main
+  git add .
+  git commit -m "first core definitions"
+  git remote add origin https://git.example.dev/[SOME_PATH/]$_REOPSITORY_NAME.git
+  git push -u origin main
+
+EOF

prepareStatesRepository.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && echo "This script prepares the user 'root' of this host and the host itself," \
+    && echo "so this script is allowed to be executed if you are root only." \
+    && exit 1
+
+# There has to be one dot at least.
+_BOOT_DOMAIN="$(hostname -b | grep -F '.' | cut -d. -f2-)"
+
+[ -z "${_BOOT_DOMAIN}" ] \
+    && echo "It was impossible to find out the domain of this host, please prepare this host first." \
+    && exit 1
+
+_REOPSITORY_NAME="cis-state-${_BOOT_DOMAIN}"
+
+
+
+#Generate README.md
+mkdir -p /tmp/skeleton/state
+cat << EOF > /tmp/skeleton/state/README.md
+This repository contains the states of the domain “$_BOOT_DOMAIN” by the Core Infrastructure System.
+EOF
+
+
+
+cat << EOF
+
+The first content for your repository for the state of the '$_BOOT_DOMAIN' domain has been created.
+
+Please create a states repository.
+To follow the naming convention name it '$_REOPSITORY_NAME'
+
+Please DO NOT use the SSH key of root for this.
+Maybe you can use https and user password for pushing the first commit.
+
+Then go to folder '/tmp/skeleton/state' and follow the instructions as your git server shows.
+For example:
+
+  cd /tmp/skeleton/state
+  git init
+  git checkout -b main
+  git add .
+  git commit -m "first state"
+  git remote add origin https://git.example.dev/[SOME_PATH/]$_REOPSITORY_NAME.git
+  git push -u origin main
+
+EOF

prepareThisHostBeforeCloning.sh

@@ -8,46 +8,66 @@
 function setNeededHostnameOrExit() {
     _FQDN="${1:?"Missing unique long hostname (fqdn, eg.: host1.example.net) for this host as first parameter."}"
 
-    echo "${_FQDN}" | grep '\.' &> /dev/null \
+    echo "${_FQDN}" | grep -F '.' &> /dev/null \
         && hostnamectl set-hostname "${_FQDN}" \
         && return 0
 
-    echo "FAILED: setting full qualified domain name, given value was:"
+    echo "FAILED: setting full qualified domain name does not contain a domain,"
-    echo "  - ${_FQDN}"
+    echo "        given value was: ${_FQDN}"
     exit 1
 }
 
-function prepare() {
+function prepareThisHost() {
     git --version > /dev/null || (apt update; apt upgrade -y; apt install git)
 
     echo
     echo "Public SSH-Key for root@$(hostname -b):"
+    cat "/root/.ssh/id_ed25519.pub" \
+        && return 0
+
     # -t    type of the key pair
     # -f    defines the filenames (we use the standard for the selected type here)
     # -q    quiet, no output or interaction
     # -N "" means the private key will not be secured by a passphrase
     # -C    defines a comment
-    cat "/root/.ssh/id_ed25519.pub" \
+    ssh-keygen \
-        || (ssh-keygen \
         -t ed25519 \
         -f "/root/.ssh/id_ed25519" -q -N "" \
-            -C "$(date +%Y%m%d)-root@$(hostname -b)" \
+        -C "$(date +%Y%m%d)-root@$(hostname -b)"
-        && cat "/root/.ssh/id_ed25519.pub")
+
+    cat "/root/.ssh/id_ed25519.pub" \
+        && return 0
 
     echo
-    echo "Now you have to register the public ssh-key from above into your git-server to grant these access rights:"
+    echo "FAILED: somthing went wrong during the generation the ssh keys."
+    echo "        These keys are mandantory. You can try to restart this script."
+    echo
+}
+
+function showFurtherSteps() {}
+    echo
+    echo "IMPORTANT: It is assumed that repositories for definitions and states already exist"
+    echo "           and comply with the naming convention."
+    echo "  Otherwise, these repositories must be created first!"
+    echo
+    echo "To grant the correct access rights, you have to register the above-mentioned ssh key,"
+    echo "as deploy key in these repositories of the Git server:"
     echo "  - scripts repository     (allow readonly access only),"
     echo "  - definitions repository (allow readonly access only),"
     echo "  - states repository      (allow writable access)."
     echo
-    echo "After all access rights are granted you can clone the Infrastructure System:"
+    echo "After all access rights are granted you can clone the Core Infrastructure System:"
-    echo "  e.g.: git clone ssh://git@git.example.dev:22448/iss.git /iss"
+    echo "  e.g.: git clone ssh://git@git.example.dev:22448/cis.git /cis"
     echo
-    echo "Finally call 'setupCoreOntoThisHost.sh' from the root directory of the repository:"
+    echo "Finally call 'setupCoreOntoThisHost.sh' from the root directory:"
-    echo "  e.g.: /iss/setupCoreOntoThisHost.sh"
+    echo "  e.g.: /cis/setupCoreOntoThisHost.sh"
     echo
 }
 
 # sanitizes all parameters
 setNeededHostnameOrExit "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    && prepare
+    && prepareThisHost \
+    && showFurtherSteps \
+    && exit 0
+
+exit 1

script/check/README.md

@@ -0,0 +1,4 @@
+runAllChecks.sh
+===============
+
+This script processes all checks of a host to verify the right configuration.

script/check/host/all/app_docker-compose_is_installed.check.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+_CURRENT_APP='docker compose version'
+
+${_CURRENT_APP} > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/app_docker_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+docker --version > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/app_nginx_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+nginx -v > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/app_nginx_starts_reliable.check.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# Fail because of unnecessary custom config
+grep "Wants=network-online.target" /lib/systemd/system/nginx.service > /dev/null 2>&1 \
+    && [ -f "/etc/systemd/system/nginx.service" ] \
+    && exit 1
+
+# Success if system config is ok
+grep "Wants=network-online.target" /lib/systemd/system/nginx.service > /dev/null 2>&1 \
+    && exit 0
+
+# Success if custom config fixes system config
+grep "Wants=network-online.target" /etc/systemd/system/nginx.service > /dev/null 2>&1 \
+    && exit 0
+
+exit 1

script/check/host/all/core_cron_starts_setup_as_fallback.check.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+crontab -l | grep -E "[0-9]{1,2}[ \*]{8}[[:blank:]]*\/cis\/setupCoreOntoThisHost.sh" > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/core_git_app_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+git --version > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/core_hostname_is_long.check.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/hostname'
+
+#The file must be readable, then
+#the number of lines containing a '.' must be zero.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "$(grep -cF '.' "${_CURRENT_FILE}")" -gt 0 ] \
+    && exit 0
+exit 1

script/check/host/all/core_ssh_app_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+ssh -V > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/core_ssh_authorized_keys_of_jenkins_points_to_definitions.check.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+_CURRENT_FILE='/home/jenkins/.ssh/authorized_keys'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#File has to be readable, then
+#search for '/definitions/' in the path of current file, after readlink expanded a potential symlink.
+[ -r "${_CURRENT_FILE}" ] \
+    && readlink -f "${_CURRENT_FILE}" | grep -q "/definitions/" \
+    && exit 0
+
+exit 1

script/check/host/all/core_ssh_authorized_keys_of_root_is_empty_or_points_to_definitions.check.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+_CURRENT_FILE='/root/.ssh/authorized_keys'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#No file is ok
+[ ! -e "${_CURRENT_FILE}" ] \
+    && exit 0
+
+#The file must be readable, then
+#all comments and all blank lines are removed, after which the number of remaining lines must be zero.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "0" == "$(cat "${_CURRENT_FILE}" | sed 's/[[:blank:]]*#.*//' | sed '/^$/d' | grep -c .)" ] \
+    && exit 0
+
+#File has to be readable, then
+#search for '/definitions/' in the path of current file, after readlink expanded a potential symlink.
+[ -r "${_CURRENT_FILE}" ] \
+    && readlink -f "${_CURRENT_FILE}" | grep -q "/definitions/" \
+    && exit 0
+
+exit 1

script/check/host/all/core_ssh_key_of_root_exists_as_expected.check.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_CURRENT_FILE='/root/.ssh/id_ed25519'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#File has to be readable and no passphrase should be needed.
+ssh-keygen -y -P "" -f "${_CURRENT_FILE}" &> /dev/null \
+    && exit 0
+
+exit 1

script/check/host/all/core_sudoers_file_of_jenkins_points_to_definitions.check.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/sudoers.d/allow-jenkins-updateRepositories'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+#File has to be readable, then
+#search for '/definitions/' in the path of current file, after readlink expanded a potential symlink.
+[ -r "${_CURRENT_FILE}" ] \
+    && readlink -f "${_CURRENT_FILE}" | grep -q "/definitions/" \
+    && exit 0
+
+exit 1

script/check/host/all/core_user_jenkins_exists.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_USER='jenkins'
+
+[ "$(id -u)" != "0" ] \
+    && printf "(INSUFFICENT RIGHTS) " \
+    && exit 1
+
+id -u "${_CURRENT_USER}" > /dev/null 2>&1 \
+    && exit 0
+
+exit 1

script/check/host/all/system_localtime_contains_cet_and_cest.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/localtime'
+
+#The file must be readable, then
+#the number of lines containing "CET" must be greater than zero, and
+#the number of lines containing "CEST" must also be greater than zero.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "$(zdump -v "${_CURRENT_FILE}" | head -n 10 | grep 'CET' | grep -c .)" -gt "0" ] \
+    && [ "$(zdump -v "${_CURRENT_FILE}" | head -n 10 | grep 'CEST' | grep -c .)" -gt "0" ] \
+    && exit 0
+exit 1

script/check/host/all/system_timezone_is_berlin.check.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+_CURRENT_FILE='/etc/timezone'
+
+#The file must be readable, then
+#the number of lines containing "Europe/Berlin" must be one.
+[ -r "${_CURRENT_FILE}" ] \
+    && [ "1" == "$(cat "${_CURRENT_FILE}" | grep 'Europe/Berlin' | grep -c .)" ] \
+    && exit 0
+exit 1

script/check/host/all/system_unattended_upgrades_are_disabled.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+! systemctl is-enabled unattended-upgrades.service > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_app_is_installed.check.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+zfs --version > /dev/null 2>&1 \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_atime_of_rootfs_zpool1.check.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+_CURRENT_POOL='zpool1'
+
+#Check if the tool 'zfs' is available, then
+#retrieve the property 'atime' from 'zpool1', without header and compare the result with 'off'
+#because this the feature 'atime' logs each access, there are many avoidable writes.
+
+#Set with: 'zfs set atime=off zpool1'
+zfs version &> /dev/null  \
+    && [ "$(zfs get atime -Ho value ${_CURRENT_POOL} 2> /dev/null)" == "off" ] \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_compression_of_rootfs_zpool1.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_POOL='zpool1'
+
+#Check if the tool 'zfs' is available, then
+#retrieve the property 'compression' from 'zpool1', without header and compare the result with 'lz4'
+
+#Set with: 'zfs set compression=lz4 zpool1'
+zfs version &> /dev/null  \
+    && [ "$(zfs get compression -Ho value ${_CURRENT_POOL} 2> /dev/null)" == "lz4" ] \
+    && exit 0
+exit 1

script/check/host/all/system_zfs_mountpoint_of_rootfs_zpool1.check.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+_CURRENT_ZFS='zpool1'
+
+#Check if the tool 'zfs' is available, then
+#retrieve the property 'mountpoint' from 'zpool1', without header and compare the result with '/zpool1'
+
+#Set with: 'zfs set mountpount=default'
+zfs version &> /dev/null \
+    && [ "$(zfs get mountpoint -Ho value ${_CURRENT_ZFS} 2> /dev/null)" == "/${_CURRENT_ZFS}" ] \
+    && exit 0
+exit 1

script/check/host/all/system_zpool_alignment_of_pool.check.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+_CURRENT_POOL='zpool1'
+
+#Check if the tool 'zpool' is available, then
+#retrieve the property 'ashift' from 'zpool1', without header and compare the result with '12'
+zpool version &> /dev/null \
+    && [ "$(zpool get ashift -Ho value ${_CURRENT_POOL} 2> /dev/null)" == "12" ] \
+    && exit 0
+exit 1

script/check/runAllChecks.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SCRIPT%%/script/check/*}/"               #Removes longest  matching pattern '/script/check/*' from the end
+_SCRIPT_PATH="${_CIS_ROOT:?"Missing CIS_ROOT"}script/"
+_OWN_DOMAIN="$(${_CIS_ROOT}core/printOwnDomain.sh)"
+_OWN_DEFINITIONS="${_CIS_ROOT}definitions/${_OWN_DOMAIN:?"Missing OWN_DOMAIN"}/"
+
+
+
+function run_as_root() {
+    [ "0" == "$(id -u)" ] \
+        && echo OK \
+        && return 0
+
+    echo FAIL
+    return 1
+}
+
+function scripts_are_updateable_by_git() {
+        git -C "${_SCRIPT_PATH:?"Missing SCRIPT_PATH"}" pull > /dev/null 2>&1 \
+        && echo OK \
+        && return 0
+
+    echo FAIL
+    return 1
+}
+
+function allChecks() {
+    local _CHECK_PATH _MODE_PATH
+    _CHECK_PATH="${1:?"allChecks(): Missing first parameter CHECK_PATH"}check/"
+    _MODE_PATH="${2:-all}/"
+    readonly _CHECK_PATH _MODE_PATH
+
+    echo "  - ${_CHECK_PATH}host/${_MODE_PATH}*.check.sh"
+    [ "$(ls -1 ${_CHECK_PATH}host/${_MODE_PATH}*.check.sh 2> /dev/null | grep -cE '.*')" == "0"  ] \
+       && echo "  nothing to do" \
+       && return 0
+
+    for _CURRENT_CHECK in ${_CHECK_PATH}host/${_MODE_PATH}*.check.sh; do
+        _NAME="$(basename ${_CURRENT_CHECK} | cut -d'.' -f1)"
+        _CONTEXT="$(echo ${_NAME} | cut -d'_' -f1)"
+        _CHECK="$(echo ${_NAME} | cut -d'_' -f2- | tr '_' ' ')"
+        _RESULT="$("${_CURRENT_CHECK}" && echo OK || echo FAIL)"
+        echo "  ${_CONTEXT^^} ${_CHECK}: ${_RESULT}"
+    done
+}
+
+echo "PRECONDITION run as root: $(run_as_root)"
+echo "PRECONDITION scripts are updateable by git: $(scripts_are_updateable_by_git)"
+echo
+echo "Check all (common):"
+allChecks "${_SCRIPT_PATH}"
+echo "Check all (own):"
+allChecks "${_OWN_DEFINITIONS}"
+echo "Check this host:"
+allChecks "${_OWN_DEFINITIONS}" "$(hostname -s)"

script/docker/printAllBridgedNetworksWithRange.sh

@@ -0,0 +1,4 @@
+#/bin/bash
+
+docker network inspect $(docker network ls | grep -F 'bridge' | cut -d' ' -f1) \
+    | jq -r '.[] | .Name + " " + .IPAM.Config[0].Subnet' -

script/docker/printContainerNamesOfComposition.sh

@@ -0,0 +1,23 @@
+#/bin/bash
+
+_COMPOSITION_FILE="${1:-./docker-compose.yml}"
+
+[ -d "${_COMPOSITION_FILE}" ] \
+    && echo "A valid composition file ('docker-compose.yml') is needed. Given parameter was: ${_COMPOSITION_FILE}" >&2 \
+    && exit 1
+
+_DOCKER_COMPOSE_CMD=""
+
+[ "${_DOCKER_COMPOSE_CMD}" = "" ] \
+    && docker compose version 2> /dev/null | grep -q version \
+    && _DOCKER_COMPOSE_CMD="docker compose"
+
+[ "${_DOCKER_COMPOSE_CMD}" = "" ] \
+    && docker-compose version 2> /dev/null | grep -q version \
+    && _DOCKER_COMPOSE_CMD="docker-compose"
+
+[ "${_DOCKER_COMPOSE_CMD}" = "" ] \
+    && echo "Command 'docker compose' not found" >&2 \
+    && exit 1
+
+${_DOCKER_COMPOSE_CMD} -f "${_COMPOSITION_FILE}" images | tail -n +2 | cut -d' ' -f1

script/host/net/printAllShortManagedHostnamesFromHostsFile.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# Select just lines containing 'managedHost'.
+# 1.) Remove everything after a '#' (including the #).
+# 2.) Remove every indenting.
+# 3.) Remove blanks (spaces or tabs) at the end of lines.
+# 4.) Replace blanks (spaces or tabs) with one ';' between the values.
+# 5.) Delete empty lines.
+# Then cut the second field
+# Then cut the first field to get the short hostname
+grep 'managedHost' /etc/hosts \
+    | sed -e 's/#.*//' \
+        -e 's/^[[:blank:]]*//' \
+        -e 's/[[:blank:]]*$//' \
+        -e 's/\s\+/;/g' \
+        -e '/^$/d' \
+    | cut -d';' -f2 \
+    | cut -d'.' -f1
+

script/host/net/printOwnDefaultMACAdress.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+cat /sys/class/net/e*/address \
+    | head -n 1

script/host/net/printOwnIPv4Adress.sh

@@ -0,0 +1,108 @@
+#!/bin/bash
+
+#grep -E '(:|^(127|169\.254|10|172\.(1(6|7|8|9)|2[0-9]|30|31)|192\.168|(22(4|5|6|7|8|9)|23(0|1|2|3|4|5|6|7|8|9))).*)' findet:
+#  loopback:  127.0.0.0/8
+#  linklocal: 169.254.0.0/16
+#  private:   10.0.0.0/8,
+#             172.16.0.0/12,   (172.16… bis 172.31…)
+#             192.168.0.0/16
+# multicast:  224.0.0.0/4      (224… bis 239…)
+
+
+function all() {
+    # Select just lines containing 'inet'.
+    # 1.) Remove every indenting.
+    # 2.) Remove 'inet '.
+    # 3.) Remove everything after a '/' (including the /).
+    ip -4 addr \
+        | grep 'inet' \
+        | sed -e 's/^[[:blank:]]*//' \
+            -e 's/inet //' \
+            -e 's/\/.*//'
+}
+
+function routed() {
+    local _DEVICE
+    _DEVICE="$(ip -4 route show default | xargs -n 1 | grep -A1 -i dev | tail -n 1)"
+    readonly _DEVICE
+
+    ip -4 addr show dev "${_DEVICE:?"Missing DEVICE"}" scope global \
+        | grep 'inet' | xargs -n 1 \
+        | grep -A1 'inet' \
+        | tail -n 1 \
+        | cut -d/ -f1
+}
+
+function public() {
+    hostname -I | xargs -n 1 \
+        | grep -vE '(:|^(127|169\.254|10|172\.(1(6|7|8|9)|2[0-9]|30|31)|192\.168|(22(4|5|6|7|8|9)|23(0|1|2|3|4|5|6|7|8|9))).*)'
+}
+
+# Maybe use "resolvectl status" to get DNS Server and specify 'nslookup'
+function published() {
+    local _BOOT_HOSTNAME
+    _BOOT_HOSTNAME="$(hostname -b)"
+    readonly _BOOT_HOSTNAME
+
+    nslookup -type=A "${_BOOT_HOSTNAME:?"Missing BOOT_HOSTNAME"}" | xargs -n 1 \
+        | grep -A2 -i "${_BOOT_HOSTNAME}" \
+        | grep -A1 -i 'address' \
+        | tail -n1
+}
+
+function verified() {
+    local _PUBLISHED_IP
+    _PUBLISHED_IP="$(published)"
+    readonly _PUBLISHED_IP
+
+    [ -z "${_PUBLISHED_IP}" ] \
+        && return 0
+
+    all | grep "${_PUBLISHED_IP}"
+}
+
+function usage() {
+    echo "Use one of the following options:"
+    echo "  --all       : prints all IPv4 addresses"
+    echo "  --routed    : prints the IPv4 address used to send traffic to the default gateway"
+    echo "  --public    : prints all IPv4 addresses direct accessable from the internet"
+    echo "  --published : prints the IPv4 address provided by DNS using this host's name"
+    echo "  --verified  : prints the IPv4 included in 'all' und respended by 'published'"
+}
+
+
+
+function main(){
+
+    case "${1}" in
+        --all)
+            all
+            return 0
+            ;;
+        --routed)
+            routed
+            return 0
+            ;;
+        --public)
+            public
+            return 0
+            ;;
+        --published)
+            published
+            return 0
+            ;;
+        --verified)
+            verified
+            return 0
+            ;;
+        *)
+            usage
+            return 1
+            ;;
+    esac
+
+    return 1
+
+}
+
+ main "$@" && exit 0 || exit 1

script/host/net/printOwnIPv6Adress.sh

@@ -0,0 +1,109 @@
+#!/bin/bash
+
+#grep -E '(^::1|(^fc.*|^fd.*)|^fe80::.*|^ff.*)' findet:
+#  loopback: ::1/128
+#  uniquelocal: fc00::/7   (fc00… bis fdff…)
+#  linklocal:   fe80::/64
+#  multicast:   ff00::/8   (ff…)
+
+
+
+function all() {
+    # Select just lines containing 'inet6'.
+    # 1.) Remove every indenting.
+    # 2.) Remove 'inet6 '.
+    # 3.) Remove everything after a '/' (including the /).
+    ip -6 addr \
+        | grep 'inet6' \
+        | sed -e 's/^[[:blank:]]*//' \
+            -e 's/inet6 //' \
+            -e 's/\/.*//'
+}
+
+function routed() {
+    local _DEVICE
+    _DEVICE="$(ip -6 route show default | xargs -n 1 | grep -A1 -i dev | tail -n 1)"
+    readonly _DEVICE
+
+    ip -6 addr show dev "${_DEVICE:?"Missing DEVICE"}" scope global \
+        | grep 'inet6' \
+        | xargs -n 1 \
+        | grep -A1 'inet6' \
+        | grep ':' \
+        | cut -d/ -f1
+}
+
+function public() {
+    hostname -I | xargs -n 1 \
+        | grep ':' \
+        | grep -vE '(^::1|(^fc.*|^fd.*)|^fe80::.*|^ff.*)'
+}
+
+# Maybe use "resolvectl status" to get DNS Server and specify 'nslookup'
+function published() {
+    local _BOOT_HOSTNAME
+    _BOOT_HOSTNAME="$(hostname -b)"
+    readonly _BOOT_HOSTNAME
+
+    nslookup -type=AAAA "${_BOOT_HOSTNAME:?"Missing BOOT_HOSTNAME"}" | xargs -n 1 \
+        | grep -A2 -i "${_BOOT_HOSTNAME}" \
+        | grep -A1 -i address \
+        | tail -n1
+}
+
+function verified() {
+    local _PUBLISHED_IP
+    _PUBLISHED_IP="$(published)"
+    readonly _PUBLISHED_IP
+
+    [ -z "${_PUBLISHED_IP}" ] \
+        && return 0
+
+    all | grep "${_PUBLISHED_IP}"
+}
+
+function usage() {
+    echo "Use one of the following options:"
+    echo "  --all       : prints all IPv6 addresses"
+    echo "  --routed    : prints the IPv6 address used to send traffic to the default gateway"
+    echo "  --public    : prints all IPv6 addresses direct accessable from the internet"
+    echo "  --published : prints the IPv6 address provided by DNS using this host's name"
+    echo "  --verified  : prints the IPv6 included in 'all' und respended by 'published'"
+}
+
+
+
+function main(){
+
+    case "${1}" in
+        --all)
+            all
+            return 0
+            ;;
+        --routed)
+            routed
+            return 0
+            ;;
+        --public)
+            public
+            return 0
+            ;;
+        --published)
+            published
+            return 0
+            ;;
+        --verified)
+            verified
+            return 0
+            ;;
+        *)
+            usage
+            return 1
+            ;;
+    esac
+
+    return 1
+
+}
+
+ main "$@" && exit 0 || exit 1

script/host/net/printOwnMACAdresses.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+cat /sys/class/net/e*/address

script/host/net/printOwnShortHostnameFromHostsFile.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+
+# Select just lines containing 'inet'.
+# 1.) Remove every indenting.
+# 2.) Remove 'inet '.
+# 3.) Remove everything after a '/' (including the /).
+# Search each IP of the IPv4-list in file '/etc/hosts' 
+# Select just lines containing 'managedHost'.
+# 1.) Remove everything after a '#' (including the #).
+# 2.) Remove every indenting.
+# 3.) Remove blanks (spaces or tabs) at the end of lines.
+# 4.) Replace blanks (spaces or tabs) with one ';' between the values.
+# 5.) Delete empty lines.
+# Then cut the second field
+# Then cut the first field to get the short hostname
+ip -4 addr \
+    | grep 'inet' \
+    | sed -e 's/^[[:blank:]]*//' \
+        -e 's/inet //' \
+        -e 's/\/.*//' \
+    | xargs -i grep {} /etc/hosts \
+    | grep 'managedHost' \
+    | sed -e 's/#.*//' \
+        -e 's/^[[:blank:]]*//' \
+        -e 's/[[:blank:]]*$//' \
+        -e 's/\s\+/;/g' \
+        -e '/^$/d' \
+    | cut -d';' -f2 \
+    | cut -d'.' -f1
+

script/host/nginx/restartIfConfigurationIsValid.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+nginx -t &> /dev/null \
+    && systemctl restart nginx.service \
+    && exit 0
+
+exit 1

script/host/nginx/setup.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+function main() {
+    local _SCRIPTPATH _DH_PATH _SELF_SIGNED_PATH
+    _SCRIPTPATH="$(cd -- "$(dirname "$0")" > /dev/null 2>&1; pwd -P)"
+    _DH_PATH="/etc/ssl/private"
+    _SELF_SIGNED_PATH="/etc/ssl/private"
+    readonly _SCRIPTPATH _DH_PATH _SELF_SIGNED_PATH
+
+    ! dpkg -s nginx > /dev/null 2>&1 \
+        && apt-get --yes install nginx-full \
+        && echo "Nginx erfolgreich installiert." \
+        || echo "Nginx ist bereits installiert."
+
+    ! dpkg -s openssl > /dev/null 2>&1 \
+        && apt-get --yes install openssl \
+        && echo "OpenSSL erfolgreich installiert." \
+        || echo "OpenSSL ist bereits installiert."
+
+    ! [ -f "${_DH_PATH}/dhparam4096.pem" ] \
+        && mkdir -p "${_DH_PATH}" \
+        && chmod go-rwx "${_DH_PATH}" \
+        && openssl dhparam -out "${_DH_PATH}/dhparam4096.pem" 4096 \
+        && echo "Diffie-Hellman-Parameters erfolgreich erstellt." \
+        || echo "Diffie-Hellman-Parameters bereits vorhanden."
+
+    ! [ -f "${_SELF_SIGNED_PATH}/selfsigned-private.key" ] \
+        && mkdir -p "${_SELF_SIGNED_PATH}" \
+        && chmod go-rwx "${_SELF_SIGNED_PATH}" \
+        && openssl req -x509 -days 36524 -nodes -newkey rsa:4096 \
+             -keyout "${_SELF_SIGNED_PATH}/selfsigned-private.key" \
+             -out "${_SELF_SIGNED_PATH}/selfsigned-fullchain.crt" \
+        && echo "Selbstsignierte Standardschlüssel erfolgreich erstellt." \
+        || echo "Selbstsignierte Standardschlüssel bereits vorhanden."
+
+#TODO Links erstellen
+#    [ -d "/etc/nginx/" ] \
+#        && cp "${_SCRIPTPATH}/etc_nginx_conf.d/"* "/etc/nginx/conf.d/" \
+#        && mkdir -p /etc/nginx/ssl-trusted \
+#        && cp "${_SCRIPTPATH}/etc_nginx_ssl-trusted/"* "/etc/nginx/ssl-trusted/" \
+#        && mkdir -p /var/www/letsencrypt/.well-known/acme-challenge \
+#        && echo "Basis-Konfiguration erfolgreich erstellt." \
+#        || echo "Basis-Konfiguration bereits vorhanden."
+
+    echo \
+        && echo "Nginx neu starten:" \
+        && nginx -t \
+        && systemctl restart nginx.service \
+        && return 0
+
+    return 1
+}
+
+main "$@" && exit 0 || exit 1

script/host/user/addUserToSudoGroup.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+sudo usermod --append --groups sudo "${1:?"Missing first parameter USER"}"

script/host/user/removeUserFromSudoGroup.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+sudo usermod --remove --groups sudo "${1:?"Missing first parameter USER"}"

script/monitor/README.md

@@ -0,0 +1,33 @@
+
+Monitoring - How it works
+=========================
+
+Basics
+------
+
+You have to set up the monitoring host first. That host will monitor your other machines.
+Execute `/cis/script/monitor/setupMonitoringHost.sh` to start the process.
+
+As usual you can configure this feature via definitions.
+```
+# Path of this feature's scripts       : '/cis/script                 /monitor'
+# Path of the corresponding definitions: '/cis/definitions/YOUR.DOMAIN/monitor'
+ls -lha '/cis/script/monitor'
+ls -lha '/cis/definitions/YOUR.DOMAIN/monitor'
+```
+
+You can modify the appearance and place your own `check.css` or `logo.png` into the definitions folder:
+
+ - /cis/definitions/YOUR.DOMAIN/monitor/check.css
+ - /cis/definitions/YOUR.DOMAIN/monitor/logo.png
+
+After the change, you have to call `/cis/script/monitor/setupMonitoringHost.sh` again,  
+because it creates links in '/var/www/html/' and gives the definitions priority over the script.  
+Additional you need to configure a webserver to publish the site. 
+
+
+
+Dashboard
+---------
+
+You can set up an dashboard following this manual [SETUP_DASHBOARD.md](SETUP_DASHBOARD.md)

script/monitor/SETUP_DASHBOARD.md

@@ -0,0 +1,126 @@
+
+How to setup a monitoring dashboard
+===================================
+
+Inspired by: https://pimylifeup.com/ubuntu-chromium-kiosk/
+
+Steps
+-----
+
+
+
+### 1.) Install Ubuntu Server (no desktop) on your computer than set hostname and timezone.
+
+```sh
+hostnamectl set-hostname check.local
+timedatectl set-timezone Europe/Berlin
+```
+
+
+
+### 2.) Install minimal GUI and Tools.
+
+```sh
+apt install ubuntu-desktop-minimal
+apt install language-pack-gnome-de
+apt install xdotool
+apt install dbus-x11
+```
+
+
+
+### 3.) Create a kiosk user with home-directory.
+
+```sh
+useradd -m kiosk
+```
+
+and disable Welocme-Screen
+```sh
+echo "yes" > /home/kiosk/.config/gnome-initial-setup-done
+```
+
+
+
+### 4.) Edit following file `nano /etc/gdm3/custom.conf` to turn of wayland and turn on autologin for user 'kiosk'.
+
+```
+[daemon]
+# Uncomment the line below to force the login screen to use Xorg
+#WaylandEnable=false
+
+WaylandEnable=false
+
+# Enabling automatic login
+#  AutomaticLoginEnable = true
+#  AutomaticLogin = user1
+
+AutomaticLoginEnable = true
+AutomaticLogin = kiosk
+```
+
+
+
+### 5.) Configure GUI of user kiosk to prevent monitor from sleeping
+
+```sh
+#gsettings list-recursively
+
+# Does not work
+#sudo -u kiosk gsettings set org.gnome.desktop.session idle-delay 0
+
+# Set idle-delay from "uint32 300" to "uint32 0", needs 'apt install dbus-x11'
+# You can check the value in "GUI-Session of kiosk -> Settings -> Power"
+sudo -u kiosk dbus-launch dconf write /org/gnome/desktop/session/idle-delay "uint32 0"
+```
+
+
+
+### 6.) Create custom service to start firefox loading the page.
+
+Therefore create a file `/etc/systemd/system/kiosk.service` with this content:
+
+```
+[Unit]
+Description=Firefox Kiosk
+Wants=graphical.target
+After=graphical.target
+
+[Service]
+Environment=DISPLAY=:0
+# Set firefox language, needs 'apt install language-pack-gnome-de' 
+Environment=LANG=de_DE.UTF-8
+Type=simple
+# Always a fresh firefox ('-' allow error if common does not exist)
+ExecStartPre=-/usr/bin/rm -r /home/kiosk/snap/firefox/common
+# Move Mouse (should also work on small screens), needs 'apt install dbus-x11'
+ExecStartPre=/usr/bin/xdotool mousemove 4096 2160
+# See: https://wiki.mozilla.org/Firefox/CommandLineOptions (just -kiosk URL => Start-Assistant, so use -url too)
+ExecStart=/usr/bin/firefox -fullscreen -kiosk -url http://monitor.example.net/check.html
+Restart=always
+RestartSec=30
+User=kiosk
+Group=kiosk
+
+[Install]
+WantedBy=graphical.target
+```
+
+
+
+### 7.) Enable the service and reboot
+
+```sh
+systemctl enable kiosk
+reboot
+```
+
+
+
+Troubleshouting
+---------------
+
+```
+systemctl disable pd-mapper.service
+apt purge cloud-init -y && apt autoremove --purge -y
+```

script/monitor/check.css

@@ -0,0 +1,77 @@
+html, body {
+    --background-theme-color: #001EA0;
+    --cell-space: 20px;
+    --logo-height: 50px;
+
+    background-color: #cccccc;
+    font-family: Verdana;
+    font-size: 14pt;
+    color: #ffffff;
+    height: 100%;
+    margin: 0;
+}
+@media screen and (orientation: portrait) {
+    body {
+        zoom: 200%
+    }
+}
+#header {
+    background-color: var(--background-theme-color); 
+    position: sticky;
+    top: 0;
+    height: calc(var(--logo-height) + (2 * var(--cell-space)));
+    width: 100%;
+}
+#header img {
+    height: var(--logo-height);
+    margin: var(--cell-space);
+    vertical-align: middle;
+}
+#header h1 {
+    display: inline;
+    font-weight: normal;
+    vertical-align: middle;
+}
+#content {
+    min-height: 100%;
+}
+#footer {
+    background-color: var(--background-theme-color);
+    position: sticky;
+    bottom: 0px;
+    padding: var(--cell-space);
+    text-align: center;
+    vertical-align: middle;
+    font-size: 22pt;
+}
+#checks {
+    display: grid;
+    grid-template-columns: repeat(auto-fit, minmax(350px, 1fr));
+    padding: var(--cell-space);
+    grid-gap: var(--cell-space);
+}
+#checks > div {
+    border: 1px solid black;
+    border-radius: 10px;
+    padding: 10px;
+    text-align: center;
+    box-shadow: 0 2px 5px 0 rgba(0, 0, 0, 0.3), 0 2px 10px 0 rgba(0, 0, 0, 0.2);
+}
+#checks > div.ok {
+    background-color: #66aa22;
+    color: #222222;	
+}
+#checks > div.info {
+    background-color: #88cc44;
+    color: #222222;	
+}
+#checks > div.warn {
+    background-color: #ffdd00;
+    color: #222222;	
+}
+#checks > div.fail {
+    background-color: #ff0000;
+}
+#checks > div.timeout {
+    background-color: var(--background-theme-color);
+}

script/monitor/check.html

@@ -0,0 +1,122 @@
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Monitoring Dashboard</title>
+    <link rel="stylesheet" href="check.css">
+</head>
+<body>
+    <div id="header">
+        <img src="logo.png"></img>
+        <h1>Monitoring</h1>
+    </div>
+    <div id="content">
+        <div id="checks">
+            <div class="check warn">Loading...</div>
+        </div>
+    </div>
+    <div id="footer">
+        Köln, <span id="datetime"></span>
+    </div>
+
+    <script>
+        var connectionAlive = true;
+
+        function downloadCheckFile(callback) {
+            var xmlHttp = new XMLHttpRequest();
+            xmlHttp.open('GET', "check.txt", false);
+            xmlHttp.onreadystatechange=function() {
+                if(xmlHttp.readyState==4) {
+                    callback(xmlHttp.responseText);
+                }
+            }
+            try {
+                xmlHttp.send(null);
+                if (xmlHttp.status >= 200 && xmlHttp.status < 304) {
+                    return true;
+                } else {
+                    return false;
+                }
+            } catch (e) {
+                return false;
+            }
+        }
+        function convertToHtml(checkText) {
+            if (!connectionAlive) {
+                return '<div class="fail">CONNECTION FAILED</div>';
+            }
+
+            var html = "";
+            var lines = checkText.split(/\n/);
+
+            for(var lineNo = 2; lineNo < lines.length; lineNo++) {
+                var line = lines[lineNo];
+
+                var parts = line.split('?');
+                if (parts.length > 1) {
+
+                    var name = parts[0].trim().split("_").join(" ");
+                    var resultParts = parts[1].trim().split('#');
+                    var result = resultParts[0];
+                    var message = resultParts[1];
+
+                    if(name == 'MISSED') {
+                        var fileTimeParts = message.split('-');
+                        var fileTime = new Date();
+                        fileTime.setHours(parseInt(fileTimeParts[0]));
+                        fileTime.setMinutes(parseInt(fileTimeParts[1]));
+                        fileTime.setSeconds(parseInt(fileTimeParts[2]));
+
+                        var scriptTime = new Date();
+                        scriptTime.setMinutes(scriptTime.getMinutes() - 2);
+                        if (scriptTime.getTime() < fileTime.getTime()){
+                            if (result == "0") {
+                                html += '<div class="ok">EVERYTHING OK<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                            } else {
+                                html += '<div class="fail">FAILED: ' + result + '<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                            }
+                        } else {
+                            html += '<div class="check fail">CHECKS TOO OLD<br/>' + fileTime.toLocaleTimeString() + '</div>';
+                        }
+                    } else {
+                        if(result.indexOf('OK') >= 0) {
+                            html += '<div class="ok">'+ name;
+                        } else if(result.indexOf('INFO') >= 0) {
+                            html += '<div class="info">'+ name;
+                        } else if(result.indexOf('TIMEOUT') >= 0) {
+                            html += '<div class="timeout">'+ name;
+                        } else if(result.indexOf('WARN') >= 0) {
+                            html += '<div class="warn">'+ name;
+                        } else {
+                            html += '<div class="fail">' + name;
+                        }
+                        if(message) {
+                            html += '<br/>' + message.trim();
+                        }
+                        html += '</div>';
+                    }
+                }
+            }
+            return html;
+        }
+        function exchangeChecks(text) {
+            document.getElementById("checks").innerHTML = convertToHtml(text);
+        }
+        function refreshTime() {
+            document.getElementById("datetime").innerHTML = new Date().toLocaleString("de-DE", {timeZone: "Europe/Berlin"});
+        }
+        function refreshChecks() {
+            connectionAlive = downloadCheckFile(exchangeChecks);
+        }
+        function reloadIfAlive() {
+            if (connectionAlive) {
+                location.reload();
+            }
+        }
+        setInterval(refreshTime, 1000);
+        setInterval(refreshChecks, 5000);
+        setInterval(reloadIfAlive, 300000);
+        refreshTime();
+        refreshChecks();
+    </script>
+</body>
+</html>

script/monitor/check.sh

@@ -0,0 +1,88 @@
+#!/bin/bash
+
+# Folders always ends with an tailing '/'
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SCRIPT%%/script/monitor/*}/"               #Removes longest  matching pattern '/script/monitor/*' from the end
+_CORE_SCRIPTS="${_CIS_ROOT:?"Missing CIS_ROOT"}core/"
+_CURRENT_DOMAIN="$("${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}printOwnDomain.sh")"
+_DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_CURRENT_DOMAIN:?"Missing CURRENT_DOMAIN"}/"
+
+# Checks for the entire domain
+_DOMAIN_CHECKS="${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/checks/"
+
+
+
+function doChecks(){
+    local readonly _TMPDIR="${1:?"doChecks(): Missing parameter TMPDIR:"}"
+
+    local _DATETIME=$(date +%H-%M-%S)
+
+    mkdir -p ${_TMPDIR}
+    rm ${_TMPDIR}/* > /dev/null 2>&1
+
+    for check in ${_DOMAIN_CHECKS}*.on
+    do
+        local _CHECK_FILENAME="${check##*/}"
+        echo -n "${_CHECK_FILENAME%%.on}?" > "${_TMPDIR}/${_CHECK_FILENAME}"
+        timeout -k 10s 20s bash ${check} >> "${_TMPDIR}/${_CHECK_FILENAME}" 2> /dev/null || echo "TIMEOUT#Timeout" >> "${_TMPDIR}/${_CHECK_FILENAME}" &
+    done
+    wait
+
+    local _FAILED=0
+    echo "CHECK?RESULT[#MESSAGE]:"
+    echo "-----------------------"
+    for resultFile in ${_TMPDIR}/*
+    do
+        cat "${resultFile}"
+        grep -q "FAIL" ${resultFile} && _FAILED=$(expr ${_FAILED} + 1)
+    done
+    echo "MISSED?${_FAILED}#${_DATETIME}"
+
+    rm -r ${_TMPDIR} > /dev/null 2>&1
+    return 0
+}
+
+function usage(){
+    printf "\nUsage: /monitoring/check.sh <command> <options>"
+    echo
+    echo "possible commands:"
+    echo
+    echo "- all"
+    echo "    Executes all checks."
+    echo "- auto <out_file>"
+    echo "    Executes quiet all checks and saves the result in the given out_file."
+    echo "      (e.g. add the following line to crontab: '* * * * * /cis/script/monitor/check.sh auto /var/www/html/check.txt'"
+    echo "       to update the file '/var/www/html/check.txt' every minute as 'check.html' needs it.)"
+    return 0
+}
+
+main(){
+    case "${1:-""}" in
+        all)
+            echo "Checks werden ausgeführt..." \
+                && echo \
+                && doChecks "/tmp/checks" color \
+                && echo \
+                && echo "Success" \
+                && return 0
+            ;;
+        auto)
+            # If just a filename is given it is created in /tmp, because of 'cd /tmp' 
+            cd /tmp \
+                && doChecks "/tmp/checks$(date +%N)" > "$2.new" \
+                && mv -f "$2.new" "$2" \
+                && return 0
+            return 1
+            ;;
+		*)
+            [ "${1:+isset}" == "isset" ] \
+                && echo "Parameter '${1}' ist kein gültiger Befehl."
+            usage
+            return 0
+            ;;
+    esac
+
+    return 1
+}
+
+main "$@" || exit 1

script/monitor/checks/EXAMPLE_CHECK.off

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+_CHECK="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_CHECK%%/definitions/*}/"             #Removes longest  matching pattern '/definitions/*' from the end
+_GENERIC_CHECKS="${_CIS_ROOT:?"Missing CIS_ROOT"}script/monitor/generic/"
+
+${_GENERIC_CHECKS:?"Missing GENERIC_CHECKS"}OVERRIDDEN_DOMAIN_CHECK.sh "your-host.your-domain.net"

script/monitor/generic/CIS_OWN_DOMAIN_CHECK.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 1 ssh -O check -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 2>&1 | grep -q -F 'Master running' \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function testDomain(){
+    checkOrStartSSHMaster \
+        || return 1
+
+    local _RESULT="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'bash /cis/core/printOwnDomain.sh' 2>&1 1>/dev/null)"
+
+    [ -z "${_RESULT}" ] \
+        && echo "OK" \
+        && return 0
+
+    echo "WARNING#Check hosts '/cis/core/printOwnDomain'"
+    return 0
+}
+
+testDomain && exit 0

script/monitor/generic/NGINX_CHECK.sh

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 1 ssh -O check -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 2>&1 | grep -q -F 'Master running' \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function checkViaHTTP() {
+    _STATUS="$(curl -I http://${_REMOTE_HOSTNAME_FQDN} 2>/dev/null | head -n 1 | cut -d$' ' -f2)"
+    [ "${_STATUS}" == "200" ] \
+        && echo "OK" \
+        && return 0
+
+    return 1
+}
+
+function checkViaHTTPS() {
+    _STATUS="$(curl -k -I https://${_REMOTE_HOSTNAME_FQDN} 2>/dev/null | head -n 1 | cut -d$' ' -f2)"
+    [ "${_STATUS}" == "200" ] \
+        && echo "OK" \
+        && return 0
+
+    return 1
+}
+
+#grep:
+# -E                         Use regexp, '.*' => any chars between 'Active:' and '(running)', the round brackets are escaped.
+
+#cut:
+# -d                         Delimiter, marker where to cut (here ;)
+# -f                         Index of column to show (One based, so there is no -f0)
+function checkViaSSH() {
+    checkOrStartSSHMaster \
+        || return 1
+
+    _RESULT=$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'systemctl status nginx.service' | grep -E 'Active:.*\(running\)' | cut -d';' -f2)
+    ! [ -z "${_RESULT}" ] && echo "OK#UPTIME:${_RESULT}" || echo "FAIL"
+}
+
+#checkViaHTTP && exit 0
+#checkViaHTTPS && exit 0
+checkViaSSH && exit 0
+
+exit 1

script/monitor/generic/PING_CHECK.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+_SERVER="${1:?"FQDN of server missing"}"
+
+# -4                         Use IPv4
+# -W            SECONDS      Wait seconds for an answer
+# -c            COUNT_VALUE  Count of pings being executed
+_RESULT="$(ping -4 -W 1 -c 1 "${_SERVER}" | grep "time=" | cut -d'=' -f4)"
+! [ -z "${_RESULT}" ] && echo "OK#RTT: ${_RESULT}" || echo "FAIL#PLEASE USE FALLBACK!"

script/monitor/generic/POSTGRES_CERTIFICATE_CHECK.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+
+function checkPostgresSSLCertificate() {
+    local _SERVER
+    _SERVER="${1:?"FQDN of server missing"}"
+    readonly _SERVER
+
+    local _RESULT
+    _RESULT="$(echo | openssl s_client -starttls postgres -connect "${_SERVER}":5432 -servername "${_SERVER}" 2> /dev/null | openssl x509 -noout -enddate | grep -F 'notAfter=' | cut -d'=' -f2)"
+    readonly _RESULT
+
+    [ -z "${_RESULT}" ] \
+        && echo "FAIL#Unable to get cert's end date from ${_SERVER}:5432" \
+        && return 1
+
+    local _ENDDATE
+    _ENDDATE="$(date --date="${_RESULT}" --utc +%s)"
+    readonly _ENDDATE
+
+    ! echo "${_ENDDATE}" | grep -q -E "^[0-9]*$" \
+        && echo "FAIL#Unable to parse end date of certificate" \
+        && return 1
+
+    local _NOW _REMAINING_DAYS
+    _NOW="$(date --date now +%s)"
+    _REMAINING_DAYS="$(( (_ENDDATE - _NOW) / 86400 ))"
+    readonly _NOW _REMAINING_DAYS
+
+    [ -z "${_REMAINING_DAYS}" ] \
+        && echo "WARN#Only ${_REMAINING_DAYS} days left" \
+        && return 1
+
+    echo "OK#${_REMAINING_DAYS} days remaining"
+    return 0
+}
+
+checkPostgresSSLCertificate "${@}" && exit 0 || exit 1

script/monitor/generic/URL_CHECK.sh

@@ -0,0 +1,62 @@
+#!/bin/bash
+
+#curl:
+# --connect-timeout SECONDS  Maximum time allowed for connection
+# -k                         Allow connections to SSL sites without certs (H)
+# -L                         Follow redirects (H)
+# --max-time        SECONDS  Maximum time allowed for the transfer
+# -s                         Silent mode. Don't output anything
+# --head                     Show head information only
+# --no-progress-meter        Clean output for grep
+
+#grep:
+# -q                         Quite, no output just status codes
+# -F                         Interpret search term as plain text
+function checkUrl() {
+    local _URL _SEARCH_STRING
+    _URL="${1:?"URL of site missing"}"
+    _SEARCH_STRING="${2}"
+    readonly _URL _SEARCH_STRING
+
+    local _RESULT
+    if [ -z "${_SEARCH_STRING}" ]; then
+        _RESULT="$(curl --connect-timeout 10 --max-time 10 --no-progress-meter --verbose "${_URL}" 2>&1 | grep -o -E "(expire.*|HTTP.*200 OK)")"
+    else
+        _RESULT="$(curl --connect-timeout 10 --max-time 10 --no-progress-meter --verbose "${_URL}" 2>&1 | grep -o -E "(expire.*|HTTP.*200 OK|${_SEARCH_STRING})")"
+    fi
+    readonly _RESULT
+
+    ! echo "${_RESULT}" | grep -q -F '200 OK' \
+        && echo "FAIL#Status code 200 not found" \
+        && return 1
+
+    ! [ -z "${_SEARCH_STRING}" ] \
+        && ! echo "${_RESULT}" | grep -q -F "${_SEARCH_STRING}" \
+        && echo "FAIL#Search string not found" \
+        && return 1
+
+    local _ENDDATE
+    _ENDDATE="$(echo "${_RESULT}" | grep -F 'expire' | cut -d':' -f2-)"
+    _ENDDATE="$(date --date="${_ENDDATE}" --utc +%s)"
+    readonly _ENDDATE
+
+    ! echo "${_ENDDATE}" | grep -q -E "^[0-9]*$" \
+        && echo "FAIL#Unable to parse end date of certificate" \
+        && return 1
+
+    local _NOW _REMAINING_DAYS
+    _NOW="$(date --date now +%s)"
+    _REMAINING_DAYS="$(( (_ENDDATE - _NOW) / 86400 ))"
+    readonly _NOW _REMAINING_DAYS
+
+    # less than 30 days remaining => should be warned
+    [ "${_REMAINING_DAYS}" -le "30" ] \
+        && echo "WARN#Certificate: only ${_REMAINING_DAYS} days left" \
+        && return 1
+
+    echo "OK#Certificate: ${_REMAINING_DAYS} days remaining"
+    return 0
+}
+
+#((curl --connect-timeout 10 --max-time 10 -k -s --head --no-progress-meter "${_URL}" | grep -qF '200 OK') && echo OK) || echo FAIL
+checkUrl "${1}" "${2}" && exit 0 || exit 1

script/monitor/generic/ZFS_POOL_CHECK.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_ZFS_POOL="${2:?"Name of zfs pool missing: e.g. zpool1"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 1 ssh -O check -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 2>&1 | grep -q -F 'Master running' \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function testPool(){
+    checkOrStartSSHMaster \
+        || return 1
+
+    local _RESPONSE="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'zpool status ${_ZFS_POOL} | grep -F scrub')"
+    local _RESULT=$(echo "${_RESPONSE}" | grep -F 'scrub repaired 0B' | grep -F '0 errors')
+    _RESULT="${_RESULT#*on}"  #Removes shortest matching pattern '*on' from the begin
+
+    [ -z "${_RESULT}" ] \
+        && echo "FAIL#CHECK POOL: ${_ZFS_POOL}" \
+        && return 0
+
+    echo "OK#Scrubbed on ${_RESULT}."
+    return 0
+}
+
+testPool && exit 0
+
+exit 1

script/monitor/generic/ZFS_SYNC_CHECK.sh

@@ -0,0 +1,106 @@
+#!/bin/bash
+
+_SCRIPT="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SCRIPT%%/script/monitor/*}/"             #Removes longest  matching pattern '/script/monitor/*' from the end
+_DOMAIN="$("${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh")"
+_COMPOSITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/compositions/"
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+# This is crucial:
+#  - default value for the filter part is extracted from the first parameter (FQDN)
+#  - but you can override this part to to adapt the test during a change of the domain.
+#    (e.g. the short hostname can be an option - or even a better default in the future)
+_ZFS_SNAPSHOT_FILTER="@SYNC_${2:-"${_REMOTE_HOSTNAME_FQDN:?"Missing REMOTE_HOSTNAME_FQDN"}"}"
+
+_MODE="${3:-"normal"}"
+_NOW_UTC_UNIXTIME=$(date -u +%s)
+_DEBUG_PATH="/tmp/monitor/"
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 1 ssh -O check -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 2>&1 | grep -q -F 'Master running' \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function checkSync() {
+    checkOrStartSSHMaster \
+        || return 1
+
+    [ "${_MODE}" == "debug" ] \
+        && mkdir -p "${_DEBUG_PATH}" > /dev/null \
+        && echo "Now: ${_NOW_UTC_UNIXTIME}" > ${_DEBUG_PATH}SECONDS_BEHIND_${_REMOTE_HOSTNAME_FQDN}.txt
+
+    ! [ -d "${_COMPOSITIONS:?"Missing COMPOSITIONS"}" ] \
+        && echo "WARN#no compositions" \
+        && return 0
+
+    [ "${_MODE}" == "debug" ] \
+        && echo "Snapshot filter: ${_ZFS_SNAPSHOT_FILTER}" >> ${_DEBUG_PATH}SECONDS_BEHIND_${_REMOTE_HOSTNAME_FQDN}.txt
+
+    # This retrieves the list of the interesting snapshots including creation timestamp
+    _SNAPSHOTS="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} zfs list -po creation,name -r -t snapshot zpool1/persistent | grep -F ${_ZFS_SNAPSHOT_FILTER})"
+    [ "${_MODE}" == "debug" ] \
+        && echo "${_SNAPSHOTS}" > ${_DEBUG_PATH}SNAPSHOTS_${_REMOTE_HOSTNAME_FQDN}.txt
+
+    [ -z "${_SNAPSHOTS}" ] \
+        && echo "FAIL#no snapshots" \
+        && return 1
+
+    echo "OK#Checks running"
+
+    for _COMPOSITION_PATH in ${_COMPOSITIONS}*; do
+
+        # If remote host is found than it is responsible for this container-composition, otherwise skip
+        #   (grep -E "^[[:blank:]]*something" means. Line has to start with "something", leading blank chars are ok.)
+        grep -E "^[[:blank:]]*${_REMOTE_HOSTNAME_SHORT}" "${_COMPOSITION_PATH}/zfssync-hosts" &> /dev/null \
+            || continue;
+
+        _COMPOSITION_NAME="${_COMPOSITION_PATH##*/}"   #Removes longest  matching pattern '*/' from the begin
+        _LAST_SNAPSHOT_UNIXTIME="$(echo "${_SNAPSHOTS}" | grep ${_COMPOSITION_NAME} | tail -n 1 | cut -d' ' -f1)"
+        _SECONDS_BEHIND=$[ ${_NOW_UTC_UNIXTIME} - ${_LAST_SNAPSHOT_UNIXTIME} ]
+
+        [ "${_MODE}" == "debug" ] \
+            && echo "${_LAST_SNAPSHOT_UNIXTIME} ${_COMPOSITION_NAME} on ${_REMOTE_HOSTNAME_FQDN} behind: ${_SECONDS_BEHIND}s" >> ${_DEBUG_PATH}SECONDS_BEHIND_${_REMOTE_HOSTNAME_FQDN}.txt
+
+        [ "${_SECONDS_BEHIND}" -lt 40 ] \
+            && continue
+
+        [ "${_SECONDS_BEHIND}" -lt 60 ] \
+            && echo "ZFSSYNC_of_${_REMOTE_HOSTNAME_SHORT}_LAGGING?WARN#${_COMPOSITION_NAME} ${_SECONDS_BEHIND}s" \
+            && continue
+
+        echo "ZFSSYNC_of_${_REMOTE_HOSTNAME_SHORT}_LAGGING?FAIL#${_COMPOSITION_NAME} ${_SECONDS_BEHIND}s"
+    done
+}
+
+
+
+RESULTS="$(checkSync)"
+
+[ "${_MODE}" == "debug" ] \
+    && echo "$RESULTS" > ${_DEBUG_PATH}RESULTS_${_REMOTE_HOSTNAME_FQDN}.txt
+
+echo "$RESULTS"

script/monitor/generic/ZFS_USAGE_CHECK.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+_REMOTE_HOST="${1:?"FQDN of server missing: e.g. host.example.net[:port]"}"
+_REMOTE_HOSTNAME_FQDN="${_REMOTE_HOST%%:*}"            #Removes longest  matching pattern ':*' from the end
+_REMOTE_HOSTNAME_SHORT="${_REMOTE_HOSTNAME_FQDN%%.*}"  #Removes longest  matching pattern '.*' from the end
+_REMOTE_PORT="${_REMOTE_HOST}:"
+_REMOTE_PORT="${_REMOTE_PORT#*:}"                      #Removes shortest matching pattern '*:' from the begin
+_REMOTE_PORT="${_REMOTE_PORT%%:*}"                     #Removes longest  matching pattern ':*' from the end
+_REMOTE_PORT="${_REMOTE_PORT:-"22"}"
+_REMOTE_USER="monitoring"
+_SOCKET='~/.ssh/%r@%h:%p'
+
+
+
+function checkOrStartSSHMaster() {
+    timeout --preserve-status 1 ssh -O check -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 2>&1 | grep -q -F 'Master running' \
+        && return 0
+
+    ssh -O stop -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} &> /dev/null
+    ssh -o ControlMaster=auto \
+        -o ControlPath=${_SOCKET} \
+        -o ControlPersist=65 \
+        -p ${_REMOTE_PORT} \
+        -f ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} exit &> /dev/null \
+        && return 0
+
+    echo "FAIL#SSH connection (setup ok?)"
+    return 1
+}
+
+function testSpace(){
+    checkOrStartSSHMaster \
+        || return 1
+
+    local _RESULT="$(ssh -S ${_SOCKET} -p ${_REMOTE_PORT} ${_REMOTE_USER}@${_REMOTE_HOSTNAME_FQDN} 'zpool list -H -o capacity,name')"
+    local _SPACE_USED=$(echo "${_RESULT}" | /usr/bin/tail -n 1 | /usr/bin/cut -f1)
+    local _POOL=$(echo "${_RESULT}" | /usr/bin/tail -n 1 | /usr/bin/cut -f2)
+
+    [ -z "${_SPACE_USED}" ] \
+        && echo "FAIL#NO value" \
+        && return 0
+
+    [ "${1:?"Missing OK_THRESHOLD"}" -ge "${_SPACE_USED%\%*}" ] \
+        && echo "OK#${_SPACE_USED} used ${_POOL}." \
+        && return 0
+
+    [ "${2:?"Missing INFO_THRESHOLD"}" -ge "${_SPACE_USED%\%*}" ] \
+        && echo "INFO#${_SPACE_USED} already used ${_POOL}." \
+        && return 0
+
+    echo "FAIL#${_SPACE_USED} used ${_POOL}!"
+    return 0
+}
+
+testSpace 80 90 && exit 0
+
+exit 1

script/monitor/setupMonitoringHost.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && sudo "${0}" \
+    && exit 0
+
+
+
+_SETUP="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SETUP%%/script/monitor/*}/"             #Removes longest  matching pattern '/script/monitor/*' from the end
+_DOMAIN="$("${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh")"
+_DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/"
+
+
+
+function checkPreconditions() {
+    [ -d "${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/checks" ] \
+        && return 0
+
+    echo "No folder for your defined checks found: ${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/checks"
+    echo "Please create it and add all your custom monitoring checks there, following this convention: 'NAME_OF_THE_CHECK.on'"
+    echo "A check has to be switched 'on' to be executed, so you can rename a check to 'NAME_OF_THE_CHECK.off' and it will be ignored."
+    echo
+    echo "You can copy the file '/cis/script/monitor/checks/EXAMPLE_CHECK.off' to your check definitions folder and modify it."
+    return 1
+}
+
+
+
+function printSelectedDefinition() {
+    local _FILE_DEFINED_DOMAIN _FILE_DEFINED_DEFAULT
+    _FILE_DEFINED_DOMAIN="${_DEFINITIONS:?"Missing DEFINITIONS"}monitor/${1:?"Missing CURRENT_FULLFILE"}"
+    _FILE_DEFINED_DEFAULT="${_CIS_ROOT:?"Missing CIS_ROOT"}script/monitor/${1:?"Missing CURRENT_FULLFILE"}"
+    readonly _FILE_DEFINED_DOMAIN _FILE_DEFINED_DEFAULT
+
+    [ -s "${_FILE_DEFINED_DOMAIN}" ] \
+        && echo "${_FILE_DEFINED_DOMAIN}" \
+        && return 0
+
+    [ -s "${_FILE_DEFINED_DEFAULT}" ] \
+        && echo "${_FILE_DEFINED_DEFAULT}" \
+        && return 0
+
+    return 1
+}
+
+function setupPublicFile() {
+    ! [ -d "/var/www/html" ] \
+        && echo "Missing folder '/var/www/html'. Is a webserver installed?" \
+        && return 1
+
+    [ -L "/var/www/html/${1:?"Missing filename"}" ] \
+        && [ "$(readlink -f /var/www/html/${1:?"Missing filename"})" == "$(printSelectedDefinition ${1:?"Missing filename"})" ] \
+        && echo "Link '/var/www/html/${1:?"Missing filename"}' already exists pointing to the expected file:" \
+        && echo "  - '$(readlink -f /var/www/html/${1:?"Missing filename"})'" \
+        && return 0
+
+    ln -f -s "$(printSelectedDefinition ${1:?"Missing filename"})" "/var/www/html/${1:?"Missing filename"}" \
+        && echo "Link '/var/www/html/${1:?"Missing filename"}' created successfully:" \
+        && echo "  - '$(readlink -f /var/www/html/${1:?"Missing filename"})'" \
+        && return 0
+}
+
+echo "Setup the monitoring host that monitors the others ... " \
+    && checkPreconditions \
+    && setupPublicFile "check.html" \
+    && setupPublicFile "check.css" \
+    && setupPublicFile "logo.png" \
+    && exit 0
+
+exit 1

script/monitor/setupServiceProvidingHost.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+[ "$(id -u)" != "0" ] \
+    && sudo "${0}" \
+    && exit 0
+
+
+
+_SETUP="$(readlink -f "${0}" 2> /dev/null)"
+
+# Folders always ends with an tailing '/'
+_CIS_ROOT="${_SETUP%%/script/monitor/*}/"             #Removes longest  matching pattern '/script/monitor/*' from the end
+_CORE_SCRIPTS="${_CIS_ROOT:?"Missing CIS_ROOT"}core/"
+_DOMAIN="$("${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh")"
+_DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}/"
+
+
+
+echo "Setup the user and permission to enable the monitoring this host ... " \
+    && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addNormalUser.sh" monitoring \
+    && echo \
+    && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}defineAuthorizedKeysOfUser.sh" "${_DEFINITIONS}" monitoring \
+    && exit 0
+
+exit 1

script/ssl/Dockerfile

@@ -0,0 +1,20 @@
+###########################################################################
+# Dockerfile to build a Container to update TLS Certificates automatically.
+# Based on latest Ubuntu LTS
+###########################################################################
+# See https://hub.docker.com/_/ubuntu
+FROM ubuntu:latest
+
+# Update repositories
+RUN echo Version 20251030v1
+RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y apt-utils
+
+#### BEGIN INSTALLATION ###################################################
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cron curl dnsutils openssh-client
+
+ADD acme.sh-3.1.1.tar.gz /tmp/acme.sh-setup/
+COPY renewCerts.sh /renewCerts.sh
+COPY start.sh /start.sh 
+
+# Run the command on container startup
+CMD ["bash", "/start.sh"]

script/ssl/README.md

@@ -0,0 +1,52 @@
+Issuing SSL certificates
+========================
+There are two modes you can use the script `renewCerts.sh`.
+
+1. dns
+2. http
+
+
+
+Dns mode
+--------
+This mode is meant to use inside a docker container defined by the `Dockerfile`.
+To configure, build and run the Container there is a file `docker-compose.yml.template`.
+You can copy this file to `docker-compose.yml` and set the needed environment variables there.
+
+- __AUTOACME_CONTAINER_HOSTNAME__  
+  is used to enable the use of the host name within the container.
+  For example, for meaningful commit messages.
+- __AUTOACME_GIT_REPOSITORY_VIA_SSH__ (optional)  
+  is used to specify a Git repository to which the keys and certificates are transferred.
+  Therefore, SSH keys are generated on first launch (`docker compose up -d`) and the repository is cloned to `~/acmeResults/`.
+  The public key must be granted __write access__ to the repository  
+  (e.g. as repository's deploy key).
+  The key can be viewed via `docker compose logs`.
+- __AUTOACME_PATH_IN_GIT_REPOSITORY__  (optional)  
+  specifies a path inside the repository were the certiticates are saved.  
+  (e.g. AUTOACME_PATH_IN_GIT_REPOSITORY="/foo/bar/" => /root/autoACME/foo/bar/your-domain.net/fullchain.crt)
+- __AUTOACME_DNS_PROVIDER__  
+  sets the provider modul of acme.sh used to communicate with your domain provider.  
+  (For further information see: https://github.com/acmesh-official/acme.sh/wiki/dnsapi)
+
+You may have to set additional environment variables depending on your provider...
+
+
+
+### Manual docker commands
+Instead of using `docker compose` you can build and run the container manually:
+```
+docker build -t cis/autoacme .
+docker run --name autoacme -d cis/autoacme
+```
+This may be useful for investiagtion...
+
+
+Http mode
+---------
+If you plan to use `renewCerts.sh` directly on your host computer this mode may fit your needs.
+Here you need a `nginx` webserver. The domain have to point to it and following configuration is needed:
+
+1. The content of folder `/var/www/letsencrypt/.well-known/acme-challenge/` has to be accessable via `http://your-domain.net/.well-known/acme-challenge/`
+2. The certificates are stored to `/etc/nginx/ssl`. If this folder is a git repository then changes will be commited and pushed.
+3. An entry into the crontab is needed to do automatic updates.

script/ssl/docker-compose.yml.template

@@ -0,0 +1,18 @@
+services:
+  autoacme:
+    container_name: autoacme
+    image: cis/autoacme
+    build: .
+    restart: unless-stopped
+    volumes:
+      - /etc/timezone:/etc/timezone:ro
+      - /etc/localtime:/etc/localtime:ro
+    environment:
+      AUTOACME_CONTAINER_HOSTNAME: ${HOSTNAME:?"HINT - You may run 'export HOSTNAME' first."}
+      AUTOACME_GIT_REPOSITORY_VIA_SSH: 'ssh://git@git.your-domain.net/your-repo.git'
+      # Optionally you can set a path inside the git repository, requires a repository.
+      AUTOACME_PATH_IN_GIT_REPOSITORY: 'hosts/all/etc/ssl/domains/'
+      # See: https://github.com/acmesh-official/acme.sh/wiki/dnsapi   and search your provider like 'hetzner' e.g. 
+      AUTOACME_DNS_PROVIDER: 'dns_hetzner'
+      HETZNER_Token: 'your-token'
+

script/ssl/renewCerts.sh

@@ -0,0 +1,550 @@
+#!/bin/bash
+
+# curl http://your.domain.net/.well-known/acme-challenge/test
+# curl http://85.183.145.8/.well-known/acme-challenge/test
+# /var/www/letsencrypt/.well-known/acme-challenge
+
+function checkConfigViaHttp() {
+    local _DOMAIN _MODE _LOCAL_FOLDER
+    _MODE="${1:?"checkConfigViaHttp(): Missing first parameter MODE"}"
+    _DOMAIN="${2:?"checkConfigViaHttp(): Missing second parameter DOMAIN"}"
+    _LOCAL_FOLDER="/var/www/letsencrypt/.well-known/acme-challenge/"
+    readonly _DOMAIN _MODE _LOCAL_FOLDER
+
+    local _LOCAL_FILE _LOCAL_URL _PUBLIC_URL
+    _LOCAL_FILE="${_LOCAL_FOLDER}${_DOMAIN}"
+    _LOCAL_URL="http://localhost/.well-known/acme-challenge/${_DOMAIN}"
+    _PUBLIC_URL="http://${_DOMAIN}/.well-known/acme-challenge/${_DOMAIN}"
+    readonly _LOCAL_FILE _LOCAL_URL _PUBLIC_URL
+
+    # Skip check if mode is not http
+    [ "${_MODE}" != "http" ] \
+        && return 0
+
+    # Fail because wildcard certificate
+    [ "${_MODE}" == "http" ] \
+        && isWildcardCertificate "${_DOMAIN}" \
+        && echo "Wildcard certificates are not supported via HTTP." \
+        && return 1
+
+    _CHECK="Available on $(hostname)@$(date)"
+
+    echo -n "Check domain '${_DOMAIN}'..." \
+        && [ -d "${_LOCAL_FOLDER}" ] \
+        && echo "${_CHECK}" > "${_LOCAL_FILE}" \
+        && curl -4s "${_PUBLIC_URL}" | grep -q "${_CHECK}" \
+        && echo " Done" \
+        && return 0
+
+    echo
+    echo "FAILED: configuration of domain '${_DOMAIN}' is INCORRECT:"
+    echo -n "  ${_PUBLIC_URL} was not found."
+
+    curl -4s "${_LOCAL_URL}" | grep -q "${_CHECK}" \
+        && echo " (check DNS first)" \
+        || echo " (check Webserver first)"
+
+    return 1
+}
+
+function isActive() {
+    local _DOMAIN _MODE _RESULT_CERTS
+    _RESULT_CERTS="${RESULT_CERTS:?"isActive(): Missing global parameter RESULT_CERTS"}"
+    _MODE="${1:?"isActive(): Missing first parameter MODE"}"
+    _DOMAIN="${2:?"isActive(): Missing second parameter DOMAIN"}"
+    readonly _DOMAIN _MODE _RESULT_CERTS
+
+    # If mode is dns the domain is active always
+    [ "${_MODE}" == "dns" ] \
+        && return 0
+
+    nginx -T 2> /dev/null | grep -q "${_RESULT_CERTS}${_DOMAIN}/fullchain.crt" \
+        && return 0
+
+    echo "Domain '${_DOMAIN}' is inaktiv and therefore it will be skipped."
+    return 1
+}
+
+function isGitRepository() {
+    local _FOLDER
+    _FOLDER="${1:?"isGitRepository(): Missing first parameter FOLDER"}"
+    readonly _FOLDER
+
+    git -C "${_FOLDER}" ls-tree main &> /dev/null \
+        && return 0
+
+    return 1
+}
+
+function isWildcardCertificate() {
+    local _DOMAIN
+    _DOMAIN="${1:?"isWildcardCertificate(): Missing first parameter DOMAIN"}"
+    readonly _DOMAIN
+
+    echo "${_DOMAIN}" | grep -q -F "_." \
+        && return 0
+
+    echo "${_DOMAIN}" | grep -q -F "*." \
+        && return 0
+
+    return 1
+}
+
+function tryGitPush() {
+    local _DOMAIN _NOW _RESULT_CERTS
+    _RESULT_CERTS="${RESULT_CERTS:?"tryGitPush(): Missing global parameter RESULT_CERTS"}"
+    _DOMAIN="${1:?"tryGitPush(): Missing first parameter DOMAIN"}"
+    _NOW="$(date +%Y%m%d_%H%M)"
+    readonly _DOMAIN _NOW _RESULT_CERTS
+
+    ! isGitRepository "${_RESULT_CERTS}" \
+        && echo \
+        && echo "Folder '${_RESULT_CERTS}' is not part of a git repository, therefore nothing will be pushed." \
+        && return 1
+
+    pushd "${_RESULT_CERTS}" > /dev/null
+    git pull > /dev/null
+    git add * > /dev/null
+    git commit -m "${_NOW} - Certificate for '${_DOMAIN}' was updated." \
+        && git push > /dev/null \
+        && popd > /dev/null \
+        && echo "SUCCESS: certificate for '${_DOMAIN}' pushed." \
+        && return 0
+
+   popd > /dev/null
+   echo "FAILED: unable to push certificate for '${_DOMAIN}'."
+   return 0
+}
+
+function own() {
+    ! [ -d "${RESULT_CERTS:?"own(): Missing global parameter RESULT_CERTS"}" ] \
+        && echo "Trying to derive domain names from subfolders of '${RESULT_CERTS}', but it is not a folder!" \
+        && return 1
+
+    local _DOMAINS _MODE
+    _DOMAINS=("${RESULT_CERTS}"*)
+    _MODE="${1:?"own(): Missing first parameter MODE"}"
+    readonly _DOMAINS _MODE
+
+    local _domain
+    for _domain in "${_DOMAINS[@]}"; do
+        # just take names of folders
+        ! [ -d "${_domain}" ] && continue
+        # cut pfad (like basename)
+        _domain="${_domain##*/}"
+
+        # folder default => skip
+        [ "${_domain}" == "default" ] && continue
+
+        case "${_MODE}" in
+            dns)
+                # dns supports all options
+                ;;
+            http)
+                # http and wildcard certifikate => skip
+                isWildcardCertificate "${_domain}" && continue
+                # ssl on domain inaktiv => skip
+                ! isActive "{_MODE}" "${_domain}" && continue
+                ;;
+            *)
+                echo "Unknown mode: ${_MODE}"
+                return 1
+                ;;
+        esac
+
+        single "${_MODE}" "${_domain}" "${2}"
+    done
+
+    return 0
+}
+
+function continueIssuingCertificate() {
+    local _CERT_FILE_FULLCHAIN _DOMAIN
+    _CERT_FILE_FULLCHAIN="${1:?"continueIssuingCertificate(): Missing first parameter CERT_FILE_FULLCHAIN"}"
+    _DOMAIN="${2:?"continueIssuingCertificate(): Missing second parameter DOMAIN"}"
+    local _CERT_FILE_FULLCHAIN _DOMAIN
+
+    local _PRETTY_DOMAIN
+    _PRETTY_DOMAIN="$(printPrettyDomain ${_DOMAIN})"
+    readonly _PRETTY_DOMAIN
+
+    # forced => should be issued
+    [ "${3:-""}" == "--force" ] \
+        && echo "Certificate for domain '${_PRETTY_DOMAIN}' is forced to be issued." \
+        && return 0
+
+    # no cert => should be issued
+    ! [ -f "${_CERT_FILE_FULLCHAIN}" ] \
+        && echo "No certificate for domain '${_PRETTY_DOMAIN}', so it will be issued." \
+        && return 0
+
+    local _ENDDATE _NOW _REMAINING_DAYS
+    _ENDDATE="$(openssl x509 -enddate -noout -in ${_CERT_FILE_FULLCHAIN} | cut -d= -f2)"
+    _ENDDATE="$(date --date="${_ENDDATE}" --utc +%s)"
+
+    _NOW="$(date --date now +%s)"
+    _REMAINING_DAYS="$(( (_ENDDATE - _NOW) / 86400 ))"
+    readonly _ENDDATE _NOW _REMAINING_DAYS
+
+    # less than 30 days remaining => should be issued
+    [ "${_REMAINING_DAYS}" -le "30" ] \
+        && echo "Certificate for domain '${_PRETTY_DOMAIN}' (${_REMAINING_DAYS} days remaining) will be issued." \
+        && return 0
+
+    echo "Certificate for domain '${_PRETTY_DOMAIN}' (${_REMAINING_DAYS} days remaining) will be skipped."
+    return 1
+}
+
+function printBaseDomain() {
+    local _DOMAIN
+    _DOMAIN="${1:?"printBaseDomain(): Missing first parameter DOMAIN"}"
+    readonly _DOMAIN
+
+    local _BASE_DOMAIN
+    # cut front '*.' or '_.'
+    _BASE_DOMAIN="${_DOMAIN#\*.}"
+    _BASE_DOMAIN="${_BASE_DOMAIN#_.}"
+    readonly _BASE_DOMAIN
+
+    echo "${_BASE_DOMAIN}" \
+        && return 0
+
+    return 1
+}
+
+function printPrettyDomain() {
+    local _BASE_DOMAIN _DOMAIN
+    _DOMAIN="${1:?"printPrettyDomain(): Missing first parameter DOMAIN"}"
+    _BASE_DOMAIN="$(printBaseDomain ${_DOMAIN})"
+    readonly _BASE_DOMAIN _DOMAIN
+
+    isWildcardCertificate "${_DOMAIN}" \
+        && echo "*.${_BASE_DOMAIN}" \
+        && return 0
+
+    echo "${_DOMAIN}" \
+        && return 0
+
+    return 1
+}
+
+function printFullDomainFolder() {
+    local _BASE_DOMAIN _DOMAIN _RESULT_CERTS
+    _RESULT_CERTS="${RESULT_CERTS:?"printFullDomainFolder(): Missing global parameter RESULT_CERTS"}"
+    _DOMAIN="${1:?"printFullDomainFolder(): Missing first parameter DOMAIN"}"
+    _BASE_DOMAIN="$(printBaseDomain ${_DOMAIN})"
+    readonly _BASE_DOMAIN _DOMAIN _RESULT_CERTS
+
+    isWildcardCertificate "${_DOMAIN}" \
+        && echo "${_RESULT_CERTS}_.${_BASE_DOMAIN}/" \
+        && return 0
+
+    echo "${_RESULT_CERTS}${_DOMAIN}/" \
+        && return 0
+
+    return 1
+}
+
+function prepareFullDomainFolder() {
+    local _DOMAIN _DOMAIN_FOLDER
+    _DOMAIN="${1:?"prepareFullDomainFolder(): Missing first parameter DOMAIN"}"
+    _DOMAIN_FOLDER="$(printFullDomainFolder "${_DOMAIN}")"
+    readonly _DOMAIN _DOMAIN_FOLDER
+
+    [ -d "${_DOMAIN_FOLDER}" ] \
+        && return 0
+
+    # create folder for results
+    echo -n "Creating folder '${_DOMAIN_FOLDER}'... " \
+        && mkdir -p "${_DOMAIN_FOLDER}" \
+        && echo "Done"
+
+    [ -d "${_DOMAIN_FOLDER}" ] \
+        && return 0
+
+    return 1
+}
+
+function prepareAndCheckAliasDomain() {
+    local _ALIAS_DOMAIN _DOMAIN
+    _DOMAIN="${1:?"prepareAndCheckAliasDomain(): Missing first parameter DOMAIN"}"
+    _ALIAS_DOMAIN="${2:?"prepareAndCheckAliasDomain(): Missing second parameter ALIAS_DOMAIN"}"
+    readonly _ALIAS_DOMAIN _DOMAIN
+
+    local _BASE_DOMAIN _CHALLENGE_ALIAS_DOMAIN_FILE _DOMAIN_FOLDER
+    _BASE_DOMAIN="$(printBaseDomain ${_DOMAIN})"
+    _DOMAIN_FOLDER="$(printFullDomainFolder ${_DOMAIN})"
+    _CHALLENGE_ALIAS_DOMAIN_FILE="${_DOMAIN_FOLDER}challenge-alias-domain"
+    readonly _BASE_DOMAIN _CHALLENGE_ALIAS_DOMAIN_FILE _DOMAIN_FOLDER
+
+    [ -d "${_DOMAIN_FOLDER}" ] \
+        && [ "$(dig +short _acme-challenge.${_BASE_DOMAIN} CNAME)" == "_acme-challenge.${_ALIAS_DOMAIN}." ] \
+        && echo "${_ALIAS_DOMAIN}" > "${_CHALLENGE_ALIAS_DOMAIN_FILE}" \
+        && echo "SUCCESS: alias domain '${_ALIAS_DOMAIN}' is used when issuing certificates for '${_BASE_DOMAIN}' via DNS." \
+        && return 0
+
+    echo "FAILED: unable to use alias domain '${_ALIAS_DOMAIN}' to issue certificates for '${_BASE_DOMAIN}'."
+    echo "        You have to configure your domain '${_BASE_DOMAIN}' first before you can use the alias domain as proof."
+    echo "        So check if there is a CNAME entry '_acme-challenge.${_BASE_DOMAIN}' pointing to:"
+    echo "          - '_acme-challenge.${_ALIAS_DOMAIN}'"
+    return 1
+}
+
+function single() {
+    local _ACME_FILE _DOMAIN _MODE _RESULT_CERTS
+    _RESULT_CERTS="${RESULT_CERTS:?"single(): Missing global parameter RESULT_CERTS"}"
+    _ACME_FILE="${ACME_FILE:?"single(): Missing global parameter ACME_FILE"}"
+    _MODE="${1:?"single(): Missing first parameter MODE"}"
+    _DOMAIN="${2:?"single(): Missing second parameter DOMAIN"}"
+    readonly _ACME_FILE _DOMAIN _MODE _RESULT_CERTS
+
+    local _BASE_DOMAIN _CHALLENGE_ALIAS_DOMAIN_FILE _DOMAIN_FOLDER _PRETTY_DOMAIN
+    _BASE_DOMAIN="$(printBaseDomain ${_DOMAIN})"
+    _DOMAIN_FOLDER="$(printFullDomainFolder ${_DOMAIN})"
+    _PRETTY_DOMAIN="$(printPrettyDomain ${_DOMAIN})"
+    _CHALLENGE_ALIAS_DOMAIN_FILE="${_DOMAIN_FOLDER}challenge-alias-domain"
+    readonly _BASE_DOMAIN _CHALLENGE_ALIAS_DOMAIN_FILE _DOMAIN_FOLDER _PRETTY_DOMAIN
+
+    ! [ -f "${_ACME_FILE}" ] \
+        && echo "Program 'acme.sh' seams not to be installed. Try run 'renewCerts.sh --setup'." \
+        && return 1
+
+    # cancel on broken configuration
+    ! checkConfigViaHttp "${_MODE}" "${_DOMAIN}" \
+        && return 1
+
+    # cancel if folder is not prepared
+    ! [ -d "${_DOMAIN_FOLDER}" ] \
+	    && echo "Certificate of domain '${_PRETTY_DOMAIN}' skipped because of missing folder:" \
+        && echo "  - '${_DOMAIN_FOLDER}'" \
+        && return 1
+
+    # check enddate if third parameter is not --force
+    ! continueIssuingCertificate "${_DOMAIN_FOLDER}fullchain.crt" "${_DOMAIN}" "${3:-""}" \
+        && return 0
+
+    # backup the keys
+    [ -f "${_DOMAIN_FOLDER}fullchain.crt" ] \
+        && cp "${_DOMAIN_FOLDER}fullchain.crt" "${_DOMAIN_FOLDER}fullchain.crt.bak"
+    [ -f "${_DOMAIN_FOLDER}private.key" ] \
+        && cp --preserve=mode,ownership "${_DOMAIN_FOLDER}private.key" "${_DOMAIN_FOLDER}private.key.bak"
+
+    local _OPTIONS
+    # always --force because we check expiring on ourself
+    # _OPTIONS="--issue --force --test"
+    _OPTIONS="--issue --force"
+    if [ "${_MODE}" == "dns" ]; then
+        _OPTIONS="${_OPTIONS} --dns ${AUTOACME_DNS_PROVIDER:?"single(): Missing global parameter AUTOACME_DNS_PROVIDER"}"
+        [ -f "${_CHALLENGE_ALIAS_DOMAIN_FILE}" ] \
+            && _OPTIONS="${_OPTIONS} --challenge-alias $(cat "${_CHALLENGE_ALIAS_DOMAIN_FILE}")"
+        isWildcardCertificate "${_DOMAIN}" \
+            && _OPTIONS="${_OPTIONS} --domain ${_PRETTY_DOMAIN}"
+    elif [ "${_MODE}" == "http" ]; then
+        _OPTIONS="${_OPTIONS} --webroot /var/www/letsencrypt"
+    fi
+    readonly _OPTIONS
+
+    ${_ACME_FILE} ${_OPTIONS} \
+        --domain "${_BASE_DOMAIN}" \
+        --server "letsencrypt" \
+        --keylength "ec-384" \
+        --fullchain-file "${_DOMAIN_FOLDER}fullchain.crt" \
+        --key-file "${_DOMAIN_FOLDER}private.key" \
+        && openssl pkcs12 -export -in "${_DOMAIN_FOLDER}fullchain.crt" -inkey "${_DOMAIN_FOLDER}private.key" -out "${_DOMAIN_FOLDER}bundle.pkx" -passout pass: \
+        && echo "Certificate of domain '${_PRETTY_DOMAIN}' was updated." \
+        && tryGitPush "${_PRETTY_DOMAIN}" \
+        && return 0
+
+    echo "Certificate of domain '${_PRETTY_DOMAIN}' remains unchanged."
+    return 0
+}
+
+function isInstalled() {
+    local _ACME_FILE
+    _ACME_FILE="${ACME_FILE:?"isInstalled(): Missing global parameter ACME_FILE"}"
+    readonly _ACME_FILE
+
+    [ -f "${_ACME_FILE}" ] \
+        && echo "Following version of acme.sh is installed:" \
+        && echo "------------------------------------------" \
+        && ${_ACME_FILE} --version | tail -n 1 \
+        && return 0
+
+    return 1
+}
+
+function extractTarArchive() {
+    local _ACME_SETUP_FILE _ACME_TAR_FILE
+    _ACME_SETUP_FILE="${ACME_SETUP_FILE:?"extractTarArchive(): Missing global parameter ACME_SETUP_FILE"}"
+    _ACME_TAR_FILE="${ACME_TAR_FILE:?"extractTarArchive(): Missing global parameter ACME_TAR_FILE"}"
+    readonly _ACME_SETUP_FILE _ACME_TAR_FILE
+
+    # extracted file already exists
+    [ -f "${_ACME_SETUP_FILE}" ] \
+        && return 0
+
+    [ -f "${_ACME_TAR_FILE}" ] \
+        && mkdir -p "/tmp/acme.sh-setup/" \
+        && tar -xzf "${_ACME_TAR_FILE}" -C "/tmp/acme.sh-setup/" \
+        && [ -f "${_ACME_SETUP_FILE}" ] \
+        && return 0
+
+    echo "Missing setup file '${_ACME_SETUP_FILE}' after trying to extract '${_ACME_TAR_FILE}'"
+    return 1
+}
+
+function setup() {
+    local _ACME_SETUP_FILE
+    _ACME_SETUP_FILE="${ACME_SETUP_FILE:?"setup(): Missing global parameter ACME_SETUP_FILE"}"
+    readonly _ACME_SETUP_FILE
+
+    isInstalled \
+        && return 0
+
+    ! [ $(id -u) == 0 ] \
+        && echo "Setup requires execution as user 'root'." \
+        && exit 1
+
+    ! [ "$(echo $HOME)" == "/root" ] \
+        && echo "The setup is executed with 'root' privileges but not in the 'root' user environment." \
+        && exit 1
+
+    ! extractTarArchive \
+        && exit 1
+
+    echo "Starting install of acme.sh:"
+    echo "----------------------------"
+    pushd "${_ACME_SETUP_FILE%/*}" > /dev/null 2>&1    #Removes shortest matching pattern '/*' from the end
+    ./acme.sh --install --no-cron --no-profile 2>&1
+    popd > /dev/null 2>&1
+    isInstalled \
+        && echo \
+        && echo 'Now this script can be added into cron-tab (crontab -e), like this e.g.:' \
+        && echo \
+        && echo '# Each day at 6:00am renew certificates:' \
+        && echo '0 6 * * * /renewCerts.sh --http --own > /var/log/renewCerts.sh.log 2>&1' \
+        && return 0
+
+    echo "Something went wrong during setup."
+    return 1
+}
+
+function usage() {
+    echo
+    echo 'Commands:'
+    echo '  --prepare DOMAIN --usingAlias ALIAS-DOMAIN  : Prepares a domain to issue certificate using an alias domain in DNS mode.'
+    echo '                                                    See: https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mode'
+    echo '  --dns            --single DOMAIN [--force]  : Issues a certificate for the given domain using DNS mode.'
+    echo '  --http           --single DOMAIN [--force]  : Issues a certificate for the given domain using HTTP mode.'
+    echo
+    echo ' (--dns|--http)    --own [--force]            : Iterates all domains found in RESULT_CERTS.'
+    echo
+    echo 'Current environment:'
+    echo "    Full name of this script:                      OWN_FULLNAME='${OWN_FULLNAME}'"
+    echo "  Configuration:"
+    echo "    Version of 'acme.sh' that will be installed:   ACME_VERSION='${ACME_VERSION}'"
+    echo "    Tar file containing the setup of 'acme.sh':    ACME_TAR_FILE='${ACME_TAR_FILE}'"
+    echo "    Setup file of 'acme.sh' after extraction:      ACME_SETUP_FILE='${ACME_SETUP_FILE}'"
+    echo "    Full name of the installed script 'acme.sh':   ACME_FILE='${ACME_FILE}'"
+    echo "  Output:"
+    echo "    Path were the issued certificate are saved:    RESULT_CERTS='${RESULT_CERTS}'"
+
+    return 0
+}
+
+function main() {
+
+    echo
+
+    [ -f "/autoACME.env" ] \
+        && source "/autoACME.env" \
+        && echo "[$(date)] Environment '/autoACME.env' loaded."
+
+    local ACME_FILE ACME_VERSION OWN_FULLNAME
+    OWN_FULLNAME="$(readlink -e ${0})"
+    ACME_FILE="/root/.acme.sh/acme.sh"
+    ACME_VERSION="acme.sh-3.1.1"
+    readonly ACME_FILE ACME_VERSION OWN_FULLNAME
+
+    local ACME_SETUP_FILE ACME_TAR_FILE RESULT_CERTS
+    ACME_SETUP_FILE="/tmp/acme.sh-setup/${ACME_VERSION}/acme.sh"
+    ACME_TAR_FILE="${OWN_FULLNAME%/*}/${ACME_VERSION}.tar.gz"
+    RESULT_CERTS="${AUTOACME_RESULT_CERTS%/}"    #Removes shortest matching pattern '/' from the end
+    RESULT_CERTS="${RESULT_CERTS:-"/etc/nginx/ssl"}/"
+    readonly ACME_SETUP_FILE ACME_TAR_FILE RESULT_CERTS
+
+    local REPOSITORY_URL
+    isGitRepository "${RESULT_CERTS}" \
+        && REPOSITORY_URL="$(git -C ${RESULT_CERTS} config --get remote.origin.url)"
+    readonly REPOSITORY_URL
+
+    case "${1}" in
+        --dns)
+            case "${2}" in
+                --single)
+                    echo "[$(date)] Issue single certificate '${3}' via DNS:" \
+                        && prepareFullDomainFolder "${3}" \
+                        && single "dns" "${3}" "${4}" \
+                        && return 0
+                    ;;
+                --own)
+                    echo "[$(date)] Renewing own certificates via DNS:"
+                    own "dns" "${3}" \
+                        && echo "Finished successfully." \
+                        && return 0
+                    ;;
+            esac
+            ;;
+        --http)
+            case "${2}" in
+                --single)
+                    echo "[$(date)] Issue single certificate '${3}' via HTTP:" \
+                        && prepareFullDomainFolder "${3}" \
+                        && single "http" "${3}" "${4}" \
+                        && echo \
+                        && echo "Checking configuration of nginx and restart the webserver:" \
+                        && echo "==========================================================" \
+                        && nginx -t && systemctl reload nginx \
+                        && return 0
+                    ;;
+                --own)
+                    echo "[$(date)] Renewing own certificates via HTTP:" \
+                        && own "http" "${3}" \
+                        && echo \
+                        && echo "Checking configuration of nginx and restart the webserver:" \
+                        && echo "==========================================================" \
+                        && nginx -t && systemctl reload nginx \
+                        && return 0
+                    ;;
+            esac
+            ;;
+        --prepare)
+            case "${3}" in
+                --usingAlias)
+                    echo "[$(date)] Prepare domain '${2}' using the alias-domain '${4}' via DNS:" \
+                        && prepareFullDomainFolder "${2}" \
+                        && prepareAndCheckAliasDomain "${2}" "${4}" \
+                        && return 0
+                    ;;
+                *)
+                    echo "Unknown command '${1}' '${2}' '${3}' '${4}'"
+                    usage
+                    return 1
+                    ;;
+            esac
+            ;;
+        --setup)
+            setup \
+                && return 0
+            ;;
+        *)
+            echo "Unknown command '${1}' '${2}'"
+            usage
+            return 1
+            ;;
+    esac
+
+    return 1
+}
+
+main "$@" && exit 0 || exit 1

script/ssl/start.sh

@@ -0,0 +1,161 @@
+#/bin/bash
+
+function createEnvironmentFile() {
+    local _ENVIRONMENT_FILE _REPOSITORY_FOLDER
+    _ENVIRONMENT_FILE="${ENVIRONMENT_FILE:?"createEnvironmentFile(): Missing global parameter ENVIRONMENT_FILE"}"
+    _REPOSITORY_FOLDER="${AUTOACME_REPOSITORY_FOLDER:?"createEnvironmentFile(): Missing global parameter AUTOACME_REPOSITORY_FOLDER"}"
+    readonly _ENVIRONMENT_FILE _REPOSITORY_FOLDER
+
+    # Save environment for cronjob
+    export -p | grep -v -E "(HOME|OLDPWD|PWD|SHLVL)" > "${_ENVIRONMENT_FILE}" \
+        && echo "SUCCESS: there values were exported into file: '${_ENVIRONMENT_FILE}'" \
+        && echo "  - AUTOACME_CONTAINER_HOSTNAME: ${AUTOACME_CONTAINER_HOSTNAME}" \
+        && echo "  - AUTOACME_DNS_PROVIDER: ${AUTOACME_DNS_PROVIDER}" \
+        && echo "  - AUTOACME_GIT_REPOSITORY_VIA_SSH: ${AUTOACME_GIT_REPOSITORY_VIA_SSH}" \
+        && echo "  - AUTOACME_PATH_IN_GIT_REPOSITORY: ${AUTOACME_PATH_IN_GIT_REPOSITORY}"
+
+    [ "${AUTOACME_GIT_REPOSITORY_VIA_SSH}" == "" ] \
+        && echo "declare -x AUTOACME_RESULT_CERTS=\"${AUTOACME_REPOSITORY_FOLDER#/}\"" >> "${_ENVIRONMENT_FILE}" \
+        && echo "SUCCESS: added AUTOACME_RESULT_CERTS (without git) into file '${_ENVIRONMENT_FILE}'." \
+        && echo "  - AUTOACME_RESULT_CERTS: ${AUTOACME_REPOSITORY_FOLDER#/}" \
+        && echo "        (depends on if there is a git repo and the path for the certs in it)"
+
+    ! [ "${AUTOACME_GIT_REPOSITORY_VIA_SSH}" == "" ] \
+        && echo "declare -x AUTOACME_RESULT_CERTS=\"${AUTOACME_REPOSITORY_FOLDER}${AUTOACME_PATH_IN_GIT_REPOSITORY#/}\"" >> "${_ENVIRONMENT_FILE}" \
+        && echo "SUCCESS: added AUTOACME_RESULT_CERTS (with git) into file '${_ENVIRONMENT_FILE}'." \
+        && echo "  - AUTOACME_RESULT_CERTS: ${AUTOACME_REPOSITORY_FOLDER}${AUTOACME_PATH_IN_GIT_REPOSITORY#/}" \
+        && echo "        (depends on if there is a git repo and the path for the certs in it)"
+
+    return 0
+}
+
+function ensureThereAreSSHKeys() {
+    grep -F 'ssh' "/root/.ssh/id_ed25519.pub" &> /dev/null \
+        && echo "SUCCESS: ssh-keys found, printing public key:" \
+        && cat "/root/.ssh/id_ed25519.pub" \
+        && return 0
+
+    # -t    type of the key pair
+    # -f    defines the filenames (we use the standard for the selected type here)
+    # -q    quiet, no output or interaction
+    # -N "" means the private key will not be secured by a passphrase
+    # -C    defines a comment
+    ssh-keygen \
+        -t ed25519 \
+        -f "/root/.ssh/id_ed25519" -q -N "" \
+        -C "$(date +%Y%m%d)-root@$(hostname -s)_onHost_${AUTOACME_CONTAINER_HOSTNAME%%.*}"
+
+    grep -F 'ssh' "/root/.ssh/id_ed25519.pub" &> /dev/null \
+        && echo "SUCCESS: ssh-keys generated, printing public key:" \
+        && cat "/root/.ssh/id_ed25519.pub" \
+        && return 0
+
+    echo
+    echo "FAILED: something went wrong during the generation of the ssh keys..."
+    echo "        These keys are mandantory to access the git repository."
+    echo "You can try to restart this script."
+    echo
+    return 1
+}
+
+function ensureGitIsInstalled() {
+    git --version &> /dev/null \
+        && return 0
+
+    echo \
+        && echo "Installing Git in 30s (ensure the SSH-Key is trusted and has write pemissions)... " \
+        && sleep 30 \
+        && DEBIAN_FRONTEND=noninteractive \
+        && apt-get install git -y &> /dev/null \
+        && echo "SUCCESS: $(git --version) is usable now." \
+        && return 0
+
+    echo
+    echo "FAILED: something went wrong during the installation of Git..."
+    echo "        Git is mandantory to push the keys into the specified repository."
+    echo "You can try to install git manually (apt install git)."
+    echo
+    return 1
+}
+
+function ensureRepositoryIsAvailableAndWritable() {
+    local _REPOSITORY_FOLDER
+    _REPOSITORY_FOLDER="${AUTOACME_REPOSITORY_FOLDER:?"ensureRepositoryIsAvailableAndWritable(): Missing global parameter AUTOACME_REPOSITORY_FOLDER"}"
+    readonly _REPOSITORY_FOLDER
+
+    [ -d "${_REPOSITORY_FOLDER}.git" ] \
+        && echo \
+        && git -C "${_REPOSITORY_FOLDER}" pull &> /dev/null \
+        && git -C "${_REPOSITORY_FOLDER}" push --dry-run &> /dev/null \
+        && echo "Writable repository found in folder '${_REPOSITORY_FOLDER}'." \
+        && return 0
+
+    ! [ -d "${_REPOSITORY_FOLDER}.git" ] \
+        && echo \
+        && echo "Cloning repository '${AUTOACME_GIT_REPOSITORY_VIA_SSH}'... " \
+        && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git clone "${AUTOACME_GIT_REPOSITORY_VIA_SSH}" "${_REPOSITORY_FOLDER}" &> /dev/null \
+        && git -C "${_REPOSITORY_FOLDER}" config user.name "autoacme on ${AUTOACME_CONTAINER_HOSTNAME%%.*}" \
+        && git -C "${_REPOSITORY_FOLDER}" config user.email "autoacme@${AUTOACME_CONTAINER_HOSTNAME%%.*}" \
+        && git -C "${_REPOSITORY_FOLDER}" push --dry-run &> /dev/null \
+        && echo "SUCCESS: repository cloned into folder '${_REPOSITORY_FOLDER}' and it is writable." \
+        && return 0
+
+    echo
+    echo "FAILED: something went wrong during cloning the repository to '${_REPOSITORY_FOLDER}' from:"
+    echo "        - ${AUTOACME_GIT_REPOSITORY_VIA_SSH}"
+    echo
+    echo "1.) You can try to clone it manually into:  git clone ${AUTOACME_GIT_REPOSITORY_VIA_SSH} '${_REPOSITORY_FOLDER}'"
+    echo "2.) Check if the repositoty is writable:    git -C '${_REPOSITORY_FOLDER}' push --dry-run"
+    return 1
+}
+
+function prepareThisRuntimeForUsingGitOrIgnore() {
+    createEnvironmentFile \
+        || return 1
+
+    [ "${AUTOACME_GIT_REPOSITORY_VIA_SSH}" == "" ] \
+        && echo "There is no git repository specified." \
+        && echo "To distribute all keys and certificates via a git repository set environment variable:" \
+        && echo "  - AUTOACME_GIT_REPOSITORY_VIA_SSH" \
+        && echo \
+        && echo "FIRST AND ONLY WARNING: DO NOT USE ANY PUBLIC GIT SERVICE FOR THAT!" \
+        && echo \
+        && return 0
+
+    echo \
+        && ensureThereAreSSHKeys \
+        && ensureGitIsInstalled \
+        && ensureRepositoryIsAvailableAndWritable \
+        && return 0
+
+    echo "No job will run inside this container because there is an issue."
+    echo "The container keeps running for 10min, please check your setup..."
+    return 1
+}
+
+AUTOACME_REPOSITORY_FOLDER="/root/acmeResults/"
+ENVIRONMENT_FILE="/autoACME.env"
+
+echo
+echo '################################################################################'
+echo "# Container started at $(date +%F_%T) on host ${AUTOACME_CONTAINER_HOSTNAME}"
+echo '################################################################################'
+echo
+
+# Log start and truncate file: /autoACME.log
+echo > /autoACME.log
+
+# Generate SSH keys and setup Git if a repository is specified, on failure keep the container running
+prepareThisRuntimeForUsingGitOrIgnore \
+    || timeout --preserve-status 10m tail -f /autoACME.log
+
+# Ensure acme.sh ist installed
+/renewCerts.sh --setup >> /autoACME.log \
+    && echo >> /autoACME.log
+
+echo "Register following entry to crontab:" >> /autoACME.log
+echo "------------------------------------" >> /autoACME.log
+_CRON_ENTRY="$((RANDOM % 59)) $((RANDOM % 5)) * * * /renewCerts.sh --dns --own >> /autoACME.log 2>&1"
+echo "${_CRON_ENTRY}" | tee -a /autoACME.log | crontab -
+
+cron && tail -n 100 -f /autoACME.log

setupCoreOntoThisHost.sh

@@ -6,6 +6,12 @@
 
 
 
+# Folders always ends with an tailing '/'
+_SETUP="$(readlink -f "${0}" 2> /dev/null)"
+_CIS_ROOT="${_SETUP%/setupCoreOntoThisHost.sh}/"             #Removes shortest matching pattern '/setupCoreOntoThisHost.sh' from the end
+_CORE_SCRIPTS="${_CIS_ROOT:?"Missing CIS_ROOT"}core/"
+
+
 
 function checkPathsAreAvaiable() {
     grep --version &> /dev/null \
@@ -31,22 +37,21 @@ function checkGitIsAvailable() {
 }
 
 function checkPreconditions() {
-    local _ROOT _DOMAIN
+    local _DOMAIN
-    _ROOT="${1:?"Missing parameter ROOT"}"
+    _DOMAIN="${1}" # Optional parameter DOMAIN
-    _DOMAIN="${2}" # Optional parameter DOMAIN
+    readonly _DOMAIN
-    readonly _ROOT _DOMAIN
 
     ! [ -z "${_DOMAIN}" ] \
         && [ "$(hostname -d)" != "${_DOMAIN}" ] \
         && echo \
-        && echo "WARNING: system-domain DOES NOT MATCH domainOfHostOwner: '$(hostname -d)' != '${_DOMAIN}'" \
+        && echo "WARNING: system-domain DOES NOT MATCH overrideOwnDomain: '$(hostname -d)' != '${_DOMAIN}'" \
         && echo
 
     # Given domain verfügbar (nicht leer)
     ! [ -z "${_DOMAIN}" ] \
         && checkPathsAreAvaiable \
         && checkGitIsAvailable \
-        && git -C "${_ROOT}" pull &> /dev/null \
+        && git -C "${_CIS_ROOT:?"Missing CIS_ROOT"}" pull &> /dev/null \
         && return 0
 
     echo
@@ -69,63 +74,65 @@ function checkPreconditions() {
 }
 
 function getOrSetDomain() {
-    local _ROOT _DOMAIN_FILE _GIVEN_DOMAIN
+    local _CURRENT_DOMAIN _GIVEN_DOMAIN _OVERRIDE_DOMAIN_FILE
-    _ROOT="${1:?"Missing parameter ROOT"}"
+    _CURRENT_DOMAIN="$("${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}printOwnDomain.sh")"
-    _DOMAIN_FILE="${_ROOT:?"Missing ROOT"}domainOfHostOwner"
+    _GIVEN_DOMAIN="${1}" # Optional parameter DOMAIN
-    _GIVEN_DOMAIN="${2}" # Optional parameter DOMAIN
+    _OVERRIDE_DOMAIN_FILE="${_CIS_ROOT:?"Missing CIS_ROOT"}overrideOwnDomain"
-    readonly _ROOT _DOMAIN_FILE _GIVEN_DOMAIN
+    readonly _CURRENT_DOMAIN _GIVEN_DOMAIN _OVERRIDE_DOMAIN_FILE
 
-    # Wenn DOMAIN_FILE enhält lesbare Daten
+    ! [ -z "${_CURRENT_DOMAIN}" ] \
-    grep '[^[:space:]]' "${_DOMAIN_FILE:?"Missing DOMAIN_FILE"}" &> /dev/null \
+        && [ -z "${_GIVEN_DOMAIN}" ] \
-        && cat "${_DOMAIN_FILE}" \
+        && echo "${_CURRENT_DOMAIN}" \
         && return 0
 
-    # Der boot-hostname muss mindestens einen Punkt enthalten, dann wird die hintere Hälfte als Domain genommen
+    ! [ -z "${_CURRENT_DOMAIN}" ] \
-    hostname -b | grep "\." | cut -d. -f2- > "${_DOMAIN_FILE}"
+        && [ "${_CURRENT_DOMAIN}" == "${_GIVEN_DOMAIN}" ] \
-    grep '[^[:space:]]' "${_DOMAIN_FILE}" &> /dev/null \
+        && echo "${_CURRENT_DOMAIN}" \
-        && cat "${_DOMAIN_FILE}" \
         && return 0
 
-    # Given domain is set (nicht leer)
+    # If there is a given domain it will be set or it will override the current one
-    ! [ -z "${_GIVEN_DOMAIN}" ] \
+    [ -z "${_CURRENT_DOMAIN}" ] \
+        && ! [ -z "${_GIVEN_DOMAIN}" ] \
         && [ "$(id -u)" == "0" ] \
+        && echo "Setting hostname to: $(hostname -s).${_GIVEN_DOMAIN}" >&2 \
         && hostnamectl set-hostname "$(hostname -s).${_GIVEN_DOMAIN}" \
-        && hostname -b | grep "\." | cut -d. -f2- > "${_DOMAIN_FILE}" \
+        && echo "${_GIVEN_DOMAIN}" \
-        && grep '[^[:space:]]' "${_DOMAIN_FILE}" &> /dev/null \
+        && return 0
-        && cat "${_DOMAIN_FILE}" \
+
+    ! [ -z "${_GIVEN_DOMAIN}" ] \
+        && echo "Overwriting domain to: ${_GIVEN_DOMAIN}" >&2 \
+        && echo "${_GIVEN_DOMAIN}" > "${_OVERRIDE_DOMAIN_FILE}" \
+        && echo "${_GIVEN_DOMAIN}" \
         && return 0
 
     return 1
 }
 
 function getRemoteRepositoryPath() {
-    local _ROOT
+    _REPOSITORY="$(git -C "${_CIS_ROOT:?"Missing CIS_ROOT"}" config --get remote.origin.url 2> /dev/null | grep -i 'git@')"
-    _ROOT="${1:?"Missing parameter ROOT"}"
+    _PATH="${_REPOSITORY%/*}"                        #Removes shortest matching pattern '/*' from the end
-    readonly _ROOT
+    ! [ -z "${_PATH}" ] \
-
+        && echo "${_PATH}/" \
-    _RESULT="$(git -C "${_ROOT:?"Missing ROOT"}" remote show origin | grep -i 'fetch' | xargs -n 1 | grep -i 'ssh://')"
-    _RESULT="${_RESULT%/*}"                        #Removes shortest matching pattern '/*' from the end
-    ! [ -z "${_RESULT}" ] \
-        && echo "${_RESULT}" \
         && return 0
 
     return 1
 }
 
 function addDefinition(){
-    local _ROOT _CORE_SCRIPTS _DEFINITIONS _REPOSITORY
+    local _DEFINITIONS _REPOSITORY
-    _DEFINITIONS="${1:?"Missing parameter DEFINITIONS"}"
+    _DEFINITIONS="${1:?"Missing first parameter DEFINITIONS"}"
-    _REPOSITORY="${2:?"Missing parameter REPOSITORY"}"
+    _REPOSITORY="$(getRemoteRepositoryPath)cis-definition-${2:?"Missing second parameter DOMAIN"}.git"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    readonly _DEFINITIONS _REPOSITORY
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
+
-    readonly _ROOT _CORE_SCRIPTS _DEFINITIONS _REPOSITORY
     [ "$(id -u)" == "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" "${_REPOSITORY}" readonly \
+        && echo "Running setup as 'root' trying to add definition repository:" \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" readonly "${_REPOSITORY}" \
         && echo "  - definitions are usable for this host." \
         && return 0
 
     [ "$(id -u)" != "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" "${_REPOSITORY}" writable \
+        && echo "Running setup as 'user' trying to add definition repository:" \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_DEFINITIONS}" writable "${_REPOSITORY}" \
         && echo "  - definitions are usable, as working copy." \
         && return 0
 
@@ -133,20 +140,22 @@ function addDefinition(){
 }
 
 function addState() {
-    local _ROOT _CORE_SCRIPTS _STATES _REPOSITORY
+    local _STATES _REPOSITORY
-    _STATES="${1:?"Missing parameter STATES"}"
+    _STATES="${1:?"Missing first parameter STATES"}"
-    _REPOSITORY="${2:?"Missing parameter REPOSITORY"}"
+    _REPOSITORY="$(getRemoteRepositoryPath)cis-state-${2:?"Missing second parameter DOMAIN"}.git"
-    _ROOT="${_STATES%%/states/*}/"             #Removes longest  matching pattern '/states/*' from the end
+    readonly _STATES _REPOSITORY
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
-    readonly _ROOT _CORE_SCRIPTS _STATES _REPOSITORY
 
     [ "$(id -u)" == "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" "${_REPOSITORY}" writable \
+        && echo "Running setup as 'root' trying to add state repository:" \
+        && echo \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" writable "${_REPOSITORY}" \
         && echo "  - states are usable for this host." \
         && return 0
 
     [ "$(id -u)" != "0" ] \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" "${_REPOSITORY}" writable \
+        && echo "Running setup as 'user' trying to add state repository:" \
+        && echo \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addAndCheckGitRepository.sh" "${_STATES}" writable "${_REPOSITORY}" \
         && echo "  - states are usable, as working copy." \
         && return 0
 
@@ -154,13 +163,10 @@ function addState() {
 }
 
 function setupCoreFunctionality() {
-    local _ROOT _CORE_SCRIPTS _DEFINITIONS _MINUTE_FROM_OWN_IP _SETUP
+    local _DEFINITIONS _MINUTE_FROM_OWN_IP
     _DEFINITIONS="${1:?"Missing DEFINITIONS: 'ROOT/definitions/DOMAIN'"}"
-    _ROOT="${_DEFINITIONS%%/definitions/*}/"   #Removes longest  matching pattern '/definitions/*' from the end
+    _MINUTE_FROM_OWN_IP="$(hostname -I | xargs -n 1 | grep -F '.' | head -n 1 | cut -d. -f4 || echo 0)" #uses last value from first own ipv4 or 0 as minute value
-    _CORE_SCRIPTS="${_ROOT:?"Missing ROOT"}core/"
+    readonly _DEFINITIONS _MINUTE_FROM_OWN_IP
-    _MINUTE_FROM_OWN_IP="$(hostname -I | xargs -n 1 | grep -F . | head -n 1 | cut -d. -f4 || echo 0)" #uses last value from first own ipv4 or 0 as minute value
-    _SETUP="${2:?"Missing SETUP"}"
-    readonly _ROOT _CORE_SCRIPTS _DEFINITIONS _MINUTE_FROM_OWN_IP _SETUP
 
     [ "$(id -u)" != "0" ] \
         && echo "Configuration of host skipped because of insufficient rights." \
@@ -176,45 +182,39 @@ function setupCoreFunctionality() {
         && echo \
         && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}ensureUsageOfDefinitions.sh" "${_DEFINITIONS}" /etc/sudoers.d/allow-jenkins-updateRepositories \
         && echo \
-        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addToCrontabEveryHour.sh" "${_SETUP}" "${_MINUTE_FROM_OWN_IP}" \
+        && "${_CORE_SCRIPTS:?"Missing CORE_SCRIPTS"}addToCrontabEveryHour.sh" "${_SETUP:?"Missing SETUP"}" "${_MINUTE_FROM_OWN_IP}" \
         && return 0
 
     return 1
 }
 
 function setup() {
-    local _ROOT _DEFINITIONS _DEFINITIONS_REPOSITORY _DOMAIN _REPOSITORY_PATH _SETUP _STATES _STATES_REPOSITORY
+    local _DEFINITIONS _DOMAIN _STATES
-    _SETUP="$(readlink -f "${0}" 2> /dev/null)"
+    _DOMAIN="$(getOrSetDomain "${1}")"
-    _ROOT="$(dirname ${_SETUP:?"Missing SETUP"} 2> /dev/null || echo "/iss")/"
-    _DOMAIN="$(getOrSetDomain "${_ROOT:?"Missing ROOT"}" "${1}")"
-    _REPOSITORY_PATH="$(getRemoteRepositoryPath "${_ROOT:?"Missing ROOT"}")"
 
-    ! checkPreconditions "${_ROOT:?"Missing ROOT"}" "${_DOMAIN}" \
+    ! checkPreconditions "${_DOMAIN}" \
         && return 1
 
-    _DEFINITIONS="${_ROOT:?"Missing ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
+    _DEFINITIONS="${_CIS_ROOT:?"Missing CIS_ROOT"}definitions/${_DOMAIN:?"Missing DOMAIN"}"
-    _DEFINITIONS_REPOSITORY="${_REPOSITORY_PATH:?"Missing REPOSITORY_PATH"}/iss-definition-${_DOMAIN:?"Missing DOMAIN"}.git"
+    _STATES="${_CIS_ROOT:?"Missing CIS_ROOT"}states/${_DOMAIN:?"Missing DOMAIN"}"
-    _STATES="${_ROOT:?"Missing ROOT"}states/${_DOMAIN:?"Missing DOMAIN"}"
+    readonly _DEFINITIONS _DOMAIN _STATES
-    _STATES_REPOSITORY="${_REPOSITORY_PATH:?"Missing REPOSITORY_PATH"}/iss-state-${_DOMAIN:?"Missing DOMAIN"}.git"
-    readonly _ROOT _DEFINITIONS _DEFINITIONS_REPOSITORY _DOMAIN _REPOSITORY_PATH _SETUP _STATES _STATES_REPOSITORY
 
     echo \
-        && echo "Running setup using repositories of: '${_REPOSITORY_PATH:?"Missing REPOSITORY_PATH"}' ..." \
+        && addDefinition "${_DEFINITIONS:?"Missing DEFINITIONS"}" "${_DOMAIN:?"Missing DOMAIN"}" \
         && echo \
-        && addDefinition "${_DEFINITIONS:?"Missing DEFINITIONS"}" "${_DEFINITIONS_REPOSITORY:?"Missing DEFINITIONS_REPOSITORY"}" \
+        && addState "${_STATES:?"Missing STATES"}" "${_DOMAIN:?"Missing DOMAIN"}" \
-        && echo \
-        && addState "${_STATES:?"Missing STATES"}" "${_STATES_REPOSITORY:?"Missing STATES_REPOSITORY"}" \
         && echo \
         && echo "Using definitions: '${_DEFINITIONS:?"Missing DEFINITIONS"}' ..." \
-        && setupCoreFunctionality "${_DEFINITIONS:?"Missing DEFINITIONS"}" "${_SETUP:?"Missing SETUP"}" \
+        && setupCoreFunctionality "${_DEFINITIONS:?"Missing DEFINITIONS"}" \
         && return 0
 
-    echo "FAIL: setup is incomplete:                        ("$(readlink -f ${0})")"
+    echo "FAIL: setup is incomplete:                         ("$(readlink -f ${0})")" >&2
-    echo "  - due to an error or insufficient rights."
+    echo "  - due to an error or insufficient rights." >&2
     return 1
 }
 
 # sanitizes all parameters
-setup \
+setup "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
+    && exit 0
-    && exit 0 || exit 1
+
+exit 1

updateRepositories.sh

@@ -20,54 +20,61 @@
 
 
 function update_repositories() {
-    local _ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
+    local _CIS_ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
     _UPDATE_REPOSITORIES="$(readlink -f "${0}" 2> /dev/null)"
-    _MODE="${1:-"all"}"
+    _CIS_ROOT="${_UPDATE_REPOSITORIES%/updateRepositories.sh}/"               #Removes shortest matching pattern '/updateRepositories.sh' from the end
-    _ROOT="$(dirname ${_UPDATE_REPOSITORIES:?"Missing UPDATE_REPOSITORIES"} 2> /dev/null || echo "/iss")/"
+    _MODE="${1:-"--core"}"
-    _DOMAIN="$(cat ${_ROOT:?"Missing ROOT"}domainOfHostOwner)"
+    _DOMAIN="$(${_CIS_ROOT:?"Missing CIS_ROOT"}core/printOwnDomain.sh)"
-    _DEFINITIONS="${_ROOT}definitions/${_DOMAIN:?"Missing DOMAIN from file: ${_ROOT}domainOfHostOwner"}/"
+    _DEFINITIONS="${_CIS_ROOT}definitions/${_DOMAIN:?"Missing DOMAIN from file: ${_CIS_ROOT}domainOfHostOwner"}/"
-    _STATES="${_ROOT}states/${_DOMAIN:?"Missing DOMAIN from file: ${_ROOT}domainOfHostOwner"}/"
+    _STATES="${_CIS_ROOT}states/${_DOMAIN:?"Missing DOMAIN from file: ${_CIS_ROOT}domainOfHostOwner"}/"
-    readonly _ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
+    readonly _CIS_ROOT _DEFINITIONS _DOMAIN _MODE _STATES _UPDATE_REPOSITORIES
 
     [ "${_MODE}" == "--repair" ] \
-        && (git -C "${_ROOT}" reset --hard origin/master; \
+        && (git -C "${_CIS_ROOT}" reset --hard origin/main; \
-            git -C "${_DEFINITIONS}" reset --hard origin/master; \
+            git -C "${_DEFINITIONS}" reset --hard origin/main; \
-            git -C "${_STATES}" reset --hard origin/master; \
+            git -C "${_STATES}" reset --hard origin/main; \
             echo "Run repairs") \
         && return 0
 
     [ "${_MODE}" == "--test" ] \
-        && git -C "${_ROOT}" pull \
+        && git -C "${_CIS_ROOT}" pull \
         && git -C "${_DEFINITIONS}" pull \
         && git -C "${_STATES}" pull \
         && echo "Run in testMode successfully." \
         && return 0
 
     [ "${_MODE}" == "--scripts" ] \
-        && echo "Host $HOSTNAME updating scripts: ${_ROOT} ..." \
+        && printf "Host $HOSTNAME updating scripts: ${_CIS_ROOT} ... " \
-        && (git -C "${_ROOT}" pull &> /dev/null &) \
+        && (git -C "${_CIS_ROOT}" pull &> /dev/null) \
+        && echo "(done)" \
         && return 0
 
     [ "${_MODE}" == "--definitions" ] \
-        && echo "Host ${HOSTNAME} updating definitions: ${_DEFINITIONS} ..." \
+        && echo "Host ${HOSTNAME} updating definitions: ${_DEFINITIONS} ... " \
-        && (git -C "${_DEFINITIONS}" pull &> /dev/null &) \
+        && (git -C "${_DEFINITIONS}" pull &> /dev/null) \
+        && echo "(done)" \
         && return 0
 
     [ "${_MODE}" == "--states" ] \
-        && echo "Host ${HOSTNAME} updating states: ${_STATES} ..." \
+        && echo "Host ${HOSTNAME} updating states: ${_STATES} ... " \
-        && (git -C "${_STATES}" pull &> /dev/null &) \
+        && (git -C "${_STATES}" pull &> /dev/null) \
+        && echo "(done)" \
         && return 0
 
-    echo "Host ${HOSTNAME} updating ${_MODE}:" \
+    [ "${_MODE}" == "--core" ] \
-        && echo "  - ${_ROOT}" \
+        && echo "Host ${HOSTNAME} updating core including scripts, definitions and states: ${_STATES} ... " \
-        && echo "  - ${_DEFINITIONS}" \
+        && (git -C "${_CIS_ROOT}" pull &> /dev/null) \
-        && echo "  - ${_STATES}"
+        && (git -C "${_DEFINITIONS}" pull &> /dev/null) \
-    git -C "${_ROOT}" pull &> /dev/null
+        && (git -C "${_STATES}" pull &> /dev/null) \
-    git -C "${_DEFINITIONS}" pull &> /dev/null
+        && echo "(done)" \
-    git -C "${_STATES}" pull &> /dev/null
+        && return 0
+
+    echo "FAILED: an error occurred during an update."
+    return 1
 }
 
 # sanitizes all parameters
-update_repositories \
+update_repositories "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
-    "$(echo ${1} | sed -E 's|[^a-zA-Z0-9/:@._-]*||g')" \
+    && exit 0
-    && exit 0 || exit 1
+
+exit 1