FAP-log-viewer-devops

Setup

Local k3d cluster

k3d cluster create --config k3d-local.yaml

Add to hosts file (e.g. C:\Windows\System32\drivers\etc\hosts in Windows): 127.0.0.1 fap.local

Build local images:

cd ../FAP-log-viewer/frontend; make docker-k3d tag=0.0.1; cd -;
cd ../FAP-log-viewer/backend/data-analyser; make docker-k3d tag=0.0.1; cd -;
cd ../FAP-log-viewer/backend/email-receiver; make docker-k3d tag=0.0.1; cd -;
cd ../FAP-log-viewer/backend/http-backend; make docker-k3d tag=0.0.1; cd -;

Cloud k3s cluster

On master node:

vim /etc/rancher/k3s/config.yaml # Put content from k3s-prod-config.yaml
curl -sfL https://get.k3s.io | sh -

On worker nodes:

server_0_ip=10.0.0.23 # IP of the master node
token= # Get from master node from /var/lib/rancher/k3s/server/token
curl -sfL https://get.k3s.io | K3S_URL=https://${server_0_ip}:6443 K3S_TOKEN=${token} sh -

cd ../FAP-log-viewer/frontend; make docker-prod tag=1.0.0; cd -;
cd ../FAP-log-viewer/backend/data-analyser; make docker-prod tag=1.0.0; cd -;
cd ../FAP-log-viewer/backend/email-receiver; make docker-prod tag=1.0.1; cd -;
cd ../FAP-log-viewer/backend/http-backend; make docker-prod tag=1.0.0; cd -;

Generate kubectl token to access the k3s remotly: https://docs.k3s.io/cluster-access

Deployment using ArgoCD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
kubectl -n argocd port-forward svc/argocd-server 8081:443
kubectl apply -f FAP-log-viewer-devops/argocd/sealed-secrets.yaml
kubectl apply -f FAP-log-viewer-devops/argocd/cert-manager.yaml

For local development:

kubectl apply -f FAP-log-viewer-devops/argocd/prometheus-stack-test.yaml
kubectl apply -f FAP-log-viewer-devops/argocd/fap-log-viewer-test.yaml

For prod:

kubectl apply -f FAP-log-viewer-devops/argocd/prometheus-stack-prod.yaml
kubectl apply -f FAP-log-viewer-devops/argocd/fap-log-viewer-prod.yaml

Useful procedures

Generate sealed secret

Prerequisite: Install kubeseal on local Linux host

• Obtain kube-seal public key

kubeseal --fetch-cert --controller-name=sealed-secrets > public-key-cert.pem

• Create sealed secret

kubectl create secret generic fap-secrets \
  --from-literal=DB_PASSWORD=$(tr -dc 'A-Za-z0-9!?%=' < /dev/urandom | head -c 20) \
  -n fap-log-viewer \
  --dry-run=client \
  -o yaml | \
kubeseal --format yaml --cert=public-key-cert.pem > \
  FAP-log-viewer-devops/k8s/fap-log-viewer/base/common/sealed_secret.yaml

kubectl create secret generic grafana-cloud-secret \
  --from-literal=username="***" \
  --from-literal=password="***" \
  -n monitoring \
  --dry-run=client \
  -o yaml | \
kubeseal --format yaml --cert=public-key-cert.pem > \
  grafana-cloud-sealed-secret.yaml

kubectl create secret generic email-credentials \
  --from-file=FAP-log-viewer/backend/email-receiver/token.json \
  -n fap-log-viewer \
  --dry-run=client \
  -o yaml | \
kubeseal --format yaml --cert=public-key-cert.pem > \
  FAP-log-viewer-devops/k8s/fap-log-viewer/base/email-receiver/sealed_email_credentials.yaml

Craete debug curl pod

kubectl -n fap-log-viewer run curl-test --image=alpine/curl -- sleep infinity

Delete docker image on k3d cluster

docker exec k3d-fap-server-0 sh -c "ctr image rm \$(ctr image list -q | grep fap-log-viewer-backend | head -1)"
docker exec k3d-fap-server-0 sh -c "ctr image list -q | grep fap-log-viewer-backend"

Backup & restore Postgres content

kubectl -n fap-log-viewer exec -t postgres-0 -- pg_dump -U postgres -d fap_analysis --format=custom --file=/tmp/fap_backup.dump
kubectl -n fap-log-viewer cp postgres-0:/tmp/fap_backup.dump ./fap_backup.dump

docker exec -t <src_db_container> pg_dump -U <db_user> -d <db_name> --format=custom --file=/tmp/fap_backup.dump # dump prod
docker cp <src_db_container>:/tmp/fap_backup.dump /abs/path/fap_backup.dump # copy to host
docker exec <src_db_container> rm /tmp/fap_backup.dump # clean prod temp
docker exec -it <dst_db_container> psql -U <db_user> -c "DROP DATABASE IF EXISTS <db_name>; CREATE DATABASE <db_name>;" # reset local DB
docker cp /abs/path/fap_backup.dump <dst_db_container>:/tmp/fap_backup.dump # copy dump in
docker exec -it <dst_db_container> pg_restore --clean --if-exists --no-owner --no-privileges -U <db_user> -d <db_name> /tmp/fap_backup.dump # restore local
rm /abs/path/fap_backup.dump # remove local dump