The setup is created for the DNB, so it will likely not run in your environment, but maybe you can reuse parts of it, and you may be able to play with it in a minikube.
These are Helm charts.
Requirements:
- A Kubernetes cluster, e.g. minikube.
- kubectl configured to access your cluster.
- Helm to install the charts.
- Task to make some things easier. (optional)
The structure of the repository should in a way represent the structure of the setup. For now, it would start as a mono repository with the overall setup with services and might be split up in smaller repositories resp. charts using chart dependencies.
$ task setup:minikube
# Installed the mock data (task install:mock:example-data)
# Enables the ingress addon (minikube addons enable ingress)
$ task install
# Install the mocked ils and the playback to the minikube
# Once it is running add the Airflow DAGs
$ task install:copy-example-dags
Note: If you need to interoperate with the docker-host inside the minikube, run eval $(minikube docker-env).
This assumes you have an aras interface (which exists probably only at the DNB).
$ ENV=dnb task install
Pywb is served from the pywb-service respectively through the pywb-ingress.
The downloading, recompression, and indexing are done as Jobs, that are orchestrated in an RDF graph and executed by Apache Airflow. Files are stored in an S3. Currently, the WARC files are all recompressed which is required for our data as it is not compressed record based.
- Software
- Job overview
- Workflow Steps
- Validation
- Metadata Extraction
- Best Practices:
- liveness check
- readiness check
- Ingress for airflow, fuseki, versitygw
- Error handling:
- For failing jobs report job as failed with the following task in airflow
- Store errors in graph
- Validation, see above
- Graph handling:
- combine fuseki and metadata graph
Containers with * are init containers
(block the init process but are cleaned after they have exited).
┌──────────────────┐ ┌──────────────────┐
│📦 PWID resolver │.>│📦 Playback │ pywb-app
└──────────────────┘ └──────────────────┘
∧
│
┌──────────────────┐
│📁Playback Cache │ .Values.webarchiveDirectory
└──────────────────┘
∧
│
┌──────────────────┐
│📦 Index* │ wb-manager-add
└──────────────────┘
∧
│
┌──────────────────┐
│📁 WARC Cache │ .Values.warcDirectory
└──────────────────┘
∧
│
┌──────────────────┐ wacli-recompress-warcs
│📦 Recompress* │ $ wacli recompress-warcs
└──────────────────┘
∧
│
┌──────────────────┐
│📁 RAW Cache │ .Values.rawWarcDirectory
└──────────────────┘
∧
│
┌──────────────────┐ ┌──────────────────┐ wacli-load-warcs
│ Repository (aras)│─>│📦 Fetch* │ $ wacli load-warcs
└──────────────────┘ └──────────────────┘
∧
│
┌──────────────────┐
│📄️ Graph Cache │ .Values.websiteGraphFile
└──────────────────┘
∧
│
┌──────────────────┐ ┌──────────────────┐ wacli-load-graph
│ Catalog/SPARQL │─>│📦 Query* │ $ wacli load-graph
└──────────────────┘ └──────────────────┘
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Crawling │ │ Playback │<..│ PWID resolver │
└──────────────────┘ └──────────────────┘ └──────────────────┘
│ ∧
∨ │
┌──────────────────┐ ┌──────────────────┐
│ Validation │ │ Index │
└──────────────────┘ └──────────────────┘
│ ∧
∨ │
┌──────────────────┐ ┌──────────────────┐
│ Cache │─>│ Cache │
└──────────────────┘ └──────────────────┘
│ ∧
∨ │
┌──────────────────┐ ┌──────────────────┐
│ Ingest │ │ (Validation) │
└──────────────────┘ └──────────────────┘
│ ∧
∨ │
┌────────────────────────────────────────┐─>┌──────────────────┐
│ Repository │ │ Validation │
└────────────────────────────────────────┘<─└──────────────────┘
The repository is a helm chart with dependencies and subcharts.
The setup uses the helm chart provided by the apache airflow project and extends it according to the documentation.
To nicely integrate secrets and connections we need to follow the documentation.
The setup uses the helm chart provided by versity.
The setup includes an Apache Jena Fuseki RDF Graph Store using the helm chart provided by zazuko.
These need to be implemented:
- https://www.redhat.com/en/blog/9-best-practices-for-deploying-highly-available-applications-to-openshift
- https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
To access some internal port, you can use port forwarding. In the Taskfile some respective tasks are specified.