Skip to content

BREAKING CHANGE: drop all sqlite support #1986

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
grokspawn wants to merge 1 commit into
base: master
Choose a base branch
from
Open

BREAKING CHANGE: drop all sqlite support #1986

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 24 additions & 175 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,15 @@ The Operator Registry is a Kubernetes/OpenShift component that provides operator
## Key Components

### Binaries
- **`opm`**: Main CLI tool for generating and updating registry databases and index images
- **`initializer`**: **(Deprecated)** Converts operator manifests to SQLite database format
- **`registry-server`**: **(Deprecated)** Exposes gRPC interface to SQLite databases
- **`configmap-server`**: Parses ConfigMaps into SQLite databases and exposes gRPC interface
- **`opm`**: Main CLI tool for rendering and serving file-based catalogs

### Libraries
- **`pkg/client`**: High-level client interface for gRPC API
- **`pkg/api`**: Low-level client libraries for gRPC interface
Comment thread
grokspawn marked this conversation as resolved.
- **`pkg/registry`**: Core registry types (Packages, Channels, Bundles)
Comment thread
grokspawn marked this conversation as resolved.
- **`pkg/sqlite`**: **(Deprecated)** SQLite database interfaces for manifests
- **`pkg/lib`**: External interfaces and standards for operator bundles
- **`pkg/containertools`**: Container tooling integration
- **`pkg/cache`**: Caching backends for file-based catalogs

### Alpha Features
- **`alpha/declcfg`**: Declarative configuration format
Expand All @@ -30,10 +27,10 @@ The Operator Registry is a Kubernetes/OpenShift component that provides operator
## Development Guidelines

### Code Structure
- **Go 1.24.4**: Minimum Go version required
- **Go 1.26.3**: Minimum Go version required
- **Cobra CLI**: Command-line interface framework
- **gRPC**: Primary API communication protocol
- **SQLite**: Database backend for registry data
- **Declarative Config (FBC)**: File-based catalog format for registry data
- **OCI Images**: Container image format for bundles

### Testing
Expand Down Expand Up @@ -93,11 +90,11 @@ podman build -t quay.io/my-namespace/my-bundle:latest -f bundle.Dockerfile .
# IMPORTANT: Bundle images must be published to a registry before they can be consumed
podman push quay.io/my-namespace/my-bundle:latest

# Add bundle to index (deprecated - use file-based catalogs instead)
opm index add --bundles quay.io/my-namespace/my-bundle:latest --from-index quay.io/my-namespace/my-index:latest --tag quay.io/my-namespace/my-index:latest
# Render bundle to declarative config
opm render quay.io/my-namespace/my-bundle:latest --output yaml > catalog.yaml

# Generate index image (deprecated - use file-based catalogs instead)
opm index add --bundles quay.io/my-namespace/my-bundle:latest --tag quay.io/my-namespace/my-index:latest
# Generate catalog from multiple bundles
opm render quay.io/my-namespace/my-bundle:v1.0.0 quay.io/my-namespace/my-bundle:v1.1.0 --output yaml
```

**⚠️ Critical Requirements:**
Expand All @@ -108,28 +105,17 @@ opm index add --bundles quay.io/my-namespace/my-bundle:latest --tag quay.io/my-n
### 3. Working with Declarative Config
```go
// Load declarative config
cfg, err := declcfg.LoadFS(os.DirFS("path/to/catalog"))
cfg, err := declcfg.LoadFS(context.TODO(), os.DirFS("path/to/catalog"))

// Convert to model
model, err := declcfg.ConvertToModel(cfg)
model, err := declcfg.ConvertToModel(*cfg)

// Write declarative config
err = declcfg.Write(cfg, "output.yaml")
// Write declarative config as YAML
f, err := os.Create("output.yaml")
err = declcfg.WriteYAML(*cfg, f)
```

### 4. Database Operations **(Deprecated)**
```go
// Create SQLite database
db, err := sqlite.Open("registry.db")

// Add bundle to database
err = db.AddBundle(bundle)

// Query packages
packages, err := db.ListPackages()
```

### 5. Serving Catalog Content with `opm serve`
### 4. Serving Catalog Content with `opm serve`
The `opm serve` command exposes operator catalog data via a gRPC interface for consumption by OLM.

```bash
Expand Down Expand Up @@ -214,121 +200,18 @@ cache, err := cache.New("/cache/dir",
- Cache directory should be persistent for production deployments
- Use `--cache-only` for pre-building cache in CI/CD pipelines

## Deprecated SQLite Commands and Operations

⚠️ **IMPORTANT DEPRECATION NOTICE**: SQLite-based catalogs and their related subcommands are deprecated. Support for them will be removed in a future release. Please migrate your catalog workflows to the new file-based catalog format.

### Deprecated Commands

#### `opm registry` Commands (All Deprecated)
- **`opm registry serve`** - **(Deprecated)** Serve SQLite database via gRPC
- **`opm registry add`** - **(Deprecated)** Add bundles to SQLite database
- **`opm registry rm`** - **(Deprecated)** Remove packages from SQLite database
- **`opm registry prune`** - **(Deprecated)** Prune SQLite database
- **`opm registry prune-stranded`** - **(Deprecated)** Prune stranded bundles from SQLite database
- **`opm registry deprecatetruncate`** - **(Deprecated)** Deprecate bundles in SQLite database
- **`opm registry mirror`** - **(Deprecated)** Mirror SQLite-based catalogs

#### `opm index` Commands (All Deprecated)
- **`opm index add`** - **(Deprecated)** Add bundles to SQLite-based index
- **`opm index rm`** - **(Deprecated)** Delete operators from SQLite-based index
- **`opm index export`** - **(Deprecated)** Export SQLite-based index
- **`opm index prune`** - **(Deprecated)** Prune SQLite-based index
- **`opm index prune-stranded`** - **(Deprecated)** Prune stranded bundles from SQLite-based index
- **`opm index deprecatetruncate`** - **(Deprecated)** Deprecate bundles in SQLite-based index

### Deprecated Libraries and Interfaces

#### `pkg/sqlite` Package (Deprecated)
- **`SQLQuerier`** - **(Deprecated)** Query interface for SQLite databases
- **`SQLLoader`** - **(Deprecated)** Load bundles into SQLite databases
- **`DeprecationAwareLoader`** - **(Deprecated)** Handle bundle deprecations in SQLite
- **`SQLDeprecator`** - **(Deprecated)** Deprecate bundles in SQLite databases

#### `pkg/lib/registry` Package (Deprecated)
- **`RegistryUpdater`** - **(Deprecated)** Update SQLite-based registries
- **`RegistryDeleter`** - **(Deprecated)** Delete from SQLite-based registries
- **`RegistryDeprecator`** - **(Deprecated)** Deprecate bundles in SQLite-based registries

### Migration Path

**From SQLite to File-Based Catalogs:**

1. **Replace `opm registry serve`** → **Use `opm serve`**
```bash
# Old (deprecated)
opm registry serve --database bundles.db

# New (recommended)
opm serve ./catalog-directory
```

2. **Replace `opm index add`** → **Use `opm alpha generate dockerfile` + `opm serve`**
```bash
# Old (deprecated)
opm index add --bundles quay.io/my/bundle:latest --tag quay.io/my/index:latest

# New (recommended)
opm alpha generate dockerfile ./catalog-directory
# Build and serve the generated Dockerfile
```

3. **Replace SQLite database operations** → **Use declarative config files**
```bash
# Old (deprecated) - SQLite database manipulation
opm registry add --database bundles.db --bundles quay.io/my/bundle:latest

# New (recommended) - File-based catalog
# Create catalog files directly in ./catalog-directory/
```

4. **Convert existing SQLite catalogs** → **Use migration tools**
```bash
# Convert SQLite index image to FBC
opm migrate quay.io/my-namespace/my-index:latest ./fbc-output --output yaml

# Convert SQLite database to FBC
opm render ./bundles.db --output yaml > catalog.yaml

# Convert FBC to a basic catalog template for simpler maintenance
opm alpha render-template basic ./catalog.yaml
```

### Deprecation Warnings

When using deprecated commands, you will see warnings like:
```
DEPRECATION NOTICE:
Sqlite-based catalogs and their related subcommands are deprecated. Support for
them will be removed in a future release. Please migrate your catalog workflows
to the new file-based catalog format.
```

**Action Required**: Plan your migration to file-based catalogs to avoid future compatibility issues.

### Migration from SQLite to File-Based Catalogs (FBC)
## Removed SQLite Support

#### Using `opm migrate`
The `opm migrate` command converts SQLite-based index images or database files to file-based catalogs:
⚠️ **NOTICE**: SQLite-based catalogs and their related commands (`opm registry`, `opm index`) have been removed from this project. All catalog workflows now use the file-based catalog (FBC) format exclusively.

```bash
# Migrate from SQLite index image to FBC
opm migrate quay.io/my-namespace/my-index:latest ./fbc-output --output yaml

# Migrate from SQLite database file to FBC
opm migrate ./bundles.db ./fbc-output --output json
**Migration**: For older versions that supported SQLite-to-FBC migration, refer to previous releases or documentation versions.

# Migrate with specific migration level
opm migrate quay.io/my-namespace/my-index:latest ./fbc-output
```
### Creating File-Based Catalogs

#### Using `opm render`
The `opm render` command can convert various sources to FBC format:
The `opm render` command converts various sources to FBC format:

```bash
# Render from SQLite database to FBC
opm render ./bundles.db --output yaml > catalog.yaml

# Render from bundle images to FBC
opm render quay.io/my/bundle:v1.0.0 quay.io/my/bundle:v1.1.0 --output json

Expand All @@ -350,30 +233,6 @@ opm alpha render-template semver ./semver-template.yaml --output json
opm alpha render-template ./template-file.yaml --output yaml
```

#### Combined Migration Workflow
For complex migrations, combine multiple tools:

```bash
# Step 1: Migrate SQLite to FBC
opm migrate quay.io/my-namespace/my-index:latest ./fbc-output --output yaml

# Step 2: Convert to basic template format (if needed)
opm alpha render-template basic ./fbc-output/package.yaml --output yaml > template.yaml

# Step 3: Modify template as needed, then render final FBC
opm alpha render-template basic ./template.yaml --output yaml > final-catalog.yaml

# Step 4: Serve the new FBC catalog
opm serve ./final-catalog.yaml
```

#### Migration Considerations
- **Bundle images must exist** in registries before they can be referenced in FBC
- **Image references are required** - use `--alpha-image-ref-template` if bundle images don't exist yet
- **Migration levels** control which schema transformations are applied
- **Output formats** include JSON (streamable) and YAML (human-readable)
- **Templates provide flexibility** for custom catalog generation workflows

## File Organization

### Key Directories
Expand Down Expand Up @@ -448,23 +307,17 @@ func TestMyFunction(t *testing.T) {
### Common Issues
1. **Bundle validation errors**: Check bundle format and annotations
2. **Image pull failures**: Verify image references and registry access
3. **Database corruption**: Rebuild database from source bundles
4. **Template rendering errors**: Check template syntax and schema
5. **Cache integrity failures**: Rebuild cache or disable integrity checks
6. **gRPC connection issues**: Verify port availability and firewall settings
7. **Memory issues with large catalogs**: Use persistent cache directory
8. **Slow startup times**: Pre-build cache with `--cache-only` flag
9. **SQLite deprecation warnings**: Migrate to file-based catalogs using `opm serve`
10. **Hidden commands**: Some deprecated commands are hidden but still accessible
3. **Template rendering errors**: Check template syntax and schema
4. **Cache integrity failures**: Rebuild cache or disable integrity checks
5. **gRPC connection issues**: Verify port availability and firewall settings
6. **Memory issues with large catalogs**: Use persistent cache directory
7. **Slow startup times**: Pre-build cache with `--cache-only` flag

### Debug Commands
```bash
# Validate bundle
opm alpha bundle validate ./bundle

# Inspect index
opm index export --from-index quay.io/my-namespace/my-index:latest

# Debug template rendering
opm alpha render-template --help

Expand All @@ -477,10 +330,6 @@ opm serve ./catalog --pprof-addr localhost:6060 --pprof-capture-profiles
# Test gRPC connectivity
grpcurl -plaintext localhost:50051 list
grpcurl -plaintext localhost:50051 api.Registry/ListPackages

# Deprecated SQLite commands (avoid using these)
opm registry serve --database bundles.db # Use 'opm serve' instead
opm index add --bundles quay.io/my/bundle:latest --tag quay.io/my/index:latest # Use file-based catalogs
```

## Contributing Guidelines
Expand Down
8 changes: 4 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ $(PROTOC):
null :=
space := $(null) #
comma := ,
# default to json1 for sqlite3 and containers_image_openpgp for containers/image
TAGS := json1,containers_image_openpgp
# default to containers_image_openpgp for containers/image
TAGS := containers_image_openpgp

# Cluster to use for e2e testing
CLUSTER ?= ""
Expand Down Expand Up @@ -62,8 +62,8 @@ build: clean $(CMDS) $(OPM)
cross: opm_version_flags=-ldflags "-X '$(PKG)/cmd/opm/version.gitCommit=$(GIT_COMMIT)' -X '$(PKG)/cmd/opm/version.opmVersion=$(OPM_VERSION)' -X '$(PKG)/cmd/opm/version.buildDate=$(BUILD_DATE)'"
cross:
ifeq ($(shell go env GOARCH),amd64)
GOOS=darwin CC=o64-clang CXX=o64-clang++ CGO_ENABLED=1 CGO_LDFLAGS='-Wl,-undefined,dynamic_lookup' $(GO) build $(opm_version_flags) -tags=$(TAGS) -o "bin/darwin-amd64-opm" --ldflags "-extld=o64-clang -extldflags=-Wl,-undefined,dynamic_lookup" ./cmd/opm
GOOS=windows CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ CGO_ENABLED=1 $(GO) build $(opm_version_flags) -tags=$(TAGS) -o "bin/windows-amd64-opm" --ldflags "-extld=x86_64-w64-mingw32-gcc" -buildmode=exe ./cmd/opm
GOOS=darwin CGO_ENABLED=0 $(GO) build $(opm_version_flags) -tags=$(TAGS) -o "bin/darwin-amd64-opm" ./cmd/opm
GOOS=windows CGO_ENABLED=0 $(GO) build $(opm_version_flags) -tags=$(TAGS) -o "bin/windows-amd64-opm" -buildmode=exe ./cmd/opm
Comment thread
grokspawn marked this conversation as resolved.
endif

.PHONY: static
Expand Down
40 changes: 22 additions & 18 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,13 @@ contribution. See the [DCO](DCO) file for details.

This project provides the following binaries:

* `opm`, which generates and updates registry databases as well as the index images that encapsulate them.
* `initializer`, which takes as an input a directory of operator manifests and outputs a sqlite database containing the same data for querying
* Deprecated - use `opm registry|index add` instead
* `registry-server`, which takes a sqlite database loaded with manifests, and exposes a gRPC interface to it.
* Deprecated - use `opm registry serve` instead
* `configmap-server`, which takes a kubeconfig and a configmap reference, and parses the configmap into the sqlite database before exposing it via the same interface as `registry-server`.
* `opm`, which manages file-based catalog (FBC) content and the catalog images that encapsulate them.

And libraries:

* `pkg/client` - providing a high-level client interface for the gRPC api.
* `pkg/api` - providing low-level client libraries for the gRPC interface exposed by `registry-server`.
* `pkg/api` - providing low-level client libraries for the gRPC interface.
* `pkg/registry` - providing basic registry types like Packages, Channels, and Bundles.
* `pkg/sqlite` - providing interfaces for building sqlite manifest databases from `ConfigMap`s or directories, and for querying an existing sqlite database.
* `pkg/lib` - providing external interfaces for interacting with this project as an api that defines a set of standards for operator bundles and indexes.
* `pkg/containertools` - providing an interface to interact with and shell out to common container tooling binaries (if installed on the environment)

Expand Down Expand Up @@ -73,26 +67,36 @@ Of course, this build step can be done with any other OCI spec container tools l

Note that you do not need to create your bundle manually. [Operator-SDK](https://github.com/operator-framework/operator-sdk) provide features and helpers to build, to update, to validate and to test bundles for projects which follows the SDK layout or not. For more information check its documentations over [Integration with OLM](https://sdk.operatorframework.io/docs/olm-integration)

# Building an index of Operators using `opm`
# Building a catalog of Operators using `opm`

Now that you have published the container image containing your manifests, how do you actually make that bundle available to other users' Kubernetes clusters so that the Operator Lifecycle Manager can install the operator? This is where the meat of the `operator-registry` project comes in. OLM has the concept of [CatalogSources](https://operator-framework.github.io/olm-book/docs/glossary.html#catalogsources) which define a reference to what packages are available to install onto a cluster. To make your bundle available, you can add the bundle to a container image which the CatalogSource points to. This image contains a database of pointers to bundle images that OLM can pull and extract the manifests from in order to install an operator. So, to make your operator available to OLM, you can generate an index image via opm with your bundle reference included:
Now that you have published the container image containing your manifests, how do you actually make that bundle available to other users' Kubernetes clusters so that the Operator Lifecycle Manager can install the operator? This is where the meat of the `operator-registry` project comes in. OLM has the concept of [CatalogSources](https://operator-framework.github.io/olm-book/docs/glossary.html#catalogsources) which define a reference to what packages are available to install onto a cluster.

To make your bundle available, you build a file-based catalog (FBC) that references your bundle images. You can render bundle images into FBC format and then serve the catalog via a container image.

First, render your bundle into a file-based catalog:

```sh
opm index add --bundles quay.io/my-container-registry-namespace/my-manifest-bundle:0.0.1 --tag quay.io/my-container-registry-namespace/my-index:1.0.0
podman push quay.io/my-container-registry-namespace/my-index:1.0.0
mkdir my-catalog
opm render quay.io/my-container-registry-namespace/my-manifest-bundle:0.0.1 -o yaml > my-catalog/catalog.yaml
```

The resulting image is referred to as an "Index". It is an image which contains a database of pointers to operator manifest content that is easily queriable via an included API that is served when the container image is run.
Then add a channel entry and package definition to your catalog (see the [FBC documentation](https://olm.operatorframework.io/docs/reference/file-based-catalogs/) for the full schema). Once your catalog is assembled, validate it:

Now that image is available for clusters to use and reference with CatalogSources on their cluster.
```sh
opm validate my-catalog
```

Index images are additive, so you can add a new version of your operator bundle when you publish a new version:
Generate a Dockerfile and build the catalog image:

```sh
opm index add --bundles quay.io/my-container-registry-namespace/my-manifest-bundle:0.0.2 --from-index quay.io/my-container-registry-namespace/my-index:1.0.0 --tag quay.io/my-container-registry-namespace/my-index:1.0.1
opm generate dockerfile my-catalog
podman build -t quay.io/my-container-registry-namespace/my-catalog:1.0.0 -f my-catalog.Dockerfile .
podman push quay.io/my-container-registry-namespace/my-catalog:1.0.0
```

For more detail on using `opm` to generate index images, take a look at the [documentation](docs/design/opm-tooling.md).
The resulting image is a catalog image containing file-based catalog content that is served via a gRPC API when the container image is run.

For more detail on using `opm`, see the [file-based catalog documentation](https://olm.operatorframework.io/docs/tasks/creating-a-catalog/).

# Using the index with Operator Lifecycle Manager

Expand All @@ -114,7 +118,7 @@ This will download the referenced image and start a pod in the designated namesp
```sh
$ kubectl logs example-manifests-wfh5h -n default

time="2019-03-18T10:20:14Z" level=info msg="serving registry" database=bundles.db port=50051
time="2019-03-18T10:20:14Z" level=info msg="serving registry" port=50051
```

Once the catalog has been loaded, your Operators package definitions are read by the `package-server`, a component of OLM. Watch your Operator packages become available:
Expand Down
Loading
Loading