ENH Increase performance of enum obsolete values for large tables

[beerbohmdo]

Description

I have a database with relative large tables. When I run db:build its hangs multiple minutes for checking if the database still contains old enum values.

I found the origin and patched it for me in the past. As we use postgres I think it be a niche issue.

I am not completly sure why, but for large tables this can be realy slow:

QUERY PLAN
Unique  (cost=1000.46..588711.55 rows=1 width=10) (actual time=2764.849..2786.626 rows=1 loops=1)
  ->  Gather Merge  (cost=1000.46..588711.55 rows=2 width=10) (actual time=2764.848..2786.615 rows=3 loops=1)
        Workers Planned: 2
        Workers Launched: 2
        ->  Unique  (cost=0.44..587711.29 rows=1 width=10) (actual time=0.062..1831.997 rows=1 loops=3)
"              ->  Parallel Index Only Scan using ix_e3dc4f5497f7676548ca90e713c5e001 on ""Container_Versions""  (cost=0.44..559543.66 rows=11267055 width=10) (actual time=0.060..1187.245 rows=9052704 loops=3)"
                    Heap Fetches: 6232848
Planning Time: 0.160 ms
JIT:
  Functions: 4
"  Options: Inlining true, Optimization true, Expressions true, Deforming true"
"  Timing: Generation 0.591 ms, Inlining 150.968 ms, Optimization 23.279 ms, Emission 23.194 ms, Total 198.031 ms"
Execution Time: 2786.999 ms

A group by does't help either.

My solution:

I fetch the current schema of the enum column, compare it to the new schema.

Than I check for the obsolete classes if any of them exists in the table. I don't know why but a EXISTS-Query is fast 🤷

Issues

• Enum obsolete value handling maybe slow with many records #11986

Pull request checklist

• The target branch is correct
  • See picking the right version
• All commits are relevant to the purpose of the PR (e.g. no debug statements, unrelated refactoring, or arbitrary linting)
  • Small amounts of additional linting are usually okay, but if it makes it hard to concentrate on the relevant changes, ask for the unrelated changes to be reverted, and submitted as a separate PR.
• The commit messages follow our commit message guidelines
• The PR follows our contribution guidelines
• Code changes follow our coding conventions
• This change is covered with tests (or tests aren't necessary for this change)
• Any relevant User Help/Developer documentation is updated; for impactful changes, information is added to the changelog for the intended release
• CI is green

[michalkleiner]

Thank you for opening the PR @beerbohmdo, appreciate it!

The test failures seem to be related to your change, could you have a look what might be going on?

Improving the performance on enums on large tables is a good area of focus 👍

[beerbohmdo]

Yes it does.

The problem is how the enum currently works and that it may contains empty values (There is a issue for that).

My code is optimized for that it explicitly add the "" value. Without the db:build may shows that the enum is changed (even if it is not).

But the test don't expect that value. I am currently not sure what todo.

[beerbohmdo]

I just removed the addition of "".