Skip to content

Revert Fastify 5 upgrade to fix WebSocket crash and 415 PATCH errors - #7384

Closed
lucferbux wants to merge 1 commit into
opendatahub-io:stable-2.xfrom
lucferbux:fix/revert-fastify5-stable-2.x
Closed

Revert Fastify 5 upgrade to fix WebSocket crash and 415 PATCH errors#7384
lucferbux wants to merge 1 commit into
opendatahub-io:stable-2.xfrom
lucferbux:fix/revert-fastify5-stable-2.x

Conversation

@lucferbux

@lucferbux lucferbux commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Reverts the Fastify 4→5 upgrade introduced in commit 452e6d230 ("Fix security vulnerabilities for rhoai-2.25.4") which shipped in v2.25.5. That upgrade caused two regressions:

1. WebSocket crash (RHOAISUP-24, RHOAIENG-59422)

@fastify/websocket v8→v11 removed the SocketStream wrapper. The WebSocket proxy handler does connection.socket which returns undefined in v11 — every WebSocket connection is broken. For non-admin users, the orphaned target WS connection gets a 403 from K8s API, emitting an unhandled error event that crashes the Node.js process (exit code 1) or causes OOMKill from accumulated leaked connections (exit code 137).

Why e2e with cluster-admin didn't catch it: the TypeError is inside a .then() (unhandled rejection, doesn't crash), and admin users' WS upgrades succeed (no error event), so the proxy is silently broken but the UI loads fine via REST.

2. 415 on PATCH requests

Fastify 5 content-type parser behavior change causes FST_ERR_CTP_INVALID_MEDIA_TYPE for application/merge-patch+json, preventing workbench start/stop operations.

Fix

Revert to fastify@4.29.1 (latest 4.x, fixes CVE-2025-32442) and matching @fastify/* plugin versions:

Package v2.25.5 (broken) This PR
fastify ^5.8.2 ^4.29.1
@fastify/websocket ^11.2.0 ^8.3.1
@fastify/http-proxy ^11.4.1 ^9.5.0
@fastify/static ^9.0.0 ^6.10.2
Other @fastify/* v5/v6 v4/v5

Also reverts loggerInstancelogger (Fastify 4 API) and FastifyHttpProxyOptionsFastifyReplyFromHooks type imports.

CVE note

Fastify 4.29.1 fixes CVE-2025-32442 (High). Three Fastify-5-only CVEs remain unpatched (CVE-2026-3635, CVE-2026-25224, CVE-2026-33806) — all are Content-Type validation bypass. The dashboard backend doesn't use Fastify's built-in content-type schema validation and runs behind oauth-proxy, so practical risk is very low.

Test plan

  • Build succeeds (npm run build)
  • Type-check passes (npm run type-check)
  • All 17 unit tests pass (npm run test:jest)
  • Lint passes (npm run test:lint)
  • Deploy to cluster and verify:
    • Non-admin user can open dashboard without pod crashes
    • WebSocket watches work (browser Network tab shows wss:// connections)
    • Workbenches can be started/stopped (PATCH with merge-patch+json)
    • No memory leak in pod metrics

@coderabbitai

coderabbitai Bot commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b2c21ef9-4288-41c2-a5e0-68c6b85963e6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci

openshift-ci Bot commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mturley for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@lucferbux
lucferbux force-pushed the fix/revert-fastify5-stable-2.x branch from beda146 to aa6d3c1 Compare April 24, 2026 16:35
Commit 452e6d2 upgraded Fastify 4→5 and all @fastify/* plugins for
security fixes. This introduced two regressions in v2.25.5:

1. @fastify/websocket v8→v11 removed the SocketStream wrapper.
   The WebSocket proxy code does `connection.socket` which returns
   undefined in v11, breaking all WebSocket connections and crashing
   the pod for non-admin users (RHOAISUP-24).

2. Fastify 5 content-type parser behavior change causes
   FST_ERR_CTP_INVALID_MEDIA_TYPE for application/merge-patch+json,
   preventing workbench start/stop operations.

This reverts to Fastify 4.29.1 (latest 4.x, fixes CVE-2025-32442)
and matching @fastify/* plugin versions compatible with Fastify 4.

RHOAIENG-59422
@lucferbux
lucferbux force-pushed the fix/revert-fastify5-stable-2.x branch from aa6d3c1 to 94de8f9 Compare April 24, 2026 16:59
@codecov

codecov Bot commented Apr 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.92%. Comparing base (98cf02e) to head (94de8f9).

Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff               @@
##           stable-2.x    #7384      +/-   ##
==============================================
- Coverage       67.93%   67.92%   -0.01%     
==============================================
  Files            2238     2238              
  Lines           50847    50847              
  Branches        14252    14252              
==============================================
- Hits            34541    34540       -1     
- Misses          16306    16307       +1     
Files with missing lines Coverage Δ
backend/src/server.ts 0.00% <ø> (ø)
backend/src/utils/proxy.ts 15.59% <100.00%> (ø)

... and 9 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 98cf02e...94de8f9. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lucferbux

Copy link
Copy Markdown
Contributor Author

Closing this in favor of #7387

@lucferbux lucferbux closed this Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant