xorgrdp: XF86VidModeSetGamma() -> BadValue #271

xrdp version

0.9.17

Detailed xrdp version, build options

xrdp 0.9.17
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2020 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --enable-ipv6
      --enable-jpeg
      --enable-fuse
      --enable-rfxcodec
      --enable-opus
      --enable-painter
      --enable-vsock
      --build=x86_64-linux-gnu
      --prefix=/usr
      --includedir=${prefix}/include
      --mandir=${prefix}/share/man
      --infodir=${prefix}/share/info
      --sysconfdir=/etc
      --localstatedir=/var
      --disable-silent-rules
      --libdir=${prefix}/lib/x86_64-linux-gnu
      --libexecdir=${prefix}/lib/x86_64-linux-gnu
      --disable-maintainer-mode
      --disable-dependency-tracking
      --with-socketdir=/run/xrdp/sockdir
      build_alias=x86_64-linux-gnu
      CFLAGS=-g -O2 -ffile-prefix-map=/build/xrdp-Doma5Z/xrdp-0.9.17=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security 
      LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed
      CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2 -Wno-error=deprecated-declarations
      PKG_CONFIG_PATH=/build/xrdp-Doma5Z/xrdp-0.9.17/pkgconfig

  Compiled with OpenSSL 3.0.2 15 Mar 2022

Operating system & version

Ubuntu 22.04

Installation method

dnf / apt / zypper / pkg / etc

Which backend do you use?

xorgxrdp

What desktop environment do you use?

gnome-flashback

Environment xrdp running on

Physical or VM

What's your client?

No response

Area(s) with issue?

Graphic glitches, Other

Steps to reproduce

Compile and execute the following program in an xorgrdp session:

gamma.c.zip

✔️ Expected Behavior

Either the XF86VidMode extension not to be present or for the call to XF86VidModeSetGamma() to not deliver an asynchronous BadValue error.

❌ Actual Behavior

XF86VidModeSetGamma() mode crashes the program with a BadValue error.

Anything else?

Many X screensaver programs use XF86VidModeSetGamma() calls to implement a fade-to-black effect before starting the screensaver, but these appear to crash in an xorgrdp session with an asynchronous BadValue error. I couldn't find anything in terms of a specification on XF86VidModeSetGamma(), but these programs are written to not guard against an asynchronous error from calling XF86VidModeSetGamma() and, since they use Xlib, crash when they receive it. However, before calling XF86VidModeSetGamma(), they do check for the presence of the XF86VidMode extension and that it is at least version 2.0, but both of these conditions appear true in an xorgrdp session, and so they fatally call XF86VidModeSetGamma() anyways.