Float32 -> BFloat16 conversion flushes subnormal on avx512bf16 CPUs (1.12+)

[AntonOresten]

On x86-64 CPUs with avx512bf16 (Zen 4/5, Sapphire Rapids, Cooper Lake), Julia 1.12+ (LLVM 18+) lowers fptrunc float -> bfloat to vcvtneps2bf16, which per the Intel SDM always flushes denormals: "Output denormals are always flushed to zero and input denormals are always treated as zero. MXCSR is not consulted nor updated."

julia> narrow(x::Float32) = reinterpret(UInt16, BFloat16(x));

julia> narrow(2f0^-127)      # exactly representable as bf16 subnormal 0x0040
0x0000

julia> widen(u::UInt16) = Float32(reinterpret(BFloat16, u));

julia> widen(0x0001)         # widening (fpext) is exact — narrowing only
9.1835f-41

The test suite catches it as the two next/prevfloat failures (BFloat16(0.0) < BFloat16(0.0)) on 1.12 and 1.13-rc1; 1.10 (software path) and 1.11 (LLVM 16 didn't select vcvtneps2bf16 for scalar fptrunc) pass. Reproduced on AMD znver4 and Intel Sapphire Rapids. Unlike #107's crash on these CPUs (gone as of LLVM 19), this is silent and persists — verified through LLVM 22.

The root cause is the LLVM lowering: it contradicts LangRef's default FP environment (subnormals preserved; denormal-fp-math defaults to ieee), and GCC refuses this instruction for scalar conversions in default mode for exactly this reason (commit de867e8da30b). Upstream is aware but unresolved: llvm/llvm-project#138424 (closed on other grounds; denormals left at "no consensus"), llvm/llvm-project#90425 (stalled). No dedicated upstream issue exists yet; everything needed to file one is below.

Julia-free reproducers (LLVM IR, C, and an MXCSR sweep of the raw instruction)

1. IR — the lowering itself (reproduces on llc 18 through 22, on any host — llc cross-compiles, only -mattr matters):

define bfloat @trunc_f32_to_bf16(float %x) {
  %r = fptrunc float %x to bfloat
  ret bfloat %r
}

$ llc -mtriple=x86_64 -mattr=+avx2 trunc.ll -o -
        callq   __truncsfbf2@PLT          # correct: preserves subnormals
$ llc -mtriple=x86_64 -mattr=+avx512bf16,+avx512vl trunc.ll -o -
        vcvtneps2bf16 %xmm0, %xmm0        # flushes subnormals unconditionally

2. C — default FP environment, no fast-math (compiles anywhere; running it needs bf16 hardware — elsewhere, inspect clang -S -march=znver4 output for vcvtneps2bf16 instead). 2^-127 (f32 bits 0x00400000) is exactly representable as bf16 0x0040:

__attribute__((noinline)) unsigned short trunc_bits(float x) {
    __bf16 b = (__bf16)x;
    unsigned short u; __builtin_memcpy(&u, &b, sizeof u); return u;
}

$ clang -O2 -march=x86-64 t.c && ./a.out   ->  0x0040   OK
$ clang -O2 -march=znver4 t.c && ./a.out   ->  0x0000   WRONG

The emitted IR is a bare fptrunc — no fast-math flags, no denormal-fp-math attribute anywhere in the module (so the default ieee mode applies).

3. MXCSR sweep — the flush is architectural, not a mode (bf16 hardware only: it executes the raw-encoded instruction). Builds with plain cc -O2:

#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <immintrin.h>
static uint16_t cvt(uint32_t f) {
    float x; memcpy(&x, &f, 4);
    register float v __asm__("xmm0") = x;
    __asm__ volatile(".byte 0x62,0xf2,0x7e,0x08,0x72,0xc0" : "+x"(v));  // vcvtneps2bf16 xmm0,xmm0
    float r = v; uint32_t o; memcpy(&o, &r, 4); return (uint16_t)o;
}
int main(void) {  // run only on CPUs with AVX512-BF16+VL
    uint32_t def = _mm_getcsr();
    uint32_t modes[] = {def, def & ~((1u<<15)|(1u<<6)), def | (1u<<15)|(1u<<6)};
    for (int i = 0; i < 3; i++) {
        _mm_setcsr(modes[i]);
        printf("MXCSR=0x%04x  0x00400000 -> 0x%04x (want 0x0040)\n", _mm_getcsr(), cvt(0x00400000u));
    }
}

MXCSR=0x1f80  0x00400000 -> 0x0000 (want 0x0040)     # default (FZ=0, DAZ=0)
MXCSR=0x1f80  0x00400000 -> 0x0000 (want 0x0040)     # FZ/DAZ explicitly cleared
MXCSR=0x9fc0  0x00400000 -> 0x0000 (want 0x0040)     # FZ/DAZ set

No MXCSR state produces the correct result, matching the SDM. The unguarded lowering is in X86TargetLowering::LowerFP_TO_BF16 (X86ISelLowering.cpp): X86ISD::CVTNEPS2BF16 is emitted whenever hasBF16()+hasVLX() (or hasAVXNECONVERT()), with no denormal-mode or fast-math check.

At the package level

The conversion itself is fixable here: on avx512bf16 hosts, route the into-bf16 constructors through the software round-to-nearest-even bit ops (the else branch that already exists) instead of Base.fptrunc. Every conversion the package controls — explicit and broadcast — then produces correct subnormals, and it still vectorizes (integer SIMD). Dropping llvm_storage entirely is not proposed: that would replace Core.BFloat16 with the software type and cost GPU users native bf16 codegen (fma.rn.bf16 vs. integer emulation); with the constructor-only change, device arithmetic keeps fma.rn.bf16 and only device conversion kernels drop to integer ops (recoverable via a CUDA.jl @device_override).

Two things remain broken regardless, because they convert outside package methods: arithmetic results in the subnormal range (add_float narrows internally), and values transported through top-level/boxed paths — the latter is what the two next/prevfloat failures actually exercise, so those become @test_broken referencing this issue. The full fix has to land in the LLVM x86 backend (or Julia's lowering).

Workaround meanwhile: julia -C "native,-avx512bf16".

Diagnosed and drafted together with Fable.