Comparing to infinity is buggy on x87

[eduardosm]

When code is compiled using the X87 FPU, comparing to infinity can lead to unexpected behavior.

I tried this code:

#[inline(never)]
fn get_num() -> f64 {
    let num: f64 = 1.0e300;
    // volatile is to avoid optimizations
    unsafe { std::ptr::read_volatile(&num) }
}

fn main() {
    let x = get_num();
    let y = get_num();
    let z = x * y;
    if z != f64::INFINITY && z != f64::NEG_INFINITY && !z.is_nan() {
        let exp = (z.to_bits() >> 52) & 0x7FF;
        assert!(exp != 0x7FF);
        println!("is finite, exp = {}", exp);
    } else {
        println!("is not finite");
    }
}

I expected to see this happen:

This should either print "infinite or nan" or "is finite, exp = ...". The assert! should never fail because the exponent bits are 0x7FF if and only if the number is infinity or nan, so the if condition should have been false.

Instead, this happened:

When compiled for X86 without SSE, the assert will fail. This could also cause safety issues if unsafe code relies on it.

The following command can be used to compile without SSE:

RUSTFLAGS="-C target-cpu=pentium" cargo run --target i686-unknown-linux-gnu

Meta

rustc --version --verbose:

rustc 1.43.1 (8d69840ab 2020-05-04)
binary: rustc
commit-hash: 8d69840ab92ea7f4d323420088dd8c9775f180cd
commit-date: 2020-05-04
host: x86_64-unknown-linux-gnu
release: 1.43.1
LLVM version: 9.0

and

rustc 1.45.0-nightly (a74d1862d 2020-05-14)
binary: rustc
commit-hash: a74d1862d4d87a56244958416fd05976c58ca1a8
commit-date: 2020-05-14
host: x86_64-unknown-linux-gnu
release: 1.45.0-nightly
LLVM version: 9.0

[hanna-kruppe]

For reference, can you post the machine code this results in? I'm particularly curious if there's any calls to functions already built in the distributed libstd, since those would potentially suffer from ABI mismatches (see #63597) that could explain the weird behavior.

[eduardosm]

For reference, can you post the machine code this results in? I'm particularly curious if there's any calls to functions already built in the distributed libstd, since those would potentially suffer from ABI mismatches (see #63597) that could explain the weird behavior.

Sure, the assembly of the main function is (after removing the println!s to simplify it):

_ZN10float_test4main17h3d6b0b215bf83903E:
	.cfi_startproc
	pushl	%ebx
	.cfi_def_cfa_offset 8
	subl	$24, %esp
	.cfi_def_cfa_offset 32
	.cfi_offset %ebx, -8
	calll	.L9$pb
	.cfi_adjust_cfa_offset 4
.L9$pb:
	popl	%ebx
	.cfi_adjust_cfa_offset -4
.Ltmp6:
	addl	$_GLOBAL_OFFSET_TABLE_+(.Ltmp6-.L9$pb), %ebx
	calll	_ZN10float_test7get_num17hedf1e4650ddf3052E
	fstpl	16(%esp)
	calll	_ZN10float_test7get_num17hedf1e4650ddf3052E
	fldl	16(%esp)
	fmulp	%st, %st(1)
	fucom	%st(0)
	fnstsw	%ax
	sahf
	jp	.LBB9_5
	flds	.LCPI9_0@GOTOFF(%ebx)
	fucomp	%st(1)
	fnstsw	%ax
	sahf
	jae	.LBB9_5
	flds	.LCPI9_1@GOTOFF(%ebx)
	fxch	%st(1)
	fucom	%st(1)
	fstp	%st(1)
	fnstsw	%ax
	sahf
	jae	.LBB9_5
	fstpl	8(%esp)
	movl	12(%esp), %eax
	notl	%eax
	testl	$2146435072, %eax
	fldz
	je	.LBB9_4
.LBB9_5:
	fstp	%st(0)
	addl	$24, %esp
	.cfi_def_cfa_offset 8
	popl	%ebx
	.cfi_def_cfa_offset 4
	retl
.LBB9_4:
	.cfi_def_cfa_offset 32
	fstp	%st(0)
	calll	_ZN3std9panicking11begin_panic17h261cc8b487132e56E
	ud2

...

.LCPI9_0:
	.long	4286578688
.LCPI9_1:
	.long	2139095040

Generated with:

RUSTFLAGS="-C target-cpu=pentium" cargo rustc --release --target i686-unknown-linux-gnu -- --emit asm

I think I forgot to mention in the OP that this happens in both debug and release mode.

[RalfJung]

Uh-oh... should this be labelled "unsound"? Any time codegen'd behavior does not match the spec, that can be considered a soundness issue as well.

[eduardosm]

I just found out that this can be reproduced without RUSTFLAGS="-C target-cpu=pentium" using --target i586-unknown-linux-gnu .

[ecstatic-morse]

I was searching for open floating point issues related to #73328 and stumbled upon this one, which seems to have slipped through the cracks. Marking as "unsound" and "needs prioritization". I'm still not sure exactly what causes this.

cc @rust-lang/wg-prioritization

[ranma42]

I believe this happens because x87 internally works on 80-bits numbers.
This format can handle numbers up to about 10^4932, which means that the result of the multiplication is not infinity (when it is on the FP stack), hence the comparisons do not succeed.
On the other hand, the code that extracts the exponent casts it to 64-bits, so the number is rounded (truncated?) to the positive infinity.

[ranma42]

In a way, this is what is happening behind the scenes:

#[inline(never)]
fn get_num() -> f64 {
    let num: f64 = 1.0e30;
    // volatile is to avoid optimizations
    unsafe { std::ptr::read_volatile(&num) }
}

fn main() {
    let x = get_num();
    let y = get_num();
    let z = x * y;
    if z != f64::INFINITY && z != f64::NEG_INFINITY && !z.is_nan() {
        let exp = ((z as f32).to_bits() >> 23) & 0x7F;
        assert!(exp != 0x7F);
        println!("is finite, exp = {}", exp);
    } else {
        println!("is not finite");
    }
}

Note that in this case the downcast as f32 is explicit.

[ecstatic-morse]

Indeed. The underlying cause is clear. I wonder what we should do here, though? Does Rust currently guarantee that extended precision is not used for operations on f64? If so, this is technically a miscompilation. However, I don't know whether it's worth fixing. Maybe we should just document the status quo and move on?