Miscompile with `-Copt-level=1` after update to LLVM 17

[djkoloski]

Minimized from code in the Fuchsia tree:

fn main() {
    enum Bits {
        None = 0x00,
        Low = 0x40,
        High = 0x80,
        Both = 0xC0,
    }

    let value = Box::new(0x40u8);
    let mut out = Box::new(0u8);

    let bits = match *value {
        0x00 => Bits::None,
        0x40 => Bits::Low,
        0x80 => Bits::High,
        0xC0 => Bits::Both,
        _ => return,
    };

    match bits {
        Bits::None | Bits::Low => {
            *out = 1;
        }
        _ => (),
    }

    assert_eq!(*out, 1);
}

This issue was introduced in the nightly-2023-08-09 build, and so it is likely to be a result of the upgrade to LLVM 17 (#114048).

$ RUSTFLAGS="-Copt-level=1" cargo +nightly-2032-08-09 run
thread 'main' panicked at src/main.rs:27:5:
assertion `left == right` failed
  left: 0
 right: 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
$ RUSTFLAGS="-Copt-level=1" cargo +nightly-2032-08-08 run
<passes>

[djkoloski]

I have bisected the LLVM pass to the instcombine pass on main. I'm trying to get a diff of before/after so I can start tracking down the exact location of the incorrect optimization.

[djkoloski]

The optimization transforms this switch lookup which appears to correspond to the match:

switch.lookup:                                    ; preds = %"_ZN5alloc5boxed12Box$LT$T$GT$3new17h3cb0a1b6f587dddeE.exit4"
  %switch.idx.mult = mul nsw i8 %8, 64, !dbg !682
  %switch.offset = add nsw i8 %switch.idx.mult, -128, !dbg !682
  call void @llvm.dbg.value(metadata i64 poison, metadata !531, metadata !DIExpression()), !dbg !719
  switch i8 %switch.offset, label %bb10 [
    i8 0, label %bb9
    i8 64, label %bb9
  ], !dbg !720

with only the calculation of %switch.idx.mult and %switch.offset changing from:

%switch.idx.mult = mul nsw i8 %8, 64, !dbg !682
%switch.offset = add nsw i8 %switch.idx.mult, -128, !dbg !682

to:

%switch.idx.mult = shl nsw i8 %8, 6, !dbg !682
%switch.offset = or i8 %switch.idx.mult, -128, !dbg !682

with the rest of the IR remaining unchanged.

The transform of %switch.idx.mult appears to be okay (mul by 64 -> shl by 6). The transform of %switch.offset introduces the issue because although the add instruction is marked as nsw, %switch.idx.mult is equal to 192 which is negative in the signed representation and so subtracting 128 does cause signed underflow.