InstCombine: Fold absorbing fmul of compared 0 into select #172381
+46
−34
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is similar to the select-bin-op identity case, except in this case we are looking for the absorbing value for the binary operator. If the compared value is a floating-point 0, and the fmul is implied to return a +0, put the 0 directly into the select operand. This pattern appears in scale-if-denormal sequences after optimizations assume denormals are treated as 0. Fold: %fabs.x = call float @llvm.fabs.f32(float %x) %mul.fabs.x = fmul float %fabs.x, known_positive %x.is.zero = fcmp oeq float %x, 0.0 %select = select i1 %x.is.zero, float %mul.fabs.x, float %fabs.x To: %fabs.x = call float @llvm.fabs.f32(float %x) %mul.fabs.x = fmul float %fabs.x,known_positive %x.is.zero = fcmp oeq float %x, 0.0 %select = select i1 %x.is.zero, float 0.0, float %fabs.x https://alive2.llvm.org/ce/z/Qcy56Z
Contributor
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
arsenm
added
floating-point
Graphite App
Member
|
@llvm/pr-subscribers-llvm-transforms Author: Matt Arsenault (arsenm) ChangesThis is similar to the select-bin-op identity case, except If the compared value is a floating-point 0, and the fmul is Fold: To: https://alive2.llvm.org/ce/z/Qcy56Z Full diff: https://github.com/llvm/llvm-project/pull/172381.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp b/llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp
index f52bac5e600cb..521cccc6c9ae1 100644
--- a/llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp
+++ b/llvm/lib/Transforms/InstCombine/InstCombineSelect.cpp
@@ -83,40 +83,60 @@ static Instruction *foldSelectBinOpIdentity(SelectInst &Sel,
if (!match(Sel.getOperand(IsEq ? 1 : 2), m_BinOp(BO)))
return nullptr;
- // The compare constant must be the identity constant for that binop.
- // If this a floating-point compare with 0.0, any zero constant will do.
- Type *Ty = BO->getType();
- Constant *IdC = ConstantExpr::getBinOpIdentity(BO->getOpcode(), Ty, true);
- if (IdC != C) {
- if (!IdC || !CmpInst::isFPPredicate(Pred))
- return nullptr;
- if (!match(IdC, m_AnyZeroFP()) || !match(C, m_AnyZeroFP()))
- return nullptr;
- }
+ // For absorbing values, we can fold to the compared value.
+ bool IsAbsorbingValue = false;
// Last, match the compare variable operand with a binop operand.
Value *Y;
if (BO->isCommutative()) {
- if (!match(BO, m_c_BinOp(m_Value(Y), m_Specific(X))))
+ // Recognized 0 as an absorbing value for fmul, but we need to be careful
+ // about the sign. This could be more aggressive, by handling arbitrary sign
+ // bit operations as long as we know the fmul sign matches (and handling
+ // arbitrary opcodes).
+ if (match(BO, m_c_FMul(m_FAbs(m_Specific(X)), m_Value(Y))) &&
+ match(C, m_AnyZeroFP()) &&
+ IC.fmulByZeroIsZero(Y, BO->getFastMathFlags(), &Sel))
+ IsAbsorbingValue = true;
+ else if (!match(BO, m_c_BinOp(m_Value(Y), m_Specific(X))))
return nullptr;
} else {
if (!match(BO, m_BinOp(m_Value(Y), m_Specific(X))))
return nullptr;
}
- // +0.0 compares equal to -0.0, and so it does not behave as required for this
- // transform. Bail out if we can not exclude that possibility.
- if (const auto *FPO = dyn_cast<FPMathOperator>(BO))
- if (!FPO->hasNoSignedZeros() &&
- !cannotBeNegativeZero(Y,
- IC.getSimplifyQuery().getWithInstruction(&Sel)))
- return nullptr;
+ // The compare constant must be the identity constant for that binop.
+ // If this a floating-point compare with 0.0, any zero constant will do.
+ Type *Ty = BO->getType();
+
+ Value *FoldedVal;
+ if (IsAbsorbingValue) {
+ FoldedVal = C;
+ } else {
+ Constant *IdC = ConstantExpr::getBinOpIdentity(BO->getOpcode(), Ty, true);
+ if (IdC != C) {
+ if (!IdC || !CmpInst::isFPPredicate(Pred))
+ return nullptr;
+
+ if (!match(IdC, m_AnyZeroFP()) || !match(C, m_AnyZeroFP()))
+ return nullptr;
+ }
+
+ // +0.0 compares equal to -0.0, and so it does not behave as required for
+ // this transform. Bail out if we can not exclude that possibility.
+ if (const auto *FPO = dyn_cast<FPMathOperator>(BO))
+ if (!FPO->hasNoSignedZeros() &&
+ !cannotBeNegativeZero(Y,
+ IC.getSimplifyQuery().getWithInstruction(&Sel)))
+ return nullptr;
+
+ FoldedVal = Y;
+ }
// BO = binop Y, X
// S = { select (cmp eq X, C), BO, ? } or { select (cmp ne X, C), ?, BO }
// =>
// S = { select (cmp eq X, C), Y, ? } or { select (cmp ne X, C), ?, Y }
- return IC.replaceOperand(Sel, IsEq ? 1 : 2, Y);
+ return IC.replaceOperand(Sel, IsEq ? 1 : 2, FoldedVal);
}
/// This folds:
diff --git a/llvm/test/Transforms/InstCombine/select-fcmp-fmul-zero-absorbing-value.ll b/llvm/test/Transforms/InstCombine/select-fcmp-fmul-zero-absorbing-value.ll
index 660d2a0c0784e..1390cef970abe 100644
--- a/llvm/test/Transforms/InstCombine/select-fcmp-fmul-zero-absorbing-value.ll
+++ b/llvm/test/Transforms/InstCombine/select-fcmp-fmul-zero-absorbing-value.ll
@@ -5,9 +5,8 @@ define float @select_oeq_fmul_fabs_or_fabs_src(float %x) {
; CHECK-LABEL: define float @select_oeq_fmul_fabs_or_fabs_src(
; CHECK-SAME: float [[X:%.*]]) {
; CHECK-NEXT: [[FABS_X:%.*]] = call float @llvm.fabs.f32(float [[X]])
-; CHECK-NEXT: [[MUL_FABS_X:%.*]] = fmul float [[FABS_X]], 0x4170000000000000
; CHECK-NEXT: [[X_IS_ZERO:%.*]] = fcmp oeq float [[X]], 0.000000e+00
-; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_ZERO]], float [[MUL_FABS_X]], float [[FABS_X]]
+; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_ZERO]], float 0.000000e+00, float [[FABS_X]]
; CHECK-NEXT: ret float [[SELECT]]
;
%fabs.x = call float @llvm.fabs.f32(float %x)
@@ -21,9 +20,8 @@ define float @select_oeq_fmul_fabs_or_fabs_src_cmp_neg0(float %x) {
; CHECK-LABEL: define float @select_oeq_fmul_fabs_or_fabs_src_cmp_neg0(
; CHECK-SAME: float [[X:%.*]]) {
; CHECK-NEXT: [[FABS_X:%.*]] = call float @llvm.fabs.f32(float [[X]])
-; CHECK-NEXT: [[MUL_FABS_X:%.*]] = fmul float [[FABS_X]], 0x4170000000000000
; CHECK-NEXT: [[X_IS_ZERO:%.*]] = fcmp oeq float [[X]], 0.000000e+00
-; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_ZERO]], float [[MUL_FABS_X]], float [[FABS_X]]
+; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_ZERO]], float 0.000000e+00, float [[FABS_X]]
; CHECK-NEXT: ret float [[SELECT]]
;
%fabs.x = call float @llvm.fabs.f32(float %x)
@@ -37,9 +35,8 @@ define float @select_oeq_fdiv_fabs_or_fabs_src(float %x) {
; CHECK-LABEL: define float @select_oeq_fdiv_fabs_or_fabs_src(
; CHECK-SAME: float [[X:%.*]]) {
; CHECK-NEXT: [[FABS_X:%.*]] = call float @llvm.fabs.f32(float [[X]])
-; CHECK-NEXT: [[MUL_FABS_X:%.*]] = fmul float [[FABS_X]], 0x3E70000000000000
; CHECK-NEXT: [[X_IS_ZERO:%.*]] = fcmp oeq float [[X]], 0.000000e+00
-; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_ZERO]], float [[MUL_FABS_X]], float [[FABS_X]]
+; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_ZERO]], float 0.000000e+00, float [[FABS_X]]
; CHECK-NEXT: ret float [[SELECT]]
;
%fabs.x = call float @llvm.fabs.f32(float %x)
@@ -169,10 +166,8 @@ define float @select_olt_fmul_fabs_or_fabs_src(float %x) {
define float @select_fmul_fabs_or_src(float %x) {
; CHECK-LABEL: define float @select_fmul_fabs_or_src(
; CHECK-SAME: float [[X:%.*]]) {
-; CHECK-NEXT: [[FABS_X:%.*]] = call float @llvm.fabs.f32(float [[X]])
-; CHECK-NEXT: [[MUL_FABS_X:%.*]] = fmul float [[FABS_X]], 0x4170000000000000
; CHECK-NEXT: [[X_IS_ZERO:%.*]] = fcmp oeq float [[X]], 0.000000e+00
-; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_ZERO]], float [[MUL_FABS_X]], float [[X]]
+; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_ZERO]], float 0.000000e+00, float [[X]]
; CHECK-NEXT: ret float [[SELECT]]
;
%fabs.x = call float @llvm.fabs.f32(float %x)
@@ -285,9 +280,8 @@ define float @select_une_fmul_fabs_or_fabs_src(float %x) {
; CHECK-LABEL: define float @select_une_fmul_fabs_or_fabs_src(
; CHECK-SAME: float [[X:%.*]]) {
; CHECK-NEXT: [[FABS_X:%.*]] = call float @llvm.fabs.f32(float [[X]])
-; CHECK-NEXT: [[MUL_FABS_X:%.*]] = fmul float [[FABS_X]], 0x4170000000000000
; CHECK-NEXT: [[X_IS_NOT_ZERO:%.*]] = fcmp une float [[X]], 0.000000e+00
-; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_NOT_ZERO]], float [[FABS_X]], float [[MUL_FABS_X]]
+; CHECK-NEXT: [[SELECT:%.*]] = select i1 [[X_IS_NOT_ZERO]], float [[FABS_X]], float 0.000000e+00
; CHECK-NEXT: ret float [[SELECT]]
;
%fabs.x = call float @llvm.fabs.f32(float %x)
@@ -440,9 +434,8 @@ define <2 x float> @select_oeq_fmul_fabs_or_fabs_src_vector(<2 x float> %x) {
; CHECK-LABEL: define <2 x float> @select_oeq_fmul_fabs_or_fabs_src_vector(
; CHECK-SAME: <2 x float> [[X:%.*]]) {
; CHECK-NEXT: [[FABS_X:%.*]] = call <2 x float> @llvm.fabs.v2f32(<2 x float> [[X]])
-; CHECK-NEXT: [[MUL_FABS_X:%.*]] = fmul <2 x float> [[FABS_X]], <float 2.000000e+01, float 4.000000e+01>
; CHECK-NEXT: [[X_IS_ZERO:%.*]] = fcmp oeq <2 x float> [[X]], zeroinitializer
-; CHECK-NEXT: [[SELECT:%.*]] = select <2 x i1> [[X_IS_ZERO]], <2 x float> [[MUL_FABS_X]], <2 x float> [[FABS_X]]
+; CHECK-NEXT: [[SELECT:%.*]] = select <2 x i1> [[X_IS_ZERO]], <2 x float> zeroinitializer, <2 x float> [[FABS_X]]
; CHECK-NEXT: ret <2 x float> [[SELECT]]
;
%fabs.x = call <2 x float> @llvm.fabs.v2f32(<2 x float> %x)
@@ -456,9 +449,8 @@ define <3 x float> @select_oeq_fmul_fabs_or_fabs_src_vector_mixed_sign_zero(<3 x
; CHECK-LABEL: define <3 x float> @select_oeq_fmul_fabs_or_fabs_src_vector_mixed_sign_zero(
; CHECK-SAME: <3 x float> [[X:%.*]]) {
; CHECK-NEXT: [[FABS_X:%.*]] = call <3 x float> @llvm.fabs.v3f32(<3 x float> [[X]])
-; CHECK-NEXT: [[MUL_FABS_X:%.*]] = fmul <3 x float> [[FABS_X]], splat (float 0x4170000000000000)
; CHECK-NEXT: [[X_IS_ZERO:%.*]] = fcmp oeq <3 x float> [[X]], zeroinitializer
-; CHECK-NEXT: [[SELECT:%.*]] = select <3 x i1> [[X_IS_ZERO]], <3 x float> [[MUL_FABS_X]], <3 x float> [[FABS_X]]
+; CHECK-NEXT: [[SELECT:%.*]] = select <3 x i1> [[X_IS_ZERO]], <3 x float> zeroinitializer, <3 x float> [[FABS_X]]
; CHECK-NEXT: ret <3 x float> [[SELECT]]
;
%fabs.x = call <3 x float> @llvm.fabs.v3f32(<3 x float> %x)
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is similar to the select-bin-op identity case, except
in this case we are looking for the absorbing value for the
binary operator.
If the compared value is a floating-point 0, and the fmul is
implied to return a +0, put the 0 directly into the select
operand. This pattern appears in scale-if-denormal sequences
after optimizations assume denormals are treated as 0.
Fold:
To:
https://alive2.llvm.org/ce/z/Qcy56Z