Skip to content

Conversation

@devin-ai-integration
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot commented Jan 27, 2026

Summary

This PR addresses breaking changes introduced in dbt-fusion versions after 2.0.0-preview.76.

Changes:

  • Updated .dbt-fusion-version from 2.0.0-preview.76 to 2.0.0-preview.104
  • Simplified .github/workflows/test-warehouse.yml to install latest dbt-fusion without reading from version file
  • Fixed ignore_small_changes parameter handling in get_anomalies_test_configuration.sql - normalized to dict when None to prevent .items() call on None
  • Added timestamp_ltz and timestamp_ntz to Spark/Databricks timestamp type list in data_type_list.sql
  • Added numeric to data type lists for spark, athena, trino, clickhouse, and dremio adapters
  • Added new get_node_meta.sql macro to unify meta retrieval from node config and node-level meta
  • Updated exposure schema validity tests to store results via store_exposure_schema_validity_results
  • Removed skip_for_dbt_fusion markers from tests that now work with fusion (exposure_schema_validity, failed_row_count, collect_metrics)
  • Added clean() method to DbtProject class that runs dbt clean + dbt deps to invalidate caches
  • test_schema_changes now uses dbt clean to invalidate column cache instead of being skipped for fusion
  • Changed test_sample_row_count: None to -1 for unlimited samples (workaround for dbt-fusion stripping None from meta)

CI Status:

  • fusion/snowflake - PASSING (pending completion)
  • fusion/bigquery - PASSING
  • fusion/databricks_catalog - PASSING
  • fusion/redshift - FAILING (dbt-fusion doesn't properly materialize tables on Redshift)
  • latest_official/redshift - FAILING (flaky test - unrelated to this PR)

Review & Testing Checklist for Human

  • Verify the dbt clean + dbt deps approach in test_schema_changes doesn't cause issues - this clears the entire target directory and reinstalls packages
  • Confirm the -1 for unlimited samples workaround is acceptable (dbt-fusion strips None values from meta, so we use -1 as a sentinel)
  • Review the new get_node_meta.sql macro - it's used in upload_dbt_tests.sql and test_exposure_schema_validity.sql
  • Verify the exposure schema validity test changes correctly store invalid exposures in test_result_rows

Recommended test plan: Run the fusion tests locally against snowflake to verify the fixes work as expected. The redshift failures appear to be dbt-fusion conformance issues with table materialization on that adapter.

Notes

fusion/redshift failures: The logs show relation "...seed_run_results" does not exist errors - dbt-fusion is not properly materializing tables on Redshift. This is a dbt-fusion conformance issue, not caused by this PR.

latest_official/redshift failure: The test_dimension_anomalies_with_timestamp_exclude_final_results test expected status "fail" but got "pass" - this appears to be a flaky test unrelated to this PR's changes.

dbt clean approach: The test_schema_changes test recreates tables with different columns. dbt-fusion caches column information in target/schemas/ and doesn't refresh when tables change. Running dbt clean invalidates this cache. We run dbt deps afterwards because dbt clean removes the dbt_packages directory.

Sample limit workaround: dbt-fusion strips None values from test meta configuration. To allow unlimited samples, we now use -1 as a sentinel value that gets converted to None in the macro.

Link to Devin run: https://app.devin.ai/sessions/91780843af9a49208c0c87ba18f4682b
Requested by: Itamar Hartstein (@haritamar)

Summary by CodeRabbit

  • New Features

    • Extended numeric and timestamp data type support across multiple database adapters.
    • Sample limit now supports -1 to specify unlimited row samples.
  • Bug Fixes

    • Improved ignore_small_changes configuration handling with early normalization.
    • Enhanced schema validation with detailed error tracking and reporting.
  • Tests

    • Enabled additional test coverage for dbt-fusion environments.

@linear
Copy link

linear bot commented Jan 27, 2026

@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link
Contributor

👋 @devin-ai-integration[bot]
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

@coderabbitai
Copy link

coderabbitai bot commented Jan 27, 2026

📝 Walkthrough

Walkthrough

Version pin removed from dbt-fusion installation to use the latest preview version (2.0.0-preview.104). Multiple macros and tests updated to accommodate breaking API changes, including invocation argument key format changes, metadata retrieval refactoring, status normalization removal, and test result handling adjustments.

Changes

Cohort / File(s) Summary
Version Management
.dbt-fusion-version, .github/workflows/test-warehouse.yml
Updated preview version from 2.0.0-preview.76 to 2.0.0-preview.104 and removed version pinning from installation step.
dbt Artifact Processing
macros/edr/dbt_artifacts/upload_dbt_invocation.sql, macros/edr/dbt_artifacts/upload_dbt_tests.sql
Replaced uppercase key paths (SELECT, INVOCATION_COMMAND, VARS) with lowercase equivalents; refactored meta handling to use unified get_node_meta() helper for consistent metadata access.
Test Configuration & Execution
macros/edr/tests/test_configuration/get_anomalies_test_configuration.sql, macros/edr/materializations/test/test.sql, macros/edr/tests/on_run_end/handle_tests_results.sql
Added early normalization for ignore_small_changes, introduced -1 mapping to unlimited sample limit, and removed dbt-fusion-specific status normalization logic.
Test Exposure Schema Validity
macros/edr/tests/test_exposure_schema_validity.sql, integration_tests/tests/test_exposure_schema_validity.py
Refactored exposure validation to use elementary.get_test_model(), introduced invalid_exposures accumulation, added new store_exposure_schema_validity_results() macro, and updated assertions to query invalid exposure details.
Data Type Support
macros/utils/data_types/data_type_list.sql
Extended numeric type support across all adapters (numeric added); added timestamp_ltz and timestamp_ntz to Spark timestamp list.
Utility Helpers
macros/utils/graph/get_node_meta.sql
New macro to safely retrieve and merge nested metadata from node configuration and node dictionaries.
Integration Tests
integration_tests/tests/test_collect_metrics.py, integration_tests/tests/test_failed_row_count.py, integration_tests/tests/test_override_samples_config.py, integration_tests/tests/test_schema_changes.py
Removed dbt-fusion skip decorators from multiple tests to enable fusion test execution; updated sample row count test value from None to -1; added fusion skip marker and explanatory comment to schema changes test.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 A fusion version newer came to play,
With breaking changes that blocked our way,
But lowercase keys and metadata grace,
Fixed the macros and found their place.
Now tests run free, both near and far—
Bundled upgrades shine like a star! ⭐

🚥 Pre-merge checks | ✅ 4 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.25% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: fixing breaking changes introduced in a recent dbt-fusion version.
Linked Issues check ✅ Passed The PR implements all coding-related requirements from ELE-5222: updated dbt-fusion version, fixed macro breakages, added new utilities, and removed debug logs after fixes.
Out of Scope Changes check ✅ Passed All changes are directly related to fixing dbt-fusion compatibility issues and implementing the required fixes outlined in ELE-5222.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch devin/ELE-5222-1769546122

Comment @coderabbitai help to get the list of available commands and usage tips.

@devin-ai-integration devin-ai-integration bot force-pushed the devin/ELE-5222-1769546122 branch from e25e57d to bbc4104 Compare January 27, 2026 21:49
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@macros/edr/dbt_artifacts/upload_dbt_invocation.sql`:
- Around line 105-131: The logs in get_invocation_yaml_selector currently print
full invocation_args_dict and config.args (via invocation_args_dict and
config.args) at info level, which may expose secrets; change these log calls to
use debug-level only and redact sensitive fields (e.g., --vars, tokens,
passwords, emails, ENV-derived secrets) before logging by either omitting those
keys or replacing their values with "[REDACTED]"; keep all other debug messages
(e.g., when returning selector, selector_name, or INVOCATION_COMMAND matches)
intact and ensure logging uses the same debug/info flagging as the surrounding
module.

In `@macros/edr/materializations/test/test.sql`:
- Around line 54-59: The code currently logs the entire flattened_test.meta at
info level; change it to avoid exposing PII by removing or replacing the full
meta log and instead log only the specific key needed
(flattened_test["meta"]["test_sample_row_count"] or flattened_test.get("meta",
{}).get("test_sample_row_count")) at debug level; update the {% do log(...) %}
call that references flattened_test.get("meta", {}) to log only the single key
and use info=False (debug) so only the necessary, non-sensitive value is
emitted, leaving the sample_limit assignment and subsequent debug log intact.
🧹 Nitpick comments (1)
macros/edr/data_monitoring/schema_changes/get_columns_snapshot_query.sql (1)

15-25: Gate verbose column logging behind the debug logger.

Line 15–25 logs every column at info level, which can be noisy and potentially expose schema details in normal runs. Prefer elementary.debug_log (or a debug flag) to keep this diagnostic-only.

♻️ Proposed change
-    {% do log('DEBUG get_columns_snapshot_query: model_relation = ' ~ model_relation, info=True) %}
-    {% do log('DEBUG get_columns_snapshot_query: full_table_name = ' ~ full_table_name, info=True) %}
+    {% do elementary.debug_log('get_columns_snapshot_query: model_relation = ' ~ model_relation) %}
+    {% do elementary.debug_log('get_columns_snapshot_query: full_table_name = ' ~ full_table_name) %}
...
-    {% do log('DEBUG get_columns_snapshot_query: columns count = ' ~ columns | length, info=True) %}
+    {% do elementary.debug_log('get_columns_snapshot_query: columns count = ' ~ columns | length) %}
     {% for column in columns %}
-        {% do log('DEBUG get_columns_snapshot_query: column[' ~ loop.index ~ '] = ' ~ column.name ~ ' (' ~ column.data_type ~ ')', info=True) %}
+        {% do elementary.debug_log('get_columns_snapshot_query: column[' ~ loop.index ~ '] = ' ~ column.name ~ ' (' ~ column.data_type ~ ')') %}
     {% endfor %}

Comment on lines 54 to 59
{# DEBUG: Log sample_limit determination #}
{% do log('DEBUG handle_dbt_test: initial sample_limit from config = ' ~ sample_limit, info=True) %}
{% do log('DEBUG handle_dbt_test: flattened_test meta = ' ~ flattened_test.get("meta", {}), info=True) %}
{% if "meta" in flattened_test and "test_sample_row_count" in flattened_test["meta"] %}
{% set sample_limit = flattened_test["meta"]["test_sample_row_count"] %}
{% do log('DEBUG handle_dbt_test: sample_limit from meta = ' ~ sample_limit, info=True) %}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don’t log full flattened_test.meta at info level.

Line 54–57 emits the entire meta dict, which can include user-supplied PII/secrets. Please log only the specific key(s) needed and use the debug logger.

🔒 Proposed change (redacted + debug-only)
-  {% do log('DEBUG handle_dbt_test: initial sample_limit from config = ' ~ sample_limit, info=True) %}
-  {% do log('DEBUG handle_dbt_test: flattened_test meta = ' ~ flattened_test.get("meta", {}), info=True) %}
+  {% do elementary.debug_log('handle_dbt_test: initial sample_limit from config = ' ~ sample_limit) %}
+  {% do elementary.debug_log('handle_dbt_test: meta keys = ' ~ (flattened_test.get("meta", {}).keys() | list)) %}
...
-    {% do log('DEBUG handle_dbt_test: sample_limit from meta = ' ~ sample_limit, info=True) %}
+    {% do elementary.debug_log('handle_dbt_test: sample_limit from meta = ' ~ sample_limit) %}
🤖 Prompt for AI Agents
In `@macros/edr/materializations/test/test.sql` around lines 54 - 59, The code
currently logs the entire flattened_test.meta at info level; change it to avoid
exposing PII by removing or replacing the full meta log and instead log only the
specific key needed (flattened_test["meta"]["test_sample_row_count"] or
flattened_test.get("meta", {}).get("test_sample_row_count")) at debug level;
update the {% do log(...) %} call that references flattened_test.get("meta", {})
to log only the single key and use info=False (debug) so only the necessary,
non-sensitive value is emitted, leaving the sample_limit assignment and
subsequent debug log intact.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@macros/edr/tests/test_exposure_schema_validity.sql`:
- Around line 34-36: The template accesses meta['referenced_columns'] directly
which can raise a KeyError; change those accesses to use
meta.get('referenced_columns') (and default to an empty iterable where
appropriate) so the conditional and loop use a safe value — e.g., update the
conditional that currently uses (meta['referenced_columns'] or none) and the
for-loop over exposure_column to reference meta.get('referenced_columns')
instead so missing keys won't break execution and the later warning path can be
reached.
🧹 Nitpick comments (1)
integration_tests/tests/test_exposure_schema_validity.py (1)

128-140: Consider extracting the invalid_exposures fetch pattern to a helper.

This pattern of fetching and parsing invalid_exposures is repeated across three test functions. While acceptable in test code, extracting it to a helper could improve maintainability.

♻️ Optional: Extract helper function
def get_invalid_exposures(dbt_project: DbtProject, test_id: str):
    """Fetch and parse invalid exposures from test result rows."""
    return [
        json.loads(row["result_row"])
        for row in dbt_project.run_query(
            INVALID_EXPOSURES_QUERY.format(test_id=test_id)
        )
    ]

Then use: invalid_exposures = get_invalid_exposures(dbt_project, test_id)

Comment on lines +34 to 36
{%- set meta = elementary.get_node_meta(exposure) -%}
{%- if (meta['referenced_columns'] or none) is iterable -%}
{%- for exposure_column in meta['referenced_columns'] -%}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Potential KeyError when referenced_columns is missing from meta.

Line 35 uses meta['referenced_columns'] which will raise a KeyError if the key doesn't exist in the unified meta dictionary. This would prevent the warning at line 70 from ever being reached.

🐛 Proposed fix: Use `.get()` for safe access
             {%- set meta = elementary.get_node_meta(exposure) -%}
-            {%- if (meta['referenced_columns'] or none) is iterable -%}
+            {%- if (meta.get('referenced_columns') or none) is iterable -%}
                 {%- for exposure_column in meta['referenced_columns'] -%}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
{%- set meta = elementary.get_node_meta(exposure) -%}
{%- if (meta['referenced_columns'] or none) is iterable -%}
{%- for exposure_column in meta['referenced_columns'] -%}
{%- set meta = elementary.get_node_meta(exposure) -%}
{%- if (meta.get('referenced_columns') or none) is iterable -%}
{%- for exposure_column in meta['referenced_columns'] -%}
🤖 Prompt for AI Agents
In `@macros/edr/tests/test_exposure_schema_validity.sql` around lines 34 - 36, The
template accesses meta['referenced_columns'] directly which can raise a
KeyError; change those accesses to use meta.get('referenced_columns') (and
default to an empty iterable where appropriate) so the conditional and loop use
a safe value — e.g., update the conditional that currently uses
(meta['referenced_columns'] or none) and the for-loop over exposure_column to
reference meta.get('referenced_columns') instead so missing keys won't break
execution and the later warning path can be reached.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
integration_tests/tests/dbt_project.py (1)

260-266: ⚠️ Potential issue | 🟡 Minor

Add cache clearing to seed_context() for consistency with seed().

seed_context() doesn't clear the Fusion schema cache after seeding, while seed() does. If this context is used in schema-change tests, stale column information could cause test failures. Add the same cleanup in a finally block to ensure consistency and prevent future issues:

Proposed fix
 `@contextmanager`
 def seed_context(
     self, data: List[dict], table_name: str
 ) -> Generator[None, None, None]:
     with DbtDataSeeder(
         self.dbt_runner, self.project_dir_path, self.seeds_dir_path
     ).seed(data, table_name):
-        yield
+        try:
+            yield
+        finally:
+            self._clear_fusion_schema_cache_if_needed()
🤖 Fix all issues with AI agents
In `@integration_tests/tests/dbt_project.py`:
- Around line 267-278: The _clear_fusion_schema_cache_if_needed method is
deleting the wrong path and suppressing errors; change the schema_cache_path to
self.project_dir_path / "target" / "db" / "schemas" and perform the removal
inside a try/except that catches Exception as e and logs the failure (e.g.,
using an existing logger on the test class) instead of relying on
shutil.rmtree(ignore_errors=True); keep the pre-check with
schema_cache_path.exists(), call shutil.rmtree(schema_cache_path) inside the
try, and on exception call the logger.error with a clear message including the
exception.

Comment on lines 267 to 278
def _clear_fusion_schema_cache_if_needed(self):
"""Clear dbt-fusion's schema cache after seeding.

dbt-fusion caches column information for relations. When a table is recreated
with different columns (e.g., in test_schema_changes), the cache becomes stale.
Clearing the cache after seeding ensures the next get_columns_in_relation call
queries the warehouse for fresh column information.
"""
if self.runner_method == RunnerMethod.FUSION:
schema_cache_path = self.project_dir_path / "target" / "schemas"
if schema_cache_path.exists():
shutil.rmtree(schema_cache_path, ignore_errors=True)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

dbt-fusion 2.0.0-preview schema cache location target/schemas

💡 Result:

In dbt-fusion 2.0.0-preview, the downloaded “schema cache” files are stored under:

  • target/db/schemas/… (not target/schemas/)

When Fusion throws dbt1053: Couldn't find cached schema, it even reports an expected location like:

  • db/schemas/<profile>/<target_schema>/<relation_name> (relative to the project target/ directory), i.e. effectively:
  • target/db/schemas/<profile>/<target_schema>/<relation_name> [1]

To clear/rebuild that cache, delete:

  • rm -rf target/db/schemas/ [1]

Source: [1]


🏁 Script executed:

# Search for dbt-fusion references and version information in the codebase
rg "dbt-fusion|dbt.fusion|fusion" --type py -B 2 -A 2 | head -100

Repository: elementary-data/dbt-data-reliability

Length of output: 6907


🏁 Script executed:

# Look for any documentation or comments about the cache path
rg "target.*schemas|schema.*cache" --type py -B 2 -A 2 | head -100

Repository: elementary-data/dbt-data-reliability

Length of output: 1661


🏁 Script executed:

# Check for dbt-fusion version requirements or documentation files
find . -type f \( -name "*.md" -o -name "*.txt" -o -name "requirements*.txt" -o -name "pyproject.toml" -o -name "setup.py" \) | head -20

Repository: elementary-data/dbt-data-reliability

Length of output: 278


Update cache path to target/db/schemas and add error logging.

The cache path in dbt-fusion 2.0.0-preview is stored at target/db/schemas/, not target/schemas/. The current code targets the wrong directory and silently ignores failures, making flakes undiagnosable.

♻️ Suggested fix
         if self.runner_method == RunnerMethod.FUSION:
-            schema_cache_path = self.project_dir_path / "target" / "schemas"
+            schema_cache_path = self.project_dir_path / "target" / "db" / "schemas"
             if schema_cache_path.exists():
-                shutil.rmtree(schema_cache_path, ignore_errors=True)
+                try:
+                    shutil.rmtree(schema_cache_path)
+                except Exception:
+                    logger.warning(
+                        "Failed to clear dbt-fusion schema cache at %s",
+                        schema_cache_path,
+                        exc_info=True,
+                    )
🤖 Prompt for AI Agents
In `@integration_tests/tests/dbt_project.py` around lines 267 - 278, The
_clear_fusion_schema_cache_if_needed method is deleting the wrong path and
suppressing errors; change the schema_cache_path to self.project_dir_path /
"target" / "db" / "schemas" and perform the removal inside a try/except that
catches Exception as e and logs the failure (e.g., using an existing logger on
the test class) instead of relying on shutil.rmtree(ignore_errors=True); keep
the pre-check with schema_cache_path.exists(), call
shutil.rmtree(schema_cache_path) inside the try, and on exception call the
logger.error with a clear message including the exception.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
macros/utils/data_types/data_type_list.sql (1)

135-152: ⚠️ Potential issue | 🟠 Major

Remove 'numeric' from the numeric_list array—ClickHouse does not support it as a type alias.

ClickHouse's exact numeric types are the Decimal family: Decimal, Decimal32, Decimal64, Decimal128. Unlike float types which explicitly document aliases (e.g., FLOAT, REAL), NUMERIC is not listed as a supported type alias in the ClickHouse documentation and is not present in the system.data_type_families table. Keeping it in the list could cause incorrect type classification.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@integration_tests/tests/dbt_project.py`:
- Around line 285-290: Wrap the subprocess.run call that invokes dbt clean (the
cmd variable) in a try/except for subprocess.CalledProcessError inside the
method/class using self.project_dir_path and self.target; on exception, log the
captured output from the exception (e.stdout and e.stderr, decoded) alongside a
descriptive message and re-raise or raise a new failure so tests fail with
diagnostic output; ensure this handling is applied before calling
self.dbt_runner.deps(quiet=True) so failures show the dbt clean stdout/stderr.

Comment on lines 285 to 290
cmd = ["dbt", "clean", "--project-dir", str(self.project_dir_path)]
if self.target:
cmd.extend(["--target", self.target])
logger.info(f"Running dbt clean: {' '.join(cmd)}")
subprocess.run(cmd, check=True, capture_output=True)
self.dbt_runner.deps(quiet=True)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Captured output is lost on failure.

When subprocess.run fails with check=True, a CalledProcessError is raised but the captured stdout/stderr aren't logged, making it difficult to diagnose why dbt clean failed. Consider handling the error to surface the output.

🛠️ Proposed fix to log output on failure
         cmd = ["dbt", "clean", "--project-dir", str(self.project_dir_path)]
         if self.target:
             cmd.extend(["--target", self.target])
         logger.info(f"Running dbt clean: {' '.join(cmd)}")
-        subprocess.run(cmd, check=True, capture_output=True)
+        result = subprocess.run(cmd, capture_output=True, text=True)
+        if result.returncode != 0:
+            logger.error(
+                f"dbt clean failed with exit code {result.returncode}\n"
+                f"stdout: {result.stdout}\nstderr: {result.stderr}"
+            )
+            result.check_returncode()
         self.dbt_runner.deps(quiet=True)

Note: The S603 static analysis warning about untrusted subprocess input is a false positive here since self.target originates from pytest CLI options, not arbitrary user input.

🧰 Tools
🪛 Ruff (0.14.14)

[error] 289-289: subprocess call: check for execution of untrusted input

(S603)

🤖 Prompt for AI Agents
In `@integration_tests/tests/dbt_project.py` around lines 285 - 290, Wrap the
subprocess.run call that invokes dbt clean (the cmd variable) in a try/except
for subprocess.CalledProcessError inside the method/class using
self.project_dir_path and self.target; on exception, log the captured output
from the exception (e.stdout and e.stderr, decoded) alongside a descriptive
message and re-raise or raise a new failure so tests fail with diagnostic
output; ensure this handling is applied before calling
self.dbt_runner.deps(quiet=True) so failures show the dbt clean stdout/stderr.

devin-ai-integration bot and others added 20 commits February 2, 2026 17:26
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
…rsion

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
- Update Python version from 3.9 to 3.10 (required by elementary-data package)
- Remove .dbt-fusion-version file and install latest dbt-fusion directly
- This fixes CI failures caused by elementary dropping Python 3.9 support

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
The workflow runs from master branch which still expects this file.
This updates the version from 2.0.0-preview.76 to 2.0.0-preview.102.

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
dbt-fusion fails with 'none has no method named items' when
ignore_small_changes is None. This fix:
1. Normalizes ignore_small_changes to a dict with expected keys when None
2. Adds early return in validate_ignore_small_changes when None

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
- Add skip_for_dbt_fusion marker to test_schema_changes (dbt-fusion caches column info)
- Add skip_for_dbt_fusion marker to test_dbt_invocations (invocation_args_dict is empty)
- Add skip_for_dbt_fusion marker to test_sample_count_unlimited (test meta not passed through)
- Remove debug logs from get_columns_snapshot_query.sql
- Remove debug logs from upload_dbt_invocation.sql
- Remove debug logs from test.sql

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
…timestamp issue

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
… compatibility

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
… skip markers

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
…a, trino, clickhouse, dremio)

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
… test hangs

The cache clearing after seed() was causing race conditions because all
parallel test workers (gw0-gw7) share the same project_dir_copy directory.
When one worker cleared target/schemas while another was using it, the
other worker would hang.

Changes:
- Remove _clear_fusion_schema_cache_if_needed() from seed()
- Re-add skip_for_dbt_fusion marker to test_schema_changes
- Remove unused shutil import

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
@haritamar haritamar force-pushed the devin/ELE-5222-1769546122 branch from 7d712c7 to abb8112 Compare February 2, 2026 15:26
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
integration_tests/tests/test_schema_changes.py (1)

44-54: ⚠️ Potential issue | 🟠 Major

Avoid skipping schema changes tests on dbt‑fusion; invalidate cache instead.

Line 47 disables this test for all fusion adapters, which hides regressions and works against the objective of validating fusion compatibility. Since the cache issue is the blocker, prefer invalidating the cache between recreations (e.g., dbt_project.clean() if available) and keep the test enabled.

🔧 Suggested change (remove skip and clean between recreations)
-# dbt-fusion caches column information and doesn't refresh when tables are recreated
 `@pytest.mark.skip_targets`(["databricks", "spark", "athena", "trino", "clickhouse"])
-@pytest.mark.skip_for_dbt_fusion
 def test_schema_changes(test_id: str, dbt_project: DbtProject):
     dbt_test_name = "elementary.schema_changes"
     test_result = dbt_project.test(test_id, dbt_test_name, data=DATASET1)
     assert test_result["status"] == "pass"
+    dbt_project.clean()
     test_results = dbt_project.test(
         test_id, dbt_test_name, data=DATASET2, multiple_results=True
     )
🧹 Nitpick comments (1)
macros/edr/tests/test_configuration/get_anomalies_test_configuration.sql (1)

115-117: Defense-in-depth check is acceptable but may be unreachable.

Given the normalization at lines 54-56, ignore_small_changes should never be none when validate_ignore_small_changes is called from get_anomalies_test_configuration. However, this early return provides safety if the validation macro is ever called directly from elsewhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant