Add lineage query parser type configuration #24941

mohittilala · 2025-12-21T04:16:24Z

Describe your changes:

Ingestion Changes:

Add lineage query parser config to select between different parser options in database lineage and dashboard metadata services

UI Changes:

I worked on adding support of both complete and only parameter name in service doc panel for focus/scroll functionality when input is selected.
It will search for complete name first and then fallback to parameter name.
This is required when there are parameters with same name and different parents.

Type of change:

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes <issue-number>: <short explanation>
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

- Add lineage query parser config to select between different parser options in database lineage and dashboard metadata services

github-actions · 2025-12-21T04:20:00Z

TypeScript types have been updated based on the JSON schema changes in the PR

github-actions · 2025-12-21T04:26:46Z

🛡️ TRIVY SCAN RESULT 🛡️

Target: `openmetadata-ingestion-base-slim:trivy (debian 12.12)`

Vulnerabilities (6)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
`libpng-dev`	CVE-2025-64720	🚨 HIGH	1.6.39-2	1.6.39-2+deb12u1
`libpng-dev`	CVE-2025-65018	🚨 HIGH	1.6.39-2	1.6.39-2+deb12u1
`libpng-dev`	CVE-2025-66293	🚨 HIGH	1.6.39-2	1.6.39-2+deb12u1
`libpng16-16`	CVE-2025-64720	🚨 HIGH	1.6.39-2	1.6.39-2+deb12u1
`libpng16-16`	CVE-2025-65018	🚨 HIGH	1.6.39-2	1.6.39-2+deb12u1
`libpng16-16`	CVE-2025-66293	🚨 HIGH	1.6.39-2	1.6.39-2+deb12u1

🛡️ TRIVY SCAN RESULT 🛡️

Target: `Java`

Vulnerabilities (33)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
`com.fasterxml.jackson.core:jackson-core`	CVE-2025-52999	🚨 HIGH	2.12.7	2.15.0
`com.fasterxml.jackson.core:jackson-core`	CVE-2025-52999	🚨 HIGH	2.13.4	2.15.0
`com.fasterxml.jackson.core:jackson-databind`	CVE-2022-42003	🚨 HIGH	2.12.7	2.12.7.1, 2.13.4.2
`com.fasterxml.jackson.core:jackson-databind`	CVE-2022-42004	🚨 HIGH	2.12.7	2.12.7.1, 2.13.4
`com.google.code.gson:gson`	CVE-2022-25647	🚨 HIGH	2.2.4	2.8.9
`com.google.protobuf:protobuf-java`	CVE-2021-22569	🚨 HIGH	3.3.0	3.16.1, 3.18.2, 3.19.2
`com.google.protobuf:protobuf-java`	CVE-2022-3509	🚨 HIGH	3.3.0	3.16.3, 3.19.6, 3.20.3, 3.21.7
`com.google.protobuf:protobuf-java`	CVE-2022-3510	🚨 HIGH	3.3.0	3.16.3, 3.19.6, 3.20.3, 3.21.7
`com.google.protobuf:protobuf-java`	CVE-2024-7254	🚨 HIGH	3.3.0	3.25.5, 4.27.5, 4.28.2
`com.google.protobuf:protobuf-java`	CVE-2021-22569	🚨 HIGH	3.7.1	3.16.1, 3.18.2, 3.19.2
`com.google.protobuf:protobuf-java`	CVE-2022-3509	🚨 HIGH	3.7.1	3.16.3, 3.19.6, 3.20.3, 3.21.7
`com.google.protobuf:protobuf-java`	CVE-2022-3510	🚨 HIGH	3.7.1	3.16.3, 3.19.6, 3.20.3, 3.21.7
`com.google.protobuf:protobuf-java`	CVE-2024-7254	🚨 HIGH	3.7.1	3.25.5, 4.27.5, 4.28.2
`com.nimbusds:nimbus-jose-jwt`	CVE-2023-52428	🚨 HIGH	9.8.1	9.37.2
`com.squareup.okhttp3:okhttp`	CVE-2021-0341	🚨 HIGH	3.12.12	4.9.2
`commons-beanutils:commons-beanutils`	CVE-2025-48734	🚨 HIGH	1.9.4	1.11.0
`commons-io:commons-io`	CVE-2024-47554	🚨 HIGH	2.8.0	2.14.0
`dnsjava:dnsjava`	CVE-2024-25638	🚨 HIGH	2.1.7	3.6.0
`io.netty:netty-codec-http2`	CVE-2025-55163	🚨 HIGH	4.1.96.Final	4.2.4.Final, 4.1.124.Final
`io.netty:netty-codec-http2`	GHSA-xpw8-rcwv-8f8p	🚨 HIGH	4.1.96.Final	4.1.100.Final
`io.netty:netty-handler`	CVE-2025-24970	🚨 HIGH	4.1.96.Final	4.1.118.Final
`net.minidev:json-smart`	CVE-2021-31684	🚨 HIGH	1.3.2	1.3.3, 2.4.4
`net.minidev:json-smart`	CVE-2023-1370	🚨 HIGH	1.3.2	2.4.9
`org.apache.avro:avro`	CVE-2024-47561	🔥 CRITICAL	1.7.7	1.11.4
`org.apache.avro:avro`	CVE-2023-39410	🚨 HIGH	1.7.7	1.11.3
`org.apache.derby:derby`	CVE-2022-46337	🔥 CRITICAL	10.14.2.0	10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
`org.apache.ivy:ivy`	CVE-2022-46751	🚨 HIGH	2.5.1	2.5.2
`org.apache.mesos:mesos`	CVE-2018-1330	🚨 HIGH	1.4.3	1.6.0
`org.apache.thrift:libthrift`	CVE-2019-0205	🚨 HIGH	0.12.0	0.13.0
`org.apache.thrift:libthrift`	CVE-2020-13949	🚨 HIGH	0.12.0	0.14.0
`org.apache.zookeeper:zookeeper`	CVE-2023-44981	🔥 CRITICAL	3.6.3	3.7.2, 3.8.3, 3.9.1
`org.eclipse.jetty:jetty-server`	CVE-2024-13009	🚨 HIGH	9.4.56.v20240826	9.4.57.v20241219
`org.lz4:lz4-java`	CVE-2025-12183	🚨 HIGH	1.8.0	1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: `Node.js`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `Python`

Vulnerabilities (3)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
`starlette`	CVE-2025-62727	🚨 HIGH	0.48.0	0.49.1
`urllib3`	CVE-2025-66418	🚨 HIGH	1.26.20	2.6.0
`urllib3`	CVE-2025-66471	🚨 HIGH	1.26.20	2.6.0

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/etc/ssl/private/ssl-cert-snakeoil.key`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/ingestion/pipelines/extended_sample_data.yaml`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/ingestion/pipelines/lineage.yaml`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/ingestion/pipelines/sample_data.json`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/ingestion/pipelines/sample_data.yaml`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/ingestion/pipelines/sample_data_aut.yaml`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/ingestion/pipelines/sample_usage.json`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/ingestion/pipelines/sample_usage.yaml`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/ingestion/pipelines/sample_usage_aut.yaml`

No Vulnerabilities Found

github-actions · 2025-12-21T04:29:03Z

🛡️ TRIVY SCAN RESULT 🛡️

Target: `openmetadata-ingestion:trivy (debian 12.12)`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `Java`

Vulnerabilities (33)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
`com.fasterxml.jackson.core:jackson-core`	CVE-2025-52999	🚨 HIGH	2.12.7	2.15.0
`com.fasterxml.jackson.core:jackson-core`	CVE-2025-52999	🚨 HIGH	2.13.4	2.15.0
`com.fasterxml.jackson.core:jackson-databind`	CVE-2022-42003	🚨 HIGH	2.12.7	2.12.7.1, 2.13.4.2
`com.fasterxml.jackson.core:jackson-databind`	CVE-2022-42004	🚨 HIGH	2.12.7	2.12.7.1, 2.13.4
`com.google.code.gson:gson`	CVE-2022-25647	🚨 HIGH	2.2.4	2.8.9
`com.google.protobuf:protobuf-java`	CVE-2021-22569	🚨 HIGH	3.3.0	3.16.1, 3.18.2, 3.19.2
`com.google.protobuf:protobuf-java`	CVE-2022-3509	🚨 HIGH	3.3.0	3.16.3, 3.19.6, 3.20.3, 3.21.7
`com.google.protobuf:protobuf-java`	CVE-2022-3510	🚨 HIGH	3.3.0	3.16.3, 3.19.6, 3.20.3, 3.21.7
`com.google.protobuf:protobuf-java`	CVE-2024-7254	🚨 HIGH	3.3.0	3.25.5, 4.27.5, 4.28.2
`com.google.protobuf:protobuf-java`	CVE-2021-22569	🚨 HIGH	3.7.1	3.16.1, 3.18.2, 3.19.2
`com.google.protobuf:protobuf-java`	CVE-2022-3509	🚨 HIGH	3.7.1	3.16.3, 3.19.6, 3.20.3, 3.21.7
`com.google.protobuf:protobuf-java`	CVE-2022-3510	🚨 HIGH	3.7.1	3.16.3, 3.19.6, 3.20.3, 3.21.7
`com.google.protobuf:protobuf-java`	CVE-2024-7254	🚨 HIGH	3.7.1	3.25.5, 4.27.5, 4.28.2
`com.nimbusds:nimbus-jose-jwt`	CVE-2023-52428	🚨 HIGH	9.8.1	9.37.2
`com.squareup.okhttp3:okhttp`	CVE-2021-0341	🚨 HIGH	3.12.12	4.9.2
`commons-beanutils:commons-beanutils`	CVE-2025-48734	🚨 HIGH	1.9.4	1.11.0
`commons-io:commons-io`	CVE-2024-47554	🚨 HIGH	2.8.0	2.14.0
`dnsjava:dnsjava`	CVE-2024-25638	🚨 HIGH	2.1.7	3.6.0
`io.netty:netty-codec-http2`	CVE-2025-55163	🚨 HIGH	4.1.96.Final	4.2.4.Final, 4.1.124.Final
`io.netty:netty-codec-http2`	GHSA-xpw8-rcwv-8f8p	🚨 HIGH	4.1.96.Final	4.1.100.Final
`io.netty:netty-handler`	CVE-2025-24970	🚨 HIGH	4.1.96.Final	4.1.118.Final
`net.minidev:json-smart`	CVE-2021-31684	🚨 HIGH	1.3.2	1.3.3, 2.4.4
`net.minidev:json-smart`	CVE-2023-1370	🚨 HIGH	1.3.2	2.4.9
`org.apache.avro:avro`	CVE-2024-47561	🔥 CRITICAL	1.7.7	1.11.4
`org.apache.avro:avro`	CVE-2023-39410	🚨 HIGH	1.7.7	1.11.3
`org.apache.derby:derby`	CVE-2022-46337	🔥 CRITICAL	10.14.2.0	10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
`org.apache.ivy:ivy`	CVE-2022-46751	🚨 HIGH	2.5.1	2.5.2
`org.apache.mesos:mesos`	CVE-2018-1330	🚨 HIGH	1.4.3	1.6.0
`org.apache.thrift:libthrift`	CVE-2019-0205	🚨 HIGH	0.12.0	0.13.0
`org.apache.thrift:libthrift`	CVE-2020-13949	🚨 HIGH	0.12.0	0.14.0
`org.apache.zookeeper:zookeeper`	CVE-2023-44981	🔥 CRITICAL	3.6.3	3.7.2, 3.8.3, 3.9.1
`org.eclipse.jetty:jetty-server`	CVE-2024-13009	🚨 HIGH	9.4.56.v20240826	9.4.57.v20241219
`org.lz4:lz4-java`	CVE-2025-12183	🚨 HIGH	1.8.0	1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: `Node.js`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `Python`

Vulnerabilities (6)

Package	Vulnerability ID	Severity	Installed Version	Fixed Version
`Werkzeug`	CVE-2024-34069	🚨 HIGH	2.2.3	3.0.3
`deepdiff`	CVE-2025-58367	🔥 CRITICAL	7.0.1	8.6.1
`ray`	CVE-2025-62593	🔥 CRITICAL	2.47.1	2.52.0
`starlette`	CVE-2025-62727	🚨 HIGH	0.48.0	0.49.1
`urllib3`	CVE-2025-66418	🚨 HIGH	1.26.20	2.6.0
`urllib3`	CVE-2025-66471	🚨 HIGH	1.26.20	2.6.0

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/etc/ssl/private/ssl-cert-snakeoil.key`

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: `/home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO`

No Vulnerabilities Found

github-actions · 2025-12-21T04:43:06Z

Jest test Coverage

UI tests summary

Lines	Statements	Branches	Functions
	64.3% (50894/79151)	41.9% (24776/59137)	45.38% (7807/17204)

… add auto as default parser option

github-actions · 2025-12-23T08:15:58Z

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

sonarqubecloud · 2025-12-23T08:39:36Z

Quality Gate passed for 'open-metadata-ui'

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

harshach · 2025-12-24T22:13:01Z

...ta-spec/src/main/resources/json/schema/metadataIngestion/parserconfig/queryParserConfig.json

+        "Auto",
+        "SqlGlot",
+        "SqlFluff",
+        "SqlParse"


@mohittilala lets remove sqlparse as selectable parser, it should only be used when all else fails

harshach · 2025-12-24T22:13:27Z

...ta-spec/src/main/resources/json/schema/metadataIngestion/parserconfig/queryParserConfig.json

+  "properties": {
+    "type": {
+      "title": "Query Parser Type",
+      "description": "Choose the SQL parser for lineage extraction:\n• Auto (default): Automatically tries SqlGlot first, falls back to SqlFluff, then SqlParse. Recommended for best results.\n• SqlGlot: High-performance parser with excellent dialect support. Falls back to SqlParse on failure.\n• SqlFluff: Comprehensive parser with strong dialect support. Falls back to SqlParse on failure.\n• SqlParse: Generic ANSI SQL parser with limited dialect support. No fallback.",


shouldn't the options be sqlglot -> sqlfluff -> sqlparser? instead of sqlglot -> sqlparser?

Copilot

Pull request overview

This PR adds configuration support for selecting different SQL query parsers (SqlGlot, SqlFluff, SqlParse, or Auto) for lineage extraction in database and dashboard metadata ingestion workflows. Additionally, it improves the UI's ServiceDocPanel to support both complete field name paths and parameter-only names for focus/scroll functionality when form inputs are selected.

Key Changes

Query Parser Configuration: Introduces a new QueryParserConfig schema with a QueryParserType enum (Auto, SqlGlot, SqlFluff, SqlParse) that can be configured in database lineage and dashboard metadata pipelines to control which SQL parser is used for lineage extraction
Ingestion Framework Updates: Threads the parser_type parameter through all lineage processing paths (database services, dashboard services, stored procedures, views, and queries) to respect the configured parser type
UI Field Name Handling: Refactors ServiceDocPanel to try both complete field names (with full path) and simple parameter names when locating elements for highlighting and scrolling, solving issues where parameters share names but have different parent paths

Reviewed changes

Copilot reviewed 35 out of 41 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`openmetadata-spec/src/main/resources/json/schema/metadataIngestion/parserconfig/queryParserConfig.json`	New JSON schema defining QueryParserConfig with QueryParserType enum for lineage parser selection
`openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceQueryLineagePipeline.json`	Adds queryParserConfig property to database lineage pipeline configuration
`openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json`	Adds queryParserConfig property to dashboard metadata pipeline configuration
`openmetadata-ui/src/main/resources/ui/src/generated/*/.ts`	Auto-generated TypeScript types from JSON schemas including QueryParserConfig and QueryParserType
`openmetadata-ui/src/main/resources/ui/src/utils/ServiceUtils.tsx`	Renames getActiveFieldNameForAppDocs to getCompleteActiveFieldName for clarity
`openmetadata-ui/src/main/resources/ui/src/utils/ServiceUtils.test.ts`	Updates tests to use renamed function
`openmetadata-ui/src/main/resources/ui/src/components/common/ServiceDocPanel/ServiceDocPanel.tsx`	Implements fallback logic to try complete field name first, then parameter name for element selection
`openmetadata-ui/src/main/resources/ui/src/components/common/ServiceDocPanel/ServiceDocPanel.test.tsx`	Updates tests for renamed function and new fallback behavior
`openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/workflows/lineage.md`	Adds documentation for Query Parser Type configuration option
`openmetadata-ui/src/main/resources/ui/public/locales/en-US/Dashboard/workflows/metadata.md`	Adds documentation for Query Parser Type configuration option
`ingestion/src/metadata/ingestion/lineage/parser.py`	Adds parser_type parameter to LineageParser and implements conditional parser selection based on type
`ingestion/src/metadata/ingestion/lineage/sql_lineage.py`	Threads parser_type parameter through get_lineage_by_query and get_lineage_via_table_entity functions
`ingestion/src/metadata/utils/db_utils.py`	Adds parser_type parameter to get_view_lineage function
`ingestion/src/metadata/ingestion/source/database/query_parser_source.py`	Adds get_query_parser_type method to retrieve parser config from source config
`ingestion/src/metadata/ingestion/source/database/lineage_source.py`	Passes parser_type from config to lineage processors
`ingestion/src/metadata/ingestion/source/database/stored_procedures_mixin.py`	Passes parser_type from config to procedure lineage processor
`ingestion/src/metadata/ingestion/source/database/lineage_processors.py`	Adds parser_type parameter to all lineage processor functions
`ingestion/src/metadata/ingestion/source/dashboard/dashboard_service.py`	Adds get_query_parser_type method for dashboard services
`ingestion/src/metadata/ingestion/source/dashboard/*/metadata.py`	Updates all dashboard connectors (Tableau, Superset, Redash, PowerBI, Quicksight, Mode, Metabase, Microstrategy, Looker, Hex, Grafana) to use parser_type from config
`ingestion/src/metadata/ingestion/source/dashboard/powerbi/databricks_parser.py`	Adds parser_type parameter to parse_databricks_native_query_source function
`ingestion/src/metadata/ingestion/processor/query_parser.py`	Adds parser_type parameter to parse_sql_statement function
`ingestion/src/metadata/cli/lineage.py`	Adds parserType field to LineageWorkflow config model
`ingestion/tests/unit/*/.py`	Updates all unit tests to pass QueryParserType.Auto parameter where required

Copilot · 2025-12-24T22:17:07Z

ingestion/src/metadata/ingestion/source/dashboard/dashboard_service.py

+    def get_query_parser_type(self) -> Optional[QueryParserType]:
+        """
+        Get the query parser type from config.
+        Returns Auto as default if not specified.
+        """
+        return (
+            self.source_config.queryParserConfig.type
+            if self.source_config.queryParserConfig
+            else None
+        )


Similar to the query_parser_source, this method returns Optional[QueryParserType] which can be None. However, when None is passed to LineageParser or other functions expecting QueryParserType, it will cause a TypeError. Consider returning QueryParserType.Auto instead of None when queryParserConfig is not set.

Suggested change

def get_query_parser_type(self) -> Optional[QueryParserType]:

"""

Get the query parser type from config.

Returns Auto as default if not specified.

"""

return (

self.source_config.queryParserConfig.type

if self.source_config.queryParserConfig

else None

)

def get_query_parser_type(self) -> QueryParserType:

"""

Get the query parser type from config.

Returns Auto as default if not specified.

"""

if (

getattr(self.source_config, "queryParserConfig", None)

and getattr(self.source_config.queryParserConfig, "type", None) is not None

):

return self.source_config.queryParserConfig.type

return QueryParserType.Auto

Copilot · 2025-12-24T22:17:07Z

openmetadata-ui/src/main/resources/ui/src/components/common/ServiceDocPanel/ServiceDocPanel.tsx

+      // eslint-disable-next-line @typescript-eslint/no-unused-vars
    } catch (error) {


The eslint-disable comment for unused-vars is unnecessary here. The error variable is intentionally not used in the catch block as per the design (errors are handled by setting markdownContent to empty string). If you want to suppress the warning, consider using an underscore prefix for the parameter name instead of a disable comment.

Copilot · 2025-12-24T22:17:07Z

ingestion/src/metadata/ingestion/source/dashboard/dashboard_service.py

+        return (
+            self.source_config.queryParserConfig.type
+            if self.source_config.queryParserConfig
+            else None


The documentation comment states "Returns Auto as default if not specified" but the method actually returns None. Either update the documentation to match the implementation, or change the implementation to return QueryParserType.Auto when queryParserConfig is not set (which would fix the related bug).

Suggested change

else None

and self.source_config.queryParserConfig.type

else QueryParserType.Auto

Copilot · 2025-12-24T22:17:08Z

ingestion/src/metadata/ingestion/source/database/query_parser_source.py

+    def get_query_parser_type(self) -> Optional[QueryParserType]:
+        """
+        Get the query parser type from source config.
+
+        Returns None to use default Auto parser type.
+        """
+        if (
+            hasattr(self.source_config, "queryParserConfig")
+            and self.source_config.queryParserConfig
+        ):
+            return self.source_config.queryParserConfig.type
+        return None


The documentation comment states "Returns None to use default Auto parser type" but this is misleading. When None is passed explicitly as a parameter to functions, it overrides their default values. The code would be clearer and more robust if this method returned QueryParserType.Auto instead of None when queryParserConfig is not set.

Suggested change

def get_query_parser_type(self) -> Optional[QueryParserType]:

"""

Get the query parser type from source config.

Returns None to use default Auto parser type.

"""

if (

hasattr(self.source_config, "queryParserConfig")

and self.source_config.queryParserConfig

):

return self.source_config.queryParserConfig.type

return None

def get_query_parser_type(self) -> QueryParserType:

"""

Get the query parser type from source config.

Returns the configured parser type, or QueryParserType.Auto when not set.

"""

if (

hasattr(self.source_config, "queryParserConfig")

and self.source_config.queryParserConfig

):

return self.source_config.queryParserConfig.type

return QueryParserType.Auto

Copilot · 2025-12-24T22:17:08Z

ingestion/src/metadata/ingestion/source/database/query_parser_source.py

+    def get_query_parser_type(self) -> Optional[QueryParserType]:
+        """
+        Get the query parser type from source config.
+
+        Returns None to use default Auto parser type.
+        """
+        if (
+            hasattr(self.source_config, "queryParserConfig")
+            and self.source_config.queryParserConfig
+        ):
+            return self.source_config.queryParserConfig.type
+        return None


The get_query_parser_type method returns Optional[QueryParserType] which can be None, but the LineageParser expects QueryParserType.Auto as the default. When this method returns None, it will cause a TypeError when the parser_type is used in comparisons within LineageParser. Consider returning QueryParserType.Auto instead of None when queryParserConfig is not set.

Suggested change

def get_query_parser_type(self) -> Optional[QueryParserType]:

"""

Get the query parser type from source config.

Returns None to use default Auto parser type.

"""

if (

hasattr(self.source_config, "queryParserConfig")

and self.source_config.queryParserConfig

):

return self.source_config.queryParserConfig.type

return None

def get_query_parser_type(self) -> QueryParserType:

"""

Get the query parser type from source config.

Returns QueryParserType.Auto if queryParserConfig is not set.

"""

if (

hasattr(self.source_config, "queryParserConfig")

and self.source_config.queryParserConfig

):

return self.source_config.queryParserConfig.type

return QueryParserType.Auto

Copilot · 2025-12-24T22:17:08Z

ingestion/src/metadata/utils/db_utils.py

 import time
 import traceback
-from typing import Iterable, List, Union
+from typing import Iterable, List, Optional, Union


Import of 'Optional' is not used.

Suggested change

from typing import Iterable, List, Optional, Union

from typing import Iterable, List, Union

Add lineage query parser type configuration

77b3def

- Add lineage query parser config to select between different parser options in database lineage and dashboard metadata services

mohittilala self-assigned this Dec 21, 2025

mohittilala requested review from a team as code owners December 21, 2025 04:16

mohittilala added Ingestion safe to test Add this label to run secure Github workflows on PRs lineage To release Will cherry-pick this PR into the release branch labels Dec 21, 2025

mohittilala had a problem deploying to test December 21, 2025 04:16 — with GitHub Actions Error

mohittilala temporarily deployed to test December 21, 2025 04:16 — with GitHub Actions Inactive

Update generated TypeScript types

5587b58

Merge branch 'main' into query-parser-config

2729c22

mohittilala temporarily deployed to test December 21, 2025 19:27 — with GitHub Actions Inactive

mohittilala added 2 commits December 23, 2025 01:03

Add query hash to the debug error traces

3557358

Make parser_type arg mandatory in all methods to avoid missing it and…

8ba8209

… add auto as default parser option

mohittilala had a problem deploying to test December 23, 2025 08:08 — with GitHub Actions Failure

mohittilala temporarily deployed to test December 23, 2025 08:08 — with GitHub Actions Inactive

harshach requested a review from Copilot December 24, 2025 22:12

Copilot started reviewing on behalf of harshach December 24, 2025 22:12 View session

harshach reviewed Dec 24, 2025

View reviewed changes

Copilot AI reviewed Dec 24, 2025

View reviewed changes

		// eslint-disable-next-line @typescript-eslint/no-unused-vars
		} catch (error) {

	else None
	and self.source_config.queryParserConfig.type
	else QueryParserType.Auto

	from typing import Iterable, List, Optional, Union
	from typing import Iterable, List, Union

Add lineage query parser type configuration #24941

Are you sure you want to change the base?

Add lineage query parser type configuration #24941

Uh oh!

Conversation

mohittilala commented Dec 21, 2025

Describe your changes:

Ingestion Changes:

UI Changes:

Type of change:

Checklist:

Uh oh!

github-actions bot commented Dec 21, 2025

Uh oh!

github-actions bot commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion-base-slim:trivy (debian 12.12)

Vulnerabilities (6)

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (3)

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/extended_sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/lineage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data_aut.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage_aut.yaml

No Vulnerabilities Found

Uh oh!

github-actions bot commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion:trivy (debian 12.12)

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (6)

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO

No Vulnerabilities Found

Uh oh!

github-actions bot commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jest test Coverage

github-actions bot commented Dec 21, 2025 •

edited

Loading

Target: `openmetadata-ingestion-base-slim:trivy (debian 12.12)`

Target: `Java`

Target: `Node.js`

Target: `Python`

Target: `/etc/ssl/private/ssl-cert-snakeoil.key`

Target: `/ingestion/pipelines/extended_sample_data.yaml`

Target: `/ingestion/pipelines/lineage.yaml`

Target: `/ingestion/pipelines/sample_data.json`

Target: `/ingestion/pipelines/sample_data.yaml`

Target: `/ingestion/pipelines/sample_data_aut.yaml`

Target: `/ingestion/pipelines/sample_usage.json`

Target: `/ingestion/pipelines/sample_usage.yaml`

Target: `/ingestion/pipelines/sample_usage_aut.yaml`

github-actions bot commented Dec 21, 2025 •

edited

Loading

Target: `openmetadata-ingestion:trivy (debian 12.12)`

Target: `Java`

Target: `Node.js`

Target: `Python`

Target: `/etc/ssl/private/ssl-cert-snakeoil.key`

Target: `/home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO`

github-actions bot commented Dec 21, 2025 •

edited

Loading