feat(spark): Add SessionStateBuilderSpark to datafusion-spark

[cht42]

Which issue does this PR close?

• Closes [datafusion-spark] Add method to register udf and expr planner in one go #19843

Rationale for this change

Currently, combining DataFusion's default features with Spark features is awkward because:

1. Expression planners must be registered before calling with_default_features().build() to take precedence
2. UDFs must be registered after the state is built (if using register_all)

This requires splitting the setup into multiple phases, which is verbose and error-prone.

What changes are included in this PR?

• Added SessionStateBuilderSpark extension trait in datafusion-spark that provides with_spark_features() method to register both the Spark expression planner (with correct precedence) and all Spark UDFs in one call
• Added core feature flag to datafusion-spark with datafusion as an optional dependency (this avoids having datafusion-core depend on datafusion-spark)
• Updated datafusion-spark crate documentation with usage example
• Simplified test context setup in datafusion-sqllogictest to use the new extension trait

Are these changes tested?

Yes, there is a unit test in datafusion-spark/src/session_state.rs plus the existing Spark SQLLogicTest suite validates that all Spark functions work correctly. The test context in datafusion-sqllogictest now uses the SessionStateBuilderSpark extension trait, serving as both a usage example and integration test.

Are there any user-facing changes?

Yes, this adds a new public API: SessionStateBuilderSpark extension trait (behind the core feature flag in datafusion-spark).

[Jefffrey]

Maybe its better to introduce a new trait (e.g. SessionStateBuilderSparkExt, though with a better name) to datafusion-spark containing the new with_spark_features method and impl this onto SessionStateBuilder to avoid needing having core depend on datafusion-spark

[cht42]

Maybe its better to introduce a new trait (e.g. SessionStateBuilderSparkExt, though with a better name) to datafusion-spark containing the new with_spark_features method and impl this onto SessionStateBuilder to avoid needing having core depend on datafusion-spark

Souds good, updated the code to use that approach

[Jefffrey]

fyi @comphead

[Jefffrey]

Would prefer to avoid ignore here if possible

[cht42]

fixed it