[SPARK-55367][PYTHON] Use venv for run-pip-tests

[gaogaotiantian]

What changes were proposed in this pull request?

Use venv instead of conda or virtualenv for run-pip-tests. Remove the conda dependency in our CI.

Why are the changes needed?

run-pip-tests require a virtual environment which we used to achieve with conda or virtualenv. However, venv(https://docs.python.org/3/library/venv.html) is the recommended way to create a virtual environment since python 3.5. It's a standard library so we don't need any new dependency. It just require python to work.

In this way we can just remove the conda part which is messing with our CI when it installs the same version of python as our docker image.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I tried it locally and it worked. Let's wait for CI results.

Was this patch authored or co-authored using generative AI tooling?

No.

[github-actions]

JIRA Issue Information

=== Bug SPARK-55367 ===
Summary: Use venv for run-pip-test
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions

[zhengruifeng]

It is a nice change if we can drop the usage of conda.
BTW, I am upgrading the ubuntu version to 24.04, and we will also use venv in it

[gaogaotiantian]

Yeah it's an unnecessary layer of complexity to setup conda just for pip test. We should be able to do it with pure python.

[LuciferYang]

Can this fix resolve the following issues?

• https://github.com/apache/spark/actions/runs/21698816615/job/62574979772

[gaogaotiantian]

@LuciferYang yes and the PR that made the error above is already reverted.

[zhengruifeng]

it seems we cannot install packages in this way, deadsnakes doesn't support such package.

And for venv, do we need to additionally install it? It should be available after install python

[gaogaotiantian]

Huh? Could you post the failure? It's only not supported in 3.14 or 3.14t? That's how I install venv.

So venv, like many other "standard libraries", is actually optional to Python package. How to package it is determined by the distribution. On Linux I believe venv is not always packaged with python.

We can use uv, but this should in theory work. deadsnake should have it.

[zhengruifeng]

https://github.com/zhengruifeng/spark/actions/runs/21749252022/job/62742834544

it acutually failed with python3.14-pip

When upgrading the os to ubuntu 24, it seems the venv is already available after intsalling python3.14

[gaogaotiantian]

Hmm, okay. In that case I'll remove this line.

[gaogaotiantian]

@zhengruifeng the notification failed. Build result in https://github.com/gaogaotiantian/spark/actions/runs/21883385818

I removed python*-pip and used python*-venv. When python -m venv executes, it installs pip if pip does not exist. It at least worked on the PR. The rest images should work too.

[zhengruifeng]

merged to master

[zhengruifeng]

@gaogaotiantian it seems python 3.14t workflow is broken https://github.com/apache/spark/actions/runs/21880964892/job/63163219324

I guess it needs a non-gil version venv?