Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 7 additions & 19 deletions .github/Dockerfile.ci
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Dockerfile for CI/CD with Poetry and Python pre-installed
FROM python:3.12-slim
# Dockerfile for CI/CD with uv and Python pre-installed
FROM ghcr.io/astral-sh/uv:python3.12-trixie

LABEL org.opencontainers.image.source="https://github.com/OpenSecFlow/netdriver"
LABEL org.opencontainers.image.description="CI/CD image with Python 3.12 and Poetry"
LABEL org.opencontainers.image.description="CI/CD image with Python 3.12 and uv"
LABEL org.opencontainers.image.licenses="Apache-2.0"

# Install system dependencies
Expand All @@ -11,22 +11,10 @@ RUN apt-get update && apt-get install -y \
curl \
&& rm -rf /var/lib/apt/lists/*

# Install Poetry
ENV POETRY_VERSION=1.8.3
ENV POETRY_HOME=/opt/poetry
ENV POETRY_NO_INTERACTION=1
ENV POETRY_VIRTUALENVS_IN_PROJECT=false
ENV POETRY_VIRTUALENVS_CREATE=true

RUN curl -sSL https://install.python-poetry.org | python3 - \
&& ln -s /opt/poetry/bin/poetry /usr/local/bin/poetry

# Install Poetry plugins
RUN poetry self add poetry-multiproject-plugin \
&& poetry self add poetry-polylith-plugin

# Verify installation
RUN poetry --version && poetry self show plugins
# Set uv environment variables
ENV UV_PYTHON=python3.12
ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy

WORKDIR /workspace

Expand Down
19 changes: 10 additions & 9 deletions .github/workflows/QUICKSTART.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ The `publish-pypi.yml` workflow uses a pre-built Docker image for faster executi
**Or build locally:**

```bash
docker build -t ghcr.io/opensecflow/netdriver/python-poetry:3.12 -f .github/Dockerfile.ci .
docker push ghcr.io/opensecflow/netdriver/python-poetry:3.12
docker build -t ghcr.io/opensecflow/netdriver/python-uv:3.12 -f .github/Dockerfile.ci .
docker push ghcr.io/opensecflow/netdriver/python-uv:3.12
```

**Note**: This only needs to be done once. The image will be cached and reused.
Expand Down Expand Up @@ -103,8 +103,8 @@ Before publishing to production PyPI, test with TestPyPI:

```bash
# 1. Update version numbers
poetry version -P projects/agent 0.3.1
poetry version -P projects/simunet 0.3.1
sed -i 's/^version = ".*"/version = "0.3.1"/' projects/agent/pyproject.toml
sed -i 's/^version = ".*"/version = "0.3.1"/' projects/simunet/pyproject.toml

# 2. Commit changes
git add projects/*/pyproject.toml
Expand Down Expand Up @@ -156,20 +156,21 @@ The `release.yml` workflow will automatically:
**Solution:** Version already exists on PyPI. Bump the version:

```bash
poetry version -P projects/agent patch
poetry version -P projects/simunet patch
# Manually update version in pyproject.toml files
sed -i 's/^version = ".*"/version = "0.3.2"/' projects/agent/pyproject.toml
sed -i 's/^version = ".*"/version = "0.3.2"/' projects/simunet/pyproject.toml
# Then rebuild and publish
```

### Workflow fails with "Poetry not found" or image pull error
### Workflow fails with "uv not found" or image pull error

**Solution:** Build the CI Docker image first

```bash
# Go to Actions → "Build CI Image" → Run workflow
```

Or check the image name matches: `ghcr.io/opensecflow/netdriver/python-poetry:3.12`
Or check the image name matches: `ghcr.io/opensecflow/netdriver/python-uv:3.12`

### Package shows as "0 B" or malformed

Expand Down Expand Up @@ -198,7 +199,7 @@ unzip -l projects/agent/dist/netdriver_agent-*.whl
### Release Process

1. Develop features on branches
2. Test locally: `poetry build-project -C projects/agent`
2. Test locally: `uv build --directory projects/agent`
3. Create PR and verify build test passes
4. Merge to master
5. Test on TestPyPI
Expand Down
42 changes: 20 additions & 22 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ This directory contains GitHub Actions workflows for automated building, testing

**What it does**:

- Uses pre-built Docker container with Poetry installed
- Uses pre-built Docker container with uv installed
- Builds wheel packages for selected projects
- Publishes to PyPI or TestPyPI
- Uploads build artifacts
Expand Down Expand Up @@ -68,7 +68,7 @@ The project supports independent release workflows for agent and simunet:

```bash
# Update agent version in pyproject.toml (optional, will be updated by workflow)
poetry version 1.0.0 -C projects/agent
sed -i 's/^version = ".*"/version = "1.0.0"/' projects/agent/pyproject.toml

# Commit version changes (optional)
git add projects/agent/pyproject.toml
Expand Down Expand Up @@ -98,7 +98,7 @@ git push origin agent-1.0.0

```bash
# Update simunet version in pyproject.toml (optional, will be updated by workflow)
poetry version 2.5.0 -C projects/simunet
sed -i 's/^version = ".*"/version = "2.5.0"/' projects/simunet/pyproject.toml

# Commit version changes (optional)
git add projects/simunet/pyproject.toml
Expand Down Expand Up @@ -165,7 +165,7 @@ Use this when you only need to release the agent:
1. **Update version number** (optional):

```bash
poetry version 1.0.0 -C projects/agent
sed -i 's/^version = ".*"/version = "1.0.0"/' projects/agent/pyproject.toml
```

2. **Commit changes** (optional):
Expand Down Expand Up @@ -197,7 +197,7 @@ Use this when you need to release simunet:
1. **Update version number** (optional):

```bash
poetry version 2.5.0 -C projects/simunet
sed -i 's/^version = ".*"/version = "2.5.0"/' projects/simunet/pyproject.toml
```

2. **Commit changes** (optional):
Expand Down Expand Up @@ -235,8 +235,8 @@ To test publishing before official release:
2. **Or use CLI**:

```bash
poetry publish -P projects/agent -r testpypi
poetry publish -P projects/simunet -r testpypi
uv publish --directory projects/agent --publish-url https://test.pypi.org/legacy/ --token $TESTPYPI_TOKEN
uv publish --directory projects/simunet --publish-url https://test.pypi.org/legacy/ --token $TESTPYPI_TOKEN
```

3. **Verify on TestPyPI**:
Expand Down Expand Up @@ -382,12 +382,12 @@ The project uses prefixed tag patterns for independent releases:

### 5. Build CI Image (`build-ci-image.yml`)

**Purpose**: Creates a Docker image with Poetry and Python pre-installed for faster CI/CD
**Purpose**: Creates a Docker image with uv and Python pre-installed for faster CI/CD

**What it includes**:

- Python 3.12
- Poetry with multiproject and polylith plugins
- uv package manager
- Git and essential build tools

**Building the image**:
Expand All @@ -408,24 +408,24 @@ The `publish-pypi.yml` workflow uses this approach:
jobs:
publish:
runs-on: ubuntu-latest
container:
image: ghcr.io/${{ github.repository }}/python-poetry:3.12
steps:
- uses: actions/checkout@v4
# Poetry and plugins are already installed!
- run: poetry build-project -C projects/agent
- uses: astral-sh/setup-uv@v4
- run: uv python install
- run: uv sync
- run: uv build --directory projects/agent
```

**Benefits of using Docker image**:

- ⚡ **Faster**: No need to install Poetry and plugins on every run
- ⚡ **Faster**: No need to install uv on every run
- 🔒 **Consistent**: Same environment across all workflows
- 💾 **Cacheable**: Image layers are cached by Docker
- 🎯 **Reproducible**: Exact same versions every time

**Image locations**:

- GitHub Container Registry: `ghcr.io/opensecflow/netdriver/python-poetry:3.12`
- GitHub Container Registry: `ghcr.io/opensecflow/netdriver/python-uv:3.12`
- Available tags: `latest`, `master`, `<branch>-<sha>`

**Benefits**:
Expand Down Expand Up @@ -485,20 +485,18 @@ container:
password: ${{ secrets.DOCKER_PASSWORD }}
```

### 3. Verify Poetry is available
### 3. Verify uv is available

Poetry and plugins are pre-installed, so you can use them directly:
uv is pre-installed, so you can use it directly:

```yaml
- name: Verify Poetry installation
- name: Verify uv installation
run: |
poetry --version
poetry self show plugins
uv --version
```

## References

- [Poetry Documentation](https://python-poetry.org/docs/)
- [Poetry Polylith Plugin](https://github.com/DavidVujic/poetry-polylith-plugin)
- [uv Documentation](https://docs.astral.sh/uv/)
- [PyPI Publishing Guide](https://packaging.python.org/tutorials/packaging-projects/)
- [GitHub Actions - Python](https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python)
2 changes: 1 addition & 1 deletion .github/workflows/build-ci-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ on:

env:
REGISTRY: ghcr.io
IMAGE_NAME: opensecflow/netdriver/python-poetry
IMAGE_NAME: opensecflow/netdriver/python-uv

jobs:
build:
Expand Down
40 changes: 19 additions & 21 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on:
- 'components/**'
- 'projects/**'
- 'pyproject.toml'
- 'poetry.lock'
- 'uv.lock'
- 'tests/**'
push:
branches: [master, main]
Expand All @@ -18,15 +18,13 @@ on:
- 'components/**'
- 'projects/**'
- 'pyproject.toml'
- 'poetry.lock'
- 'uv.lock'
- 'tests/**'

jobs:
test:
name: Lint and Test
runs-on: ubuntu-latest
container:
image: ghcr.io/opensecflow/netdriver/python-poetry
permissions:
contents: read
packages: read
Expand All @@ -35,30 +33,31 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4

- name: Verify Poetry installation
run: |
poetry --version
poetry self show plugins
- name: Install uv
uses: astral-sh/setup-uv@v4

- name: Set up Python
run: uv python install

- name: Install dependencies
run: poetry install --no-interaction
run: uv sync

- name: Run pylint
run: |
echo "Running pylint checks..."
poetry run pylint bases/ components/ --exit-zero --output-format=colorized || true
uv run pylint bases/ components/ --exit-zero --output-format=colorized || true
continue-on-error: true

- name: Run unit test
run: |
echo "Running unit pytest..."
poetry run pytest -v --tb=short -m unit
uv run pytest -v --tb=short -m unit


- name: Run integration test
run: |
echo "Running integration pytest..."
poetry run pytest -v --tb=short --mock-dev -m integration
uv run pytest -v --tb=short --mock-dev -m integration

- name: Generate test summary
if: always()
Expand All @@ -75,8 +74,6 @@ jobs:
name: Build packages
needs: test
runs-on: ubuntu-latest
container:
image: ghcr.io/opensecflow/netdriver/python-poetry
permissions:
contents: read
packages: read
Expand All @@ -88,22 +85,23 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4

- name: Verify Poetry installation
run: |
poetry --version
poetry self show plugins
- name: Install uv
uses: astral-sh/setup-uv@v4

- name: Set up Python
run: uv python install

- name: Install dependencies
run: poetry install --no-interaction
run: uv sync

- name: Build ${{ matrix.project }}
run: |
echo "Building netdriver-${{ matrix.project }}..."
poetry build-project -C projects/${{ matrix.project }} --format wheel
uv build --directory projects/${{ matrix.project }}

- name: Check package metadata
run: |
pip install twine
uv pip install twine
twine check projects/${{ matrix.project }}/dist/*.whl

- name: Verify package contents
Expand Down
22 changes: 9 additions & 13 deletions .github/workflows/publish-pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,6 @@ jobs:
publish:
name: Publish Python packages
runs-on: ubuntu-latest
# Use pre-built container with Python and Poetry installed
container:
image: ghcr.io/opensecflow/netdriver/python-poetry
environment: ${{ github.event.inputs.environment || 'pypi' }}
permissions:
contents: read
Expand All @@ -34,13 +31,14 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4

- name: Verify Poetry installation
run: |
poetry --version
poetry self show plugins
- name: Install uv
uses: astral-sh/setup-uv@v4

- name: Set up Python
run: uv python install

- name: Install dependencies
run: poetry install --no-interaction
run: uv sync

- name: Determine projects to publish
id: projects
Expand Down Expand Up @@ -73,17 +71,15 @@ jobs:
echo "Building $project..."

# Clean and build
poetry build-project -C "projects/$project" --format wheel
uv build --directory "projects/$project"

# Publish
if [ "$TARGET" = "testpypi" ]; then
echo "Publishing $project to TestPyPI..."
poetry config pypi-token.testpypi "$TESTPYPI_TOKEN"
poetry publish -C "projects/$project" -r testpypi --skip-existing
uv publish --directory "projects/$project" --token "$TESTPYPI_TOKEN" --publish-url https://test.pypi.org/legacy/
else
echo "Publishing $project to PyPI..."
poetry config pypi-token.pypi "$PYPI_TOKEN"
poetry publish -C "projects/$project" --skip-existing
uv publish --directory "projects/$project" --token "$PYPI_TOKEN"
fi

echo "✓ Successfully published $project to $TARGET"
Expand Down
Loading