feat(python): add smoke test generation for production environment testing

[davidkonigsberg]

Description

Refs: Requested by @davidkonigsberg

Adds a new smoke test generation feature to the Python SDK generator. When enabled, the SDK generates tests in tests/smoke/ that can be run against a production environment to verify that API examples work correctly.

Link to Devin run: https://app.devin.ai/sessions/e9f4af0bad3045c0b0542ca02ac4ca06

Changes Made

• Add ValidationFilterConfig class for filtering which examples get specific validation types
• Add SmokeTestConfig class with layered validation configuration:
  • generate_smoke_tests - Enable/disable smoke test generation
  • include_examples / exclude_examples - Filter which examples to generate tests for
  • structural_validation - Filter config for type/structure validation (default: all examples)
  • strict_validation - Filter config for exact value matching (default: none)
• Add smoke_test_config option to SDKCustomConfig
• Create SmokeTestFactory class that generates pytest tests in tests/smoke/ directory
• Create smoke_test_utilities.py with validate_smoke_test_response (structural) and validate_strict_response (exact match) functions
• Wire up smoke test generation in sdk_generator.py
• Add example fixture at seed/python-sdk/examples/smoke-tests/ demonstrating the feature
• Add changelog entry for v4.47.0-rc0

Usage:

generators:
  - name: fernapi/fern-python-sdk
    config:
      smoke_test_config:
        generate_smoke_tests: true
        exclude_examples:
          - "Users.createUser"  # Skip all examples for this endpoint
        structural_validation:
          include_examples: null  # null = all (default)
          exclude_examples: []
        strict_validation:
          include_examples:
            - "Config.getSettings"  # Only strict validate this endpoint
          exclude_examples: []

Validation modes:

1. Status validation - Always runs; verifies request succeeds (no exception = 2xx response)
2. Structural validation - Validates response types/structure match the example (not exact values)
3. Strict validation - Validates exact response values match the example

Environment variables:

• SMOKE_TEST_BASE_URL - Production API base URL
• SMOKE_TEST_<PARAM> - Auth tokens and other client parameters (string types only)

Updates Since Last Revision

• Moved generate_smoke_tests flag inside smoke_test_config (previously was a separate top-level config option)
• Merged latest main branch to resolve CI compilation issues (unrelated @fern-api/ir-utils TypeScript error)

Testing

• Unit tests added/updated
• Manual lint check completed (pnpm run check passes)
• Pre-commit hooks pass (ruff, ruff-format)
• Seed test generates example fixture successfully
• All CI checks passing

Human Review Checklist

• Verify structural validation logic handles bool vs int correctly (Python's bool is a subclass of int)
• Confirm Pydantic v1/v2 compatibility (code uses .dict() which is deprecated in v2 - should use .model_dump())
• Verify _normalize_value in strict validation handles all edge cases (datetime, UUID, nested Pydantic models)
• Confirm include/exclude pattern matching works as expected for both endpoint and example-level patterns
• Note: Union types currently skip validation ("no_validate") - verify this is acceptable
• Review the generated example at seed/python-sdk/examples/smoke-tests/ to verify output looks correct

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring