Skip to content

Comments

Fix: Standardize User-Agent headers across Importers (#2122)#2185

Open
NamanmeetSingh wants to merge 1 commit intoaboutcode-org:mainfrom
NamanmeetSingh:fix-2122-user-agent
Open

Fix: Standardize User-Agent headers across Importers (#2122)#2185
NamanmeetSingh wants to merge 1 commit intoaboutcode-org:mainfrom
NamanmeetSingh:fix-2122-user-agent

Conversation

@NamanmeetSingh
Copy link

Fixes #2122

Description:
This PR implements a standardized User-Agent header across all network-calling modules to prevent upstream bot-blocking (like 403 Forbidden errors), as outlined in Issue #2122.

Note: @Kiran95021 had a great write-up for this issue previously! Since there hadn't been any recent activity on it and GSoC prep is ramping up, I went ahead and drafted the implementation to help get this unblocked. If you were already working on this locally, please let me know—I'm happy to collaborate!

Changes Made:

  • Added a VC_USER_AGENT constant to vulnerablecode/settings.py.
  • Updated requests.get, requests.post, and urllib calls inside importers, pipelines, datasources, and utils.py to securely pass the new header.
  • Injected the header into shared session objects (e.g., requests_with_5xx_retry).
  • Updated the pytest mock files (using mock_get.call_args and **kwargs in monkeypatches) to strictly assert and validate the presence of the new header.

Testing:

  • Ran docker compose exec vulnerablecode ./manage.py import nginx_importer locally on WSL2.
  • Successfully bypassed the bot-blocks, fetched the index, and cleanly inserted 43 advisories.

…rs to prevent blocking

Signed-off-by: Namanmeet Singh <singhnamanmeet@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Standardize 'User-Agent' headers across Importers to prevent blocking

1 participant