fix: prevent model mixing between LiteLLM configuration profiles #10499
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR attempts to address Issue #9169. Feedback and guidance are welcome.
Problem
When two configuration profiles are created with LiteLLM API provider but different instances (different base URLs), the list of models was always pulled from one of them, causing issues when they do not provide the same models.
Root Cause
The model cache was using only the provider name as the cache key, not distinguishing between different LiteLLM instances with different base URLs.
Solution
getCacheKey()function that generates unique cache keys including a hash of the base URL for multi-instance providers (LiteLLM, Ollama, LM Studio, Requesty, DeepInfra, Roo)writeModels()/readModels()to use unique filenames per instancegetModels()/refreshModels()to use unique cache keysflushModels()to clear both memory and file cache for the specific instancedeleteModelsCacheFile()for cleaning up disk cacheTesting
getCacheKeyreturns correct keys for different provider configurationsChanges from Previous PR #9170
This PR rescopes the fix as requested, with the following improvements:
getCacheKey()andgetCacheFilename()helpersflushModels()(not just memory cache)Fixes #9169