[Concept]: feat: introducing lazy chain loading #644
Conversation
🦋 Changeset detectedLatest commit: b71011b The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| // default - loads all chains from the networks config | ||
| chainSelectorsToLoad := cfg.Networks.ChainSelectors() | ||
|
|
||
| if loadcfg.chainSelectorsToLoad != nil { |
There was a problem hiding this comment.
We no longer care about what chains to load via chainSelectorsToLoad as we no longer load everything.
graham-chainlink
changed the title
graham-chainlink
force-pushed
the
ggoh/lazy-blockchain
branch
3 times, most recently
from
2c1b033 to
a5255df
Compare
|
| // Unlike EVMChains, this method returns an error if any chain fails to load. | ||
| // The error may contain multiple chain load failures wrapped together. | ||
| // Successfully loaded chains are still returned in the map. | ||
| func (l *LazyBlockChains) TryEVMChains() (map[uint64]evm.Chain, error) { |
There was a problem hiding this comment.
I think you can use a generic implementation here and reduce a lot of the duplication in this file:
func tryChains[T any, PT *T](l *LazyBlockChains, family string) (map[uint64]T, error) {
l.mu.RLock()
selectors := make([]uint64, 0)
for selector, f := range l.availableChains {
if f == family {
selectors = append(selectors, selector)
}
}
l.mu.RUnlock()
chains := make(map[uint64]T)
var errs []error
for _, selector := range selectors {
chain, err := l.GetBySelector(selector)
if err != nil {
errs = append(errs, fmt.Errorf("failed to load chain %d: %w", selector, err))
continue
}
switch c := chain.(type) {
case T:
chains[selector] = c
case PT:
if c != nil {
chains[selector] = *c
}
}
}
if len(errs) > 0 {
return chains, errors.Join(errs...)
}
return chains, nil
}
// then
func (l *LazyBlockChains) EVMChains() map[uint64]evm.Chain {
chains, err := tryChains[evm.Chain](l, chainsel.FamilyEVM)
if err != nil {
l.lggr.Errorw("Failed to load one or more EVM chains", "error", err)
}
return chains
}
func (l *LazyBlockChains) SolanaChains() map[uint64]solana.Chain {
chains, err := tryChains[solana.Chain](l, chainsel.FamilySolana)
if err != nil {
l.lggr.Errorw("Failed to load one or more Solana chains", "error", err)
}
return chains
}
// etc...There was a problem hiding this comment.
Oh! Good pick up, using generic cleaned it up alot, thanks! I had to use the type constraint for it to work
func tryChains[T any, PT interface {
*T
}](l *LazyBlockChains, family string) (map[uint64]T, error) {
chain/lazy_blockchains.go
Outdated
| chains := make(map[uint64]evm.Chain) | ||
| var errs []error | ||
|
|
||
| for _, selector := range selectors { |
There was a problem hiding this comment.
I think this must be done in parallel for the lazy loading to become a viable option.
There was a problem hiding this comment.
done! another good pick up, updated the other getters to be parallel too!
| ) | ||
|
|
||
| // ChainLoader is an interface for loading a blockchain instance lazily. | ||
| type ChainLoader interface { |
There was a problem hiding this comment.
did you try to do
type ChainLoader = fchain.ChainLoaderin engine/cld/chains/chains.go?
I think it could simplify a few interfaces and avoid a few map/slice copies.
There was a problem hiding this comment.
yeah i knew about this but decided to keep both interface in the separate package, there were no copying as it is just the interface instead of different struct. But type alias looks cleaner, i updated to use it!
| blockChains, err := chains.LoadChains(ctx, lggr, cfg, chainSelectorsToLoad) | ||
| // Use lazy loading for chains - they will be initialized on first access | ||
| // All chains from the network config are made available, but only loaded when accessed | ||
| lazyBlockChains, err := chains.NewLazyBlockChains(ctx, lggr, cfg) |
There was a problem hiding this comment.
I think we should feature flag this initially. Nothing too complicated, just check for an environment variable -- say, CLD_LAZY_BLOCK_CHAINS=1.
There was a problem hiding this comment.
Good idea, added a feature toggle.
Chains are loaded lazily instead of eager, this means chains will only be loaded when it is being used, this means users are not forced to provide any secrets for chains if they are not used upfront and avoid loading huge amount of chains that are never used. This change also means we can deprecate `ChainOverrides` feature as we no longer have to tell CLD what chains to load. This lazy loading is hidden under the feature flag CLD_LAZY_BLOCKCHAINS
graham-chainlink
force-pushed
the
ggoh/lazy-blockchain
branch
from
a5255df to
b71011b
Compare
2 participants
Chains are loaded lazily instead of eager, this means chains will only be loaded when it is being used, this means users are not forced to provide any secrets for chains if they are not used upfront and avoid loading huge amount of chains that are never used.
This change also means we can deprecate
ChainOverridesfeature as we no longer have to tell CLD what chains to load beforehand.