WIP+RFC add chunked mode

[oscardssmith]

based on discussion JuliaDiff/Diffractor.jl#54 and in slack. This is an initial implementation of forward and reverse chunked mode AD.

[github-actions]

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

	export RuleConfig, HasReverseMode, NoReverseMode, HasForwardsMode, NoForwardsMode, HasChunkedMode, NoChunkedMode
	export RuleConfig,
	HasReverseMode,
	NoReverseMode,
	HasForwardsMode,
	NoForwardsMode,
	HasChunkedMode,
	NoChunkedMode

[github-actions]

[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

	function frule(::RuleConfig{>:HasChunkedMode},
	(Δf, Δx)::Tuple{Any,ProductTangent}, f, args...)
	function frule(
	::RuleConfig{>:HasChunkedMode}, (Δf, Δx)::Tuple{Any,ProductTangent}, f, args...
	)

[mcabbott]

I know I suggested this RuleConfig idea, but I'm not sure it fits. If there is an un-chunked rrule which needs the configuration, then this rrule(args...) can't call it.

I think it might be cleanest to have a new function, whose fallback is chunked_rrule(config, f, args...) = rrule(config, f, args...). Any AD which may make chunks should always call this, and any AD rule to exploit chunks will overload it.

Not sure how that intersects with @opt_out. Maybe there are other weird corners to think about too.

[oscardssmith]

yeah. We might need it to be separate. It feels clunky, but this method also doesn't seem great.

[oxinabox]

Hmm ideally we want to hit the version with this particular trait dropped, and other traits preserved (like HasReverseMode etc)
We should be able to achieve that with invoke, but can we do it elegantly?

[mcabbott]

How much is gained from this kind of chunking? I mean storing several partials in some tuple. For reverse mode, nothing, I think. For forward mode, maybe derivatives_using_output has the same benefits.

The big gain seems to be from making a solid matrix to represent many vectors, and e.g. getting matrix multiplication. So I think the initial design ought to make that work.

[oscardssmith]

This design does this (I think). partials here can either be a Tuple{Tuple} or an eachrow(Matrix)

[mcabbott]

Ok, I guess a vector of vectors can have the same meaning as an EachCol. Just a different path, by dispatch on ProductTangent{<:EachSlice}.

[oxinabox]

Having some tests would prove illuminating I think