How do I register a function in the idle loop that pre-parses all the sources files under a specific directory

[nordlow]

The source file scanning under a source directory of a project using serve-d takes too long to complete interactively so it needs to be split up into parts where after each source file being parses the parallelMain loop needs to react to messages. I presume a fiber that yields after each source file parsing is completed. I see there is parallelMain and spawnFiber but I can't find any usage example of spawnFiber.

I see

	spawnFiber("Run didSave processors", {
		auto projectUri = workspace(params.textDocument.uri).folder.uri;

		auto project = &projectTests.require(projectUri, UnittestProject(projectUri));
		rescanFile(*project, params.textDocument);

		rpc.notifyMethod("coded/pushProjectTests", *project);
	}, requiredLibdparsePageCount * 2);

but I can't find any calls to yield in rescanFile.

Does the delegate passed to spawnFiber need to care about it being interrupted? If so, how does it suspend (yield) itself and how does it signal to itself when file scanning is done? Do I simply Fiber.yield at the end of the foreach-source-file scope?

It would be very nice if serve-d contained such a code example.

[WebFreak001]

you should probably do a single scan on the @onProjectAvailable event and register a file watcher on the files where you want to rerun your function if they change, for an example see the profilegc implementation:

• registration:

  serve-d/source/served/workers/profilegc.d

  Lines 152 to 156 in a14aec8

  	@onProjectAvailable
  	void onProfileGCProjectAvailable(WorkspaceD.Instance instance, string dir, DocumentUri uri)
  	{
  	profileGCCache.update(uri.chomp("/") ~ "/profilegc.log");
  	}

  + this single line and its import in extension.d:

  serve-d/source/served/extension.d

  Line 1391 in a14aec8

  served.workers.profilegc,

• watching files:

  serve-d/source/served/workers/profilegc.d

  Lines 131 to 172 in a14aec8

  	@onRegisteredComponents
  	void setupProfileGCWatchers()
  	{
  	if (capabilities
  	.workspace.orDefault
  	.didChangeWatchedFiles.orDefault
  	.dynamicRegistration.orDefault)
  	{
  	if (!rpc.registerCapabilitySync(
  	"profilegc.watchfiles",
  	"workspace/didChangeWatchedFiles",
  	DidChangeWatchedFilesRegistrationOptions(
  	[
  	FileSystemWatcher("**/profilegc.log")
  	]
  	)
  	))
  	error("Failed to dynamically register GC watchers");
  	}
  	}
  	@onProjectAvailable
  	void onProfileGCProjectAvailable(WorkspaceD.Instance instance, string dir, DocumentUri uri)
  	{
  	profileGCCache.update(uri.chomp("/") ~ "/profilegc.log");
  	}
  	@protocolNotification("workspace/didChangeWatchedFiles")
  	void onChangeProfileGC(DidChangeWatchedFilesParams params)
  	{
  	foreach (change; params.changes)
  	{
  	if (!change.uri.endsWith("profilegc.log"))
  	continue;
  	if (change.type == FileChangeType.created
  	|| change.type == FileChangeType.changed)
  	profileGCCache.update(change.uri);
  	else if (change.type == FileChangeType.deleted)
  	profileGCCache.clear(change.uri);
  	}
  	}

Note that if you want to use DCD, D-Scanner, dfmt or other components that have setup code (which are loaded lazily and on-demand), you probably should not use onProjectAvailable, which is called immediately, but instead use onAddedProject, which is only called once any file within the project is actually opened in the editor or a global search or other function caused it to load.

Inside the spawnFiber you can just call Fiber.yield - if you call any rpc reading or sending methods, they yield internally.

This can all be self-contained in its own module without the need to put the code anywhere else. There are some other events that you can hook into using the UDAs, such as config updates or just registering raw LSP methods to implement functionality.

If you need to change capabilities on the initialize call / program startup, you can actually even create a method with the @onInitialize UDA and as parameter ref InitializeParams params and mutate it. However this is barely tested since I implemented this without ever using it outside a single test and you might just end up needing to just change the reply in the initialize method in extension.d (I'm not even sure if it was on master branch or on the eventcore integration branch that I never merged to master but should be almost fully implemented and should significantly reduce CPU and RAM usage)

The available serve-d specific UDA events are documented here:

serve-d/source/served/types.d

Lines 609 to 629 in a14aec8

	/// Event called when all components have been registered but no workspaces have
	/// been setup yet.
	/// Signature: `()`
	enum onRegisteredComponents;
	/// Event called when a project is available but not intended to be loaded yet.
	/// Should not access any components, otherwise it will force a load, but only
	/// show hints in the UI. When it's accessed and actually being loaded the
	/// events `onAddingProject` and `onAddedProject` will be emitted.
	/// Signature: `(WorkspaceD.Instance, string dir, DocumentUri uri)`
	enum onProjectAvailable;
	/// Event called when a new workspaced instance is created. Called before dub or
	/// fsworkspace is loaded.
	/// Signature: `(WorkspaceD.Instance, string dir, DocumentUri uri)`
	enum onAddingProject;
	/// Event called when a new project root is finished setting up. Called when all
	/// components are loaded. DCD is loaded but not yet started at this point.
	/// Signature: `(WorkspaceD.Instance, string dir, string rootFolderUri)`
	enum onAddedProject;

The general LSP/RPC events available in all LSPs using serve-d:lsp as a library are documented here:

serve-d/serverbase/source/served/utils/events.d

Lines 7 to 65 in a14aec8

	/// Called for requests (not notifications) from the client to the server. This
	/// UDA must be used at most once per method for regular methods. For methods
	/// returning arrays (T[]) it's possible to register multiple functions with the
	/// same method. In this case, if the client supports it, partial results will
	/// be sent for each returning method, meaning the results are streamed. In case
	/// the client does not support partial methods, all results will be
	/// concatenated together and returned as one.
	struct protocolMethod
	{
	string method;
	}
	/// Called after the @protocolMethod for this method is handled. May have as
	/// many handlers registered as needed. When the actual protocol method is a
	/// partial method (multiple handlers, returning array) this will be ran on each
	/// chunk returned by every handler. In that case the handler will be run
	/// multiple times on different fibers.
	struct postProtocolMethod
	{
	string method;
	}
	/// UDA to annotate a request or notification parameter with to supress linting
	/// warnings.
	enum nonStandard;
	struct protocolNotification
	{
	string method;
	}
	struct EventProcessorConfig
	{
	deprecated("This is no longer needed after upgrading `extension.modules` to `memberModules`")
	// string[] allowedDuplicateMethods = ["object"];
	string[] allowedDuplicateMethods = ["object", "served", "std", "core", "etc", "io", "workspaced", "fs"];
	}
	/// Hooks into initialization, possibly manipulating the InitializeResponse.
	/// Called after the extension entry point `initialize()` method, but before the
	/// initialize response was sent to the client.
	///
	/// If it's desired not to stall the initialization routine, use
	/// `@postProtocolMethod("initialized")` instead of these UDAs, which runs in a
	/// separate fiber after the response has been sent. Warning: other requests and
	/// notifications may have been called within this switching time window, so
	/// if these functions depend on what is being called in the initialize hook,
	/// they will break.
	///
	/// Annotated method is expected to have this type signature:
	/// ```d
	/// @initializeHook
	/// void myInitHook(InitializeParams params, ref InitializeResult result);
	/// @onInitialize
	/// void otherHook(InitializeParams params);
	/// ```
	enum initializeHook;
	/// ditto
	enum onInitialize;

[nordlow]

Inside the spawnFiber you can just call Fiber.yield - if you call any rpc reading or sending methods, they yield internally.

With this you really mean

"inside the delegate passed to spawnFiber you can just call Fiber.yield"

, right?

[WebFreak001]

yes