feat: Add the possibility to use the dirac-cwl project's job wrapper

[Loxeris]

Adds slights changes to create_job_wrapper() to use the experimental Job Wrapper for CWL jobs.
If the job executable is set to dirac-cwl-exec in the JDL, the new job wrapper is used, else the defaultWrapperLocation is used like before.
The dirac-cwl project is cloned locally on the pilot to get the latest version of the job wrapper before being run.

See DIRACGrid/dirac-cwl#44

BEGINRELEASENOTES

*WorkloadManagementSystem
NEW: Experimental job wrapper for cwl workflows

ENDRELEASENOTES

[aldbr]

Looks good to me!
Note for the other reviewer: the new job wrapper is experimental and is not expected to be executed at a large scale. It's only meant to be used for a few jobs in parallel for now.

@Loxeris have you tested it using the SingularityComputingElement?
The JobAgent can use the SingularituComputingElement instead of the InProcessComputingElement. In that case, the job wrapper is generated in a slightly different way:

DIRAC/src/DIRAC/Resources/Computing/SingularityComputingElement.py

Lines 211 to 222 in 11bc43d

	result = createJobWrapper(
	wrapperPath=tmpDir,
	rootLocation=self.__innerdir,
	jobID=jobDesc.get("jobID", 0),
	jobParams=jobDesc.get("jobParams", {}),
	resourceParams=jobDesc.get("resourceParams", {}),
	optimizerParams=jobDesc.get("optimizerParams", {}),
	pythonPath="python",
	log=log,
	logLevel=logLevel,
	extraOptions="" if self.__installDIRACInContainer else "/tmp/pilot.cfg",
	)

[Loxeris]

@Loxeris have you tested it using the SingularityComputingElement? The JobAgent can use the SingularituComputingElement instead of the InProcessComputingElement. In that case, the job wrapper is generated in a slightly different way

@aldbr I believe the computing element used by the CTAO test instance I did end-to-end tests on is SSHComputingElement. I didn't try using a different type of computing element.

[fstagni]

Just to say that that PR must first be tested using the DIRAC certification setup (instructions not written down yet, will be there for the hackathon next week)

[aldbr]

@Loxeris managed to test the PR in certification.
There are still missing pieces to properly report the status of the jobs (status is stuck in Matched), but this will be handled later in the dirac-cwl repo.
The pilot output reported the job as Done though.

[Loxeris]

It could be interesting to pass the JobID to the cwl Job Wrapper in this PR, as it seems to be needed for job reports and future development in the dirac-cwl repo.

[fstagni]

It could be interesting to pass the JobID to the cwl Job Wrapper in this PR, as it seems to be needed for job reports and future development in the dirac-cwl repo.

Please verify if that's actually needed, and open issue if it is. Merging this one now.