QCG-PilotJob

When an experiment requires the execution of numerous jobs, particularly those with interdependencies or dynamic characteristics, the limitations of mechanisms built into traditional queuing systems become evident. Simply put, schedulers often struggle to efficiently manage a large number of jobs, leading to common configurations that impose strict limits on the number of concurrent jobs a user can run.

These challenges can be effectively addressed with QCG-PilotJob, which introduces an additional layer of scheduling within the context of an already running job in the queuing system. This approach allows task management to occur within the user’s allocation, avoiding any impact on the performance of the central scheduler.

QCG-PilotJob offers several key advantages, including support for DAG workflows, dynamic job submission via API calls, automatic recovery of prematurely stopped workflows, and the ability to generate performance reports from executions—features that are valuable across a variety of computing scenarios.

QCG-Portal facilitates the use of QCG-PilotJob either as a standalone application or, more importantly, as a background execution engine tailored for specific scenarios, enhancing flexibility and efficiency.