(workflows_tags)= # Autonomous Pipelines (jqmc_workflow) ## Overview The **jqmc_workflow** package provides an autonomous pipeline engine for **jQMC** calculations. Users define a *pipeline* — a directed acyclic graph (DAG) of workflow steps — and the engine takes care of: - **Input generation** — TOML input files are created automatically from explicit parameter values (with sensible defaults). - **Data transfer** — Files are uploaded to (and downloaded from) remote supercomputers via SSH/SFTP using [Paramiko](https://www.paramiko.org/). - **Job submission, monitoring, and collection** — Jobs are submitted through the site's scheduler (PBS / Slurm / local `bash`), polled until completion, and output files are fetched back. - **Dependency resolution** — The DAG-based `Launcher` identifies which workflow steps are *ready* (all dependencies satisfied) and executes them **in parallel** using Python `asyncio` tasks. - **Target error-bar estimation** — When a `target_error` (Ha) is specified, a small *pilot* run is executed first; the statistical error is used to estimate the number of production steps required, together with the estimated wall time. A single Python script can therefore express a full QMC pipeline — from wavefunction preparation (TREXIO → `hamiltonian_data.h5`) through VMC optimization, MCMC production sampling, and LRDMC extrapolation — and run it end-to-end with automatic restarts across interruptions. ## Architecture ```text run_pipeline.py │ ▼ ┌────────┐ │Launcher│ DAG executor (asyncio) └──┬─────┘ │ creates asyncio.Task per ready node │ ├──► Container("vmc") │ └─► VMC_Workflow.async_launch() │ ├──► Container("mcmc-prod") ← runs in parallel │ └─► MCMC_Workflow.async_launch() │ └──► Container("lrdmc-ext") ← runs in parallel └─► LRDMC_Ext_Workflow.async_launch() ├─► LRDMC_Workflow (alat=0.50) ┐ ├─► LRDMC_Workflow (alat=0.40) ├ parallel └─► LRDMC_Workflow (alat=0.25) ┘ ``` ### Key components | Class | Role | |---|---| | `Launcher` | Executes a DAG of `Container` nodes in topological order, launching independent nodes in parallel. | | `Container` | Wraps any `Workflow` subclass in a dedicated project directory with state tracking (`workflow_state.toml`). | | `FileFrom` / `ValueFrom` | Declare inter-workflow dependencies (files or computed values). | | `WF_Workflow` | Converts a TREXIO file to `hamiltonian_data.h5`. | | `VMC_Workflow` | Jastrow / orbital optimization (`job_type=vmc`). | | `MCMC_Workflow` | Production energy sampling (`job_type=mcmc`). | | `LRDMC_Workflow` | Lattice-regularized diffusion Monte Carlo for a single $a$ value. | | `LRDMC_Ext_Workflow` | Runs multiple `LRDMC_Workflow` instances at different lattice spacings and performs $a^2 \to 0$ extrapolation. | ### Target error-bar estimation When `target_error` is set, each workflow (MCMC and LRDMC) operates in a pilot + production cycle. A separate queue may be used for the pilot via `pilot_queue_label` (defaults to `queue_label`). #### MCMC / VMC 1. **Pilot** — A short run of `pilot_steps` steps. 2. **Production** — Step count estimated from the pilot error bar. #### LRDMC LRDMC has an additional calibration stage to automatically determine `num_mcmc_per_measurement` (GFMC projections per measurement) from a `target_survived_walkers_ratio` (default 0.97): 1. **Calibration** (`_pilot_a/_pilot1` – `_pilot_a/_pilot3`, parallel) — Three short LRDMC runs with `num_mcmc_per_measurement = Ne × k × (0.3/alat)²` (k=2,4,6; $N_e$ is the total electron count). A quadratic is fit to the observed survived-walkers ratio and the optimal `num_mcmc_per_measurement` is determined. 2. **Error-bar pilot** (`_pilot_b`) — A run with the calibrated `num_mcmc_per_measurement`; its error bar estimates the production step count. 3. **Production** (`_1`, `_2`, …) — Start from scratch, accumulate statistics until `target_error` is achieved. If `num_mcmc_per_measurement` is given explicitly, the calibration stage is skipped and only the error-bar pilot is executed. For `LRDMC_Ext_Workflow` (multi-alat extrapolation), every alat value independently runs its own calibration, error-bar pilot, and production in parallel. There is no inter-alat interaction until the final extrapolation step. #### Step estimation formula The required number of production steps is estimated via $$ N_{\text{eff,prod}} = N_{\text{eff,pilot}} \times \left(\frac{\sigma_{\text{pilot}}}{\sigma_{\text{target}}}\right)^2 \times \frac{W_{\text{pilot}}}{W_{\text{prod}}} $$ where $N_{\text{eff}} = N_{\text{total}} - N_{\text{warmup}}$ is the number of steps that actually contribute to statistics (warmup steps are discarded during post-processing via `-w`), and $W = \text{walkers\_per\_MPI} \times \text{num\_MPI}$ is the total number of walkers. The ratio $W_{\text{pilot}} / W_{\text{prod}}$ (the *walker ratio*) accounts for the pilot queue using fewer (or more) MPI processes than the production queue. The number of MPI processes is read from `queue_data.toml` (`num_cores`, or `mpi_per_node × nodes` as fallback). The total production steps are then $N_{\text{prod}} = N_{\text{eff,prod}} + N_{\text{warmup}}$. The estimated production wall time is based on the **net** computation time parsed from the pilot output (``Net GFMC time`` for LRDMC, ``Net total time for MCMC`` for MCMC/VMC). Only this net portion scales with the step count; overhead (JIT compilation, file I/O, queue wait) is treated as a constant: $$ T_{\text{prod}} \approx (T_{\text{wall,pilot}} - T_{\text{net,pilot}}) + T_{\text{net,pilot}} \times \frac{N_{\text{prod}}}{N_{\text{pilot}}} $$ ```text -- LRDMC Step Estimation Summary (a=0.25) -------- pilot steps = 100 warmup steps = 10 pilot error = 0.00393172 Ha target error = 0.001 Ha nmpm = 32 pilot MPI procs = 48 prod. MPI procs = 480 walker ratio = 0.1 estimated steps = 155 pilot wall time = 5m 12s pilot net time = 2m 42s est. prod. time = 6m 43s -------------------------------------------------- ``` The first production run starts from scratch (no restart from the pilot checkpoint, which lives in the ``_pilot_b/`` subdirectory). After each production run, the error bar is re-evaluated. If it exceeds the target, additional steps are estimated and a continuation run is launched automatically, up to `max_continuation` times. ### Restart behavior Every job is recorded in `workflow_state.toml` with a lifecycle: ```text submitted → completed → fetched ``` On restart, the engine checks each job's status: - **`fetched`** — Input generation *and* submission are both skipped. - **`submitted`** / **`completed`** — Input is not regenerated; the job is resumed (polled or fetched). - **No record** — A fresh input file is generated and the job is submitted. This means a pipeline can be interrupted at any point (Ctrl-C, node failure, wall-time limit) and simply re-run; it will pick up exactly where it left off. ## Configuration Configuration files are managed in the following directory hierarchy. The engine looks for a project-local override first, then falls back to the user-global directory: 1. `./jqmc_setting_local/` — project-local override (if it exists in CWD) 2. `~/.jqmc_setting/` — user-global default On the very first run, if neither directory exists, the template shipped with the package is copied to `~/.jqmc_setting/` and the user is asked to edit it. ### Directory structure ```text ~/.jqmc_setting/ ├── machine_data.yaml # Server machine definitions ├── localhost/ # Settings for localhost │ ├── queue_data.toml │ └── submit_mpi.sh # (name is user-defined) ├── my-cluster/ # Settings for a remote cluster (nickname) │ ├── queue_data.toml │ └── submit_mpi.sh └── ... ``` ### `machine_data.yaml` A YAML file that defines each compute machine. The top-level keys are nicknames (arbitrary labels); they do **not** need to match the SSH host. Remote machines use the `ssh_host` field to specify the SSH connection target. #### Example ```yaml localhost: machine_type: local queuing: false workspace_root: /home/user/jqmc_work my-cluster: # nickname (freely chosen) ssh_host: pbs-cluster # Host alias in ~/.ssh/config machine_type: remote queuing: true workspace_root: /home/user/jqmc_work jobsubmit: /opt/pbs/bin/qsub jobcheck: /opt/pbs/bin/qstat jobdel: /opt/pbs/bin/qdel jobnum_index: 0 ``` #### Parameters | Key | Type | Required | Description | |---|---|---|---| | `ssh_host` | String | When `machine_type: remote` | SSH connection target. Typically a `Host` alias defined in `~/.ssh/config`. | | `machine_type` | `"local"` or `"remote"` | Yes | `local`: execute on the same host. `remote`: connect via SSH (Paramiko). | | `queuing` | Boolean | Yes | `true`: use a batch scheduler. `false`: direct execution. | | `workspace_root` | String (path) | Yes | Root directory for file management. Upload/download paths are resolved relative to this directory. Must be set on **both** localhost and the remote server. | | `jobsubmit` | String (command) | When `queuing: true` | Command to submit a job (e.g. `qsub`, `sbatch`, `pjsub`). Not needed for `queuing: false`. | | `jobcheck` | String (command) | When `queuing: true` | Command to check job status (e.g. `qstat`, `squeue --noheader`, `pjstat`). | | `jobdel` | String (command) | When `queuing: true` | Command to cancel a job (e.g. `qdel`, `scancel`, `pjdel`). | | `jobnum_index` | Integer | When `queuing: true` | 0-based column index of the job ID in the output of `jobsubmit`. For PBS (`42.server`), use `0`. For Slurm (`Submitted batch job 42`), use `3`. | | `ip` | String | No | IP address or hostname of the remote machine. Usually the SSH alias in `~/.ssh/config` is used instead. | #### SSH connection For `machine_type: remote`, the `ssh_host` field specifies the SSH connection target. This is typically a `Host` alias defined in `~/.ssh/config`. The engine reads `~/.ssh/config` via Paramiko to resolve `HostName`, `User`, `IdentityFile`, `Port`, and `ProxyJump` (multi-hop) settings. Your SSH key must be accessible without a passphrase prompt (use `ssh-agent` or a passphrase-less key). > **Note:** `CanonicalizeHostname` in `~/.ssh/config` may trigger a > Paramiko bug. > The engine automatically works around this by removing the directive > in-memory before connecting. ### `queue_data.toml` Defines batch queue presets for each machine. Written in TOML format with queue labels as table keys. #### Example ```toml [default] submit_template = 'submit_mpi.sh' max_job_submit = 10 queue = 'batch' num_cores = 120 omp_num_threads = 1 nodes = 1 mpi_per_node = 120 max_time = '24:00:00' [large] submit_template = 'submit_mpi.sh' max_job_submit = 10 queue = 'batch' num_cores = 480 omp_num_threads = 1 nodes = 4 mpi_per_node = 120 max_time = '24:00:00' ``` #### Parameters | Key | Type | Required | Description | |---|---|---|---| | `submit_template` | String | Yes | Name of the job submission script template file placed in the same directory (e.g. `"submit_mpi.sh"`, `"submit_gpu.sh"`). | | `max_job_submit` | Integer | Yes | Maximum number of concurrently submitted jobs for this queue. | | *custom keys* | Any | No | Any additional key-value pairs. These are substituted into job script templates as `_KEY_` (upper-case). Common keys: `num_cores`, `omp_num_threads`, `nodes`, `mpi_per_node`, `max_time`, `queue`, `account`, `partition`. | #### TOML format notes - Strings must be quoted: `queue = "small"`. - Booleans: lowercase `true` / `false` only. - Time values (e.g. `max_time`) **must** be quoted to avoid TOML local-time parsing: `max_time = "24:00:00"`. ### Job script templates Shell scripts placed in the per-machine directory, referenced by the `submit_template` key in `queue_data.toml`. Template variables (written as `_KEY_` with underscores) are replaced at submission time. You can name the file anything you like (e.g. `submit_gpu.sh`). #### Predefined variables | Variable | Description | |---|---| | `_INPUT_` | Path to the input file | | `_OUTPUT_` | Path to the output file | | `_JOBNAME_` | Job name | #### Custom variables All keys from `queue_data.toml` are available in upper-case with surrounding underscores. For example, `num_cores = 48` becomes `_NUM_CORES_` in the template. #### Example (`submit_mpi.sh` for PBS) ```bash #!/bin/sh #PBS -N _JOBNAME_ #PBS -q _QUEUE_ #PBS -l nodes=_NODES_:ppn=_MPI_PER_NODE_ #PBS -l walltime=_MAX_TIME_ export OMP_NUM_THREADS=_OMP_NUM_THREADS_ INPUT=_INPUT_ OUTPUT=_OUTPUT_ mpirun -np _NUM_CORES_ jqmc ${INPUT} > ${OUTPUT} 2>&1 ``` #### Example (`submit_serial.sh`) ```bash #!/bin/sh #PBS -N _JOBNAME_ #PBS -q _QUEUE_ #PBS -l nodes=_NODES_ #PBS -l walltime=_MAX_TIME_ export OMP_NUM_THREADS=_OMP_NUM_THREADS_ INPUT=_INPUT_ OUTPUT=_OUTPUT_ jqmc ${INPUT} > ${OUTPUT} 2>&1 ``` ## Pipeline example A minimal pipeline script that runs VMC → MCMC + LRDMC extrapolation: ```python from jqmc_workflow import ( Container, FileFrom, Launcher, LRDMC_Ext_Workflow, MCMC_Workflow, ValueFrom, VMC_Workflow, ) server = "pbs-cluster" h5 = "hamiltonian_data.h5" vmc = Container( label="vmc", dirname="vmc", input_files=[h5], workflow=VMC_Workflow( server_machine_name=server, hamiltonian_file=h5, queue_label="default", pilot_queue_label="small", jobname="vmc", target_error=0.001, ), ) mcmc = Container( label="mcmc-prod", dirname="mcmc_prod", input_files=[ h5, FileFrom("vmc", "hamiltonian_data_opt_step_*.h5"), ], workflow=MCMC_Workflow( server_machine_name=server, hamiltonian_file=h5, queue_label="default", pilot_queue_label="small", jobname="mcmc", target_error=0.001, ), ) lrdmc = Container( label="lrdmc-ext", dirname="lrdmc_ext", input_files=[ h5, FileFrom("vmc", "hamiltonian_data_opt_step_*.h5"), ], workflow=LRDMC_Ext_Workflow( server_machine_name=server, alat_list=[0.10, 0.20, 0.25, 0.30], hamiltonian_file=h5, queue_label="default", pilot_queue_label="small", jobname_prefix="lrdmc", target_survived_walkers_ratio=0.97, E_scf=ValueFrom("mcmc-prod", "energy"), target_error=0.001, ), ) pipeline = Launcher(workflows=[vmc, mcmc, lrdmc]) pipeline.launch() ``` In this example, `mcmc-prod` and `lrdmc-ext` depend on `vmc` (via `FileFrom`). Additionally, `lrdmc-ext` depends on `mcmc-prod` (via `ValueFrom` for `E_scf`), so the DAG becomes VMC → MCMC → LRDMC-ext. The `target_survived_walkers_ratio` triggers automatic calibration of `num_mcmc_per_measurement` independently at each lattice spacing. All alat values run their calibration, error-bar pilot, and production phases in parallel. ## Job Manager CLI (`jqmc-jobmanager`) The `jqmc-jobmanager` command-line tool monitors and manages running pipelines. It recursively discovers `workflow_state.toml` files under the current directory and displays a summary tree. ### Commands | Command | Description | |---|---| | `show` | Print the workflow tree. Add `--id N` to display full detail for one job. | | `check` | Print the tree **and** query the scheduler on the remote machine for live job status. Use `--id N` to target a specific job. | | `del` | Cancel a queued/running job and mark the workflow as `cancelled`. Requires `--id N`. | ### Common options | Option | Description | |---|---| | `--id N` | Numeric job ID shown in the tree (0-based). | | `-s` / `--server` | Server machine name (defaults to `localhost`). Used by `check` and `del` to connect to the remote scheduler. | | `--log-level` | `INFO` (default) or `DEBUG`. | ### Usage examples Show the full workflow tree: ```bash jqmc-jobmanager show ``` Show tree and detailed info for job 2: ```bash jqmc-jobmanager show --id 2 ``` Check live queue status for job 4 on `pbs-cluster`: ```bash jqmc-jobmanager check --id 4 -s pbs-cluster ``` Cancel job 4 on `pbs-cluster`: ```bash jqmc-jobmanager del --id 4 -s pbs-cluster ``` ### Example output ```text ============================================================================== Workflow Job Tree Root: /home/user/project ============================================================================== ID Status Label Type Server Job# -------------------------------------------------------------------------- 0 completed wf WF_Workflow ? - dir: 00_wf 1 completed vmc VMC_Workflow pbs-cluster 12345 dir: 01_vmc energy: -17.17 +- 0.005 Ha 2 running mcmc-prod MCMC_Workflow pbs-cluster 12367 dir: 02_mcmc 3 running lrdmc-ext LRDMC_Ext_Workflow ? - dir: 03_lrdmc_ext ``` For the full API reference, see {ref}`API reference for the pipeline (jqmc_workflow) `.