Archive experiments

[bpiwowar]

On some clusters, some partitions are faster... but time-limited. It would be good to have an archive option, where experiments and their jobs would be moved (when successful) to another location. Optionally, this folder can be used as a recovery when deleted from the main location.

In details:

1. Add folders in WorkspaceSettings with
   • a path attribute (so that we can add later others)
   • a backup attribute (false by default => when true, we copy after completion)
2. Whenever a job finishes, if backup is true, copy all the files except experimaestro related (except the dynamic outputs file which should be preserved). Have a submit option (backup=False) that prevents this behavior.
3. If at some point a job was deleted but can be recovered, copy it back from backup (and add .done to mark it as done when copy is over)
4. Add CLI options (e.g. orphans) to include backup (true by default); add also this to the TUI interface
5. We need to take care of paths: on copy, we make them relative to the destination folder (if they were relative path in the original workspace)

[VictorMorand]

This could be used when creating new experiments that may reuse results from other worskspaces.
This use-case is different from the original backup idea though, as we would not like the results to be copied back to original workspace.
-> we could fuse the two usages by allowing several backup_directories and adding tags setting the behavior (load only vs backup)

workspaces:
  - id: neuralir
    path: ~/experiments/xpmir
    triggers:
      - "neuralir-*"          

 - id: nlp
    path: ~/experiments/nlp
    triggers:
      - "nlp-*"
      - "transformer-*"
   backup_directories : 
     - path : "~/other/workspace/jobs"
        backup_to: false
     - path: "/path/to/slow/backup/dir"
       backup_to: true

[bpiwowar]

Modified the initial description; does it fullfill your needs?

This could be used when creating new experiments that may reuse results from other worskspaces. This use-case is different from the original backup idea though, as we would not like the results to be copied back to original workspace. -> we could fuse the two usages by allowing several backup_directories and adding tags setting the behavior (load only vs backup)

workspaces:
  - id: neuralir
    path: ~/experiments/xpmir
    triggers:
      - "neuralir-*"          

 - id: nlp
    path: ~/experiments/nlp
    triggers:
      - "nlp-*"
      - "transformer-*"
   backup_directories : 
     - path : "~/other/workspace/jobs"
        backup_to: false
     - path: "/path/to/slow/backup/dir"
       backup_to: true

[VictorMorand]

yes perfectly !

• one gotcha, when cleaning orphans, make sure not to include the folder if backup=false (when migrating from another xp, it would delete jobs that are not orphans in this context)