HT Condor submission script for parallel running

FCCAnalysis
1

Hi,

I’m trying to run several background processes in parallel via HT condor, and I was wondering if there exists any good pre-written submission scripts where one easily can change the processList background either via the terminal directly or via some other variable?

If not, does anyone have any good ideas how to make one? I was playing around a bit with using sys.arguments to then write a simple bash script for this, but writing here first might be a good idea.

Thanks :slight_smile:

2

Hi!

Here is the snippet that writes the HTCondor submission script in FCCAnalyses: https://github.com/HEP-FCC/FCCAnalyses/blob/master/python/FCCAnalysisRun.py#L294-L318

And here is an FCCAnslyses example submitting jobs on HTCondor where the proceeList can easily be changed: https://github.com/HEP-FCC/FCCAnalyses/blob/master/examples/FCCee/higgs/mH-recoil/ee/analysis_stage1_batch.py#L2-L6

Does this help?

Cheers,
Brieuc

3

Hi @axgallen,

Are you trying to change processList in every submission?

At the moment FCCAnalyses does not support changing of processList from command line or environment variables.

One way to change processList at the moment could be to use sed to replace some predefined marker or to use some templating engine like jinja2.

Best,
Juraj

4

Hi! This I’m aware of, but I was hoping to be able to change the processList for every submission. As @jsmiesko mentioned below, apparently this isn’t supported, but can you maybe enlighten me on what chunks does? Does this run the different processes in processList in parallel via the same HTCondor submission?

5

Hi!

Ah, yes! One might be able to do this via some sed magic! Unfortunate to hear that processList can’t be changed via the command line at the moment, but I’ll find a workaround :slight_smile: Thanks for the help!

6

Hi @axgallen,

the question is how would you like this command line interface to look like, or what kind of changes you would like to make?

For example, you would like to change just number of chunks in one of the processes or to delete one complete process and add other one?

The chunks describe number of files the analysis stage will be split into when running at the HTCondor.

Best,
Juraj

7

Hi @jsmiesko,

Managed to create a bash script which uses the sed command to submit jobs from an array of given backgrounds, so all works fine now! Thanks for the help :slight_smile:

Cheers,
Axel