Hi,
I’m trying to run several background processes in parallel via HT condor, and I was wondering if there exists any good pre-written submission scripts where one easily can change the processList
background either via the terminal directly or via some other variable?
If not, does anyone have any good ideas how to make one? I was playing around a bit with using sys.arguments
to then write a simple bash script for this, but writing here first might be a good idea.
Thanks
Hi!
Here is the snippet that writes the HTCondor submission script in FCCAnalyses: https://github.com/HEP-FCC/FCCAnalyses/blob/master/python/FCCAnalysisRun.py#L294-L318
And here is an FCCAnslyses example submitting jobs on HTCondor where the proceeList can easily be changed: https://github.com/HEP-FCC/FCCAnalyses/blob/master/examples/FCCee/higgs/mH-recoil/ee/analysis_stage1_batch.py#L2-L6
Does this help?
Cheers,
Brieuc
Hi @axgallen,
Are you trying to change processList
in every submission?
At the moment FCCAnalyses does not support changing of processList
from command line or environment variables.
One way to change processList
at the moment could be to use sed
to replace some predefined marker or to use some templating engine like jinja2
.
Best,
Juraj
Hi! This I’m aware of, but I was hoping to be able to change the processList
for every submission. As @jsmiesko mentioned below, apparently this isn’t supported, but can you maybe enlighten me on what chunks
does? Does this run the different processes in processList
in parallel via the same HTCondor submission?
Hi!
Ah, yes! One might be able to do this via some sed
magic! Unfortunate to hear that processList
can’t be changed via the command line at the moment, but I’ll find a workaround Thanks for the help!
Hi @axgallen,
the question is how would you like this command line interface to look like, or what kind of changes you would like to make?
For example, you would like to change just number of chunks in one of the processes or to delete one complete process and add other one?
The chunks describe number of files the analysis stage will be split into when running at the HTCondor.
Best,
Juraj
Hi @jsmiesko,
Managed to create a bash script which uses the sed
command to submit jobs from an array of given backgrounds, so all works fine now! Thanks for the help
Cheers,
Axel