Issue running FCCAnalysis software on HT condor

williams · February 21, 2023, 3:43pm

Dear all,
I have been able to run some FCC analysis code locally, but when I try to run it on HT condor I get some unusual errors.
I have pasted them below…
Does anyone know how I could solve this?
Best wishes,
Sarah Williams

In case you’re interested in the error… I am getting this error in the HT condor error log:

File “/usr/sbin/make-jobfeatures”, line 77
os.umask(0022)
^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
In file included from libFCCAnalyses dictionary payload:39:
/afs/cern.ch/work/w/williams/public/FCC-LLP-tutorial-Winter23/FCCAnalyses/install/include/FCCAnalyses/VertexFinderLCFIPlus.h:114:19: error: no type named ‘FCCAnalysesV0’ in namespace ‘FCCAnalyses::VertexingUtils’
VertexingUtils::FCCAnalysesV0 get_V0s( ROOT::VecOps::RVecedm4hep::TrackState np_tracks,

/afs/[cern.ch/work/w/williams/public/FCC-LLP-tutorial-Winter23/FCCAnalyses/install/include/FCCAnalyses/VertexFinderLCFIPlus.h:122:19:](http://cern.ch/work/w/williams/public/FCC-LLP-tutorial-Winter23/FCCAnalyses/install/include/FCCAnalyses/VertexFinderLCFIPlus.h:122:19:) error: no type named 'FCCAnalysesV0' in namespace 'FCCAnalyses::VertexingUtils'
VertexingUtils::FCCAnalysesV0 get_V0s( ROOT::VecOps::RVec<edm4hep::TrackState> np_tracks,
~~~~~~~~~~~~~~~~^
/afs/[cern.ch/work/w/williams/public/FCC-LLP-tutorial-Winter23/FCCAnalyses/install/include/FCCAnalyses/VertexFinderLCFIPlus.h:132:19:](http://cern.ch/work/w/williams/public/FCC-LLP-tutorial-Winter23/FCCAnalyses/install/include/FCCAnalyses/VertexFinderLCFIPlus.h:132:19:) error: no type named 'FCCAnalysesV0' in namespace 'FCCAnalyses::VertexingUtils'
VertexingUtils::FCCAnalysesV0 get_V0s_jet( ROOT::VecOps::RVec<edm4hep::ReconstructedParticleData> recoparticles,
~~~~~~~~~~~~~~~~^
python3: /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.26.06-bc7bv5ucu42pjp34ex4atpgzy4fkb3lw/spack-src/interpreter/llvm/src/tools/clang/lib/AST/DeclCXX.cpp:1391: clang::CXXMethodDecl* clang::CXXRecordDecl::getLambdaCallOperator() const: Assertion `!Calls.empty() && "Missing lambda call operator!"' failed.
*** Break *** abort

eperez · February 21, 2023, 8:26pm

Hi Sarah,

You are probably running some locally compiled code (from the LLP fork of FCCAnalyses ?) when you run locally. While when you run with the batch option, the standard FCCAnalyses library is used maybe (I’m not fully sure here, but that would explain what you see).

The code that you are using (with FCCAnalysesV0) has been merged into the central FCCAnalyses repository yesterday only. And I see that
/cvmfs/fcc.cern.ch/sw/latest/setup.sh
takes a release from 2023-01-15. Hence the central FCCAnalyses does not know indeed about FCCAnalysesV0.

If you are indeed using the central library when you run on the batch, I think you just have to wait for a new key4hep stack to be built and installed, that takes the last version of FCCAnalyses (assuming that the local fork did not contain any other functionality not integrated in the central repository).

If that is not the reason of the problem: a full A-to-Z recipe to reproduce the problem will certainly be needed for someone to take a look.

E.

brfranco · February 22, 2023, 9:07am

Hello Sarah,

Thanks for your post. Can you confirm that you correctly source the setup.sh from your local FCCAnalyses build at the beginning of the shell script which is ran on condor nodes?

Cheers,
Brieuc

williams · February 23, 2023, 12:48pm

Hi all,
Thanks for the input here- this is interesting and seems consistent…
Do you know if there’s a way to get around this for now?
Let me discuss if I can get others to check before coming back to you…
Essentially I am running instructions here:

github.com

jalimena/LLPFCCTutorial/blob/main/README.md

# LLP FCC Tutorial
Tutorial for getting started studying long-lived particles (LLPs) at the FCC-ee

## Who this tutorial is for

This tutorial is for anyone interested in working with the FCC software. It assumes some basic knowledge of ROOT, C++, linux, and (very basic knowledge of) git. It should be appropriate for beginner Masters and PhD students.

## What this tutorial will teach you

- How to run the [FCCAnalyses software](https://github.com/HEP-FCC/FCCAnalyses), using long-lived Heavy Neutral Leptons (HNLs) that decay to electrons and neutrinos as an example
- In particular, this tutorial is set up to walk you through the major analysis steps, running local jobs on lxplus.

## What this tutorial will NOT cover
- How to generate samples and run pythia8 and Delphes. If you want to do that, you can look [here](https://github.com/jalimena/FCCeePhysicsPerformance/tree/master/case-studies/BSM/LLP/DisplacedHNL/HNL_sample_creation)
- How to run batch jobs and put the output on eos on lxplus. (However, the code to do this is available in the tutorial files. It might require some additional permissions, though.)
- How to run the code on the Fermilab LPC (because the input samples are not there. In principle you can run at the LPC, but this has not been tested.)


## Step 0. Setup

This file has been truncated. show original

But get stuck running analysis_stage1.py on HT condor.
Any pointers would be greatly appreciated.
Sarah

williams · March 1, 2023, 8:15am

Hi @brfranco and @eperez ,
I am just coming back to this as I believe @jalimena and Yuliia Borysenkova have been experiencing the same problem.

I checked and my local version of setup.sh points to the same release.
Do any of you have any idea either how quickly the next version will be built, or if there’s a workaround here?
Best wishes,

Sarah

lportale · March 1, 2023, 8:43am

Hi @williams,

I came across a similar issue, and one quick fix that I found was to source my local setup script in config/FCCAnalysisRun.py instead of the default /cvmfs/sw.hsf.org/key4hep/setup.sh. This is probably not the best “long-term” solution, but maybe it can help in your case too.

Best,
Louis

jsmiesko · March 2, 2023, 8:12am

Hello @williams,

The following PR should fix the issue:

Thanks @lportale for the hint

Best,
Juraj

williams · March 20, 2023, 1:19pm

Hi @jsmiesko and @lportale ,
Thanks for the input here, and sorry this is likely going to be a stupid question. My local setup script is still pointing to the 15th January one…
can I check what I should do with the PR below? Should I be pulling it into my area? (and potentially stupid question… how?)
Help getting this working is much appreciated, as its becoming increasingly time-consuming not being able to run things on condor…
Sarah

jsmiesko · March 20, 2023, 1:49pm

Hello @williams,

I’m not entirely sure how you have set up your local copy of the FCCAnalyses. Can you post output of the command

git remote -v

and

git branch

?

Best,
Juraj

williams · March 20, 2023, 4:37pm

Hi @jsmiesko,
Thanks for this,

Here are the outputs:
[williams@lxplus756>/afs/cern.ch/work/w/williams/public/FCC-LLP-tutorial-Winter23/FCCAnalyses>]$ git remote -v
origin GitHub - FCC-LLP/FCCAnalyses: FCC Physics Analyses using RDataFrame (fetch)
origin GitHub - FCC-LLP/FCCAnalyses: FCC Physics Analyses using RDataFrame (push)
[williams@lxplus756>/afs/cern.ch/work/w/williams/public/FCC-LLP-tutorial-Winter23/FCCAnalyses>]$ git branch

master

Does this help?
Sarah

jsmiesko · March 21, 2023, 9:00am

Hello @williams,

Looking at the state of your fork of the FCCAnalyses, there are quite a few commits missing

We need to somehow add them to your fork, can you try merging the upstream version of FCCAnalyses?

Remaining on master branch try:

git remote add upstream https://github.com/HEP-FCC/FCCAnalyses.git
git fetch upstream
git merge upstream/master

Best,
Juraj

williams · March 21, 2023, 2:02pm

Hi all,
@jalimena very kindly merged these changes into the FCC-LLP repo that we were running off, and I have now been able to run on HTcondor without the same error as before.
It seems to have ran ok, but I am still getting some errors…
File “/usr/sbin/make-jobfeatures”, line 77
os.umask(0022)
^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
If you want to look at the full outputs they are here:
/afs/cern.ch/work/w/williams/public/FCC-LLP-tutorial-Winter23/FCCAnalyses/BatchOutputs/2023-03-21_14-24-03/p8_ee_Zee_ecm91/
Thanks for any input,
Sarah

jsmiesko · March 27, 2023, 11:48am

Hello @williams, @jalimena,

the problem occurs because there is no isolation between environments of the batch machine and machine where the fccanalyses commands are issued.

The following PR should fix the issue:

Best,
Juraj

Issue running FCCAnalysis software on HT condor

Dear all, I have been able to run some FCC analysis code locally, but when I try to run it on HT condor I get some unusual errors. I have pasted them below… Does anyone know how I could solve this? Best wishes, Sarah Williams

Dear all,
I have been able to run some FCC analysis code locally, but when I try to run it on HT condor I get some unusual errors.
I have pasted them below…
Does anyone know how I could solve this?
Best wishes,
Sarah Williams