Load libFCCAnalysesFlavour package when running in batch

jalimena · April 7, 2022, 8:59am

Hi,

I am trying to run my analysis code in batch mode. I can run it fine locally, but when I try it with condor, I get this error:

Warning in <TInterpreter::ReadRootmapFile>: class  edm4hep::ObjectID found in libedm4hepDict.so  is already in libedm4drDict.so 
Error in <TUnixSystem::FindDynamicLibrary>: libFCCAnalysesFlavour[.so | .dll | .dylib | .sl | .dl | .a] does not exist in /afs/cern.ch/user/h/helsens/FCCsoft/HEP-FCC/FCCeePhysicsPerformance/case-
studies/flavour/dataframe/install/lib:/cvmfs/sw.hsf.org/spackages5/py-awkward/1.4.0/x86_64-centos7-gcc11.2.0-opt/ee337/lib/python3.9/site-packages/awkward:/afs/cern.ch/user/h/helsens/FCCsoft/HEP-
FCC/FCCAnalyses/install/lib:/cvmfs/sw.hsf.org/spackages5/key4hep-stack/2022-03-30/x86_64-centos7-gcc11.2.0-opt/mrwoj/lib64:/cvmfs/sw.hsf.org/spackages5/key4hep-stack/2022-03-30/x86_64-centos7-gcc
11.2.0-opt/mrwoj/lib:/cvmfs/sw.hsf.org/spackages5/xgboost/1.5.2/x86_64-centos7-gcc11.2.0-opt/6ptqk/lib64:
... 
#not copying the whole path, but let me know if you need it
Traceback (most recent call last):
  File "/afs/cern.ch/work/j/jalimena/FCCeeLLP/FCCeePhysicsPerformance/case-studies/BSM/LLP/DisplacedHNL/analysis_general.py", line 21, in <module>
    _HNL   = ROOT.dummyLoaderFlavour #### Needed to fix undeclared selMC_leg()
  File "/cvmfs/sw.hsf.org/spackages5/root/6.26.00/x86_64-centos7-gcc11.2.0-opt/jx56q/lib/ROOT/_facade.py", line 195, in _fallback_getattr
    raise AttributeError("Failed to get attribute {} from ROOT".format(name))
AttributeError: Failed to get attribute dummyLoaderFlavour from ROOT

It seems to me that condor can’t find the libFCCAnalysesFlavour package, but when I run locally, it can find it just fine. How do I tell condor to load this package? Is there something I can do in my batch submission script?

My batch submission script is here:

github.com

jalimena/FCCeePhysicsPerformance/blob/master/case-studies/BSM/LLP/DisplacedHNL/runAnalysis_general_batch.py

#run this with:
#python runAnalysis_general_batch.py

from config.common_defaults import deffccdicts
import config.runDataFrameBatch as rdf
import os

basedir=os.path.join(os.getenv('FCCDICTSDIR', deffccdicts), '') + "yaml/FCCee/spring2021/IDEA/"
outdir="/afs/cern.ch/work/j/jalimena/FCCeeLLP/FCCeePhysicsPerformance/case-studies/BSM/LLP/DisplacedHNL/Batch_Analysis_general/"

NUM_CPUS=8
output_list=[]
fraction=1.

inputana="/afs/cern.ch/work/j/jalimena/FCCeeLLP/FCCeePhysicsPerformance/case-studies/BSM/LLP/DisplacedHNL/analysis_general.py"

process_list=['p8_ee_Zee_ecm91',
              #'p8_ee_Zbb_ecm91',
              #'p8_ee_Ztautau_ecm91',
              #'p8_ee_Zuds_ecm91',

This file has been truncated. show original

And my analysis code is here:

github.com

jalimena/FCCeePhysicsPerformance/blob/master/case-studies/BSM/LLP/DisplacedHNL/analysis_general.py

# This is a basic example showing how to read different objects like electrons, jets, ETmiss etc. from the EDM4HEP files 
# and how to access and store some simple variables in an output ntuple.
# It has been edited in order to accomodate studies of HNLs using the FCC framework

import ROOT
import os
import argparse


### TODO: see if can be simplified/improved #####
#setup of the libraries, following the example:
print ("Load cxx analyzers ... ",)
ROOT.gSystem.Load("libedm4hep")
ROOT.gSystem.Load("libpodio")
ROOT.gSystem.Load("libFCCAnalyses")
ROOT.gSystem.Load("libFCCAnalysesFlavour")
ROOT.gErrorIgnoreLevel = ROOT.kFatal
_edm  = ROOT.edm4hep.ReconstructedParticleData()
_pod  = ROOT.podio.ObjectID()
_fcc  = ROOT.dummyLoader

This file has been truncated. show original

Thanks for any suggestions.

Juliette

clement.helsens · April 7, 2022, 10:51am

hello @jalimena ,

this is very timely as I have just re-written all the procedure. For this you will have to use my branch and this will help me if you can give it a try:

Please note that the preSel.py has gone and things have been re-arranged for ease of use (at least I believe that is the case )

I have started to change the instructions, but they are not yet finalised, but they should help you getting started.
Please try to run locally some test samples before going on batch. In principle the ENV is sent to batch, thus if the PATH are locally working fine, they should also with batch (I tested this with my local FCCAnalyses, but not including an extra package as you do, thus I would be interested to see if it works out of the box)
Clement

jalimena · April 7, 2022, 11:00am

Ok thanks, @clement.helsens ! So do you suggest I check out your branch of FCCAnalyses, link my FCCeePhysicsPerformance to it with

cmake .. -DCMAKE_INSTALL_PREFIX=../install -DFCCANALYSES_INCLUDE_PATH=<my_Complete_Path_to>/FCCAnalyses/install/include/FCCAnalyses/

and try to run my scripts locally and if that works, in batch as well? Just to make sure that’s the workflow you suggest I try? thanks again!

clement.helsens · April 7, 2022, 11:03am

yes, this is correct.
Please let me know (maybe on mattermost as it has nothing to do with this post) if instructions are understandable

EDIT:
I replied too quickly, what you should do in addition to compile FCCeePP linking the new FCCAnalyses is to make sure you have the ENV var well setup like here:

github.com

HEP-FCC/FCCeePhysicsPerformance/blob/master/case-studies/flavour/dataframe/localSetup.sh

export LD_LIBRARY_PATH=$PWD/install/lib:$LD_LIBRARY_PATH
export ROOT_INCLUDE_PATH=$PWD/install/include/FCCAnalysesFlavour:$ROOT_INCLUDE_PATH

jalimena · April 7, 2022, 11:11am

ah ha, I had

export ROOT_INCLUDE_PATH=$PWD/install/include/FCCAnalyses:$ROOT_INCLUDE_PATH

instead, maybe that was my mistake, let’s see

jalimena · April 7, 2022, 3:44pm

hi @clement.helsens , I think you have a mistake in your

github.com

clementhelsens/FCCAnalyses/blob/batch/examples/FCCee/higgs/mH-recoil/mumu/analysis_stage1_batch.py

#Mandatory: List of processes
processList = {
    'p8_ee_ZZ_ecm240':{'chunks':20},#Run the full statistics in 10 jobs in output dir <outputDir>/p8_ee_ZZ_ecm240/chunk<N>.root
    'p8_ee_WW_ecm240':{'chunks':20},#Run the full statistics in 10 jobs in output dir <outputDir>/p8_ee_WW_ecm240/chunk<N>.root
    'p8_ee_ZH_ecm240':{'chunks':20} #Run the full statistics in 10 jobs in output dir <outputDir>/p8_ee_ZH_ecm240/chunk<N>.root
}

#Mandatory: Production tag when running over EDM4Hep centrally produced events, this points to the yaml files for getting sample statistics
prodTag     = "FCCee/spring2021/IDEA/"

#Optional: output directory, default is local dir
outputDir   = "ZH_mumu_recoil_batch/stage1"

#Optional: ncpus, default is 4
nCPUS       = 4

#Optional running on HTCondor, default is False
runBatch    = True

#Optional batch queue name when running on HTCondor, default is workday

This file has been truncated. show original

I try/get:

$ cd FCCAnalyses/examples/FCCee/higgs/mH-recoil/mumu
$ python /afs/cern.ch/work/j/jalimena/FCCeeLLP/FCCAnalyses/config/FCCAnalysisRun.py  analysis_stage1_batch.py 
Warning in <TInterpreter::ReadRootmapFile>: class  edm4hep::ObjectID found in libedm4hepDict.so  is already in libedm4drDict.so 
----> Load cxx analyzers from libFCCAnalyses... 
----> yaml file /afs/cern.ch/work/h/helsens/public/FCCDicts/yaml/FCCee/spring2021/IDEA/p8_ee_ZZ_ecm240/merge.yaml succesfully opened
----> Running process p8_ee_ZZ_ecm240 with fraction=1, output=p8_ee_ZZ_ecm240, chunks=20
----> Running on Batch
Traceback (most recent call last):
  File "/afs/cern.ch/work/j/jalimena/FCCeeLLP/FCCAnalyses/config/FCCAnalysisRun.py", line 463, in <module>
    sendToBatch(foo, chunkList, process, analysisFile)
  File "/afs/cern.ch/work/j/jalimena/FCCeeLLP/FCCAnalyses/config/FCCAnalysisRun.py", line 230, in sendToBatch
    localDir = os.environ["LOCAL_DIR"]
  File "/cvmfs/sw.hsf.org/spackages5/python/3.9.10/x86_64-centos7-gcc11.2.0-opt/7j5vq/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'LOCAL_DIR'

I guess you meant to define a localDir instead of an outputDir in this file?

clement.helsens · April 7, 2022, 3:49pm

hello @jalimena

no, this this an ENVVAR, see:

github.com

clementhelsens/FCCAnalyses/blob/batch/setup.sh#L6

      
        
            #!/bin/bash
            source /cvmfs/fcc.cern.ch/sw/latest/setup.sh
            export PYTHONPATH=$PWD:$PYTHONPATH
            export LD_LIBRARY_PATH=$PWD/install/lib:$LD_LIBRARY_PATH
            export ROOT_INCLUDE_PATH=$PWD/install/include/FCCAnalyses:$ROOT_INCLUDE_PATH
            export LOCAL_DIR=$PWD
            export LD_LIBRARY_PATH=`python -m awkward.config --libdir`:$LD_LIBRARY_PATH

jalimena · April 7, 2022, 4:01pm

whoops yes i just saw this, my bad. your instructions are clear, I was just going too fast and missed this…

I was able to successfully convert my analysis.py into the new RDFanalysis version, ran it locally, and submitted it to batch. Batch jobs are idle now, I’ll let you know how it goes!

clement.helsens · April 8, 2022, 12:42pm

very good! no news good news I guess?

jalimena · April 8, 2022, 12:55pm

yeah, getting there. for some reason it didn’t quite work out of the box on my script, but it works when I explicitly set more of the event variables in the .sh scripts it generates. but I’ve got it a bit hard-coded now, I’m currently working on making it not hardcoded.

One thing I can say already:
These lines:

github.com

clementhelsens/FCCAnalyses/blob/batch/config/FCCAnalysisRun.py#L266-L273

      
        
            frun.write('echo "ls -altr ZH_mumu_recoil_batch"\n')
            frun.write('ls -altr ZH_mumu_recoil_batch/\n')
            
            
frun.write('echo "ls -altr ZH_mumu_recoil_batch/stage1"\n')
            frun.write('ls -altr ZH_mumu_recoil_batch/stage1\n')
            
            
frun.write('echo "ls -altr p8_ee_ZZ_ecm240"\n')
            frun.write('ls -altr p8_ee_ZZ_ecm240/\n')

don’t work unless you are running your example.

clement.helsens · April 8, 2022, 1:13pm

Please let me know what is needed in addition as I was able to run just fine yesterday.

thanks, this was a leftover from my tests such that files are properly copied back from batch to home dir when the output path is not absolute. Will remove them.

jalimena · April 12, 2022, 9:22am

Thanks @clement.helsens . I’ve now tried everything I can think of in order to simply and generically modify your FCCAnalysisRun.py such that it works with my local versions of the necessary packages, but I can’t get it to work. Here is how I needed to hard-code it so that it works for me:

         subprocess.getstatusoutput('chmod 777 %s'%(frunname))
         frun.write('#!/bin/bash\n')
+        frun.write('export RUN_DIR=$PWD\n')
+        frun.write('cd /afs/cern.ch/work/j/jalimena/FCCeeLLP/delphes\n')
         frun.write('source /cvmfs/sw.hsf.org/key4hep/setup.sh\n')
+        frun.write('export LD_LIBRARY_PATH=$PWD/install/lib:$LD_LIBRARY_PATH\n')
+        frun.write('export CMAKE_PREFIX_PATH=$PWD/install:$CMAKE_PREFIX_PATH\n')
+        frun.write('export DELPHES_DIR=$PWD/install\n')
+        frun.write('cd ../k4simdelphes\n')
+        frun.write('export LD_LIBRARY_PATH=$PWD/install/lib64:$LD_LIBRARY_PATH\n')
+        frun.write('export PATH=$PWD/install/bin:$PATH\n')
+
+        frun.write('cd /afs/cern.ch/work/j/jalimena/FCCeeLLP/FCCAnalyses\n')
+        frun.write('source ./setup.sh\n')
+        frun.write('cd ../FCCeePhysicsPerformance/case-studies/flavour/dataframe\n')
+        frun.write('source ./localSetup.sh\n')
+
         #frun.write('export PYTHONPATH=$LOCAL_DIR:$PYTHONPATH\n')
         #frun.write('export LD_LIBRARY_PATH=$LOCAL_DIR/install/lib:$LD_LIBRARY_PATH\n')
         #frun.write('export ROOT_INCLUDE_PATH=$LOCAL_DIR/install/include/FCCAnalyses:$ROOT_INCLUDE_PATH\n')
 
+        frun.write('cd $RUN_DIR\n')
         frun.write('mkdir job{}_chunk{}\n'.format(process,ch))

With this, I can successfully run batch jobs, submitting like this:

$ cd /afs/cern.ch/work/j/jalimena/FCCeeLLP/FCCAnalyses
$ source ./setup.sh
$ python config/FCCAnalysisRun.py ../FCCeePhysicsPerformance/case-studies/BSM/LLP/DisplacedHNL/rdfanalysis_general.py

My rdfanalysis_general.py is here:

github.com

jalimena/FCCeePhysicsPerformance/blob/master/case-studies/BSM/LLP/DisplacedHNL/rdfanalysis_general.py

#Mandatory: List of processes
processList = {
        'p8_ee_Zee_ecm91':{'fraction':0.1, 'chunks':10},
        #'p8_ee_ZZ_ecm240':{},#Run the full statistics in one output file named <outputDir>/p8_ee_ZZ_ecm240.root
        #'p8_ee_WW_ecm240':{'fraction':0.5, 'chunks':2}, #Run 50% of the statistics in two files named <outputDir>/p8_ee_WW_ecm240/chunk<N>.root
        #'p8_ee_ZH_ecm240':{'fraction':0.2, 'output':'p8_ee_ZH_ecm240_out'} #Run 20% of the statistics in one file named <outputDir>/p8_ee_ZH_ecm240_out.root (example on how to change the output name)
}

#Mandatory: Production tag when running over EDM4Hep centrally produced events, this points to the yaml files for getting sample statistics
prodTag     = "FCCee/spring2021/IDEA/"

#Optional: output directory, default is local dir
outputDir = "./read_EDM4HEP/"

#Optional: ncpus, default is 4
#nCPUS       = 8 #can use for local running
nCPUS       = 4 #better for batch running

#Optional running on HTCondor, default is False
#runBatch    = False

This file has been truncated. show original

Any ideas on how to make it work in a generic way? Let me know if I can provide you with anything else. If we can’t figure out how to make it work in a generic, then I guess maybe the best thing is to push your changes as you have them to the main branch, and I make these modifications locally for myself - better than nothing.

clement.helsens · April 12, 2022, 9:48am

Thanks @jalimena , I think the only option we have is to have a set of base common commands in a config file and the possibility for users to pass a custom config file, such that the FCCAnalysisRun remains untouched by users.

jalimena · April 12, 2022, 11:02am

Yeah that sounds good, thanks. If you set up the machinery, I can test such a custom config file on my side.

jalimena · April 26, 2022, 6:14pm

thanks for all the work. with the latest changes to FCCAnalyses, particularly [1], my problems are solved. i think we can close this ticket

[1] New run analysis scheme by clementhelsens · Pull Request #140 · HEP-FCC/FCCAnalyses · GitHub