General problem running FCCAnalysis

Analysis
1

Hi,

I have a general problem with FCCAnalyses which persists across the FCCAnalyses framework.

I am trying to compile my own versions of FCCAnalyses and FCCeePhysicsPerformance locally. I can do this ok and can run preSel.py correctly. However when I try to run finalSel.py I get a segmentation violation.

As a last resort, I went back to the tutorial and didn’t download FCCAnalyses. But I get a similar error. See below.

This didn’t happen before with the tutorial. Help !

Thanks
Roy

[rlemmon@lxplus750 mytutorial]$ python finalSel.py
Warning in TInterpreter::ReadRootmapFile: class edm4hep::ObjectID found in libedm4drDict.so is already in libedm4hepDict.so
getting info from the web
EnableImplicitMT: 2
Load cxx analyzers …
Error in TUnixSystem::FindDynamicLibrary: libdatamodel[.so | .dll | .dylib | .sl | .dl | .a] does not exist in /cvmfs/sw.hsf.org/spackages4/xgboost/1.3.3/x86_64-centos7-gcc8.3.0-opt/yqy4fdg/lib64:/cvmfs/sw-nightlies.hsf.org/spackages4/py-uproot/4.0.11/x86_64-centos7-gcc8.3.0-opt/nn7ypaagp457csbgpbzwymz334thfl5s/lib:/cvmfs/sw-nightlies.hsf.org/spackages4/py-scikit-learn/1.0.1/x86_64-centos7-gcc8.3.0-opt/2v7bdlg4qa7wswx7dbznsdgd …
…/contrib/bzip2/1.0.8/x86_64-centos7-gcc4.8.5-opt/ceo6esx3tynrmqufzjhorensrssplo3m/lib:.:/cvmfs/sw-nightlies.hsf.org/spackages4/root/6.24.00/x86_64-centos7-gcc8.3.0-opt/7mjcil7w6kjmeazip3z46mjs4onxqnaq/lib:/lib64/tls/x86_64:/lib64/tls:/lib64/x86_64:/lib64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64:/usr/lib64:/cvmfs/sw-nightlies.hsf.org/spackages4/root/6.24.00/x86_64-centos7-gcc8.3.0-opt
open file /afs/cern.ch/user/r/rlemmon/mytutorial/p8_ee_ZH_ecm240.root
processed events {‘p8_ee_ZH_ecm240’: 1}
events in ttree {‘p8_ee_ZH_ecm240’: 1000}

running over process : p8_ee_ZH_ecm240
Defining snapshots and histograms
Evaluating…
*** Break *** segmentation violation

===========================================================
There was a crash.
This is the entire stack trace of all threads:

Thread 2 (Thread 0x7f0a8d2cf700 (LWP 5559) “python”):
#0 0x00007f0aa0c43de2 in pthread_cond_timedwait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0aa120e49c in PyCOND_TIMEDWAIT (us=, mut=0x7f0aa1405070 <_PyRuntime+1232>, cond=0x7f0aa1405040 <_PyRuntime+1184>) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/condvar.h:73
#2 take_gil (ceval=ceval
entry=0x7f0aa1404de8 <_PyRuntime+584>, tstate=tstate
entry=0x3e06900) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval_gil.h:206
#3 0x00007f0aa120e833 in PyEval_RestoreThread (tstate=tstate
entry=0x3e06900) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:399
#4 0x00007f0aa12ad962 in pysleep (secs=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Modules/timemodule.c:1867
#5 time_sleep (self=, obj=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Modules/timemodule.c:338
#6 0x00007f0aa116ee80 in cfunction_vectorcall_O (func=0x7f0a9fee1cc0, args=, nargsf=, kwnames=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Objects/methodobject.c:486
#7 0x00007f0aa110228a in _PyObject_Vectorcall (kwnames=, nargsf=, args=, callable=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Include/cpython/abstract.h:127
#8 call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x3e06900) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:4963
#9 _PyEval_EvalFrameDefault (f=, throwflag=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:3469
#10 0x00007f0aa120f4f1 in _PyEval_EvalCodeWithName (_co=, globals=, locals=locals
entry=0x0, args=, argcount=1, kwnames=0x0, kwargs=0x7f0a9fe81930, kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7f0a8d768ee0, qualname=0x7f0a8d76e5e0) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:4298
#11 0x00007f0aa1128a31 in _PyFunction_Vectorcall (func=func
entry=0x7f0a9fde4670, stack=, nargsf=, nargsf
entry=1, kwnames=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Objects/call.c:436
#12 0x00007f0aa112a7cc in PyVectorcall_Call (callable=0x7f0a9fde4670, tuple=, kwargs=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Objects/call.c:200
#13 0x00007f0aa10fd5a0 in do_call_core (kwdict=0x7f0a8d70abc0, callargs=0x7f0a9fe81910, func=0x7f0a9fde4670, tstate=0x3e06900) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:5010
#14 _PyEval_EvalFrameDefault (f=, throwflag=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:3559
#15 0x00007f0aa10fa93b in function_code_fastcall (co=, args=, nargs=1, globals=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Objects/call.c:284
#16 0x00007f0aa10fcb92 in _PyObject_Vectorcall (kwnames=, nargsf=, args=, callable=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Include/cpython/abstract.h:127
#17 call_function (kwnames=0x0, oparg=1, pp_stack=, tstate=0x3e06900) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:4963
#18 _PyEval_EvalFrameDefault (f=, throwflag=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:3486
#19 0x00007f0aa10fa93b in function_code_fastcall (co=, args=, nargs=1, globals=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Objects/call.c:284
#20 0x00007f0aa10fcb92 in _PyObject_Vectorcall (kwnames=, nargsf=, args=, callable=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Include/cpython/abstract.h:127
#21 call_function (kwnames=0x0, oparg=1, pp_stack=, tstate=0x3e06900) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:4963
#22 _PyEval_EvalFrameDefault (f=, throwflag=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/ceval.c:3486
#23 0x00007f0aa10fa93b in function_code_fastcall (co=, args=, nargs=1, globals=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Objects/call.c:284
#24 0x00007f0aa112c08c in _PyObject_Vectorcall (kwnames=0x0, nargsf=1, args=0x7f0a8d2cee28, callable=0x7f0a8e34a160) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Include/cpython/abstract.h:127
#25 method_vectorcall (method=method
entry=0x7f0a8d67cec0, args=, nargsf=nargsf
entry=0, kwnames=0x0) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Objects/classobject.c:67
#26 0x00007f0aa112a7cc in PyVectorcall_Call (callable=0x7f0a8d67cec0, tuple=, kwargs=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Objects/call.c:200
#27 0x00007f0aa12aebea in t_bootstrap (boot_raw=boot_raw
entry=0x7f0a8d4c2f90) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Modules/_threadmodule.c:1002
#28 0x00007f0aa125aab7 in pythread_wrapper (arg=) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-python-3.8.12-t7taq425vxxwcboo47pli3tbaezwdxoh/spack-src/Python/thread_pthread.h:232
#29 0x00007f0aa0c3fea5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007f0aa025fb0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f0aa14b3740 (LWP 5517) “python”):
#0 0x00007f0aa0226659 in waitpid () from /lib64/libc.so.6
#1 0x00007f0aa01a3f62 in do_system () from /lib64/libc.so.6
#2 0x00007f0aa01a4311 in system () from /lib64/libc.so.6
#3 0x00007f0a9fb543c3 in TUnixSystem::Exec (shellcmd=, this=0x1444f60) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.24.00-7mjcil7w6kjmeazip3z46mjs4onxqnaq/spack-src/core/unix/src/TUnixSystem.cxx:2120
#4 TUnixSystem::StackTrace (this=0x1444f60) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.24.00-7mjcil7w6kjmeazip3z46mjs4onxqnaq/spack-src/core/unix/src/TUnixSystem.cxx:2411
#5 0x00007f0a9fd02488 in (anonymous namespace)::do_trace (sig=1) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.24.00-7mjcil7w6kjmeazip3z46mjs4onxqnaq/spack-src/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:182
#6 (anonymous namespace)::TExceptionHandlerImp::HandleException (this=, sig=1) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.24.00-7mjcil7w6kjmeazip3z46mjs4onxqnaq/spack-src/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:195
#7 0x00007f0a9fb56c3d in TUnixSystem::DispatchSignals (this=0x1444f60, sig=kSigSegmentationViolation) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.24.00-7mjcil7w6kjmeazip3z46mjs4onxqnaq/spack-src/core/unix/src/TUnixSystem.cxx:3644
#8
#9 0x00007f0a783dbbe8 in ?? ()
#10 0x00007ffe142f4a90 in ?? ()
#11 0x0000000000000000 in ?? ()

The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum https://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.

#9 0x00007f0a783dbbe8 in ?? ()
#10 0x00007ffe142f4a90 in ?? ()
#11 0x0000000000000000 in ?? ()

Traceback (most recent call last):
File “finalSel.py”, line 38, in
myana.run(ncpu=NUM_CPUS, doTree=DO_TREE)
File “/cvmfs/sw-nightlies.hsf.org/spackages4/fccanalyses/commit.2a7af22c36ca02e84cc9dfebba5bf4ba15804ab2/x86_64-centos7-gcc8.3.0-opt/5pejlnr6u25er7nm2stdzwpcvp4dupbd/python/config/runDataFrameFinal.py”, line 166, in run
all_events = df.Count().GetValue()
cppyy.ll.SegmentationViolation: const ULong64_t& ROOT::RDF::RResultPtr<ULong64_t>::GetValue() =>
SegmentationViolation: segfault in C++; program state was reset
[rlemmon@lxplus750 mytutorial]$

2

Hi Roy,

this is strange, libdatamodel.so is provided by fcc-edm which shouldn’t really be needed.

What release / setup script are you using to run this? A new release was recently published, if you are using the “latest” symlink, you will need to recompile your local installations. Can you try that?
Cheers,
Valentin

3

Hi Valentin,

I was first using the tutorial setup so I don’t (think) I have any local installations, right ?

I logged out of lxplus and back in again. Then I sourced /cvmfs/fcc.cern.ch/sw/latest/setup.sh.

After that I just stepped through the tutorial to re-create p8_ee_ZH_ecm240.root and then ran python finalSel.py. But I still get the above error ?

Cheers
Roy

4

hello, this is true that this library is loaded here:

and it should be removed for installed versions
I have noticed the same thing and removed it in my branch:

that I should be merging soon as it contains a new improved way to run the analyses.
@rlemmon , I suggest you run the local code and manually remove the loading of the problematic library. Sorry for that.
Clement

5

Hi Clement,

Thanks for that and I commented it out in my local installation but it still doesn’t work.

However I noticed something strange perhaps. My cmake version is 2.8.12.2. Is this an issue and can it be upgraded on my lxplus instance without sudo ?

When I do make install in FCCAnalyses/build it tells me cmake 3.1…3.16 is required ?

Thanks
Roy

6

Hello @rlemmon ,

if you comment out this in your local code, you need to make sure the pythonpath is also set correctly such that you do not take the one installed on cvmfs. I was able to run myself just fine couple of days ago in the process of changing the running procedure, thus if you could provide a reproducer this will help with debugging.
Clement

7

Hi Clement,

What do you mean setting the pythonpath correctly ? Which one will cvmfs take ?

Also I am not sure what a reproducer is ! If you could provide some more information …

Thanks
Roy

8

Hello @rlemmon ,

what I mean with a reproducer is a set of instructions and simple code such that I can reproduce your problem. For example:

git clone mycode.git
cd mycode
source setupmycode
compile my code
run mycode

and after running those I will reproduce your problem such that a diagnostic could hopefully be made.

Cheers,
Clement

9

Hi Clement,

I see. Please forgive my unfamiliarity with ‘reproducer’.

So if I start with the simple tutorial, all I do to cause the crash is (after logging on to lxplus this morning):

source /cvmfs/fcc.cern.ch/sw/latest/setup.sh
cd mytutorial
python finalSel.py

That causes the segmentation problem. Scroll to the end …

Cheers
Roy

[rlemmon@lxplus711 mytutorial]$ python finalSel.py
Warning in <TInterpreter::ReadRootmapFile>: class       
edm4hep::ObjectID found in libedm4hepDict.so  is 
already in libedm4drDict.so 
getting info from the web
EnableImplicitMT: 2
Load cxx analyzers ... 
Error in <TUnixSystem::FindDynamicLibrary>:      
libdatamodel[.so | .dll | .dylib | .sl | .dl | .a] does not exist 
in /cvmfs/sw.hsf.org/spackages5/key4hep-
stack/2022-03-30/x86_64-centos7-gcc11.2.0-opt/mrwoj
/lib64:/cvmfs/sw.hsf.org/spackages5/key4hep-
stack/2022-03-30/x86_64-centos7-gcc11 ....
....hsf.org/spackages5/root/6.26.00/x86_64-centos7-
 gcc11.2.0-opt/jx56q/lib:/lib64/tls/x86_64:/lib64
/tls:/lib64/x86_64:/lib64:/usr/lib64/tls/x86_64:/usr/lib64
/tls:/usr/lib64/x86_64:/usr/lib64:/cvmfs/sw.hsf.org
/spackages5/root/6.26.00/x86_64-centos7-gcc11.2.0-
opt
open file  /afs/cern.ch/user/r/rlemmon/mytutorial
/p8_ee_ZH_ecm240.root
processed events  {'p8_ee_ZH_ecm240': 1}
events in ttree   {'p8_ee_ZH_ecm240': 1000}

running over process :  p8_ee_ZH_ecm240
 Defining snapshots and histograms
 Evaluating...
 *** Break *** segmentation violation


===========================================================

There was a crash.
This is the entire stack trace of all threads:

Thread 2 (Thread 0x7f0f3e04e700 (LWP 20454) “python”):
#0 0x00007f0f53149de2 in pthread_cond_timedwait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0f5372f3f7 in PyCOND_TIMEDWAIT (us=, mut=0x7f0f53941f70 <_PyRuntime+432>, cond=0x7f0f53941f40 <_PyRuntime+384>) at Python/condvar.h:73
#2 take_gil (tstate=tstate
entry=0x4931d60) at Python/ceval_gil.h:247
#3 0x00007f0f5372f83e in PyEval_RestoreThread (tstate=tstate
entry=0x4931d60) at Python/ceval.c:467
#4 0x00007f0f537d6362 in pysleep (secs=) at ./Modules/timemodule.c:2037
#5 time_sleep (self=, obj=) at ./Modules/timemodule.c:365
#6 0x00007f0f5369146a in cfunction_vectorcall_O (func=0x7f0f523e5630, args=0x7f0f3e1e6500, nargsf=, kwnames=) at Objects/methodobject.c:516
#7 0x00007f0f53602852 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7f0f3e1e6500, callable=0x7f0f523e5630, tstate=0x4931d60) at ./Include/cpython/abstract.h:118
#8 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7f0f3e1e6500, callable=0x7f0f523e5630) at ./Include/cpython/abstract.h:127
#9 call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x4931d60) at Python/ceval.c:5077
#10 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at Python/ceval.c:3489
#11 0x00007f0f5373062e in _PyEval_EvalFrame (throwflag=0, f=0x7f0f3e1e6380, tstate=0x4931d60) at ./Include/internal/pycore_ceval.h:40
#12 _PyEval_EvalCode (tstate=tstate
entry=0x4931d60, _co=_co
entry=0x7f0f3e516ea0, globals=, locals=locals
entry=0x0, args=, argcount=1, kwnames=0x0, kwargs=0x7f0f523d3ea0, kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7f0f3e515d50, qualname=0x7f0f3e505960) at Python/ceval.c:4329
#13 0x00007f0f5364d321 in _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at Objects/call.c:396
#14 0x00007f0f53600234 in do_call_core (kwdict=0x7f0f3e1dfc40, callargs=0x7f0f523d3e80, func=0x7f0f5245f4c0, tstate=0x4931d60) at Python/ceval.c:5125
#15 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at Python/ceval.c:3582
#16 0x00007f0f535fabcb in _PyEval_EvalFrame (throwflag=0, f=0x7f0f3e1d9e40, tstate=0x4931d60) at ./Include/internal/pycore_ceval.h:40
#17 function_code_fastcall (tstate=0x4931d60, co=, args=, nargs=1, globals=) at Objects/call.c:330
#18 0x00007f0f53602790 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7f0f3e1d3db8, callable=0x7f0f3ef86ca0, tstate=0x4931d60) at ./Include/cpython/abstract.h:118
#19 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7f0f3e1d3db8, callable=0x7f0f3ef86ca0) at ./Include/cpython/abstract.h:127
#20 call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x4931d60) at Python/ceval.c:5077
#21 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at Python/ceval.c:3506
#22 0x00007f0f535fabcb in _PyEval_EvalFrame (throwflag=0, f=0x7f0f3e1d3c40, tstate=0x4931d60) at ./Include/internal/pycore_ceval.h:40
#23 function_code_fastcall (tstate=0x4931d60, co=, args=, nargs=1, globals=) at Objects/call.c:330
#24 0x00007f0f53602790 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7f0f3e1d9df8, callable=0x7f0f3ef86f70, tstate=0x4931d60) at ./Include/cpython/abstract.h:118
#25 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7f0f3e1d9df8, callable=0x7f0f3ef86f70) at ./Include/cpython/abstract.h:127
#26 call_function (kwnames=0x0, oparg=, pp_stack=, tstate=0x4931d60) at Python/ceval.c:5077
#27 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at Python/ceval.c:3506
#28 0x00007f0f535fabcb in _PyEval_EvalFrame (throwflag=0, f=0x7f0f3e1d9c80, tstate=0x4931d60) at ./Include/internal/pycore_ceval.h:40
#29 function_code_fastcall (tstate=0x4931d60, co=, args=, nargs=1, globals=) at Objects/call.c:330
#30 0x00007f0f536501ac in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7f0f3e04de68, callable=0x7f0f3ef86d30, tstate=0x4931d60) at ./Include/cpython/abstract.h:118
#31 method_vectorcall (method=, args=0x7f0f539c8058, nargsf=, kwnames=0x0) at Objects/classobject.c:61
#32 0x00007f0f537d7319 in t_bootstrap (boot_raw=0x7f0f3e262600) at ./Modules/_threadmodule.c:1040
#33 0x00007f0f5377e947 in pythread_wrapper (arg=) at Python/thread_pthread.h:245
#34 0x00007f0f53145ea5 in start_thread () from /lib64/libpthread.so.0
#35 0x00007f0f52765b0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f0f53a04740 (LWP 20327) “python”):
#0 0x00007f0f5272c659 in waitpid () from /lib64/libc.so.6
#1 0x00007f0f526a9f62 in do_system () from /lib64/libc.so.6
#2 0x00007f0f526aa311 in system () from /lib64/libc.so.6
#3 0x00007f0f5207379b in TUnixSystem::Exec (shellcmd=, this=0x16601a0) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.26.00-jx56qpg245tgnins37ks2kzu4xdrinjz/spack-src/core/unix/src/TUnixSystem.cxx:2108
#4 TUnixSystem::StackTrace (this=0x16601a0) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.26.00-jx56qpg245tgnins37ks2kzu4xdrinjz/spack-src/core/unix/src/TUnixSystem.cxx:2399
#5 0x00007f0f52213f83 in (anonymous namespace)::do_trace (sig=1) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.26.00-jx56qpg245tgnins37ks2kzu4xdrinjz/spack-src/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:182
#6 (anonymous namespace)::TExceptionHandlerImp::HandleException (this=, sig=1) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.26.00-jx56qpg245tgnins37ks2kzu4xdrinjz/spack-src/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:195
#7 0x00007f0f52070da1 in TUnixSystem::DispatchSignals (this=0x16601a0, sig=kSigSegmentationViolation) at /cvmfs/tmp/spack-stage/gitlab-runner/spack-stage-root-6.26.00-jx56qpg245tgnins37ks2kzu4xdrinjz/spack-src/core/unix/src/TUnixSystem.cxx:3614
#8
#9 0x00007f0f2abcf73f in ?? ()
#10 0x0000003f00000009 in ?? ()
#11 0x000020000b4e04ec in ?? ()
#12 0x00007f0f51cc5ff0 in ?? () from /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/4omkc/lib64/libstdc++.so.6
#13 0x000000000b97b900 in ?? ()
#14 0x00007f0f51c53a90 in ?? () from /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/4omkc/lib64/libstdc++.so.6
#15 0x0000000000000000 in ?? ()

The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum https://root.cern/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.

#9 0x00007f0f2abcf73f in ?? ()
#10 0x0000003f00000009 in ?? ()
#11 0x000020000b4e04ec in ?? ()
#12 0x00007f0f51cc5ff0 in ?? () from /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/4omkc/lib64/libstdc++.so.6
#13 0x000000000b97b900 in ?? ()
#14 0x00007f0f51c53a90 in ?? () from /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/4omkc/lib64/libstdc++.so.6
#15 0x0000000000000000 in ?? ()

Traceback (most recent call last):
File “/afs/cern.ch/user/r/rlemmon/mytutorial/finalSel.py”, line 38, in
myana.run(ncpu=NUM_CPUS, doTree=DO_TREE)
File “/cvmfs/sw.hsf.org/spackages5/fccanalyses/0.3.5/x86_64-centos7-gcc11.2.0-opt/o2vb6/python/config/runDataFrameFinal.py”, line 166, in run
all_events = df.Count().GetValue()
cppyy.ll.SegmentationViolation: const ULong64_t& ROOT::RDF::RResultPtr<ULong64_t>::GetValue() =>
SegmentationViolation: segfault in C++; program state was reset
[rlemmon@lxplus711 mytutorial]$

10

Hi @rlemmon,

ok, I see. This is because the finalSel.py will call the runDataFrameFinal.py from the stack on cvmfs and not your local one, thus my comment about the PYTHONPATH.
This is a problem we have to fix, as now the cvmfs stack is not working, but in the mean time, you could get the code locally and follow the instrucitons to setup and compile:

and removing the loading of the problematic library.
Sorry for that, we will fix this asap.
Clement

11

Hi Clement,

So that works in fact. Reproducer:

git clone https://github.com/HEP-FCC/FCCAnalyses.git
cd FCCAnalyses
source ./setup.sh
mkdir build install
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=../install
make install
cd ..
python examples/FCCee/higgs/mH-recoil/mumu/preSel.py
python examples/FCCee/higgs/mH-recoil/mumu/finalSel.py
python config/doPlots.py examples/FCCee/higgs/mH-recoil/mumu/plots.py

It was important when I ran finalSel.py to remove line 57

ROOT.gSystem.Load("libdatamodel")

as you said yesterday. Otherwise it searches for it, presumably using $PYTHONPATH ?

So now that is sorted, I will try again using the original libraries I wanted to compile in FCCeePhysicsPerformance.

Don’t go anywhere ! Apart from lunch …

Cheers
Roy