CMS simulation chain for 8.16 TeV samples

This repository contains generator fragments and the commands needed to produce the following CMSSW data tiers:

generator fragment
  -> GEN-SIM
  -> GEN-SIM-RAW (DIGI + L1 + DIGI2RAW + HLT)
  -> AODSIM (reconstruction)
  -> MINIAODSIM (PAT)

The workflow documented below was tested interactively on lxplus with CMSSW_9_4_11 in an EL7 container. It is intended for the Run 2 2017 pp reference configuration (Run2_2017_ppRef).

Repository contents

• PYTHIACP5_8p16TeV_fragment.py: PYTHIA 8 CP5 weighted hard-QCD sample.
• HERWIGCH3_8p16TeV_fragment.py: HERWIG 7 CH3 weighted hard-QCD sample.

Both fragments configure a center-of-mass energy of 8160 GeV. The commands below use the PYTHIA fragment; substitute the HERWIG filename to generate the corresponding HERWIG configuration.

Metadata note: the PYTHIA fragment currently describes the sample as 5.02 TeV in configurationMetadata.annotation, while comEnergy is set to 8160 GeV. The generator therefore runs at 8.16 TeV, as confirmed in the PYTHIA initialization output. Resolve the annotation before using the sample for centrally tracked production metadata.

CMSSW releases on lxplus

CMS release information is available from the CMSSW documentation under CMSSW Release Notes. To list releases available for the active architecture, use:

scram list CMSSW

For a release compatible with the current lxplus operating system, the usual setup is:

cd /afs/cern.ch/user/g/gnigmatk/soft/cmssw
cmsrel CMSSW_15_0_12_patch1
cd CMSSW_15_0_12_patch1/src
cmsenv
scram build -j24

Examples already present in this user area include:

• CMSSW_13_3_1_patch1 for EL8 (lxplus8.cern.ch)
• CMSSW_15_0_12_patch1 for EL9 (lxplus9.cern.ch)

The simulation chain in this repository requires the older CMSSW_9_4_11 release. It cannot be created for the native EL8 architecture (el8_amd64_gcc12), so enter the legacy EL7 environment first. See the CMS container instructions for additional details.

One-time setup for this project

Log in to lxplus, enter the directory where CMSSW releases are kept, and start the EL7 container:

cd /afs/cern.ch/user/g/gnigmatk/soft/cmssw
cmssw-el7

The prompt changes to Singularity>. Create the release if it does not already exist:

cmsrel CMSSW_9_4_11
cd CMSSW_9_4_11/src
cmsenv

The release may report that slc7_amd64_gcc630 is a non-production architecture and that the production architecture was slc6_amd64_gcc630. That warning was observed during the successful test recorded here.

Install a fragment in the CMSSW package layout. Run these commands from CMSSW_9_4_11/src and replace /path/to/cmsSimChain with the location of this repository on lxplus:

mkdir -p Configuration/GenProduction/python
cp /path/to/cmsSimChain/PYTHIACP5_8p16TeV_fragment.py \
  Configuration/GenProduction/python/

Build once after adding or changing a fragment:

scram build -j24

For later sessions, re-enter the EL7 container and restore the environment:

cd /afs/cern.ch/user/g/gnigmatk/soft/cmssw
cmssw-el7
cd CMSSW_9_4_11/src
cmsenv

Run the complete PYTHIA chain

Run every command below from CMSSW_9_4_11/src after cmsenv.

There are two useful cmsDriver.py modes:

• With --no_exec, it writes the Python configuration but does not process events. Run the generated configuration explicitly with cmsRun.
• Without --no_exec, it writes the configuration and immediately starts cmsRun.

The example below uses --no_exec consistently so each generated configuration and execution step is visible. -n 17 generates a small test sample; -n -1 processes all events available in each input file. If -n is omitted, cmsDriver.py defaults to one event, which can silently reduce every downstream file to one event.

1. GEN-SIM

Generate the configuration:

cmsDriver.py Configuration/GenProduction/python/PYTHIACP5_8p16TeV_fragment.py \
  --fileout file:testCP5_GENSIM.root \
  --mc \
  --eventcontent RAWSIM \
  --datatier GEN-SIM \
  --conditions 94X_mc2017_realistic_forppRef5TeV \
  --beamspot Realistic5TeVppCollision2017 \
  --step GEN,SIM \
  --geometry DB:Extended \
  --era Run2_2017_ppRef \
  -n 17 \
  --no_exec

This creates PYTHIACP5_8p16TeV_fragment_py_GEN_SIM.py. Run it with:

cmsRun PYTHIACP5_8p16TeV_fragment_py_GEN_SIM.py

The output is testCP5_GENSIM.root.

Instead of passing -n 17, the generated configuration can be edited before running it:

process.maxEvents = cms.untracked.PSet(
    input = cms.untracked.int32(17)
)

Using -n is preferable because the requested event count remains recorded in the cmsDriver.py command embedded at the top of the generated file.

2. DIGI, L1, DIGI2RAW, and HLT

cmsDriver.py step1 \
  --filein file:testCP5_GENSIM.root \
  --fileout file:testCP5_GENSIM_DR.root \
  --mc \
  --eventcontent RAWSIM \
  --datatier GEN-SIM-RAW \
  --conditions 94X_mc2017_realistic_forppRef5TeV_v1 \
  --beamspot Realistic5TeVppCollision2017 \
  --step DIGI,L1,DIGI2RAW,HLT:PRef \
  --geometry DB:Extended \
  --era Run2_2017_ppRef \
  -n -1 \
  --no_exec

cmsRun step1_DIGI_L1_DIGI2RAW_HLT.py

The output is testCP5_GENSIM_DR.root.

CMSSW 9_4_11 prints a BMTF packer warning stating that all detector information is returned in FED 1376 rather than FEDs 1376 and 1377. This warning appeared in the successful test and did not stop processing.

3. RAW2DIGI and reconstruction to AODSIM

cmsDriver.py step2 \
  --filein file:testCP5_GENSIM_DR.root \
  --fileout file:testCP5_GENSIM_DR_AOD.root \
  --mc \
  --eventcontent AODSIM \
  --datatier AODSIM \
  --conditions 94X_mc2017_realistic_forppRef5TeV_v1 \
  --step RAW2DIGI,L1Reco,RECO,RECOSIM,EI \
  --geometry DB:Extended \
  --era Run2_2017_ppRef \
  -n -1 \
  --no_exec

cmsRun step2_RAW2DIGI_L1Reco_RECO_RECOSIM_EI.py

The output is testCP5_GENSIM_DR_AOD.root.

4. PAT to MINIAODSIM

cmsDriver.py step3 \
  --filein file:testCP5_GENSIM_DR_AOD.root \
  --fileout file:testCP5_GENSIM_DR_AOD_miniAOD.root \
  --mc \
  --eventcontent MINIAODSIM \
  --runUnscheduled \
  --datatier MINIAODSIM \
  --conditions 94X_mc2017_realistic_forppRef5TeV \
  --step PAT \
  --geometry DB:Extended \
  --era Run2_2017_ppRef \
  -n -1 \
  --no_exec

cmsRun step3_PAT.py

The output is testCP5_GENSIM_DR_AOD_miniAOD.root.

The tested transcript used step1 as the arbitrary configuration-name prefix for both the DIGI and PAT commands. This works, but distinct step1, step2, and step3 prefixes are used above to avoid confusing or overwriting generated configuration files.

Expected products

After the complete test chain, the working directory contains:

Configuration/GenProduction/python/PYTHIACP5_8p16TeV_fragment.py
PYTHIACP5_8p16TeV_fragment_py_GEN_SIM.py
step1_DIGI_L1_DIGI2RAW_HLT.py
step2_RAW2DIGI_L1Reco_RECO_RECOSIM_EI.py
step3_PAT.py
testCP5_GENSIM.root
testCP5_GENSIM_DR.root
testCP5_GENSIM_DR_AOD.root
testCP5_GENSIM_DR_AOD_miniAOD.root

The generated Python files and ROOT outputs are working products; they should not be committed to this fragment repository.

Validate each output

Check that every file exists and is readable before starting the next step:

edmFileUtil testCP5_GENSIM.root
edmFileUtil testCP5_GENSIM_DR.root
edmFileUtil testCP5_GENSIM_DR_AOD.root
edmFileUtil testCP5_GENSIM_DR_AOD_miniAOD.root

Count events explicitly:

edmFileUtil -e testCP5_GENSIM.root
edmFileUtil -e testCP5_GENSIM_DR.root
edmFileUtil -e testCP5_GENSIM_DR_AOD.root
edmFileUtil -e testCP5_GENSIM_DR_AOD_miniAOD.root

For the commands above, all four files should contain 17 events. In the recorded test, GEN-SIM contained 17 events but the three downstream commands omitted -n; their logs therefore show only Event 1 being processed.

To inspect event content without a graphical display:

edmDumpEventContent testCP5_GENSIM_DR_AOD_miniAOD.root

Starting interactive ROOT over SSH may print can't figure out DISPLAY. Use root -l -b for batch inspection, or reconnect with X11 forwarding (ssh -Y) if a graphical ROOT session is actually needed. Remember that .ls is a ROOT prompt command; at the Singularity> shell prompt, use ordinary ls.

Using DBS inputs

For centrally stored input, replace a local input such as file:testCP5_GENSIM.root with a DBS dataset specification:

dbs:/PRIMARY_DATASET/PROCESSED_DATASET/DATA_TIER

Keep the data tier and processing step consistent with the input: GEN-SIM is the input to DIGI, GEN-SIM-RAW is the input to reconstruction, and AODSIM is the input to PAT.