Repository navigation
OpenMS integration upgrade: direct vendor input, ProSE default, in-process Percolator, distributed LFQ feature finding - #740
Draft
jpfeuffer wants to merge 12 commits into
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This was referenced Oct 8, 2026
…r, distributed LFQ feature finding - Read Thermo .raw and Bruker .d directly with OpenMS (no conversion by default); --convert_raw keeps ThermoRawFileParser for mzML-only steps, with explicit errors otherwise. - Add OpenMS ProSE as default search engine with internal Percolator rescoring (no separate rescoring step; fails if ProSE did not rescore), idparquet output. - Enforce in-process PercolatorAdapter rescoring for Comet/Sage/MS-GF+. - Distribute ProteomicsLFQ feature finding per run with -detect_only/-feat_dir feature checkpoints and a combining ProteomicsLFQ task (bigbio#739); new --lfq_distributed_featurefinding parameter. - Drop unused mzTab intermediates from ProteomicsLFQ and IsobaricWorkflow. - Add test_lfq_prose profile and CI matrix entries; update schema and docs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…K m/z enforcement ProSE now runs with -Search:peptdeep:enable (new --prose_peptdeep, default true, and --prose_peptdeep_instrument) and fails if no predicted features were added. Modules reading Bruker .d input fail unless OpenMS used the Bruker TDF SDK for m/z calibration (open-source approximation is up to ~37 ppm off). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…h direct idparquet output Pins all OpenMS modules to ghcr.io/jpfeuffer/openms-tools-thirdparty:quantms-onnx-bruker (ONNX Runtime, PeptDeep models, Bruker TDF SDK; OpenMS #9975, #10449, #10450, #10452-#10455, -out, removing the external percolator binary and the IDFileConverter step. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
jpfeuffer
force-pushed
the
jpfeuffer-openms-integration-upgrade
branch
from
October 9, 2026 06:29
9ee9d10 to
a3e87c9
Compare
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
timosachsenberg
left a comment
timosachsenberg
left a comment
There was a problem hiding this comment.
Reviewed 1afcff8 against the OpenMS implementation used by the pinned image and current OpenMS develop. Four correctness issues need attention before merging.
The checkpoint/strict-uniqueness issue belongs in OpenMS; I opened OpenMS/OpenMS#10481 with the cause, a reproducer based on the existing TOPP fixtures, the metadata/memory/output tradeoffs, and the regression tests. The other three findings can be addressed in quantms.
Validation here is source-level; I have not run the pipeline or the OpenMS reproducer locally.
5 tasks done
|
HI @jpfeuffer , I was trying to update it but saw you have already worked on it. Thank you for picking this up so quickly! I tested the distributed LFQ part of this PR on real data and wanted to share the results in case they're useful. Setup
Results
So it would indeed save lots of time if we scale up. :) Thank you! |
…/decoy params) - Pass -protein_quantification to distributed detection via shared feature args (OpenMS#10481); -resume re-detects when --protein_quant changes - ProSE: honour --run_fdr_cutoff (reject <= 0), forward --decoy_string as -Search:decoy_prefix, reject non-PSM --fdr_level at startup - nf-test for custom decoy prefix/suffix with ProSE Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Heads-up from the OpenMS side for when the image pin moves to an official OpenMS build. OpenMS/OpenMS#10578 moves ProSE's Percolator rescoring into It exits with code 6.
Generated by Claude Code |
- Pin all OpenMS modules (Docker and SIF) by digest to the OpenMS nightly of 2026-10-27 (develop bd790fca41): includes #9975, #10462, #10469, #10578, #10486 and the ProteomicsLFQ/Percolator/QPX fixes; no Bruker SDK. Follow-up: switch to the bigbio retag 2026.10.27 (bigbio/quantms-containers#32). - ProSE: -rescore was removed (OpenMS#10578); pass -Search:rescore true. ProSE -out .idparquet (OpenMS#10450) is not merged, so write -out_idxml and convert to .idparquet with IDFileConverter in the same task. - Remove stale ConsensusID mentions (docs, schema help, test_localize). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
Re your heads-up (#740 (comment)): thanks, done in efd3a89, together with the pin bump.
|
- Repin all OpenMS modules to develop-2026.10.28-f940a46542 (containerdeploy run 40679593583; Docker index c9b4b871..., SIF 0cdb19e5...), which contains OpenMS#9975, #10462, #10745 and #10450. - ProSE: -out *_prose.idparquet natively; drop the interim idXML + IDFileConverter step. - Remove the vendor-file readlink workaround; the image reads symlinked .raw and .d inputs directly. - ConsensusID (removed in OpenMS#10619): fail on retired consensusid_* / min_consensus_support params, and actually call validateInputParameters. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
jpfeuffer
force-pushed
the
jpfeuffer-openms-integration-upgrade
branch
from
October 28, 2026 12:34
86400d7 to
c0b659c
Compare
… PEPs Percolator falls back to its default vector on the tiny CI phospho dataset (also with the external binary and the Oct 9 image), yielding near-constant PEPs around 0.67-0.75; a 1-PEP threshold of 0.3 flipped between platforms. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
Progress update: ProSE / ProteomicsLFQ batching (work in progress) State: the pushed head is still What is implemented (local)
Validation done locally (completed)
Running / not done
Known unrelated CI note: the |
…teomicsLFQ detection batches ProSE searches all runs sharing search settings and database in one task by default (--prose_batch_size / --prose_batches), with per-run rescoring, PSM FDR and predicted features verified for every output; search protein FDR disabled, model transfer rejected. Distributed ProteomicsLFQ feature detection can batch runs (--lfq_feature_batch_size / --lfq_feature_batches); default stays one run per task. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
Follow-up: the batching changes are now pushed as 0a404f7, and the PR body is updated. The local Comet/Sage |
Collaborator
Author
|
@timosachsenberg something is wrong. Prose is slower than MSGF PLUS COMET PLUS LOCALIZATION PLUS EXTERNAL PERCOLATOR RESCORING. |
Collaborator
Author
|
Ok looks like it is PRIDE server that is slow AF @ypriverol |
Collaborator
Author
|
@ypriverol CI timing analysis for ProSE is not slower. Most of the wall time in both profiles is spent downloading the two ~190 MB PXD007058 RAW files from PRIDE FTP.
The Proposed follow-up: stop downloading these RAWs from PRIDE in CI (cache them or use a smaller mirror in quantms-test-datasets). Optionally, ProSE could also start searching per file while other files are still downloading. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
OpenMS integration upgrade (closes #739 for the distributed feature finding part).
.rawand Bruker.dare read directly by OpenMS (ProSE, Comet, Sage, ProteomicsLFQ). Conversion is off by default;--convert_rawre-enables ThermoRawFileParser.rawextensions..rawand.dinputs directly (Thermo .raw reader fails on symlinked input files (breaks Nextflow/Galaxy staging) OpenMS/OpenMS#10451, fixed by #10452), so the earlierreadlink -fworkaround was removed.--search_engines prose)OPENMS PROSEmodule: ProSE rescores with its in-process Percolator (-Search:rescore true, [API,PARAM,TEST] ProSE: Percolator rescoring in ProSEAlgorithm, on by default OpenMS/OpenMS#10578) and there is no separate PSM rescoring step; the task fails if ProSE did not report rescoring. ProSE writes its identifications natively as.idparquet(-out *_prose.idparquet, [FEATURE,PARAM] ProSE: write per-file .idparquet via new -out (deprecate -out_idxml) OpenMS/OpenMS#10450); there is no idXML intermediate or IDFileConverter step.--prose_peptdeep true,--prose_peptdeep_instrument QE): ms2_cosine, ms2_spectral_angle, ms2_pearson, ms2_frac_pred_found and rt_abs_error are added before rescoring. The task fails unless ProSE logs[PeptDeepRescoring] Predicted features added: N>0 / M PSMs, and OpenMS itself throws if ONNX or the models are missing ([FEATURE,LOG] ProSE: peptdeep:enable fails instead of continuing without predictions OpenMS/OpenMS#10461).--prose_peptdeep falseis the only, explicit, opt-out..dneeds no Bruker SDK. OpenMS computes m/z from the.dMzCalibration table with per-frame temperature correction ([FIX,IO] BrukerTimsFile: compute m/z from the MzCalibration table OpenMS/OpenMS#10469, merged) and 1/K0 with the rational TimsCalibration model. On a public timsTOF Pro HeLa run this matched the SDK'stims_index_to_mzexactly (max 0.0000 ppm over 81,181 peaks). quantms has no SDK check, so.dinput works on amd64 and arm64. If a user supplies an SDK (OPENMS_BRUKER_SDK_PATH, [FIX,IO] BrukerTimsFile: use an optional Bruker SDK for m/z too (follow-up to #10469) OpenMS/OpenMS#10460), OpenMS uses it; the image does not ship one. Only TDF.dis supported (not TSF/BAF).-use_subprocess false, with a log check). Peptide/protein-level FDR still requires the external binary; this is an OpenMS limitation and is documented.PROTEOMICSLFQ_DETECTtask per run (-detect_only -feat_dir) writes.featureParquetcheckpoints. A combiningPROTEOMICSLFQtask (-feat_dir) then does alignment, linking, inference, FDR and quantification.-resume, changing FDR/inference/quant params re-runs only the combine task.--lfq_distributed_featurefinding(defaulttrue).--prose_batch_size N(max runs per task) or--prose_batches N(balanced task count per group, capped at the group size).-Search:FDR:protein 0), and model transfer or pooling between runs is rejected, so results do not depend on batching.--lfq_feature_batch_size Nor--lfq_feature_batches Nputs several runs into one-detect_onlycall. Combine is unchanged.-resumeunit. A change to any run or to the batch composition reruns the whole batch, and memory scales with the batch. Invalid values (both given, 0, negative, fractional, boolean) fail at startup.consensusid_algorithm,min_consensus_support,consensusid_considered_top_hitsandconsensusid_debugparams now fail at startup with an explicit message instead of an nf-schema warning. While adding this,validateInputParameters()turned out never to be called (so the existingms2features_rangecheck was dead);main.nfnow calls it.test_lfq_proseprofile (PXD007058 phospho RAW files, no conversion) added to CI. The profiles that need mzML now setconvert_raw = true.Dependencies (OpenMS PRs, all from jpfeuffer/OpenMS)
All modules are pinned by digest to the official OpenMS dev build
develop-2026.10.28-f940a46542(containerdeploy run 40679593583, develop @ f940a46542 = merge of OpenMS/OpenMS#10450, run_tests=true, all jobs passed; tools reportRevision: f940a46):ghcr.io/openms/openms-tools-thirdparty@sha256:c9b4b871f8c5edcc588ffa7759eb709eb827713e44c003066b6436bae5899637(amd64e31f0ab1…, arm64ad6e78d1…)oras://ghcr.io/openms/openms-tools-thirdparty-sif@sha256:0cdb19e52d3ff534390ffdedcb19c242fe30ae3b3dcd0e47c1ca1788030a8cac(same run, built from that Docker tag)The mutable
latestnightly predated #10450, so a dedicated build was made. It contains every PR below, plus OpenMS/OpenMS#10714 and #10745. Compared with the previous pin (2026-10-27 nightly, bd790fca41), only ProSE's interface changed (new-out,Search:percolator:max_train_psms); every other tool's INI is identical. Follow-up: bigbio/quantms-containers#32 retags this exact index (multi-arch) and SIF asghcr.io/bigbio/openms-tools-thirdparty:2026.10.28; #740 switches to that tag once it is merged.-out *.idparquet(used)-rescore→-Search:rescoreWITH_ONNX=ON), PeptDeep models; no Bruker SDKprotein_quantificationin the feature-checkpoint fingerprint (fixes #10481); quantms still passes the mode to detection, since a mismatch is now refused.d: SDK-free m/z from the MzCalibration table.raw/.d(follow-up: direct vendor input for TMT/iTRAQ)Review fixes (d75cc3f)
-protein_quantificationthrough the shared feature args, so detection and combine use the same mode and-resumere-detects when--protein_quantchanges (ProteomicsLFQ checkpoints lose theoretical peptide uniqueness when changing protein quantification mode OpenMS/OpenMS#10481). Verified with--protein_quant strictly_unique_peptides.--run_fdr_cutoffafter its internal rescoring and rejects values <= 0, since ProSE reads 0 as "off".--decoy_stringis forwarded as-Search:decoy_prefixand matched as either prefix or suffix. A new nf-test covers both positions, and a negative control without the flag fails.--fdr_levelother thanpsm_level_fdrsis rejected at startup with ProSE.Notes / blockers
test_lfqfailures were caused by GitHub raw returning HTTP 504 forBSA1_F2.mzMLduring the unchanged upstream input-existence check. Both Nextflow versions failed at the same moment, in both attempts. They were not caused by this PR. On HEAD 1afcff8 every CI check passes: all Docker and Singularity profiles on 25.10.4 and latest-everything, pre-commit, and nf-core lint.Validation
With the official dev build f940a46542 (c0b659c, test_localize threshold fix 843a60d), macOS arm64 (native arm64 image), Nextflow 25.10.4:
Revision: f940a46.-outaccepts idXML/idparquet,-out_idxmlis a deprecated alias, and-out_mergedaccepts idparquet.libonnxruntime.so.1is linked,peptdeep_{ms2,rt,ccs}_dynamic.onnxare present, ConsensusID is absent, and there is no Bruker SDK. All module flags were checked against the build's INIs.test_lfq_prose,docker(Thermo RAW read directly through the staged symlink, ProSE, distributed LFQ, QPX, pmultiqc): passed (9 tasks, 6m22s). Per run:Predicted features added: 1171 / 1171 PSMs(1144 / 1144), NCE=26Rescoring … PSMs in-process with 33 features.Percolator : rescored 1 / 1 file(s)*_prose.idparquetbundle (psms/proteins/protein_groups/search_params), no idXML1 run(s) detected; combine:Feature checkpoints: 2 reused, 0 detected.test_lfq,docker --search_engines comet(Comet → in-process PercolatorAdapter → distributed LFQ → QPX): passed (29 tasks), which also exercises the newvalidateInputParameters()call.modules/local/openms/prose/tests/main.nf.test, customQMSD_prefix and_QMSDsuffix decoys, native idparquet): 2/2 passed.--consensusid_algorithm best --min_consensus_support 1exits 1 with the explicit message.Skipping Percolator rescoring … Only 0 PSMs/Percolator : rescored 0 / 1 file(s). The quantms module then fails explicitly ("did not rescore"); this is intended..rawand a symlinked Bruker.d(timsTOF Pro HeLa, 57,003 spectra,MzCalibration table … per-frame temperature correction, rational 1/K0, no SDK). On amd64 the MzCalibration m/z previously matched the SDK exactly (0.0000 ppm, 81,181 peaks).rescored 2 / 2, and per-run PeptDeep features (170/170, 161/161).test_lfq_proseon local RAWs, default (1 ProSE task for 2 runs) vs--prose_batch_size 1: per-run idparquet PSMs, proteins and search params are identical apart from run_identifier/date timestamps. msstats and QPX feature/psm are identical; QPX pg is identical up to row order andpg_id.--lfq_feature_batch_size 2: 1 detect task, outputs identical.--feature_with_id_min_scorereruns detection only..d: exit 0,rescored 2 / 2, PeptDeep features on every PSM (189/189, 498/498).test_lfq(Comet + Sage + PSM_CLEAN, PSM_CLEAN on the host because the rescoring image pull kept failing): passed.nextflow lintand pre-commit on the changed files: clean. The pre-existing exceptions are the non-executable outdir hook and achannellint error in the untouched nf-coreutils_nfcore_pipeline, which is identical to dev.test_localize(Comet+MS-GF+ → PSM_CLEAN merge → in-process Percolator → ONSITE), mzML pre-converted with FileConverter and PSM_CLEAN 0.0.24 run on the host: passed. On CI (c0b659c) rep2 failed withHigh-scoring PSMs for training: 0. On this tiny dataset Percolator falls back to its default vector (Found 0 test set PSMs with q<0.01. Resetting score vector), which also happens with the externalpercolatorbinary and with the Oct 9 image. All PEPs are then roughly constant (0.666 / 0.673 / 0.676 on arm64, amd64 and external), so ONSITE's1-PEP > 0.3cut flipped between platforms. 843a60d lowers the profile'sonsite_modeling_score_thresholdto 0.01 and documents why. This profile is a smoke test for localization.test_dda_id_fine_tuningfailure (run 40698624889) verified as a PRIDE FTPConnection resetwhile staging the RAW input, not related to this PR.PR checklist
test_lfq_proseprofile + CI matrix)nf-core pipelines lint). (only pre-existing failures)nextflow run . -profile test,docker --outdir <OUTDIR>). (see Validation; needs the official OpenMS image pin before merge)nextflow run . -profile debug,test,docker --outdir <OUTDIR>). (not run)docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).