[workforcedeveloper]

Description

• Adding a new benchmark which allows individuals to evaluate language models on the TechWolf/JobBERT-evaluation-dataset benchmark but using the ONET taxonomy, a dataset stored in HuggingFace (workforcedeveloper/JobBERT-ONET-evaluation-dataset). Each ESCO job title has been mapped to a ONET code using the crosswalk from here (https://esco.ec.europa.eu/system/files/2023-08/ONET_%28Occupations%29_0_updated.csv).

Checklist

• Added new tests for new functionality
• Tested locally with example tasks
• Code follows project style guidelines
• Documentation updated
• No new warnings introduced
• If the rankings artifact schema changed: bumped SCHEMA_VERSION in workrb/rankings.py and updated SUPPORTED_SCHEMA_VERSIONS

[warreveys]

Hi!

Nice contribution, thanks a lot!

Some thoughts on this:

• Your indexing inside the loop is causing an unnecessary O(k*n), which we can bring down to O(n) by creating a mapping dictionary before the loop ({title: i for i, title in enumerate(TARGET_SPACE)}).
• We should consider creating an ONET class to allow for more tasks with a target space derived from ONET. We can start small here. To check whether this would work: is the JSON file from a specific ONET version that can still be retrieved online? I'd like to move toward a design similar to the ESCO class here, but I am happy to contribute this myself later.
• We should think twice about the name of this task. 'ONETJobNormRanking' is very generic, and it sounds like the ground-truth normalisation test for ONET, of which there could be loads. Could we at least call it something like 'JobBERTONetJobNormRanking'?
• I'd like to see a little docstring in the added task, just briefly explaining what the queries and targets are exactly.
• I have approved the CI workflow, so please check this when it finished

Excited to hear what you think!

[blackmlengineer]

Hi @warreveys, I accidentally pushed this under the incorrect GH account and was trying to figure out how to have the merge request moved over to this account.

All of your thoughts make sense and I will be implementing them and submitting a new request soon.

[warreveys]

Hi, thanks for the contribution, and apologies for the late reply due to some time off.

Two small comments left:

• Thanks for the name change, but it currently breaks test_jobnorm_onet.py: the test calls workrb.tasks.ONETJobNormRanking, which no longer exists. Could you update the test accordingly?
• The download writes to the current working directory. Please make it write to the cache dir instead. You can follow the pattern in src/workrb/data/esco.py:130-131, but use "workrb" rather than "wteb" as the cache name. (wteb is a stale leftover from an old project name that I'll fix in the esco file separately.)

Thanks again!

[warreveys]

Hi,

I did some final changes. If we are aligned on this implementation, we are ready to merge this feature. Thanks again for contributing your dataset!