Skip to content

TODO from discussion #4

Description

@rstojnic

TODO sotabench lib:

  • remove benchmark() function from benchmark.py
  • move deps to requirements
  • evaluation.json should be made if some ENV variable is set, otherwise pprint something
  • for each benchmark:
    • benchmark()
    • default transform
    • the dataset
    • default parameters
  • documentation:
    • dataset examples
    • default transform example
    • input fed to model, and expected output
    • link to examples of benchmarked models
  • a library of transforms (maybe)

And additional requests:

  • BenchmarkResult return value should also contain: 1) the dataset used, 2) the transform used, 3) input parameters used when invoking the function, 4) anything else - so it's a self-contained record of results

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions