Issues

• Write bash script for execution of whole Pipeline

  important
  OpenGPTX/olm-datasets#8 · Filter by author Lena-Jurkschat opened on Dec 15, 2022

• Improve logging of dataset cleaning

  invalidNice-to-have
  OpenGPTX/olm-datasets#7 · Filter by author Lena-Jurkschat opened on Dec 15, 2022

• Add german Stopwordslist

  OpenGPTX/olm-datasets#6 · Filter by author Lena-Jurkschat opened on Dec 15, 2022

• Add usage of validation split to cleaning script

  OpenGPTX/olm-datasets#5 · Filter by author Lena-Jurkschat opened on Dec 15, 2022

• Bash script for hole data processing Pipeline

  OpenGPTX/olm-datasets#4 · Filter by author Lena-Jurkschat opened on Dec 15, 2022

• Improve language seperate Datasets within get_hf_datasets_from_parquet.py

  OpenGPTX/olm-datasets#2 · Filter by author Lena-Jurkschat opened on Dec 13, 2022

• Add setup.py usage to README

  OpenGPTX/olm-datasets#1 · Filter by author Lena-Jurkschat opened on Dec 13, 2022