Integration with Huggingface datasets #4962

Huggingface Datasets has nicely gathered popularity over the last few months and it has a very simple API for accessing the most common NLP datasets. In addition, it has the potential to support multi-modal datasets as well (see related issue). At the moment, AllenNLP integrates datasets by downloading them manually and by reporting in the configuration file the path to the dataset. This scenario works most of the time but doesn't guarantee complete transparency in the training process.

Based on this issue, I was considering whether it would be possible to support this library so that AllenNLP can potentially take advantage of their caching functionalities as well. I'm aware that AllenNLP has its own way of handling tokenization and indexing but I still believe having a common entry point for dataset creation would be very handy as well as very elegant from the reproducibility point of view.

Any thoughts about this idea?

Thanks,
Alessandro