Dataset Format discussion

[PGijsbers]

Please add/correct me :)

Description.xml on MinIO

For now, the description.xml file within MinIO is only there for external users/archival.
Descriptions will continue to be queried from the Rest API.
To see if we can switch in the future, we will:

• Set up a job to check the integrity of the minio descriptions from the REST descriptions periodically to see if we can "guarantee" integrity.
• Monitor the load that querying dataset descriptions impose on the database

Dataset Metadata

• Information about how to use features and other things that inform (defaults for) task creation will be stored on a suggested_task field.
  This makes some old attributes obsolete (e.g., default_target_attribute).

• Feature descriptions should be supported and more easily added on upload, so we can store this in a structured manner (as opposed to free form in the dataset description). This should be alongside supported for linking ontologies.

[luisoala]

[luisoala]

[luisoala]

i created a digest based on our blackboard and discussion, you can edit it here https://beta.tldraw.com/r/v2_c_76Cd6eHqhprWJBadz8aTC

[luisoala]

i also made a demo notebook using lance https://github.com/eto-ai/lance which describes the depicted workflow without the db component (ie directly querying the ds tablein storage) parked here https://github.com/luisoala/lance-test

[joaquinvanschoren]

Hi all, Would it make sense to do a video call next Tuesday afternoon about this? It would be a great preparation for the Paris meeting. I'll also try to write down more before that. Cheers, Joaquin

…

On Wed, 22 Feb 2023 at 12:14, Luis Oala ***@***.***> wrote: i also made a demo notebook using lance https://github.com/eto-ai/lance which describes the depicted workflow without the db component (ie directly querying the ds tablein storage) parked here https://github.com/luisoala/lance-test — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANFAV4ACEJ7INGJ7X34GTTWYXYKTANCNFSM6AAAAAAVC4SC6A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[luisoala]

sounds good (y)

i started a glossary here https://www.overleaf.com/6899967464rntqdbcmrcxc its still in sad state but i hope to have it intelligible by early next week

feel free to add, edit, recycle

would be great to have a simple reference soon

[luisoala]

btw https://github.com/activeloopai/deeplake seems to follow a similar logic to parquet/lance/parquet what we discussed in munich