You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Passing files with .arrow extension into data_files argument, at least when streaming=True is very slow.
Steps to reproduce the bug
Basically it goes through the _get_extraction_protocol method located here
The method then looks at some base known extensions where arrow is not defined so it proceeds to determine the compression with the magic number method which is slow when dealing with a lot of files which are stored in s3 and by looking at this predefined list, I don't see arrow in there either so in the end it return None:
Describe the bug
Passing files with
.arrow
extension into data_files argument, at least whenstreaming=True
is very slow.Steps to reproduce the bug
Basically it goes through the
_get_extraction_protocol
method located hereThe method then looks at some base known extensions where
arrow
is not defined so it proceeds to determine the compression with the magic number method which is slow when dealing with a lot of files which are stored in s3 and by looking at this predefined list, I don't seearrow
in there either so in the end it return None:Expected behavior
My expectation is that
arrow
would be in the known lists so it would return None without going through the magic number method.Environment info
datasets 2.19.0
The text was updated successfully, but these errors were encountered: