PERF: improve performance of read_dataframe if a filter is used

[theroggy]

In read_dataframe without arrow, the number of rows of the result was counted first, and then the full data was read.

Especially when using a filter, counting the rows can take significant time.

This PR avoids doing the rowcount before reading to improve performance by reading the results in chunks till no more rows are returned from the dataset. For the new zealand building outlines geopackage (3.3 million rows) this gives these results:

• If the filter limits the rows a lot counting the rows can even take the same time as the subsequent reading of all data... so in this case the time taken ~halves.
  • e.g. reading the test file file with where="ST_NPOINTS(st_buffer(geom, 10)) > 2000" (returning 9 rows) took 82 s, now 45 s.
• When reading the entire file without filter, both implementations take 55-60 seconds on my windows laptop (plugged in), with the chunked implementation giving the same average timings. To be sure, if no filter is used and counting rows is fast for the file type involved, no chunking is used (=old way).

[jorisvandenbossche]

I don't know if it is worth the added complexity, but in theory the following optimization might be possible: check if the driver supports fast count, and in the case there are no filters, that count should be a cheap lookup (like GPKG without any filtering), and at that point actually populate num_features. That would avoid using the growing chunk size in those cases (and we could directly read the entire data or directly use the max chunk size)

[theroggy]

I don't know if it is worth the added complexity, but in theory the following optimization might be possible: check if the driver supports fast count, and in the case there are no filters, that count should be a cheap lookup (like GPKG without any filtering), and at that point actually populate num_features. That would avoid using the growing chunk size in those cases (and we could directly read the entire data or directly use the max chunk size)

According to my tests when I wrote the PR, I didn't measure a significant difference between chunking or not for the case where no filtering was done, but it is not difficult to shortcut this case, so I added it.

[jorisvandenbossche]

Looks good! Just two suggestions for clarifying comments for the future reader

[jorisvandenbossche]

(and needs to fix the merge conflicts)