feat: add ExternalPacked blob kind for blob v2

[wojiaodoubao]

A common practice for organizing multimodal data is to package a large number of scattered files into several large archives (such as tar or zip files), which can prevent excessive inode consumption, achieve higher read speeds via streaming, and avoid inconsistencies between metadata and the scattered stored files.

To better support such archives, I suggest we extend the ExternalPacked format for Lance BlobV2. For example linking images from the archives to a Lance table.

[codecov]

Codecov Report

❌ Patch coverage is 57.85124% with 51 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/blob.rs	17.74%	46 Missing and 5 partials ⚠️

📢 Thoughts on this report? Let us know!

[Xuanwo]

This idea is kinda interesting. So your idea focuses more on reading an external packed blob, and Lance won't generate such a blob itself, is that correct?

[wojiaodoubao]

So your idea focuses more on reading an external packed blob, and Lance won't generate such a blob itself, is that correct?

Yes.

[Xuanwo]

Yes.

Then I think we don't need to add a new type. We can simply enable reading a range from an external blob.

[wojiaodoubao]

We can simply enable reading a range from an external blob.

Do you mean we could enable External BlobFile to handle any arbitrary range(currently the position is always 0), thus eliminating the need to introduce ExternalPacked as an extra component?

It makes sense to me, let me update the pr.

[Xuanwo]

Do you mean we could enable External BlobFile to handle any arbitrary range(currently the position is always 0), thus eliminating the need to introduce ExternalPacked as an extra component?

Yes!