Core: Add metadata table for discovering metadata tables

[yangshangqing95]

close #17132

Summary

This PR adds a new Iceberg metadata table, metadata_tables, to make available metadata tables discoverable through the existing metadata table mechanism.

For example, in Spark users can query:

SELECT * FROM catalog.db.table.metadata_tables;

Result:

entries
files
data_files
delete_files
history
metadata_log_entries
snapshots
refs
manifests
partitions
all_data_files
all_delete_files
all_files
all_manifests
all_entries
position_deletes
metadata_tables

The new metadata table returns the supported metadata table names

This avoids requiring users to know all metadata table names ahead of time from documentation or source code.

Also, replace selected hard-coded metadata table suffixes in tests and metadata table implementations with MetadataTableType#tableName, enhance code robustness.

[gaborkaszab]

Hi @yangshangqing95 ,
I took a Quick Look and left some comments wrt refactors and the new metadata scan implementation.

A general note: querying static data such as metadata table names through SELECT * FROM catalog.db.table.metadata_tables seems somewhat odd to me. We'd expect the same answer regardless the table we use for the query, right?
Anyway, I recall for Iceberg V4 there are consideration to not allow some of the current metadata table and have new ones because the physical representation of a table changes. So this might make sense to have different answers based on the table's version.
Let's see what others think!

[gaborkaszab]

I randomly checked other metadata tables and they seem to use the following InputFile:
table().io().newInputFile(table().operations().current().metadataFileLocation()),

[yangshangqing95]

That’s a good point. For listing the available metadata tables, at least at this stage, we don’t actually need to read any physical file.

Using a directly constructed empty DataFile has a couple of benefits here:

1. It avoids unnecessary file operations.
2. It makes the intent clearer in the code, since otherwise it may look like the metadata table information is stored in or read from the table metadata file.

So I think using an empty DataFile is more explicit for this particular metadata table.

[gaborkaszab]

Please follow the pattern other metadata tables are using. See my comment above for constructor vs of()

[yangshangqing95]

done

[gaborkaszab]

Other metadata tables use StaticDataTask.of()

[yangshangqing95]

same reason as the above one

[gaborkaszab]

see comment above

[yangshangqing95]

done

[gaborkaszab]

Checking RefsTable instead of constructing StaticDataTask.Row[]', we can simply use Collectionwith a transform function to convert it to row viaStaticDataTask.Row.of()`

[yangshangqing95]

Hi @gaborkaszab same reason with the usage of empty Datafile, as constructor accept parameter Row[]

StaticDataTask(
      DataFile metadataFile, 
      Schema tableSchema, 
      Schema projectedSchema, 
      StructLike[] rows)

[gaborkaszab]

I don't think we are expected to use the constructor from metadata table code. They use StaticDataTask.of() instead.

[yangshangqing95]

done

[yangshangqing95]

Hi @yangshangqing95 , I took a Quick Look and left some comments wrt refactors and the new metadata scan implementation.

A general note: querying static data such as metadata table names through SELECT * FROM catalog.db.table.metadata_tables seems somewhat odd to me. We'd expect the same answer regardless the table we use for the query, right? Anyway, I recall for Iceberg V4 there are consideration to not allow some of the current metadata table and have new ones because the physical representation of a table changes. So this might make sense to have different answers based on the table's version. Let's see what others think!

Hi @gaborkaszab thanks for taking a look and leaving the comments!

Regarding the metadata table query, I did consider this initially as well. From both an implementation perspective and the current project structure, I’m still leaning toward exposing it under each table, mainly for a few reasons:

1. We don’t currently have a default metadata catalog to store this kind of information, and introducing one just for this metadata feels a bit heavy.
2. Given the current code structure, putting this in MetadataTableType keeps the change smaller and less intrusive.
3. There may be more metadata table types in the future, and the supported set could differ across Iceberg versions such as V1, V2, V3, and V4. Keeping this under each table leaves room for future extension, where each table version can expose the metadata tables that are meaningful for it.

So from that perspective, it may not necessarily be true that all tables should always return the exact same metadata tables.