[SPARK-54831][SQL] Add COMMENT ON COLUMN support to set and remove column comments

[ganeshashree]

This PR adds support for the COMMENT ON COLUMN SQL syntax to set, update, and remove comments on table columns.

The syntax follows the SQL standard:

COMMENT ON COLUMN [catalog.][[database.]table.]column IS { 'comment_text' | NULL }

Key features:

Set/Update comment: COMMENT ON COLUMN table.column IS 'text' - Sets or updates a column comment
Remove comment: COMMENT ON COLUMN table.column IS NULL - Removes the comment

Flexible qualification: Supports table.column, database.table.column, and catalog.database.table.column formats

Implementation highlights:

• Added CommentOnColumn logical plan node in v2AlterTableCommands.scala
• Parser support in SqlBaseParser.g4 and AstBuilder.scala
• Works with both V2 DataSource tables and V1/session catalog tables

What changes were proposed in this pull request?

Why are the changes needed?

Currently, Spark only supports setting column comments via:

1. CREATE TABLE with COMMENT clause
2. ALTER TABLE ... ALTER COLUMN ... COMMENT (doesn't support removal)

This PR provides a standard SQL syntax (COMMENT ON COLUMN) that:

• Follows SQL standard conventions (consistent with existing COMMENT ON TABLE)
• Allows explicit removal of column comments via IS NULL
• Provides cleaner, more intuitive syntax for managing column-level documentation

Does this PR introduce any user-facing change?

Yes. New SQL syntax:

-- Set a column comment
COMMENT ON COLUMN my_table.my_column IS 'This is a column comment';

-- Update a column comment  
COMMENT ON COLUMN my_table.my_column IS 'Updated comment';

-- Remove a column comment
COMMENT ON COLUMN my_table.my_column IS NULL;

-- Set empty string comment
COMMENT ON COLUMN my_table.my_column IS '';

-- Works with qualified names
COMMENT ON COLUMN my_catalog.my_db.my_table.my_column IS 'comment';

How was this patch tested?

Added comprehensive parser tests in DDLParserSuite.scala covering:

• Various qualification levels (table.column, database.table.column, etc.)
• NULL handling for comment removal

Added integration tests in DataSourceV2SQLSuite.scala covering:

• Setting, updating, and removing comments on V1 and V2 tables
• NULL vs. empty string behavior
• Different qualification formats
• Error cases (column not found)

Was this patch authored or co-authored using generative AI tooling?

No