Spark Connect: Kerberos authentication fails for Iceberg/Hive Metastore (GSS initiate failed)

[jusch23]

I've deployed a Spark Connect server in my Kubernetes cluster using the SparkConnectServer CR from the spark-k8s-operator. I'm attempting to query an iceberg table via pyspark, but the connection to hive metastore fails:

$ python pyspark-test.py
3.5.6
3.5.6-stackable25.7.0
Traceback (most recent call last):
  File "/home/linuxuser/pyspark-test.py", line 43, in <module>
    df.show()
  File "/home/linuxuser/pyspark-venv/lib64/python3.12/site-packages/pyspark/sql/connect/dataframe.py", line 996, in show
    print(self._show_string(n, truncate, vertical))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linuxuser/pyspark-venv/lib64/python3.12/site-packages/pyspark/sql/connect/dataframe.py", line 753, in _show_string
    ).toPandas()
      ^^^^^^^^^^
  File "/home/linuxuser/pyspark-venv/lib64/python3.12/site-packages/pyspark/sql/connect/dataframe.py", line 1663, in toPandas
    return self._session.client.to_pandas(query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linuxuser/pyspark-venv/lib64/python3.12/site-packages/pyspark/sql/connect/client/core.py", line 873, in to_pandas
    table, schema, metrics, observed_metrics, _ = self._execute_and_fetch(
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linuxuser/pyspark-venv/lib64/python3.12/site-packages/pyspark/sql/connect/client/core.py", line 1283, in _execute_and_fetch
    for response in self._execute_and_fetch_as_iterator(req):
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linuxuser/pyspark-venv/lib64/python3.12/site-packages/pyspark/sql/connect/client/core.py", line 1264, in _execute_and_fetch_as_iterator
    self._handle_error(error)
  File "/home/linuxuser/pyspark-venv/lib64/python3.12/site-packages/pyspark/sql/connect/client/core.py", line 1503, in _handle_error
    self._handle_rpc_error(error)
  File "/home/linuxuser/pyspark-venv/lib64/python3.12/site-packages/pyspark/sql/connect/client/core.py", line 1539, in _handle_rpc_error
    raise convert_exception(info, status.message) from None
pyspark.errors.exceptions.connect.SparkConnectGrpcException: (org.apache.iceberg.hive.RuntimeMetaException) Failed to connect to Hive Metastore

In the spark connect server I can see issues with kerberos authentication, but unfortunately I was not able to resolve them. From the logs it seems like initial the authentication works, but the Kerberos ticket is not being used for the hive metastore connection afterwards. The spark configuration I'm using works fine for regular SparkApplication jobs with Hive. Is this kind of setup supported with Spark Connect as well, and if so what might be missing?

Spark Connect server logs:

2026-06-17T08:45:39,918 INFO [main] org.apache.spark.deploy.k8s.SparkKubernetesClientFactory - Auto-configuring K8S client using current context from users K8S config file
2026-06-17T08:45:42,006 INFO [main] org.apache.spark.deploy.security.HadoopDelegationTokenManager - Attempting to login to KDC using principal: myuser@MY.EXAMPLE.COM
Java config name: null
Native config name: /etc/krb5.conf
Loading config file from /etc/krb5.conf
Loading krb5 profile at /etc/krb5.conf
libdefaults = {
default_realm = MY.EXAMPLE.COM
}
realms = {
MY.EXAMPLE.COM = {
kdc = MY.EXAMPLE.COM
}
}
domain_realm = {
.my.example.com=MY.EXAMPLE.COM
my.example.com=MY.EXAMPLE.COM
cluster.local = MY.EXAMPLE.COM
.cluster.local = MY.EXAMPLE.COM
}
Loaded from native config
>>> KdcAccessibility: reset
>>> KdcAccessibility: reset
>>> KeyTabInputStream, readName(): MY.EXAMPLE.COM
>>> KeyTabInputStream, readName(): myuser
>>> KeyTab: load() entry length: 88; type: 18
Looking for keys for: myuser@MY.EXAMPLE.COM
Added key: 18, version: 1
Looking for keys for: myuser@MY.EXAMPLE.COM
Added key: 18, version: 1
Getting permitted_enctypes from libdefaults
default_enctypes were null, using builtin default etypes for configuration default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 20 19.
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=MY.EXAMPLE.COM UDP:88, timeout=30000, number of retries =3, #bytes=157
>>> KDCCommunication: kdc=MY.EXAMPLE.COM UDP:88, timeout=30000,Attempt =1, #bytes=157
>>> KrbKdcReq send: #bytes read=204
>>>Pre-Authentication Data:
         PA-DATA type = 19
         PA-ETYPE-INFO2 etype = 18, salt = MY.EXAMPLE.COMmyuser, s2kparams = null

>>>Pre-Authentication Data:
         PA-DATA type = 2
         PA-ENC-TIMESTAMP
>>>Pre-Authentication Data:
         PA-DATA type = 16

>>>Pre-Authentication Data:
         PA-DATA type = 15

>>> KdcAccessibility: remove MY.EXAMPLE.COM
>>> KDCRep: init() encoding tag is 126 req type is 11
>>>KRBError:
         sTime is Wed Jun 17 08:45:42 GMT 2026 1781685942000
         suSec is 987219
         error code is 25
         error Message is Additional pre-authentication required
         sname is krbtgt/MY.EXAMPLE.COM@MY.EXAMPLE.COM
         eData provided.
         msgType is 30
>>>Pre-Authentication Data:
         PA-DATA type = 19
         PA-ETYPE-INFO2 etype = 18, salt = MY.EXAMPLE.COMmyuser, s2kparams = null

>>>Pre-Authentication Data:
         PA-DATA type = 2
         PA-ENC-TIMESTAMP
>>>Pre-Authentication Data:
         PA-DATA type = 16

>>>Pre-Authentication Data:
         PA-DATA type = 15

KrbAsReqBuilder: PREAUTH FAILED/REQ, re-send AS-REQ
Getting permitted_enctypes from libdefaults
default_enctypes were null, using builtin default etypes for configuration default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 20 19.
Looking for keys for: myuser@MY.EXAMPLE.COM
Added key: 18, version: 1
Looking for keys for: myuser@MY.EXAMPLE.COM
Added key: 18, version: 1
Getting permitted_enctypes from libdefaults
default_enctypes were null, using builtin default etypes for configuration default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 20 19.
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=MY.EXAMPLE.COM UDP:88, timeout=30000, number of retries =3, #bytes=244
>>> KDCCommunication: kdc=MY.EXAMPLE.COM UDP:88, timeout=30000,Attempt =1, #bytes=244
>>> KrbKdcReq send: #bytes read=105
>>> KrbKdcReq send: kdc=MY.EXAMPLE.COM TCP:88, timeout=30000, number of retries =3, #bytes=244
>>> KDCCommunication: kdc=MY.EXAMPLE.COM TCP:88, timeout=30000,Attempt =1, #bytes=244
>>>DEBUG: TCPClient reading 1873 bytes
>>> KrbKdcReq send: #bytes read=1873
>>> KdcAccessibility: remove MY.EXAMPLE.COM
Looking for keys for: myuser@MY.EXAMPLE.COM
Added key: 18, version: 1
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbAsRep cons in KrbAsReq.getReply myuser
2026-06-17T08:45:42,116 INFO [main] org.apache.spark.deploy.security.HadoopDelegationTokenManager - Successfully logged into KDC.
2026-06-17T08:45:42,127 INFO [main] org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider - getting token for: org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem@282aea3c with renewer myuser@MY.EXAMPLE.COM
2026-06-17T08:45:42,128 INFO [main] org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider - getting token for: org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem@282aea3c with renewer myuser@MY.EXAMPLE.COM
2026-06-17T08:45:44,398 INFO [main] org.apache.hadoop.hive.conf.HiveConf - Found configuration file null
2026-06-17T08:45:44,866 INFO [main] org.apache.spark.deploy.security.HadoopDelegationTokenManager - Calculated delay on renewal is 6917527691376622592, based on next renewal 9223372036854775807 and the ratio 0.75, and current time 1781685944863
2026-06-17T08:45:44,877 INFO [main] org.apache.spark.deploy.security.HadoopDelegationTokenManager - Scheduling renewal in 1921535469826.8 h.
2026-06-17T08:45:44,881 INFO [main] org.apache.spark.deploy.security.HadoopDelegationTokenManager - Updating delegation tokens.
2026-06-17T08:45:44,901 INFO [dispatcher-CoarseGrainedScheduler] org.apache.spark.deploy.SparkHadoopUtil - Updating delegation tokens for current user.
2026-06-17T08:45:44,963 INFO [main] org.apache.spark.deploy.SparkHadoopUtil - Updating delegation tokens for current user.
2026-06-17T08:45:45,165 INFO [kubernetes-executor-snapshots-subscribers-0] org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator - Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 1, known: 0, sharedSlotFromPendingPods: 2147483647.
2026-06-17T08:45:45,195 INFO [main] org.apache.spark.deploy.k8s.submit.KubernetesClientUtils - Spark configuration files loaded from Some(/stackable/spark/conf) : security.properties,metrics.properties,template.yaml
2026-06-17T08:45:45,698 INFO [kubernetes-executor-snapshots-subscribers-0] org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator - Found 0 reusable PVCs from 0 PVCs
2026-06-17T08:45:45,768 INFO [main] org.apache.spark.util.Utils - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33043.
2026-06-17T08:45:45,769 INFO [main] org.apache.spark.network.netty.NettyBlockTransferService - Server created on spark-connect-server-headless:33043
2026-06-17T08:45:45,772 INFO [main] org.apache.spark.storage.BlockManager - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2026-06-17T08:45:45,785 INFO [main] org.apache.spark.storage.BlockManagerMaster - Registering BlockManager BlockManagerId(driver, spark-connect-server-headless, 33043, None)
2026-06-17T08:45:45,790 INFO [dispatcher-BlockManagerMaster] org.apache.spark.storage.BlockManagerMasterEndpoint - Registering block manager spark-connect-server-headless:33043 with 413.9 MiB RAM, BlockManagerId(driver, spark-connect-server-headless, 33043, None)
2026-06-17T08:45:45,865 INFO [main] org.apache.spark.storage.BlockManagerMaster - Registered BlockManager BlockManagerId(driver, spark-connect-server-headless, 33043, None)
2026-06-17T08:45:45,868 INFO [main] org.apache.spark.storage.BlockManager - Initialized BlockManager: BlockManagerId(driver, spark-connect-server-headless, 33043, None)
2026-06-17T08:45:45,885 INFO [kubernetes-executor-snapshots-subscribers-0] org.apache.spark.deploy.k8s.submit.KubernetesClientUtils - Spark configuration files loaded from Some(/stackable/spark/conf) : security.properties,metrics.properties,template.yaml
2026-06-17T08:45:45,974 INFO [kubernetes-executor-snapshots-subscribers-0] org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep - Decommissioning not enabled, skipping shutdown script
2026-06-17T08:45:45,976 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Stopped o.s.j.s.ServletContextHandler@57b130f6{/,null,STOPPED,@Spark}
2026-06-17T08:45:45,979 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@395eb363{/jobs,null,AVAILABLE,@Spark}
2026-06-17T08:45:45,981 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@3355b8ff{/jobs/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:45,982 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@2cce10bc{/jobs/job,null,AVAILABLE,@Spark}
2026-06-17T08:45:45,984 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@12e2f5ab{/jobs/job/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:45,985 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@3bdbf8d3{/stages,null,AVAILABLE,@Spark}
2026-06-17T08:45:45,986 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@3b25709e{/stages/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,065 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@43294e9b{/stages/stage,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,070 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7672960e{/stages/stage/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,072 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5fb514c2{/stages/pool,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,074 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@22a63740{/stages/pool/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,075 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@25823192{/storage,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,077 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@64544ee4{/storage/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,078 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@4641f66c{/storage/rdd,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,079 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@2cacb445{/storage/rdd/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,081 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@2168def{/environment,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,082 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@54a6ef6{/environment/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,084 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@4f2fdf3d{/executors,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,085 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@71ee2fff{/executors/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,086 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@252d8df6{/executors/threadDump,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,088 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@774013dc{/executors/threadDump/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,089 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@391dfe7c{/executors/heapHistogram,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,166 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1c135f63{/executors/heapHistogram/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,182 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@355493bf{/static,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,185 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1169fdfd{/,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,187 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@52eca410{/api,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,189 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@549d14fe{/metrics,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,191 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@213ceb4e{/jobs/job/kill,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,192 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@4d174189{/stages/stage/kill,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,278 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@2f1b8770{/metrics/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:46,280 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@22bbbe6{/metrics/prometheus,null,AVAILABLE,@Spark}
2026-06-17T08:45:51,645 INFO [dispatcher-CoarseGrainedScheduler] org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint - No executor found for 192.168.7.223:33302
2026-06-17T08:45:52,598 INFO [dispatcher-CoarseGrainedScheduler] org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.7.223:33318) with ID 1,  ResourceProfileId 0
2026-06-17T08:45:52,606 INFO [main] org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
2026-06-17T08:45:52,794 INFO [dispatcher-BlockManagerMaster] org.apache.spark.storage.BlockManagerMasterEndpoint - Registering block manager 192.168.7.223:41661 with 413.9 MiB RAM, BlockManagerId(1, 192.168.7.223, 41661, None)
2026-06-17T08:45:53,505 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7a925ac4{/connect,null,AVAILABLE,@Spark}
2026-06-17T08:45:53,508 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@56114349{/connect/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:53,510 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@88fe9a8{/connect/session,null,AVAILABLE,@Spark}
2026-06-17T08:45:53,513 INFO [main] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@37e0614e{/connect/session/json,null,AVAILABLE,@Spark}
2026-06-17T08:45:53,566 INFO [main] org.apache.spark.sql.connect.service.SparkConnectServer - Spark Connect server started at: 0:0:0:0:0:0:0:0%0:15002
2026-06-17T08:47:33,998 INFO [grpc-default-executor-0] org.apache.spark.sql.internal.SharedState - Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
2026-06-17T08:47:34,002 INFO [grpc-default-executor-0] org.apache.spark.sql.internal.SharedState - Warehouse path is 'file:/stackable/spark/spark-warehouse'.
2026-06-17T08:47:34,173 INFO [grpc-default-executor-0] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1874a1db{/SQL,null,AVAILABLE,@Spark}
2026-06-17T08:47:34,175 INFO [grpc-default-executor-0] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@28d78cb{/SQL/json,null,AVAILABLE,@Spark}
2026-06-17T08:47:34,177 INFO [grpc-default-executor-0] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@646cfdd7{/SQL/execution,null,AVAILABLE,@Spark}
2026-06-17T08:47:34,178 INFO [grpc-default-executor-0] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@6c7c7174{/SQL/execution/json,null,AVAILABLE,@Spark}
2026-06-17T08:47:34,181 INFO [grpc-default-executor-0] org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@72f4bd86{/static/sql,null,AVAILABLE,@Spark}
2026-06-17T08:47:35,832 INFO [grpc-default-executor-0] org.apache.spark.sql.connect.service.SparkConnectExecutionManager - ExecuteHolder ExecuteKey(linuxuser,37bf356d-defd-4bad-8eaa-9d67e3269e5a,ddeaa276-b96f-40f4-ad68-80bb006d286d) is created.
2026-06-17T08:47:35,833 INFO [grpc-default-executor-0] org.apache.spark.sql.connect.service.SparkConnectExecutionManager - Starting thread for cleanup of abandoned executions every 30000 ms
2026-06-17T08:47:36,109 INFO [SparkConnectGRPCSender_opId=ddeaa276-b96f-40f4-ad68-80bb006d286d_startIndex=0] org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender - Starting for opId=ddeaa276-b96f-40f4-ad68-80bb006d286d, reattachable=true, lastConsumedStreamIndex=0
2026-06-17T08:47:37,178 INFO [SparkConnectExecuteThread_opId=ddeaa276-b96f-40f4-ad68-80bb006d286d] hive.metastore - Trying to connect to metastore with URI thrift://hive-dev.default.svc.cluster.local:9083
2026-06-17T08:47:37,298 ERROR [SparkConnectExecuteThread_opId=ddeaa276-b96f-40f4-ad68-80bb006d286d] org.apache.thrift.transport.TSaslTransport - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:228) ~[jdk.security.jgss:?]
        at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:95) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:38) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) ~[hive-shims-common-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) ~[hive-shims-common-2.3.9.jar:2.3.9]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) ~[hadoop-client-api-3.4.1-stackable25.7.0.jar:?]
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) ~[hive-shims-common-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:478) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245) ~[hive-metastore-2.3.9.jar:2.3.9]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1740) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:97) ~[hive-metastore-2.3.9.jar:2.3.9]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
        at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:60) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:72) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:185) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:63) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:34) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:143) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:70) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:65) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:107) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:88) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:71) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:49) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916) ~[?:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.CachingCatalog.loadTable(CachingCatalog.java:147) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.spark.SparkCatalog.load(SparkCatalog.java:846) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:170) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.spark.sql.connector.catalog.CatalogV2Util$.getTable(CatalogV2Util.scala:363) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:337) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$resolveRelation$5(Analyzer.scala:1315) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$resolveRelation$1(Analyzer.scala:1311) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$resolveRelation(Analyzer.scala:1296) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$14.applyOrElse(Analyzer.scala:1153) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$14.applyOrElse(Analyzer.scala:1117) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:138) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76) ~[spark-sql-api_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:138) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:32) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1117) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1076) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:222) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126) ~[scala-library-2.12.18.jar:?]
        at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122) ~[scala-library-2.12.18.jar:?]
        at scala.collection.immutable.List.foldLeft(List.scala:91) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:219) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:211) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at scala.collection.immutable.List.foreach(List.scala:431) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:211) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:240) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:236) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:187) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:236) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:202) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:182) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:89) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:182) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:223) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:222) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:77) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:138) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:219) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:546) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:219) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:218) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:77) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:74) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:66) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:91) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:89) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.planner.SparkConnectPlanner.transformShowString(SparkConnectPlanner.scala:248) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.planner.SparkConnectPlanner.transformRelation(SparkConnectPlanner.scala:104) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.SparkConnectPlanExecution.handlePlan(SparkConnectPlanExecution.scala:64) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.handlePlan(ExecuteThreadRunner.scala:197) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1(ExecuteThreadRunner.scala:165) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1$adapted(ExecuteThreadRunner.scala:138) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$2(SessionHolder.scala:189) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$1(SessionHolder.scala:189) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94) ~[spark-core_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withContextClassLoader$1(SessionHolder.scala:176) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:178) [spark-core_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.withContextClassLoader(SessionHolder.scala:175) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.withSession(SessionHolder.scala:188) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.executeInternal(ExecuteThreadRunner.scala:138) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.org$apache$spark$sql$connect$execution$ExecuteThreadRunner$$execute(ExecuteThreadRunner.scala:90) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner$ExecutionThread.run(ExecuteThreadRunner.scala:235) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:166) ~[java.security.jgss:?]
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:126) ~[java.security.jgss:?]
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:195) ~[java.security.jgss:?]
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:205) ~[java.security.jgss:?]
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:230) ~[java.security.jgss:?]
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:196) ~[java.security.jgss:?]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:209) ~[jdk.security.jgss:?]
        ... 114 more
2026-06-17T08:47:37,317 WARN [SparkConnectExecuteThread_opId=ddeaa276-b96f-40f4-ad68-80bb006d286d] hive.metastore - Failed to connect to the MetaStore Server...
org.apache.thrift.transport.TTransportException: GSS initiate failed
        at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:314) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:38) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) ~[hive-shims-common-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) ~[hive-shims-common-2.3.9.jar:2.3.9]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) ~[hadoop-client-api-3.4.1-stackable25.7.0.jar:?]
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) ~[hive-shims-common-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:478) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245) ~[hive-metastore-2.3.9.jar:2.3.9]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1740) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:97) ~[hive-metastore-2.3.9.jar:2.3.9]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
        at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:60) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:72) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:185) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:63) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:34) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:143) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:70) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:65) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:107) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:88) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:71) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:49) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916) ~[?:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.CachingCatalog.loadTable(CachingCatalog.java:147) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.spark.SparkCatalog.load(SparkCatalog.java:846) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:170) ~[org.apache.iceberg_iceberg-spark-runtime-3.5_2.12-1.10.1.jar:?]
        at org.apache.spark.sql.connector.catalog.CatalogV2Util$.getTable(CatalogV2Util.scala:363) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:337) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$resolveRelation$5(Analyzer.scala:1315) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$resolveRelation$1(Analyzer.scala:1311) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$resolveRelation(Analyzer.scala:1296) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$14.applyOrElse(Analyzer.scala:1153) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$14.applyOrElse(Analyzer.scala:1117) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:138) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76) ~[spark-sql-api_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:138) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:32) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1117) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1076) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:222) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126) ~[scala-library-2.12.18.jar:?]
        at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122) ~[scala-library-2.12.18.jar:?]
        at scala.collection.immutable.List.foldLeft(List.scala:91) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:219) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:211) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at scala.collection.immutable.List.foreach(List.scala:431) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:211) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:240) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:236) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:187) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:236) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:202) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:182) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:89) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:182) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:223) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:222) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:77) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:138) ~[spark-catalyst_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:219) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:546) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:219) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:218) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:77) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:74) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:66) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:91) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:89) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.planner.SparkConnectPlanner.transformShowString(SparkConnectPlanner.scala:248) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.planner.SparkConnectPlanner.transformRelation(SparkConnectPlanner.scala:104) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.SparkConnectPlanExecution.handlePlan(SparkConnectPlanExecution.scala:64) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.handlePlan(ExecuteThreadRunner.scala:197) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1(ExecuteThreadRunner.scala:165) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1$adapted(ExecuteThreadRunner.scala:138) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$2(SessionHolder.scala:189) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) ~[spark-sql_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$1(SessionHolder.scala:189) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94) ~[spark-core_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withContextClassLoader$1(SessionHolder.scala:176) ~[spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:178) [spark-core_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.withContextClassLoader(SessionHolder.scala:175) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.service.SessionHolder.withSession(SessionHolder.scala:188) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.executeInternal(ExecuteThreadRunner.scala:138) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner.org$apache$spark$sql$connect$execution$ExecuteThreadRunner$$execute(ExecuteThreadRunner.scala:90) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
        at org.apache.spark.sql.connect.execution.ExecuteThreadRunner$ExecutionThread.run(ExecuteThreadRunner.scala:235) [spark-connect_2.12-3.5.6-stackable25.7.0.jar:3.5.6-stackable25.7.0]
2026-06-17T08:47:37,321 INFO [SparkConnectExecuteThread_opId=ddeaa276-b96f-40f4-ad68-80bb006d286d] hive.metastore - Waiting 1 seconds before next connection attempt.
...

SparkConnect CR:

apiVersion: spark.stackable.tech/v1alpha1
kind: SparkConnectServer
metadata:
  name: spark-connect
  namespace: default
spec:
  args:
  - --repositories http://proxy-svc.default.svc.cluster.local/content/groups/public/
  - --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.1,org.apache.hadoop:hadoop-aws:3.3.4,org.apache.spark:spark-hadoop-cloud_2.12:3.5.6
  clusterOperation:
    reconciliationPaused: false
    stopped: false
  executor:
    cliOverrides: {}
    config:
      affinity:
        nodeAffinity: null
        nodeSelector: null
        podAffinity: null
        podAntiAffinity: null
      logging:
        containers:
          spark:
            custom:
              configMap: log4j-config
        enableVectorAgent: null
      resources:
        cpu:
          max: null
          min: 100m
        memory:
          limit: null
          runtimeLimits: {}
        storage: {}
    configOverrides:
      spark-defaults.conf:
        spark.driver.extraJavaOptions: ' -Djava.security.auth.login.config=/stackable/conf/jaas-config/jaas.conf
          -Djava.security.krb5.realm=MY.EXAMPLE.COM -Djava.security.krb5.kdc=MY.EXAMPLE.COM -Dsun.security.krb5.debug=true'
        spark.executor.extraJavaOptions: ' -Djava.security.auth.login.config=/stackable/conf/jaas-config/jaas.conf
          -Djava.security.krb5.realm=MY.EXAMPLE.COM -Djava.security.krb5.kdc=MY.EXAMPLE.COM -Dsun.security.krb5.debug=true'
        spark.executor.instances: "1"
        spark.hadoop.hive.metastore.kerberos.principal: hive/_HOST@MY.EXAMPLE.COM
        spark.hadoop.hive.metastore.sasl.enabled: "true"
        spark.kerberos.keytab: /stackable/conf/user/keytab
        spark.kerberos.principal: myuser@MY.EXAMPLE.COM
        spark.kubernetes.file.upload.path: s3a://mybucket-internal/.spark
        spark.kubernetes.hadoop.configMapName: hadoop-kerberos
        spark.hadoop.hadoop.security.authentication: kerberos
        spark.hadoop.hadoop.security.authorization: "true"
        spark.kubernetes.executor.hadoopConfigMapName: hadoop-kerberos
        spark.kubernetes.kerberos.krb5.configMapName: krb5-conf
        spark.rpc.message.maxSize: "1024"
        spark.sql.catalog.iceberg: org.apache.iceberg.spark.SparkCatalog
        spark.sql.catalog.iceberg.type: hive
        spark.sql.catalog.iceberg.uri: thrift://hive-dev.default.svc.cluster.local:9083
        spark.sql.catalog.iceberg.warehouse: s3a://mybucket-external/warehouse
        spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
        spark.sql.iceberg.check-ordering: "false"
        spark.sql.catalog.iceberg.hadoop.security.authentication: kerberos
        spark.sql.catalog.iceberg.hive.metastore.sasl.enabled: "true"
        spark.sql.catalog.iceberg.hive.metastore.kerberos.principal: hive/_HOST@MY.EXAMPLE.COM
        spark.log.level: "DEBUG"
    envOverrides: {}
    jvmArgumentOverrides:
      add: []
      remove: []
      removeRegex: []
    podOverrides:
      spec:
        containers:
        - name: spark
          volumeMounts:
          - mountPath: /stackable/conf/user
            name: keytab
            readOnly: true
          - mountPath: /stackable/conf/jaas-config
            name: jaas-config
            readOnly: true
        volumes:
        - name: keytab
          secret:
            secretName: keytab-product-default
        - name: jaas-config
          configMap:
            name: jaas-config
  image:
    productVersion: 3.5.6
    pullPolicy: Always
    stackableVersion: 25.7.0
  server:
    cliOverrides: {}
    config:
      logging:
        containers:
          spark:
            custom:
              configMap: log4j-config
        enableVectorAgent: null
      resources:
        cpu:
          max: null
          min: 100m
        memory:
          limit: null
          runtimeLimits: {}
        storage: {}
    configOverrides:
      spark-defaults.conf:
        spark.driver.cores: "4"
        spark.driver.extraJavaOptions: ' -Djava.security.auth.login.config=/stackable/conf/jaas-config/jaas.conf
          -Djava.security.krb5.realm=MY.EXAMPLE.COM -Djava.security.krb5.kdc=MY.EXAMPLE.COM -Dsun.security.krb5.debug=true'
        spark.executor.extraJavaOptions: ' -Djava.security.auth.login.config=/stackable/conf/jaas-config/jaas.conf
          -Djava.security.krb5.realm=MY.EXAMPLE.COM -Djava.security.krb5.kdc=MY.EXAMPLE.COM -Dsun.security.krb5.debug=true'
        spark.hadoop.hive.metastore.kerberos.principal: hive/_HOST@MY.EXAMPLE.COM
        spark.hadoop.hive.metastore.sasl.enabled: "true"
        spark.hadoop.hadoop.security.authentication: kerberos
        spark.hadoop.hadoop.security.authorization: "true"
        spark.kerberos.keytab: /stackable/conf/user/keytab
        spark.kerberos.principal: myuser@MY.EXAMPLE.COM
        spark.kubernetes.file.upload.path: s3a://mybucket-internal/.spark
        spark.kubernetes.hadoop.configMapName: hadoop-kerberos
        spark.kubernetes.executor.hadoopConfigMapName: hadoop-kerberos
        spark.kubernetes.kerberos.krb5.configMapName: krb5-conf
        spark.rpc.message.maxSize: "1024"
        spark.sql.catalog.iceberg: org.apache.iceberg.spark.SparkCatalog
        spark.sql.catalog.iceberg.type: hive
        spark.sql.catalog.iceberg.uri: thrift://hive-dev.default.svc.cluster.local:9083
        spark.sql.catalog.iceberg.warehouse: s3a://mybucket-external/warehouse
        spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
        spark.sql.catalog.iceberg.hadoop.security.authentication: kerberos
        spark.sql.catalog.iceberg.hive.metastore.sasl.enabled: "true"
        spark.sql.catalog.iceberg.hive.metastore.kerberos.principal: hive/_HOST@MY.EXAMPLE.COM
        spark.sql.iceberg.check-ordering: "false"
        spark.log.level: "DEBUG"
    envOverrides: {}
    jvmArgumentOverrides:
      add: []
      remove: []
      removeRegex: []
    podOverrides:
      spec:
        containers:
        - name: spark
          volumeMounts:
          - mountPath: /stackable/conf/user
            name: keytab
            readOnly: true
          - mountPath: /stackable/conf/jaas-config
            name: jaas-config
            readOnly: true
        volumes:
        - name: keytab
          secret:
            secretName: keytab-product-default
        - name: jaas-config
          configMap:
            name: jaas-config
    roleConfig:
      listenerClass: cluster-internal

pyspark-test.py:

from pathlib import Path
import grpc

import pyspark
from pyspark.sql.connect.client import ChannelBuilder
from pyspark.sql.connect.session import SparkSession
from pyspark.sql import Row

from datetime import datetime, date

class CustomCAChannelBuilder(ChannelBuilder):
    def __init__(self, url: str, ca_pem_path: str):
        super().__init__(url)
        self._ca_pem = Path(ca_pem_path).read_bytes()

    def toChannel(self):
        # Build TLS credentials with your custom root CA
        tls_creds = grpc.ssl_channel_credentials(root_certificates=self._ca_pem)

        # Preserve token auth if your sc:// URL includes ;token=...
        if self._token is not None:
            tls_creds = grpc.composite_channel_credentials(
                tls_creds,
                grpc.access_token_call_credentials(self._token),
            )

        return grpc.secure_channel(self.endpoint, tls_creds, options=self._channel_options)


builder = CustomCAChannelBuilder(
    "sc://spark-connect.my.example.com:443/;use_ssl=true",
    "/etc/pki/tls/certs/ca-bundle.crt",
)

spark = SparkSession.builder.channelBuilder(builder).getOrCreate()

print(pyspark.__version__)
print(spark.version)
df = spark.table('`iceberg`.`test`.`table`')
df.show()

[razvan]

I added an integration test for this usecase here.

There are a couple of important things to note that I also learned myself while doing this:

1. Your original keytab problem is solved by adding an init container that runs kinit and puts the result in a shared cache KRB5CCNAME. This is needed because Spark doesn't login automatically when running in --deploy-mode client which is what the SDP opertor is using. The logs you see are from the Hadoop libs which are not used by Iceberg/Hive.

2. You need to explicitly tell the JVM where the Kerberos config is located:

    jvmArgumentOverrides:
      add:
        - -Djava.security.krb5.conf=/stackable/kerberos/krb5.conf
        - -Djavax.security.auth.useSubjectCredsOnly=false

3. You need the latest dev version of the operator due to a bug that was recently fixed here.

4. What I also learned while implementing the test application is that there is a bug in the Spark Connect class loader that prevents you from running distributed scans on Iceberg tables (dt.collect() / count()). You will see this error:

    java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda
      to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3

I have found two upstream bug reports, so this is not an SDP operator problem:

• https://issues.apache.org/jira/browse/SPARK-46032 (still open for Spark 3)
• https://issues.apache.org/jira/browse/SPARK-51537 (fixed the bug in Spark 4)

UPDATE: I tried this. Didn't work. Claude Code suggests to try a client side spark.addArtifact() instead of server side --packages. I didn't try this.

Hope this helps,
Razvan

[razvan]

Regarding the ClassCastException : only Spark 3 is affected. Spark 4 works and I've updated the docs and client application for the test in the PR above.