ADC 0.27.1 fails with ERR_INVALID_ARG_TYPE when deleting a remote-only consumer credential during sync

[circleyu]

Description

When using APISIX Ingress Controller 2.1.0 with ADC 0.27.1 and the apisix provider, ADC fails to synchronize resources if APISIX contains a Consumer Credential that was created directly through the APISIX Admin API and is not present in the Kubernetes desired state.

ADC detects the remote-only consumer_credential and generates a delete diff for it, but the synchronization then fails with HTTP 500:

TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string
or an instance of Buffer, TypedArray, or DataView. Received undefined

This prevents the entire ADC synchronization from completing.

As a consequence, unrelated resources managed by APISIX Ingress Controller, such as Upstreams, can remain stale.

---

Environment

• APISIX Ingress Controller: 2.1.0
• ADC: 0.27.1
• ADC image: ghcr.io/api7/adc:0.27.1
• Provider: apisix
• APISIX deployment mode: traditional / etcd-backed
• Kubernetes resources are managed by APISIX Ingress Controller.
• Consumers and Credentials are dynamically managed by an external application through the APISIX Admin API.

Ingress Controller provider configuration:

provider:
  type: apisix
  syncPeriod: 1m
  initSyncDelay: 20m

---

Scenario

We have an external application that dynamically manages API keys.

The intended ownership model is:

Kubernetes
    |
    v
APISIX Ingress Controller
    |
    v
ADC
    |
    +---- Routes
    +---- Upstreams
    +---- TLS
    +---- Plugin configuration
    |
    v
APISIX

External API key management application
    |
    v
APISIX Admin API
    |
    +---- Consumers
    +---- Consumer Credentials
    |
    v
APISIX

The external application creates Consumers and key-auth Credentials using the APISIX Admin API.

There are no corresponding ApisixConsumer resources in Kubernetes.

For example:

$ kubectl get apisixconsumer -A
No resources found

The APISIX Admin API contained Consumers such as:

{
  "username": "cust_9f19e3aa0f01",
  "labels": {
    "tenant_id": "2"
  }
}

and the Consumer had a credential:

{
  "id": "cred_b4ef3e46d9d24b3abd5e6aaf66e13038",
  "plugins": {
    "key-auth": {
      "key": "<redacted>"
    }
  }
}

---

ADC behavior

With ADC debug logging enabled, ADC detected the credential as a remote-only resource.

The local desired state did not contain the credential, while the remote state contained:

consumer_credentials:
  cred_b4ef3e46d9d24b3abd5e6aaf66e13038

ADC then generated a diff similar to:

resourceType: consumer_credential
type: delete

Immediately afterwards, synchronization failed with:

HTTP 500:
{
  "message": "TypeError [ERR_INVALID_ARG_TYPE]: The \"data\" argument must be
  of type string or an instance of Buffer, TypedArray, or DataView.
  Received undefined"
}

The Ingress Controller reported:

ERROR provider.executor failed to run http sync for server
ERROR provider.client failed to execute adc command
ERROR provider.client failed to sync resources
ERROR provider failed to sync

The target server was the normal APISIX Admin API endpoint:

http://<apisix-admin-service>:9180

---

Impact

The failure appears to abort the complete ADC synchronization rather than only the Consumer Credential operation.

We observed an unrelated APISIX Upstream remaining stale.

The Kubernetes EndpointSlice contained:

172.16.112.15

while the corresponding APISIX Upstream still referenced an old Pod IP.

After removing the remote-only Consumer Credential, ADC synchronization recovered and the APISIX Upstream was automatically updated to:

172.16.112.15:4318

After that, the following errors no longer appeared:

failed to run http sync
failed to execute adc command
failed to sync resources
ERR_INVALID_ARG_TYPE

This makes the issue reproducible in our environment.

---

Steps to reproduce

1. Deploy APISIX with APISIX Ingress Controller 2.1.0.
2. Use ADC 0.27.1 with:

provider.type = apisix

3. Do not create an ApisixConsumer in Kubernetes.
4. Create a Consumer directly through the APISIX Admin API.
5. Create a key-auth Credential for that Consumer through the Admin API.
6. Trigger or wait for an APISIX Ingress Controller / ADC full synchronization.
7. ADC discovers the remote-only consumer_credential.
8. ADC generates:

resourceType: consumer_credential
type: delete

9. ADC synchronization fails with:

ERR_INVALID_ARG_TYPE
Received undefined

10. Delete the Consumer Credential manually through the APISIX Admin API.
11. Trigger another synchronization.
12. Synchronization succeeds and unrelated resources such as Upstreams are updated normally.

---

Expected behavior

At minimum, processing a remote-only Consumer Credential should not cause ADC to return HTTP 500 or abort synchronization of unrelated resources.

However, we would also like clarification about the expected resource ownership model.

APISIX officially supports creating Consumers and Credentials through the Admin API.

Is the following configuration model supported when using APISIX Ingress Controller 2.x with the apisix provider?

AIC / ADC:
  Routes
  Upstreams
  TLS
  Plugins

Admin API / external application:
  Consumers
  Consumer Credentials

Specifically:

1. Is ADC expected to delete Consumers or Consumer Credentials that exist in APISIX but are not present in the Kubernetes/AIC desired state?
2. Is there a supported way to mark Admin API-created Consumers/Credentials as unmanaged by ADC?
3. Is there a configuration option to exclude consumer and consumer_credential resources from ADC reconciliation?
4. If mixed ownership is not supported, what is the recommended architecture for applications that dynamically create, rotate, revoke, or suspend API keys through the APISIX Admin API while Routes and Upstreams are managed by APISIX Ingress Controller?

---

Additional observation

The Consumer itself did not have an APISIX Ingress Controller ownership label.

For example, it only contained application-specific labels:

{
  "labels": {
    "tenant_id": "2"
  }
}

Despite this, its Credential was still included in the ADC remote diff and was scheduled for deletion.

Therefore, it does not appear that the absence of a managed-by: apisix-ingress-controller label prevents ADC from reconciling the Consumer Credential.

---

Workaround

Our current workaround is to remove the Admin API-created Consumer Credential.

After removing the credential:

• ADC synchronization succeeds.
• ERR_INVALID_ARG_TYPE no longer occurs.
• Unrelated APISIX Upstreams are synchronized correctly.

However, this prevents us from using APISIX Admin API-managed Credentials for our dynamic API key management use case.

[circleyu]

I found a potentially relevant note in the backend-apisix README that may help clarify this issue:

• The APISIX backend is still marked as experimental and notes that some resources may always be considered changed during diff/sync because APISIX can normalize/default fields when storing configuration.
• The README also shows that unsupported resource types such as Consumer Group are explicitly excluded from dump/sync processing.

Reference: https://github.com/api7/adc/blob/main/libs/backend-apisix/README.md

This raises a related ownership question for consumer / consumer_credential resources:

1. Is there currently any supported way to exclude consumer and consumer_credential from backend-apisix dump/sync processing, similar to how unsupported Consumer Groups are excluded?
2. Could ADC support a configurable resource exclusion or ownership filter so that Admin API-managed Consumers/Credentials are preserved while AIC/ADC manages Routes, Upstreams, TLS, and plugin configuration?
3. If this mixed ownership model is intentionally unsupported, could the recommended architecture be documented explicitly?

In our case the Consumer had no managed-by: apisix-ingress-controller label, but its Credential was still included in the remote diff and scheduled for deletion, so ownership does not appear to be determined solely by that label.