Down services getting removed with Eureka discovery

[SledgeHammer01]

Spring Boot 2.3.1.RELEASE
spring-boot-admin-starter-server 2.2.3

I am using discovery first, so Admin is a Eureka client and discovering services through Eureka.

Scenario #1: I have a Spring Config server instance. When I shut the app, it is removed from Eureka. After a bit it is removed from Admin as well. When I start it back up, it comes back into Admin with a notification of Unknown -> UP.

Scenario #2: I am using Hashicorp Vault and I wanted to plug it into Eureka & Admin, so I am using Netflix Sidecar. I have a custom health indicator that looks at the Vault health endpoint and returns either UP or DOWN. Since I'm using Sidecar, the service that Eureka & Admin talk to is always up. If I shut down Vault, the Sidecar is still running, but the health indicator returns DOWN. After a while this also drops off of Admin. This scenario is a little different then #1 since the Sidecar remains in Eureka with a DOWN status.

Why are these 2 scenarios dropping off of Admin? Based on the docs, I am NOT specifying spring.boot.admin.auto-deregistration, so it should default to false, no? Shouldn't these two scenarios stay in Admin forever as DOWN (unless I restart Admin obviously). They seem to get de-registered on NON-UP status in Eureka with discovery mode enabled @ 10 - 40 seconds.

For the sidecar scenario, Vault is running on port 8200 and the Sidecar is running on port 8199. The Sidecar is what registers with Eureka:

<homePageUrl>https://vault01.xxx.com:8200</homePageUrl>
<statusPageUrl>https://vault01.xxx.com:8199/actuator/info</statusPageUrl>
<healthCheckUrl>https://vault01.xxx:8199/actuator/health</healthCheckUrl>
<secureHealthCheckUrl>https://vault01.xxx.com:8199/actuator/health</secureHealthCheckUrl>

Turning on logging:

2020-07-04 09:37:34.331 DEBUG 12336 --- [ctor-http-nio-1] d.c.b.a.s.e.ConcurrentMapEventStore : Events appended to log [InstanceStatusChangedEvent(super=InstanceEvent(instance=d16f8f0c27ed, version=3, timestamp=2020-07-04T16:37:34.331741700Z, type=STATUS_CHANGED), statusInfo=StatusInfo(status=DOWN, details={reactiveDiscoveryClients={status=UP, details={Simple Reactive Discovery Client={status=UP, details={services=[]}}, Spring Cloud Eureka Reactive Discovery Client={status=UP, details={services=[xxx-vault, xxx-config]}}}}, hystrix={status=UP}, localApplication={status=DOWN, details={error=org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://vault01.xxx.com:8200/v1/sys/health": Connect to

Not sure how you are capturing the :8200/v1/sys/health since Eureka knows nothing about that endpoint, only Sidecar knows about that, and that's inside of a try/catch.

Then a bit later, I get:

2020-07-04 09:38:01.766 INFO 12336 --- [freshExecutor-0] d.c.b.a.s.c.d.InstanceDiscoveryListener : Instance 'd16f8f0c27ed' missing in DiscoveryClient services and will be removed.
2020-07-04 09:38:01.766 DEBUG 12336 --- [freshExecutor-0] d.c.b.a.s.e.ConcurrentMapEventStore : Events appended to log [InstanceDeregisteredEvent(super=InstanceEvent(instance=d16f8f0c27ed, version=4, timestamp=2020-07-04T16:38:01.766276900Z, type=DEREGISTERED))]
2020-07-04 09:38:01.766 DEBUG 12336 --- [freshExecutor-0] d.c.b.a.s.e.InstanceEventPublisher : Event published InstanceDeregisteredEvent(super=InstanceEvent(instance=d16f8f0c27ed, version=4, timestamp=2020-07-04T16:38:01.766276900Z, type=DEREGISTERED))

EDIT:

I debugged this a bit and noticed DiscoveryClient::getServices() doesn't return the service if all instances are DOWN, that's fine, I modified the getServices() code to return the service even if all instances are down, but seems like Admin drops the app somewhere else if it has 0 instances rather then just show "Down 0 instances".

[SledgeHammer01]

Spent a few hours today debugging this further.

Looks like this is an issue with the current implementation of the EurekaDiscoveryClient. Not sure if this is a recent change, or was change at all, but it kind of defeats the purpose of Boot Admin if the services drop off when they go down :).

I was able to fix this in my Boot Admin service by overriding the DiscoveryClient bean and overriding 2 methods.

Issue #1: EurekaDiscoveryClient::getServices() will NOT return a service if getInstances().isEmpty(). I modified that behavior to always return the list of services.

Issue #2: Once the service is being returned, you then call EurekaDiscoveryClient::getInstances() for each service. I think this was a recent change in the past year or two, they changed this method to call getInstancesByVipAddress() on the EurekaClient, which once again doesn't return non "UP" services. I modified this method to call EurekaClient::getApplication() to get the service instance info, and then call Application::getInstancesAsIsFromEureka() which will return all instances regardless of status.

Now Boot Admin behaves as expected :).

[joshiste]

... if the services drop off when they go down

IIRC that was always the default behavior using eureka.

Now Boot Admin behaves as expected :).

I'm glad that you could solve this issue. I'm going to close this issue. Imho this is totally related to the Eureka Client filtering the down services.

p.s.

there is a filterOnlyUpInstances property on the EurekaClientConfigBean may be you can set this to false to get the behavior you want.