Report more information about keystore contents on reload

[DaveCTurner]

Today when invoking POST _nodes/reload_secure_settings the response includes just a list of node IDs and names, which is not all that useful for diagnosing problems related to secure settings. Could we include more information about the keystore that was loaded on each node in the response? All of the following would be helpful, as long as they do not compromise security:

• absolute path to keystore file
• last-modified date of keystore file
• list of setting keys loaded on each node

I don't see an obvious reason why keeping this information hidden has any security benefits (but ofc security has lots of nonobvious concerns too).

---

Relevant forum post

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[tvernum]

absolute path to keystore file

I wonder whether the absolute path is sensitive, my gut feel is that it is not, but we'd want to think that through. To my knowledge there is no other API that leaks the location of the config directory (I may be wrong though).

list of setting keys loaded on each node

We may need to respect filtered settings for this (maybe - I'm not sure whether we advertise that the filtering behaviour applies to key names), and I don't think we would want to report whether the value was change by the reload, just that it was read.

--

We can definitely do better than we do today - even if all we do is report the number of settings that were found of each node, that would be progress.

[DaveCTurner]

Maybe we could report the path in logs instead, if it's too sensitive to return in an API?

I do not believe that secure settings can be marked as filtered, at least org.elasticsearch.common.settings.SecureSetting#ALLOWED_PROPERTIES does not include Property.Filtered. I do agree that we don't care whether the value was changed with this particular reload or not.

I would be concerned that simply reporting the number of settings is still not sufficient for the troubleshooting cases we've had in this area in the past. Yes with sufficient effort you can work out if you're manipulating the wrong keystore by adding/removing dummy settings and seeing if the count changes, but this is asking rather a lot of our users IMO. I am struggling to think of how the keys themselves could be sensitive - you can deduce the things that should be in the keystore by looking at unprotected settings in most cases anyway.

[DaveCTurner]

with sufficient effort you can work out if you're manipulating the wrong keystore

An alternative idea that would also help without exposing too much extra info would be to report some easily-computed hash of the keystore file as loaded on each node. Still not quite as nice as reporting the names of the settings it loaded, but would at least make it easier to correlate the file that ES read against the one the user thinks it should have read.

[DaveCTurner]

#128097 is a great example of the kind of thing we cannot track down today. My hunch in that case is that ES is reloading settings from a keystore that is missing some items (for reasons outside of ES's control) but it's really hard to tell. The keystore in this case is some kind of virtual secret file managed by Kubernetes so it's pretty hard for the user to get hold of exactly the data that ES is seeing.

[DaveCTurner]

#128658 is another good example - the obvious explanation for this user's troubles is that the keystore contents are not as expected, but again there's layers of abstraction between what the user configures and what ES sees that make it hard to make progress.

[ebarlas]

@tvernum and @DaveCTurner, I started looking into this issue today.

Based on an initial investigation, including the details requested (secure setting names, keystore path, keystore update time) seems reasonable.

• The reload endpoint is only available to cluster admins (action cluster:admin/nodes/reload_secure_settings). The information disclosure would be confined to those users.
• Secure setting names are public domain and non-standard secure settings in the keystore fail validation on ES launch. Taken together, it's very unlikely that any sensitive information could be leaked by disclosing secure setting names.
• The Elasticsearch home path is already available via the node info endpoint (action cluster:monitor/nodes/info). The keystore path (typically $home/config/elasticsearch.keystore) isn't a significant disclosure beyond that.

---

Alternative approaches:

• Expand node-info response to include these details so they can be fetched independent of a reload
• Introduce a new endpoint gated by a different action to avoid expanding the scope of the reload action
• Include (1) hash and (2) count of secure settings rather than full names, as discussed above

[DaveCTurner]

Thanks @ebarlas for the analysis, makes sense to me.

Secure setting names are public domain and non-standard secure settings in the keystore fail validation on ES launch.

That's true, but do note that affix settings include arbitrary user input in their names: for instance s3.client.${CLIENT_NAME}.access_key. I do not think the user's name for their S3 clients is particularly sensitive, especially since we're only proposing to expose this to users with cluster:admin/nodes/reload_secure_settings permission (and who know the keystore password, if present) but just wanted to make sure you hadn't overlooked this case.

Expand node-info response

The node-info API is in base ES, not x-pack. We could in principle make it pluggable to support this usage if we really wanted, but it's also accessible to a much broader set of users/roles than the reload-settings API so I suspect we wouldn't want to do this.

Introduce a new endpoint

If the new endpoint read its response from the keystore then there would be doubt about whether those were the settings actually applied. But yes we could also consider a new endpoint that returns the current set of names of secure settings on each node.

[ebarlas]

Great! Thanks for the feedback.

That's true, but do note that affix settings include arbitrary user input in their names.

I did overlook that, thanks for clarifying.

The node-info API is in base ES, not x-pack.

I believe the reload API is also in base ES, adjacent to node-info. Am I misinterpreting that?