Skip to content

fix(datastore): update emulator zip checksum for 2.3.1 The checksum handed to DownloadableEmulatorRunner is still the one for the 2.0.2 archive. 9eb86f06 bumped MIN_VERSION from 2.0.2 to 2.3.1 but left MD5_CHECKSUM alone, so BaseEmulatorHelper.downloadZipFile() never finds a match and start() re-fetches the ~36 MB zip every time. Fixes #12058 - #14592

Open
rootkiller6788 wants to merge 2 commits into
googleapis:mainfrom
rootkiller6788:fix-datastore-emulator-md5

Conversation

@rootkiller6788

Copy link
Copy Markdown

Fixes #12058.

LocalDatastoreHelper still hands e0d1170519cf52e2e5f9f93892cdf70c to DownloadableEmulatorRunner, and that is the checksum of the 2.0.2 archive, not the 2.3.1 one it now points at. Found the culprit while going through the file history: e17b57b6 moved MIN_VERSION to 2.0.2 and set the checksum that matched, then 9eb86f06 (#1698) moved MIN_VERSION on to 2.3.1 and didn't bring the checksum along. curl -I on the old zip still reports ETag: e0d1170519cf52e2e5f9f93892cdf70c, which is how I spotted it.

Since downloadZipFile() compares the two and re-fetches on mismatch, the condition was true on every call and each start() pulled ~36 MB again even with a perfectly good copy sitting in java.io.tmpdir.

Verified against the real artifact rather than just eyeballing the string. The 2.3.1 zip is 37,929,131 bytes; downloading it and running md5sum gives 7c1f5a3276241a8f78cb1a837daaaa47, and GCS agrees (ETag plus the x-goog-hash md5, base64 fB9aMnYkGo94yxqDfaqqRw==). I also replayed the check from BaseEmulatorHelper over the downloaded file: the old constant makes it re-download, the new one lets it use the cached copy.

The second commit is a separate small thing I ran into in the same constructor, so it's easy to drop if you'd rather keep this to one change: the gcloud command line is built from the raw builder.consistency while the downloadable runner's uses getConsistency(). newBuilder().build() leaves builder.consistency at 0.0 while getConsistency() resolves to DEFAULT_CONSISTENCY (0.9), so the two runners disagree for the same helper. Using getConsistency() in both makes them match.

The checksum handed to DownloadableEmulatorRunner is still the one for the
2.0.2 archive. 9eb86f0 bumped MIN_VERSION from 2.0.2 to 2.3.1 but left
MD5_CHECKSUM alone, so BaseEmulatorHelper.downloadZipFile() never finds a
match:

  if (!zipFile.exists() || (md5CheckSum != null && !md5CheckSum.equals(md5(zipFile))))

The condition is true on every call and start() re-fetches the ~36 MB zip
even when the copy in java.io.tmpdir is fine.

For what it's worth, e0d1170519cf52e2e5f9f93892cdf70c is still the ETag of
cloud-datastore-emulator-2.0.2.zip, which is how I noticed. The 2.3.1 object
sits at 7c1f5a3276241a8f78cb1a837daaaa47 (37,929,131 bytes, matches the
x-goog-hash md5 too).

Fixes googleapis#12058
Spotted this while poking at the checksum above, unrelated to it.

The constructor resolves consistency once, into this.consistency:

  this.consistency = builder.consistency > 0 ? builder.consistency : DEFAULT_CONSISTENCY;

The downloadable runner's command line already reads it back via
getConsistency(). The gcloud one reads builder.consistency instead, which is
still 0.0 for anyone who just does LocalDatastoreHelper.newBuilder().build().
So the same helper starts gcloud with --consistency=0.0 but hands the other
runner 0.9, and getConsistency() reports 0.9 either way.

Use getConsistency() here as well so both paths agree. Happy to split this
out or drop it if you'd rather keep the change to just the checksum.
@rootkiller6788
rootkiller6788 marked this pull request as ready for review October 7, 2026 04:24
@rootkiller6788
rootkiller6788 requested a review from a team as a code owner October 7, 2026 04:24

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the MD5 checksum for the downloadable Datastore emulator and changes how the consistency flag is added in the LocalDatastoreHelper constructor by calling getConsistency(). The review feedback correctly points out that calling an overridable instance method like getConsistency() inside a constructor is risky and violates Java best practices, suggesting an alternative that resolves the value directly from the builder.

// At most one of --consistency | --use-firestore-in-datastore-mode can be specified.
// --consistency will be ignored with --use-firestore-in-datastore-mode.
gcloudCommand.add(CONSISTENCY_FLAG + builder.consistency);
gcloudCommand.add(CONSISTENCY_FLAG + getConsistency());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Calling the overridable instance method getConsistency() inside the constructor is risky. If LocalDatastoreHelper is subclassed, the subclass's version of getConsistency() could be invoked before the subclass is fully initialized, which violates the general Java practice of not calling overridable methods in constructors (Effective Java, Item 19). To avoid this, resolve the consistency value directly from the builder using the DEFAULT_CONSISTENCY constant.

Suggested change
gcloudCommand.add(CONSISTENCY_FLAG + getConsistency());
gcloudCommand.add(CONSISTENCY_FLAG + (builder.consistency == 0.0 ? DEFAULT_CONSISTENCY : builder.consistency));

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

java-datastore: LocalDatastoreHelper is downloading datastore emulator zip every time even when file exists

1 participant