From f0c93abada3ad6d74d21735c36a50e54115dbdfc Mon Sep 17 00:00:00 2001 From: milanmajchrak Date: Tue, 18 Aug 2026 12:46:21 +0200 Subject: [PATCH 01/10] VSB-TUO/Fix: past embargo end date no longer wipes bitstream policies Running `dspace itemupdate -a dc.date.embargoend` with an embargo end date that has already passed left the ORIGINAL bitstreams of the item with zero resource policies. Every download answered HTTP 401 and the item was shown as `accessStatus = restricted` even though its metadata said `dc.rights.access = openAccess`. Confirmed on dspace7-test.vsb.cz on item 4dde91f3-7078-4241-9938-9b8488623bb1: 401 on all 18 bitstreams. Cause ----- `ItemUpdate.syncEmbargoPolicies()` deleted before it validated. `clearExistingSafEmbargoPolicies()` ran as the very first statement and removed every dated Anonymous/READ policy on the ORIGINAL bitstreams; only afterwards did the method look at the date, find it in the past, print a warning and return without creating a replacement. Because the previous run had already removed the collection's undated default policy (`removeImmediateAnonymousReadPolicies`), that dated policy was the only one left, so the file ended up with none at all. The same wipe happened for an end date of today (the "is it past" test ran before the "+1 day"), for a removed, empty or unparseable `dc.date.embargoend`, and regardless of `dc.rights.access`. In every case the exit code was 0. Fix --- `syncEmbargoPolicies()` is now ordered objections -> validation -> mutation, and nothing is deleted before its replacement is stored: * Refuses to act on a withdrawn item, on an item that is not archived, and on any `dc.rights.access` value outside {openAccess, embargoedAccess} - a single unknown or restrictive value blocks the whole item. * Validates `dc.date.embargoend` (strict `LocalDate.parse`; `DCDate` rolls 2026-02-30 over into 2026-03-02) before touching a policy. * Computes the start date as `embargoend + 1 day` at midnight UTC, so the day boundary no longer depends on the server time zone, and an embargo ending today keeps the file closed today. * An expired embargo is a publication, not a deletion: the real, already passed start date is written and the file becomes readable. * Instead of delete + create, the existing Anonymous/READ policy is mutated in place ("survivor", the dated one with the oldest start date), then normalised to `TYPE_CUSTOM` / rpName `embargo`, then the duplicates are removed. Survivors are found by (group, action) and not by rpName, so the legacy "Standard Embargo" / "Special Case Embargo" rows already in the customer database are adopted and normalised. * A bitstream without an Anonymous/READ policy never gets one invented - that would widen access instead of re-dating it. It is reported, together with a pointer to `dspace bulk-access-control`. * Removing `dc.date.embargoend` lifts the embargo (`startDate = null`), which is how an operator is supposed to re-open a file. * `ItemUpdate` gained a real log4j2 logger and `prWarn`/`prErr`, and every refusal increments `embargoSyncFailures`, which makes `main()` exit 1 - the problems used to be invisible to a script. `ItemImportServiceImpl` carried three latent defects on the neighbouring import path, fixed here as well: * rpName "Special Case Embargo - No access rights metadata" is 48 characters and the column is `varchar(30)`; on PostgreSQL this aborts the whole import. Both scenarios now write `embargo`, the access condition name from access-conditions.xml. * The past-date guard ran before the "+1 day", so an embargo ending today was dropped. Replaced with the same UTC calendar-day arithmetic. * The created policy had no `rpType`, and `processEmbargoMetadata` ran on the workflow branch too, i.e. before the submission was approved. It is now `TYPE_CUSTOM` and only applied when `!useWorkflow`. The import path still creates the policy - there is no Anonymous/READ policy on a bitstream before `installItem` - so the survivor rule of `ItemUpdate` is deliberately not shared with it. Tests ----- New: EmbargoPastDateIT (reproduces the customer 401), EmbargoDateBoundaryIT, EmbargoSafetyIT, EmbargoLifecycleIT - 28 integration tests covering the boundary days, the "never zero READ policies" invariant, withdrawn and non-archived items, unknown access rights and the legacy rpNames. Changed: six ItemUpdateIT tests and EmbargoImportIT.testPastEmbargoDateNoPolicy asserted only the absence of a policy named "Standard Embargo" or the absence of a start date. A bitstream stripped of every policy passes those assertions just as well as a correct one, which is precisely why the bug survived them. They now assert the identity of the surviving policy and whether an anonymous visitor can actually read the file. Co-Authored-By: Claude Opus 5 (1M context) --- .../app/itemimport/ItemImportServiceImpl.java | 93 +- .../org/dspace/app/itemupdate/ItemUpdate.java | 300 ++++-- .../app/itemimport/EmbargoImportIT.java | 35 +- .../app/itemupdate/EmbargoDateBoundaryIT.java | 753 +++++++++++++++ .../app/itemupdate/EmbargoLifecycleIT.java | 884 ++++++++++++++++++ .../app/itemupdate/EmbargoPastDateIT.java | 316 +++++++ .../app/itemupdate/EmbargoSafetyIT.java | 753 +++++++++++++++ .../dspace/app/itemupdate/ItemUpdateIT.java | 238 +++-- 8 files changed, 3171 insertions(+), 201 deletions(-) create mode 100644 dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java create mode 100644 dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java create mode 100644 dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java create mode 100644 dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java index 362872edfe14..f18917ae5f02 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java @@ -33,9 +33,11 @@ import java.nio.file.Path; import java.sql.SQLException; import java.text.SimpleDateFormat; +import java.time.LocalDate; +import java.time.ZoneOffset; +import java.time.format.DateTimeParseException; import java.util.ArrayList; import java.util.Arrays; -import java.util.Calendar; import java.util.Date; import java.util.Enumeration; import java.util.GregorianCalendar; @@ -80,7 +82,6 @@ import org.dspace.content.BitstreamFormat; import org.dspace.content.Bundle; import org.dspace.content.Collection; -import org.dspace.content.DCDate; import org.dspace.content.DSpaceObject; import org.dspace.content.Item; import org.dspace.content.MetadataField; @@ -148,6 +149,14 @@ public class ItemImportServiceImpl implements ItemImportService, InitializingBean { private final Logger log = LogManager.getLogger(); + /** + * Name written on the embargo resource policies created during import. It is the {@code name} of the + * {@code embargoed} access condition in access-conditions.xml and has to fit the resourcepolicy.rpname + * column (varchar(30)): the previous "Special Case Embargo - No access rights metadata" was 48 characters + * and aborted the whole import on PostgreSQL with "value too long for type character varying(30)". + */ + private static final String EMBARGO_POLICY_NAME = "embargo"; + private DSpaceRunnableHandler handler; @Autowired(required = true) @@ -810,8 +819,6 @@ protected Item addItem(Context c, List mycollections, String path, // non-standard permissions List options = processContentsFile(c, myitem, itemPathDir, "contents"); - // Check for embargo metadata and set up embargo terms if needed - processEmbargoMetadata(c, myitem); if (useWorkflow) { // don't process handle file // start up a workflow @@ -827,6 +834,13 @@ protected Item addItem(Context c, List mycollections, String path, mapOutputString = itemname + " " + myitem.getID(); } } else { + // Check for embargo metadata and set up embargo terms if needed. + // Only on this branch, and before installItem: a workflow item must not be given an Anonymous READ + // policy before it has been approved, and the TYPE_CUSTOM embargo policy written here is what stops + // installItem from cloning the collection's undated default READ policy next to it (see + // ItemServiceImpl.addDefaultPoliciesNotInPlace), which would defeat the embargo outright. + processEmbargoMetadata(c, myitem); + // only process handle file if not using workflow system String myhandle = processHandleFile(c, myitem, itemPathDir, "handle"); @@ -2576,34 +2590,27 @@ protected void processEmbargoMetadata(Context c, Item item) throws SQLException, return; } - // Parse and validate embargo date - DCDate embargoEndDate; - Date endDate; + // Parse and validate embargo date. All arithmetic is done in UTC calendar days: neither + // Calendar.getInstance() (server time zone) nor DCDate (lenient, rolls 2026-02-30 over into + // 2026-03-02) can decide a day boundary reliably. + Date accessStartDate; try { - embargoEndDate = new DCDate(embargoEndDateStr); - endDate = embargoEndDate.toDate(); - - if (endDate == null) { - logError("ERROR: Invalid embargo end date format: " + embargoEndDateStr); - return; - } - - if (endDate.before(new Date())) { - logInfo("WARNING: Embargo end date is in the past: " + embargoEndDateStr + - ". Embargo will not be applied."); + LocalDate embargoEndDay = LocalDate.parse(embargoEndDateStr.trim()); + + // dc.date.embargoend is the inclusive last day of the embargo, so access starts the day after. + // The "already passed" test has to run on that start day and not on the end day, otherwise an + // embargo ending today would be dropped although the file must still be closed today. + LocalDate accessStartDay = embargoEndDay.plusDays(1); + if (!accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { + logInfo("Embargo: end date " + embargoEndDateStr + " has already passed, no embargo policy" + + " is created. installItem applies the collection default policies, so the files" + + " are as accessible as the collection says."); return; } - - // Resource policy start date = embargoend + 1 day - // The embargo end date is the last day of the embargo, - // so the file becomes accessible the day after. - Calendar cal = Calendar.getInstance(); - cal.setTime(endDate); - cal.add(Calendar.DAY_OF_MONTH, 1); - endDate = cal.getTime(); - } catch (Exception e) { - logError("ERROR: Failed to parse embargo end date: " + embargoEndDateStr + - ". Error: " + e.getMessage()); + accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); + } catch (DateTimeParseException e) { + logError("ERROR: Invalid embargo end date format: " + embargoEndDateStr + + ". Expected a strict ISO date (yyyy-MM-dd)."); return; } @@ -2622,15 +2629,15 @@ protected void processEmbargoMetadata(Context c, Item item) throws SQLException, if (hasEmbargoedAccess) { // Scenario 1: Standard embargo (embargoedAccess + embargoend) logInfo("Embargo: Setting standard embargo on item until " + embargoEndDateStr); - applyEmbargoToItemBitstreams(c, item, endDate, "Standard Embargo"); } else { // Scenario 2: Only embargo end date present (special case) logInfo("Embargo: SPECIAL CASE - Found dc.date.embargoend without " + "dc.rights.access=embargoedAccess"); logInfo("Embargo: Applying embargo based on end date only until " + embargoEndDateStr); - applyEmbargoToItemBitstreams(c, item, endDate, - "Special Case Embargo - No access rights metadata"); } + // Both scenarios produce the same policy. They used to differ only in the rpName, and one of + // those two names did not fit the 30 character rpname column at all. + applyEmbargoToItemBitstreams(c, item, accessStartDate, EMBARGO_POLICY_NAME); } catch (Exception e) { logError("ERROR: Failed to apply embargo to bitstreams", e); } @@ -2641,10 +2648,18 @@ protected void processEmbargoMetadata(Context c, Item item) throws SQLException, } /** - * Apply embargo ResourcePolicy to all bitstreams in the item. - * Sets READ permission for Anonymous group with the embargo end date as start date. + * Apply an embargo ResourcePolicy to all ORIGINAL bitstreams of the item. + * + *

This is the import path, where the bitstreams do not have an Anonymous READ policy yet - they only + * get one from the collection default when installItem runs - so the policy is created here. ItemUpdate + * works on already archived items and mutates the existing policy instead; the two must not be merged.

+ * + * @param c DSpace context + * @param item item being imported + * @param accessStartDate day the files become publicly readable (dc.date.embargoend + 1 day, midnight UTC) + * @param policyReason value for resourcepolicy.rpname, at most 30 characters */ - protected void applyEmbargoToItemBitstreams(Context c, Item item, Date embargoEndDate, String policyReason) + protected void applyEmbargoToItemBitstreams(Context c, Item item, Date accessStartDate, String policyReason) throws SQLException, AuthorizeException { try { @@ -2671,8 +2686,12 @@ protected void applyEmbargoToItemBitstreams(Context c, Item item, Date embargoEn ResourcePolicy policy = resourcePolicyService.create(c, null, anonymousGroup); policy.setdSpaceObject(bitstream); policy.setAction(Constants.READ); - policy.setStartDate(embargoEndDate); + policy.setStartDate(accessStartDate); policy.setRpName(policyReason); + // TYPE_CUSTOM is load bearing twice: AuthorizeServiceImpl only skips policies on a + // not-yet-installed item when they are custom, and installItem only clones the + // collection default READ policy onto a bitstream that has no custom one yet. + policy.setRpType(ResourcePolicy.TYPE_CUSTOM); // Add policy to bitstream's existing policies bitstream.getResourcePolicies().add(policy); @@ -2686,7 +2705,7 @@ protected void applyEmbargoToItemBitstreams(Context c, Item item, Date embargoEn } logInfo("Embargo: Applied embargo policy to " + bitstreamsProcessed + - " bitstreams until " + embargoEndDate.toString()); + " bitstreams, readable from " + accessStartDate.toString()); } catch (Exception e) { logError("ERROR: Failed to apply embargo to item bitstreams", e); diff --git a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java index 6ad230ab88eb..42e7ae71dafb 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java +++ b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java @@ -15,9 +15,11 @@ import java.io.IOException; import java.io.PrintWriter; import java.sql.SQLException; +import java.time.LocalDate; +import java.time.ZoneOffset; +import java.time.format.DateTimeParseException; import java.util.ArrayList; import java.util.Arrays; -import java.util.Calendar; import java.util.Date; import java.util.HashMap; import java.util.List; @@ -31,13 +33,14 @@ import org.apache.commons.cli.Option; import org.apache.commons.cli.Options; import org.apache.commons.lang3.StringUtils; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; import org.dspace.authorize.AuthorizeException; import org.dspace.authorize.ResourcePolicy; import org.dspace.authorize.factory.AuthorizeServiceFactory; import org.dspace.authorize.service.ResourcePolicyService; import org.dspace.content.Bitstream; import org.dspace.content.Bundle; -import org.dspace.content.DCDate; import org.dspace.content.Item; import org.dspace.content.MetadataValue; import org.dspace.content.factory.ContentServiceFactory; @@ -103,12 +106,23 @@ public class ItemUpdate { protected static final ResourcePolicyService resourcePolicyService = AuthorizeServiceFactory.getInstance() .getResourcePolicyService(); + /** API logging; the operator console output is written by {@link #pr(String)} and its siblings. */ + private static final Logger log = LogManager.getLogger(ItemUpdate.class); + private static final String EMBARGO_FIELD_RIGHTS_ACCESS = "dc.rights.access"; private static final String EMBARGO_FIELD_DATE_END = "dc.date.embargoend"; + private static final String OPEN_ACCESS = "openAccess"; private static final String EMBARGOED_ACCESS = "embargoedAccess"; - private static final String STANDARD_EMBARGO_POLICY_NAME = "Standard Embargo"; - // Must fit resourcepolicy.rpname length (varchar(30)) - private static final String SPECIAL_CASE_EMBARGO_POLICY_NAME = "Special Case Embargo"; + + /** + * Name written on every embargo policy. It is the {@code name} of the {@code embargoed} access condition in + * access-conditions.xml and has to fit the resourcepolicy.rpname column (varchar(30)). + */ + private static final String EMBARGO_POLICY_NAME = "embargo"; + + /** The only supported way of changing access to a bitstream this tool refuses to touch. */ + private static final String BULK_ACCESS_CONTROL_HINT = + "Use the 'dspace bulk-access-control' script to grant or restore access to it."; static { filterAliases.put("ORIGINAL", "org.dspace.app.itemupdate.OriginalBitstreamFilter"); @@ -140,6 +154,8 @@ public boolean accept(File dir, String n) { protected ActionManager actionMgr = new ActionManager(); protected List undoActionList = new ArrayList<>(); protected String eperson; + /** Bitstreams and items whose embargo could not be synchronised; a non-zero count fails the run. */ + protected int embargoSyncFailures = 0; /** * @param argv the command line arguments given @@ -382,6 +398,13 @@ public static void main(String[] argv) { context.restoreAuthSystemState(); } + // Embargo problems are reported per item and never abort the run, but they must not be reported as + // success either: an operator scripting itemupdate has to see them in the exit code. + if (iu.embargoSyncFailures > 0) { + prErr(iu.embargoSyncFailures + " embargo synchronisation problem(s) reported above."); + status = 1; + } + if (isTest) { pr("***End of Test Run***"); } else { @@ -604,9 +627,30 @@ protected void setEPerson(Context context, String eperson) * @param s String */ static void pr(String s) { + log.info(s); System.out.println(s); } + /** + * report something the operator has to look at which does not stop the run + * + * @param s String + */ + static void prWarn(String s) { + log.warn(s); + System.out.println("WARNING: " + s); + } + + /** + * report something that made this tool refuse to do what it was asked to do + * + * @param s String + */ + static void prErr(String s) { + log.error(s); + System.out.println("ERROR: " + s); + } + /** * print if verbose flag is set * @@ -636,119 +680,205 @@ protected static boolean containsEmbargoField(String[] targetFields) { return false; } + /** + * Bring the {@code Anonymous}/{@code READ} resource policies of the ORIGINAL bitstreams in line with the + * embargo metadata of the item ({@code dc.rights.access}, {@code dc.date.embargoend}). + * + *

The order of the three phases below is the whole point of this method. A policy whose start date lies + * in the future only postpones access and is therefore harmless, but deleting a policy - or writing a start + * date that has already passed - is a publishing operation. So every objection is raised first, the target + * state is computed second, and only a run that got that far may touch a single policy. Nothing is deleted + * before its replacement has been stored, which is why the existing policy is mutated rather than replaced: + * a failure between a delete and a create would leave the file with no policy at all, i.e. HTTP 401.

+ * + * @param context DSpace context + * @param item item that has just been updated from the SAF archive + * @throws SQLException if a database error occurs + * @throws AuthorizeException if the policy update is not permitted + */ protected void syncEmbargoPolicies(Context context, Item item) throws SQLException, AuthorizeException { - clearExistingSafEmbargoPolicies(context, item); - - List embargoEndDates = itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY); - if (embargoEndDates.size() > 1) { - ItemUpdate.pr("WARNING: Multiple dc.date.embargoend values found. Using first value only."); - } - if (embargoEndDates.isEmpty()) { - List accessRights = itemService.getMetadata(item, "dc", "rights", "access", Item.ANY); - for (MetadataValue accessRight : accessRights) { - if (EMBARGOED_ACCESS.equals(accessRight.getValue())) { - ItemUpdate.pr("WARNING: Item has dc.rights.access=embargoedAccess but no dc.date.embargoend. " - + "Cannot set embargo without end date."); - break; - } - } - return; - } - - String embargoEndDateStr = embargoEndDates.get(0).getValue(); - if (StringUtils.isBlank(embargoEndDateStr)) { - ItemUpdate.pr("WARNING: dc.date.embargoend is empty. Cannot set embargo."); - return; - } - - DCDate embargoEndDate = new DCDate(embargoEndDateStr); - Date endDate = embargoEndDate.toDate(); - if (endDate == null) { - ItemUpdate.pr("ERROR: Invalid embargo end date format: " + embargoEndDateStr); + // --- phase 1: objections ------------------------------------------------------------------------- + if (item.isWithdrawn()) { + prWarn("Item " + itemLabel(item) + " is withdrawn, its bitstream policies are left untouched." + + " A withdrawn item must never regain a READ policy, or the takedown would undo itself" + + " as soon as the embargo lapses."); return; } - if (endDate.before(new Date())) { - ItemUpdate.pr("WARNING: Embargo end date is in the past: " + embargoEndDateStr - + ". Embargo will not be applied."); + if (!item.isArchived()) { + prWarn("Item " + itemLabel(item) + " is not archived (workspace or workflow submission)," + + " its bitstream policies are left untouched."); return; } - Calendar cal = Calendar.getInstance(); - cal.setTime(endDate); - cal.add(Calendar.DAY_OF_MONTH, 1); - Date accessStartDate = cal.getTime(); - List accessRights = itemService.getMetadata(item, "dc", "rights", "access", Item.ANY); boolean hasEmbargoedAccess = false; for (MetadataValue accessRight : accessRights) { - if (EMBARGOED_ACCESS.equals(accessRight.getValue())) { + String value = StringUtils.trimToEmpty(accessRight.getValue()); + if (EMBARGOED_ACCESS.equals(value)) { hasEmbargoedAccess = true; - break; + } else if (!OPEN_ACCESS.equals(value)) { + // restrictedAccess, metadataOnlyAccess or a value we do not know. A single such value blocks the + // whole item even next to an openAccess one: contradictory metadata is never resolved towards + // disclosure, and an unknown access right is not an invitation to guess. + prWarn("Item " + itemLabel(item) + " carries " + EMBARGO_FIELD_RIGHTS_ACCESS + "='" + + accessRight.getValue() + "', which is not an embargo access right (" + + OPEN_ACCESS + ", " + EMBARGOED_ACCESS + "), its bitstream policies are left" + + " untouched."); + return; } } - String policyReason = hasEmbargoedAccess ? STANDARD_EMBARGO_POLICY_NAME - : SPECIAL_CASE_EMBARGO_POLICY_NAME; - applyEmbargoToItemBitstreams(context, item, accessStartDate, policyReason); - } + // --- phase 2: validate and compute the target state ---------------------------------------------- + // null means "no embargo", i.e. the file is readable immediately + Date accessStartDate; + List embargoEndDates = itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY); - protected void clearExistingSafEmbargoPolicies(Context context, Item item) throws SQLException, AuthorizeException { - Group anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); - if (anonymousGroup == null) { - return; - } + if (embargoEndDates.isEmpty()) { + if (hasEmbargoedAccess) { + prWarn("Item " + itemLabel(item) + " has " + EMBARGO_FIELD_RIGHTS_ACCESS + "=" + EMBARGOED_ACCESS + + " but no " + EMBARGO_FIELD_DATE_END + ". Cannot set an embargo without an end date," + + " its bitstream policies are left untouched."); + embargoSyncFailures++; + return; + } + // Removing dc.date.embargoend is how an operator lifts an embargo. + pr("Item " + itemLabel(item) + " has no " + EMBARGO_FIELD_DATE_END + ", lifting the embargo."); + accessStartDate = null; + } else { + if (embargoEndDates.size() > 1) { + prWarn("Multiple " + EMBARGO_FIELD_DATE_END + " values found. Using first value only."); + } - List originalBundles = item.getBundles(Constants.CONTENT_BUNDLE_NAME); - for (Bundle bundle : originalBundles) { - for (Bitstream bitstream : bundle.getBitstreams()) { - List readPolicies = resourcePolicyService.find(context, bitstream, Constants.READ); - for (ResourcePolicy policy : readPolicies) { - if (policy.getGroup() != null - && anonymousGroup.equals(policy.getGroup()) - && policy.getStartDate() != null - && (STANDARD_EMBARGO_POLICY_NAME.equals(policy.getRpName()) - || SPECIAL_CASE_EMBARGO_POLICY_NAME.equals(policy.getRpName()))) { - resourcePolicyService.delete(context, policy); - } - } + String embargoEndDateStr = embargoEndDates.get(0).getValue(); + if (StringUtils.isBlank(embargoEndDateStr)) { + prErr(EMBARGO_FIELD_DATE_END + " is empty on item " + itemLabel(item) + ", its bitstream policies" + + " are left untouched."); + embargoSyncFailures++; + return; + } + + LocalDate embargoEndDay; + try { + // Strict ISO parsing on purpose: DCDate rolls 2026-02-30 over into 2026-03-02 and would turn a + // typo into a real embargo date. + embargoEndDay = LocalDate.parse(embargoEndDateStr.trim()); + } catch (DateTimeParseException e) { + prErr("Invalid " + EMBARGO_FIELD_DATE_END + " '" + embargoEndDateStr + "' on item " + + itemLabel(item) + ", expected a strict ISO date (yyyy-MM-dd). Its bitstream policies" + + " are left untouched."); + embargoSyncFailures++; + return; + } + + // dc.date.embargoend is the inclusive last day of the embargo, so access starts the day after, at + // midnight UTC. Calendar.getInstance() would use the server time zone and shift that boundary. + LocalDate accessStartDay = embargoEndDay.plusDays(1); + accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); + + if (!accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { + // An expired embargo is a publication, not a deletion. The real - already passed - start date is + // written, which makes the policy effective immediately. + pr("Embargo of item " + itemLabel(item) + " already expired on " + embargoEndDay + + ", its ORIGINAL bitstreams are public since " + accessStartDay + "."); } } + + // --- phase 3: mutate ----------------------------------------------------------------------------- + applyEmbargoToItemBitstreams(context, item, accessStartDate); } - protected void applyEmbargoToItemBitstreams(Context context, Item item, Date startDate, String policyReason) + /** + * Write the target embargo state onto the {@code Anonymous}/{@code READ} policy of every ORIGINAL bitstream + * of the item. + * + *

Exactly one such policy is left behind per bitstream: a second, undated one would silently defeat the + * embargo. A policy is never created - a bitstream without an {@code Anonymous}/{@code READ} policy was not + * public, and inventing one would widen access instead of re-dating it.

+ * + * @param context DSpace context + * @param item item whose ORIGINAL bitstreams are synchronised + * @param startDate day the files become publicly readable, or {@code null} to lift the embargo + * @throws SQLException if a database error occurs + * @throws AuthorizeException if the policy update is not permitted + */ + protected void applyEmbargoToItemBitstreams(Context context, Item item, Date startDate) throws SQLException, AuthorizeException { Group anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); if (anonymousGroup == null) { + prErr("Group '" + Group.ANONYMOUS + "' not found, no embargo policy was synchronised for item " + + itemLabel(item) + "."); + embargoSyncFailures++; return; } - List originalBundles = item.getBundles(Constants.CONTENT_BUNDLE_NAME); - for (Bundle bundle : originalBundles) { + for (Bundle bundle : item.getBundles(Constants.CONTENT_BUNDLE_NAME)) { for (Bitstream bitstream : bundle.getBitstreams()) { - removeImmediateAnonymousReadPolicies(context, bitstream, anonymousGroup); - - ResourcePolicy policy = resourcePolicyService.create(context, null, anonymousGroup); - policy.setdSpaceObject(bitstream); - policy.setAction(Constants.READ); - policy.setStartDate(startDate); - policy.setRpName(policyReason); - bitstream.getResourcePolicies().add(policy); - resourcePolicyService.update(context, policy); + // Deliberately located by (group, action) and not by rpName: policies written by earlier + // versions of this tool carry "Standard Embargo" or "Special Case Embargo" and have to be + // adopted and normalised instead of being left behind next to a new one. + List anonymousReadPolicies = new ArrayList<>(); + for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream, Constants.READ)) { + if (anonymousGroup.equals(policy.getGroup())) { + anonymousReadPolicies.add(policy); + } + } + + if (anonymousReadPolicies.isEmpty()) { + prErr("Bitstream '" + bitstream.getName() + "' (" + bitstream.getID() + ") of item " + + itemLabel(item) + " has no " + Group.ANONYMOUS + " READ policy, so there is" + + " nothing to re-date and its embargo could not be synchronised. No policy is" + + " created: that would grant access nobody ever granted. " + + BULK_ACCESS_CONTROL_HINT); + embargoSyncFailures++; + continue; + } + + ResourcePolicy survivor = selectSurvivorPolicy(anonymousReadPolicies); + survivor.setStartDate(startDate); + survivor.setRpType(ResourcePolicy.TYPE_CUSTOM); + survivor.setRpName(EMBARGO_POLICY_NAME); + resourcePolicyService.update(context, survivor); + + // Only now, with the replacement safely stored, may the duplicates go. Reference identity is + // used on purpose: ResourcePolicy.equals compares values, which the lines above just changed. + for (ResourcePolicy policy : anonymousReadPolicies) { + if (policy != survivor) { + resourcePolicyService.delete(context, policy); + } + } } } } - protected void removeImmediateAnonymousReadPolicies(Context context, Bitstream bitstream, Group anonymousGroup) - throws SQLException, AuthorizeException { - List readPolicies = resourcePolicyService.find(context, bitstream, Constants.READ); - for (ResourcePolicy policy : readPolicies) { - if (policy.getGroup() != null - && anonymousGroup.equals(policy.getGroup()) - && policy.getStartDate() == null) { - resourcePolicyService.delete(context, policy); + /** + * Pick the policy that has been in force the longest: the dated one with the oldest start date or, when none + * of them is dated, the first one. Adopting the newest would resurrect an obsolete embargo date. + * + * @param anonymousReadPolicies the Anonymous READ policies of a single bitstream, never empty + * @return the policy to be mutated into the embargo policy + */ + protected ResourcePolicy selectSurvivorPolicy(List anonymousReadPolicies) { + ResourcePolicy oldestDated = null; + for (ResourcePolicy policy : anonymousReadPolicies) { + if (policy.getStartDate() == null) { + continue; + } + if (oldestDated == null || policy.getStartDate().before(oldestDated.getStartDate())) { + oldestDated = policy; } } + return oldestDated == null ? anonymousReadPolicies.get(0) : oldestDated; + } + + /** + * Handle of the item, or its UUID while it has none. Only used in operator messages. + * + * @param item item to describe + * @return handle or UUID + */ + protected static String itemLabel(Item item) { + return item.getHandle() == null ? String.valueOf(item.getID()) : item.getHandle(); } } //end of class diff --git a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java index 9f536bc27b2d..14501a1d8a02 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java @@ -23,6 +23,7 @@ import org.dspace.AbstractIntegrationTestWithDatabase; import org.dspace.authorize.ResourcePolicy; import org.dspace.authorize.factory.AuthorizeServiceFactory; +import org.dspace.authorize.service.AuthorizeService; import org.dspace.authorize.service.ResourcePolicyService; import org.dspace.builder.CollectionBuilder; import org.dspace.builder.CommunityBuilder; @@ -35,6 +36,7 @@ import org.dspace.content.service.ItemService; import org.dspace.content.service.MetadataSchemaService; import org.dspace.core.Constants; +import org.dspace.eperson.EPerson; import org.dspace.eperson.Group; import org.dspace.eperson.factory.EPersonServiceFactory; import org.dspace.eperson.service.GroupService; @@ -64,6 +66,7 @@ public class EmbargoImportIT extends AbstractIntegrationTestWithDatabase { private ItemService itemService = ContentServiceFactory.getInstance().getItemService(); private ResourcePolicyService resourcePolicyService = AuthorizeServiceFactory.getInstance().getResourcePolicyService(); + private AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); private GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); private ConfigurationService configurationService = DSpaceServicesFactory.getInstance().getConfigurationService(); @@ -183,7 +186,11 @@ public void testStandardEmbargoImport() throws Exception { } /** - * Test that no embargo is applied when embargo date is in the past + * An embargo that has already expired must not be turned into a policy - but "no embargo policy" is only + * half the requirement. The original assertion ("no Anonymous policy carries a start date") is satisfied + * just as well by a bitstream that has no policy at all and answers HTTP 401, which is the failure mode + * this branch is fixing. The test therefore also asserts that the file really is publicly readable, which + * on the import path means the collection default policies installItem applies. */ @Test public void testPastEmbargoDateNoPolicy() throws Exception { @@ -226,6 +233,32 @@ public void testPastEmbargoDateNoPolicy() throws Exception { p.getStartDate() != null); assertTrue("Should not have embargo policy for past dates", !hasEmbargoPolicy); + + // The point of not writing an expired embargo policy is that the file stays available. Zero policies + // would satisfy the assertion above and leave every download at HTTP 401. + assertTrue("An expired embargo end date must leave the bitstream readable, not policy-less", + anonymousCanRead(bitstream)); + } + + /** + * What an anonymous visitor gets, with the test's own turnOffAuthorisationSystem calls temporarily unwound. + */ + private boolean anonymousCanRead(Bitstream bitstream) throws Exception { + EPerson savedUser = context.getCurrentUser(); + int popped = 0; + while (context.ignoreAuthorization()) { + context.restoreAuthSystemState(); + popped++; + } + context.setCurrentUser(null); + try { + return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); + } finally { + context.setCurrentUser(savedUser); + for (int i = 0; i < popped; i++) { + context.turnOffAuthorisationSystem(); + } + } } /** diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java new file mode 100644 index 000000000000..4f58666ed0d1 --- /dev/null +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java @@ -0,0 +1,753 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.app.itemupdate; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.PrintStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.LocalDate; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.Date; +import java.util.List; +import java.util.Locale; +import java.util.stream.Collectors; + +import org.apache.commons.io.file.PathUtils; +import org.apache.commons.io.output.TeeOutputStream; +import org.dspace.AbstractIntegrationTestWithDatabase; +import org.dspace.authorize.ResourcePolicy; +import org.dspace.authorize.factory.AuthorizeServiceFactory; +import org.dspace.authorize.service.AuthorizeService; +import org.dspace.authorize.service.ResourcePolicyService; +import org.dspace.builder.BitstreamBuilder; +import org.dspace.builder.CollectionBuilder; +import org.dspace.builder.CommunityBuilder; +import org.dspace.builder.ItemBuilder; +import org.dspace.builder.MetadataFieldBuilder; +import org.dspace.content.Bitstream; +import org.dspace.content.Collection; +import org.dspace.content.Item; +import org.dspace.content.MetadataField; +import org.dspace.content.MetadataSchema; +import org.dspace.content.MetadataValue; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.ItemService; +import org.dspace.content.service.MetadataFieldService; +import org.dspace.content.service.MetadataSchemaService; +import org.dspace.core.Constants; +import org.dspace.eperson.EPerson; +import org.dspace.eperson.Group; +import org.dspace.eperson.factory.EPersonServiceFactory; +import org.dspace.eperson.service.GroupService; +import org.dspace.handle.factory.HandleServiceFactory; +import org.dspace.handle.service.HandleService; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +/** + * Date-boundary and baseline behaviour of the VSB-TUO embargo synchronisation in {@link ItemUpdate}. + * + *

Every scenario models the customer workflow literally: a SAF archive is re-imported with + * {@code ItemUpdate -s SAFDIR -d dc.rights.access -d dc.date.embargoend -a dc.rights.access + * -a dc.date.embargoend}, which is what makes {@code processArchive} call {@code syncEmbargoPolicies}.

+ * + *

The binding rules under test (VSB-TUO embargo specification):

+ *
    + *
  • the resulting {@code Anonymous}/{@code READ} policy on every ORIGINAL bitstream starts at + * {@code dc.date.embargoend + 1 day} at midnight UTC, because {@code dc.date.embargoend} + * is the inclusive last day of the embargo;
    • + *
  • there is always exactly one such policy - never zero (the file would answer HTTP 401) and never + * two (an undated one would silently neutralise the embargo);
    • + *
  • the policy is normalised to {@code rpType=TYPE_CUSTOM} and {@code rpName="embargo"};
    • + *
  • an embargo end date that already lies in the past is a publication, not a deletion.
    • + *
+ * + *

All dates are derived from {@code LocalDate.now(ZoneOffset.UTC)}, never hard-coded, so the suite cannot + * become a time bomb (lesson of PR #1359) and cannot straddle the UTC/Europe-Dublin day boundary.

+ */ +public class EmbargoDateBoundaryIT extends AbstractIntegrationTestWithDatabase { + + /** Expected normalised policy name. Must stay within the 30 character {@code ResourcePolicy.rpname} column. */ + private static final String EMBARGO_POLICY_NAME = "embargo"; + + private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); + private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); + private final ResourcePolicyService resourcePolicyService = + AuthorizeServiceFactory.getInstance().getResourcePolicyService(); + private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); + private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); + private final MetadataSchemaService metadataSchemaService = + ContentServiceFactory.getInstance().getMetadataSchemaService(); + private final MetadataFieldService metadataFieldService = + ContentServiceFactory.getInstance().getMetadataFieldService(); + + /** Human readable trace of every policy state observed during a test; appended to failure messages. */ + private final StringBuilder diagnostics = new StringBuilder(); + + private Collection collection; + private Group anonymousGroup; + private Path tempDir; + private String previousHandlePrefix; + + @Before + @Override + public void setUp() throws Exception { + super.setUp(); + context.turnOffAuthorisationSystem(); + + parentCommunity = CommunityBuilder.createCommunity(context) + .withName("Parent Community") + .build(); + collection = CollectionBuilder.createCollection(context, parentCommunity) + .withName("Collection") + .build(); + + // Neither field exists in the test metadata registry; AddMetadataAction would fail without them. + ensureMetadataFieldExists("rights", "access"); + ensureMetadataFieldExists("date", "embargoend"); + + anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); + previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; + ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); + + context.restoreAuthSystemState(); + + tempDir = Files.createTempDirectory("embargoDateBoundaryIT"); + } + + @After + @Override + public void destroy() throws Exception { + // HANDLE_PREFIX is a mutable public static; leaking it would poison other test classes. + ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; + if (tempDir != null) { + PathUtils.deleteDirectory(tempDir); + } + super.destroy(); + } + + /** + * Row 1 of the specification: a future {@code dc.date.embargoend} closes the file and leaves exactly one + * normalised {@code Anonymous}/{@code READ} policy starting the day after the embargo end date. + */ + @Test + public void futureEmbargoEndBlocksAccess() throws Exception { + LocalDate embargoEnd = utcToday().plusMonths(6); + LocalDate expectedStartDay = embargoEnd.plusDays(1); + + Item item = createItem("Future embargo thesis"); + Bitstream bitstream = createOriginalBitstream(item, "future.pdf"); + dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + assertFreshImportBaseline(bitstream); + + runItemUpdate(item, dublinCore(item, "embargoedAccess", embargoEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B - after itemupdate with FUTURE dc.date.embargoend=" + embargoEnd, bitstream); + + assertEmbargoEndStored(item, embargoEnd.toString()); + + ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy("future embargo end " + embargoEnd, bitstream); + assertNormalisedEmbargoPolicy("future embargo end " + embargoEnd, policy, expectedStartDay); + + assertFalse("dc.date.embargoend=" + embargoEnd + " lies in the future, so resource policy #" + + policy.getID() + " must not be date-valid yet." + diagnostics, + resourcePolicyService.isDateValid(policy)); + assertFalse("dc.date.embargoend=" + embargoEnd + " lies in the future, so an anonymous visitor must NOT" + + " be able to download the ORIGINAL bitstream." + diagnostics, + anonymousCanRead(bitstream)); + } + + /** + * Row 2 of the specification: {@code dc.date.embargoend} is the inclusive last day of the embargo. + * When it equals today the file must still be closed today and open only tomorrow, so the policy start date + * is tomorrow - the "+1 day" has to be applied before, not after, any past/future comparison. + */ + @Test + public void embargoEndTodayStillBlocksToday() throws Exception { + LocalDate embargoEnd = utcToday(); + LocalDate expectedStartDay = embargoEnd.plusDays(1); + + Item item = createItem("Embargo ending today"); + Bitstream bitstream = createOriginalBitstream(item, "today.pdf"); + dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + assertFreshImportBaseline(bitstream); + + runItemUpdate(item, dublinCore(item, "embargoedAccess", embargoEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B - after itemupdate with dc.date.embargoend=TODAY=" + embargoEnd, bitstream); + + assertEmbargoEndStored(item, embargoEnd.toString()); + + ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy("embargo ending today " + embargoEnd, bitstream); + assertNormalisedEmbargoPolicy("embargo ending today " + embargoEnd, policy, expectedStartDay); + + assertFalse("dc.date.embargoend=" + embargoEnd + " is TODAY and the last day of an embargo is inclusive," + + " so resource policy #" + policy.getID() + " must start tomorrow (" + expectedStartDay + + ") and must not be date-valid yet." + diagnostics, + resourcePolicyService.isDateValid(policy)); + assertFalse("dc.date.embargoend=" + embargoEnd + " is TODAY, so the ORIGINAL bitstream must still be" + + " closed for anonymous visitors today and open only from " + expectedStartDay + "." + + diagnostics, + anonymousCanRead(bitstream)); + } + + /** + * Row 3 of the specification: an embargo that ended yesterday is expired, which means the file is published. + * The policy must survive with a start date of today, be date-valid and let anonymous visitors in. + */ + @Test + public void embargoEndYesterdayOpensAccess() throws Exception { + LocalDate embargoEnd = utcToday().minusDays(1); + LocalDate expectedStartDay = embargoEnd.plusDays(1); + + // No dc.rights.access at all - the specification treats a missing value exactly like openAccess. + Item item = createItem("Embargo ended yesterday"); + Bitstream bitstream = createOriginalBitstream(item, "yesterday.pdf"); + dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + assertFreshImportBaseline(bitstream); + + runItemUpdate(item, dublinCore(item, null, embargoEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B - after itemupdate with dc.date.embargoend=YESTERDAY=" + embargoEnd, bitstream); + + assertEmbargoEndStored(item, embargoEnd.toString()); + + ResourcePolicy policy = + assertExactlyOneAnonymousReadPolicy("embargo ended yesterday " + embargoEnd, bitstream); + assertNormalisedEmbargoPolicy("embargo ended yesterday " + embargoEnd, policy, expectedStartDay); + + assertTrue("dc.date.embargoend=" + embargoEnd + " expired yesterday, so resource policy #" + policy.getID() + + " (start=" + policy.getStartDate() + ") must already be date-valid." + diagnostics, + resourcePolicyService.isDateValid(policy)); + assertTrue("An embargo that ended yesterday publishes the file, so the ORIGINAL bitstream must be" + + " readable by anonymous visitors." + diagnostics, + anonymousCanRead(bitstream)); + } + + /** + * Main regression test - the exact record observed on dspace7-test.vsb.cz (item + * 4dde91f3-7078-4241-9938-9b8488623bb1: {@code dc.date.embargoend} in the past, + * {@code dc.rights.access=openAccess}, yet HTTP 401 on all 18 bitstreams). + * + *

The priming run with a future date is mandatory: it is what replaces the inherited collection default + * with a single dated policy, so that the following past-date run has exactly one policy left to destroy.

+ */ + @Test + public void pastEmbargoEndWithOpenAccessOpensAccess() throws Exception { + LocalDate futureEnd = utcToday().plusYears(1); + LocalDate pastEnd = utcToday().minusMonths(1); + LocalDate expectedStartDay = pastEnd.plusDays(1); + + Item item = createItem("VSB-TUO thesis published after embargo"); + Bitstream bitstream = createOriginalBitstream(item, "thesis.pdf"); + dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + assertFreshImportBaseline(bitstream); + + // (b) the state the customer confirmed as working: an embargo with a future end date + runItemUpdate(item, dublinCore(item, "embargoedAccess", futureEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B - after itemupdate with FUTURE dc.date.embargoend=" + futureEnd, bitstream); + assertFalse("fixture precondition: while embargoed until " + futureEnd + " the ORIGINAL bitstream must" + + " not be publicly readable." + diagnostics, + anonymousCanRead(bitstream)); + + // (c) the operator lets the embargo expire: past end date, item declared openAccess + runItemUpdate(item, dublinCore(item, "openAccess", pastEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP C - after itemupdate with PAST dc.date.embargoend=" + pastEnd + + " and dc.rights.access=openAccess", bitstream); + + assertEmbargoEndStored(item, pastEnd.toString()); + assertEquals("itemupdate did not store dc.rights.access=openAccess." + diagnostics, + "openAccess", firstMetadataValue(item, "rights", "access")); + + ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy("expired embargo " + pastEnd + + " with dc.rights.access=openAccess", bitstream); + assertNormalisedEmbargoPolicy("expired embargo " + pastEnd, policy, expectedStartDay); + + assertTrue("Expired embargo (" + pastEnd + ") left resource policy #" + policy.getID() + + " not date-valid, so the file stays unreachable although dc.rights.access=openAccess." + + diagnostics, + resourcePolicyService.isDateValid(policy)); + assertTrue("An expired embargo publishes the file. The ORIGINAL bitstream is still unreadable for" + + " anonymous visitors (HTTP 401) although dc.date.embargoend=" + pastEnd + + " has passed and dc.rights.access=openAccess." + diagnostics, + anonymousCanRead(bitstream)); + } + + /** + * Same as {@link #pastEmbargoEndWithOpenAccessOpensAccess()} but the item keeps + * {@code dc.rights.access=embargoedAccess}. An expired embargo publishes the file regardless: the access + * right only names the licence regime, the end date decides when it lapses. + */ + @Test + public void pastEmbargoEndWithEmbargoedAccessOpensAccess() throws Exception { + LocalDate futureEnd = utcToday().plusYears(1); + LocalDate pastEnd = utcToday().minusMonths(1); + LocalDate expectedStartDay = pastEnd.plusDays(1); + + Item item = createItem("VSB-TUO thesis with lapsed embargoedAccess"); + Bitstream bitstream = createOriginalBitstream(item, "lapsed.pdf"); + dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + assertFreshImportBaseline(bitstream); + + runItemUpdate(item, dublinCore(item, "embargoedAccess", futureEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B - after itemupdate with FUTURE dc.date.embargoend=" + futureEnd, bitstream); + assertFalse("fixture precondition: while embargoed until " + futureEnd + " the ORIGINAL bitstream must" + + " not be publicly readable." + diagnostics, + anonymousCanRead(bitstream)); + + runItemUpdate(item, dublinCore(item, "embargoedAccess", pastEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP C - after itemupdate with PAST dc.date.embargoend=" + pastEnd + + " and dc.rights.access=embargoedAccess", bitstream); + + assertEmbargoEndStored(item, pastEnd.toString()); + + ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy("expired embargo " + pastEnd + + " with dc.rights.access=embargoedAccess", bitstream); + assertNormalisedEmbargoPolicy("expired embargo " + pastEnd, policy, expectedStartDay); + + assertTrue("Expired embargo (" + pastEnd + ") left resource policy #" + policy.getID() + + " not date-valid." + diagnostics, + resourcePolicyService.isDateValid(policy)); + assertTrue("dc.rights.access=embargoedAccess must not keep an ALREADY EXPIRED embargo closed; the" + + " ORIGINAL bitstream must be readable by anonymous visitors after " + pastEnd + "." + + diagnostics, + anonymousCanRead(bitstream)); + } + + /** + * The policy start date must be built as {@code Date.from(day.atStartOfDay(ZoneOffset.UTC).toInstant())}, + * i.e. midnight UTC of a calendar day, and never midnight in the JVM time zone as produced by + * {@code Calendar.getInstance()} or {@code java.sql.Date.valueOf(LocalDate)}. + * + *

The harness pins the JVM to Europe/Dublin, which is UTC+1 during Irish Summer Time, so the two + * candidates differ by exactly one hour. The embargo end date is therefore anchored to the next 1 July - + * still computed from today, so fully dynamic - which is guaranteed to fall inside Irish Summer Time and + * makes the difference observable all year round.

+ * + *

{@code syncEmbargoPolicies} is invoked directly rather than through {@code processArchive} on purpose: + * {@code processArchive} ends with {@code context.uncacheEntity(item)}, which evicts the bitstream policies, + * so they would come back from the day-granular {@code @Temporal(DATE)} column as a {@code java.sql.Date} + * and the exact instant could no longer be inspected.

+ */ + @Test + public void startDateIsUtcMidnightNotServerZone() throws Exception { + LocalDate embargoEnd = nextIrishSummerTimeDay(); + LocalDate expectedStartDay = embargoEnd.plusDays(1); + + Date expectedUtcMidnight = Date.from(expectedStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); + Date serverZoneMidnight = Date.from(expectedStartDay.atStartOfDay(ZoneId.systemDefault()).toInstant()); + assertNotEquals("fixture precondition: the JVM time zone (" + ZoneId.systemDefault() + ") must differ from" + + " UTC on " + expectedStartDay + ", otherwise this test cannot tell midnight UTC apart" + + " from midnight in the server zone. The harness pins Europe/Dublin in" + + " AbstractDSpaceIntegrationTest.", + expectedUtcMidnight, serverZoneMidnight); + + Item item = createItem("UTC midnight embargo", + "rights", "access", "embargoedAccess", + "date", "embargoend", embargoEnd.toString()); + Bitstream bitstream = createOriginalBitstream(item, "utc-midnight.pdf"); + dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + assertFreshImportBaseline(bitstream); + Integer importedPolicyId = anonymousReadPolicies(bitstream).get(0).getID(); + + context.turnOffAuthorisationSystem(); + try { + new ItemUpdate().syncEmbargoPolicies(context, item); + } finally { + context.restoreAuthSystemState(); + } + dump("STEP B - after syncEmbargoPolicies with dc.date.embargoend=" + embargoEnd, bitstream); + + ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy("dc.date.embargoend=" + embargoEnd, bitstream); + assertNotNull("resource policy #" + policy.getID() + " must carry a start date." + diagnostics, + policy.getStartDate()); + assertEquals("the inherited Anonymous/READ policy must be MUTATED in place, not deleted and recreated:" + + " a policy may never be removed before its replacement is stored, otherwise a failure" + + " between the two leaves the file with zero policies (HTTP 401)." + diagnostics, + importedPolicyId, policy.getID()); + + long actualMillis = policy.getStartDate().getTime(); + assertEquals("dc.date.embargoend=" + embargoEnd + " must yield a start date of exactly midnight UTC on " + + expectedStartDay + " (epochMillis=" + expectedUtcMidnight.getTime() + "). Midnight in" + + " the server time zone " + ZoneId.systemDefault() + " would be epochMillis=" + + serverZoneMidnight.getTime() + ", which is what Calendar.getInstance() or" + + " java.sql.Date.valueOf(LocalDate) produce. Actual epochMillis=" + actualMillis + " (" + + new Date(actualMillis).toInstant().atZone(ZoneOffset.UTC) + ")." + diagnostics, + expectedUtcMidnight.getTime(), actualMillis); + + assertFalse("dc.date.embargoend=" + embargoEnd + " lies in the future, so an anonymous visitor must NOT" + + " be able to download the ORIGINAL bitstream." + diagnostics, + anonymousCanRead(bitstream)); + assertEquals("resource policy #" + policy.getID() + " must be normalised to rpType=" + + ResourcePolicy.TYPE_CUSTOM + "." + diagnostics, + ResourcePolicy.TYPE_CUSTOM, policy.getRpType()); + assertEquals("resource policy #" + policy.getID() + " must be normalised to rpName=\"" + + EMBARGO_POLICY_NAME + "\"." + diagnostics, + EMBARGO_POLICY_NAME, policy.getRpName()); + } + + /** + * Row 12 of the specification: several {@code dc.date.embargoend} values are a data error the operator has + * to see, but the run still completes and uses the first value. + */ + @Test + public void multipleEmbargoEndValuesUsesFirst() throws Exception { + LocalDate firstEnd = utcToday().plusDays(30); + LocalDate secondEnd = utcToday().plusDays(400); + LocalDate expectedStartDay = firstEnd.plusDays(1); + + Item item = createItem("Two embargo end dates"); + Bitstream bitstream = createOriginalBitstream(item, "two-dates.pdf"); + dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + assertFreshImportBaseline(bitstream); + + String consoleOutput = runItemUpdate(item, + dublinCore(item, "embargoedAccess", firstEnd.toString(), secondEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B - after itemupdate with dc.date.embargoend=" + firstEnd + " AND " + secondEnd, bitstream); + + List storedEndDates = metadataValues(item, "date", "embargoend"); + assertEquals("fixture precondition: itemupdate must have stored both dc.date.embargoend values, got " + + storedEndDates, 2, storedEndDates.size()); + assertEquals("fixture precondition: dublin_core.xml document order must be preserved, so the first stored" + + " dc.date.embargoend value is the first one written; got " + storedEndDates, + firstEnd.toString(), storedEndDates.get(0)); + + String lowerCaseOutput = consoleOutput.toLowerCase(Locale.ROOT); + assertTrue("itemupdate must warn the operator that dc.date.embargoend carries more than one value and" + + " that only the first one is used. Console output was:\n" + consoleOutput, + lowerCaseOutput.contains("multiple") && lowerCaseOutput.contains("embargoend")); + + ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy("two dc.date.embargoend values (" + + firstEnd + ", " + secondEnd + ")", bitstream); + assertNormalisedEmbargoPolicy("two dc.date.embargoend values", policy, expectedStartDay); + + assertNotEquals("the SECOND dc.date.embargoend value (" + secondEnd + ") must be ignored, the embargo has" + + " to follow the first one (" + firstEnd + ")." + diagnostics, + secondEnd.plusDays(1), toLocalDate(policy.getStartDate())); + assertFalse("both dc.date.embargoend values lie in the future, so the ORIGINAL bitstream must not be" + + " publicly readable." + diagnostics, + anonymousCanRead(bitstream)); + } + + // ----------------------------------------------------------------------------------------------- + // assertions + // ----------------------------------------------------------------------------------------------- + + /** + * A bitstream created by {@code BitstreamBuilder} inherits the collection DEFAULT_BITSTREAM_READ, which is + * byte-for-byte the state a fresh SAF import leaves behind: one Anonymous/READ policy without a start date. + */ + private void assertFreshImportBaseline(Bitstream bitstream) throws Exception { + List defaultBitstreamReadGroups = + authorizeService.getAuthorizedGroups(context, collection, Constants.DEFAULT_BITSTREAM_READ); + assertTrue("fixture precondition: the collection must grant DEFAULT_BITSTREAM_READ to Anonymous," + + " otherwise the imported bitstream does not model the customer's repository.", + defaultBitstreamReadGroups.contains(anonymousGroup)); + + List policies = anonymousReadPolicies(bitstream); + assertEquals("fixture precondition: a freshly imported ORIGINAL bitstream must carry exactly one" + + " Anonymous/READ policy inherited from the collection default." + diagnostics, + 1, policies.size()); + assertNull("fixture precondition: the inherited Anonymous/READ policy must have no start date." + + diagnostics, + policies.get(0).getStartDate()); + assertTrue("fixture precondition: a freshly imported ORIGINAL bitstream must be publicly readable." + + diagnostics, + anonymousCanRead(bitstream)); + } + + /** + * Zero policies means HTTP 401 on the file; more than one means an undated policy can coexist with the dated + * one and silently neutralise the embargo. Exactly one is the only acceptable outcome. + */ + private ResourcePolicy assertExactlyOneAnonymousReadPolicy(String scenario, Bitstream bitstream) + throws Exception { + List policies = anonymousReadPolicies(bitstream); + assertEquals("After " + scenario + " the ORIGINAL bitstream must carry EXACTLY ONE Anonymous/READ policy." + + " Zero policies make the file unreachable (HTTP 401); more than one lets an undated" + + " policy neutralise the embargo. Found " + policies.size() + "." + diagnostics, + 1, policies.size()); + return policies.get(0); + } + + /** + * The surviving policy has to be normalised: {@code TYPE_CUSTOM} (AuthorizeServiceImpl only honours custom + * policies on not-yet-installed items), {@code rpName="embargo"} (the access condition name used by + * access-conditions.xml, short enough for the 30 character column) and a start date of + * {@code dc.date.embargoend + 1 day}. + */ + private void assertNormalisedEmbargoPolicy(String scenario, ResourcePolicy policy, LocalDate expectedStartDay) { + assertNotNull("After " + scenario + " resource policy #" + policy.getID() + " must carry a start date." + + diagnostics, + policy.getStartDate()); + assertEquals("After " + scenario + " resource policy #" + policy.getID() + " must start on the day AFTER" + + " dc.date.embargoend, because the end date is the inclusive last day of the embargo." + + diagnostics, + expectedStartDay, toLocalDate(policy.getStartDate())); + assertEquals("After " + scenario + " resource policy #" + policy.getID() + " must be normalised to" + + " rpType=" + ResourcePolicy.TYPE_CUSTOM + "; AuthorizeServiceImpl only honours custom" + + " policies on not-yet-installed items." + diagnostics, + ResourcePolicy.TYPE_CUSTOM, policy.getRpType()); + assertEquals("After " + scenario + " resource policy #" + policy.getID() + " must be normalised to" + + " rpName=\"" + EMBARGO_POLICY_NAME + "\", replacing legacy names such as" + + " \"Standard Embargo\" or \"Special Case Embargo\"." + diagnostics, + EMBARGO_POLICY_NAME, policy.getRpName()); + } + + private void assertEmbargoEndStored(Item item, String expectedEmbargoEnd) { + assertEquals("itemupdate did not store dc.date.embargoend on the item, so the run never really reached it" + + " (ItemArchive.create may have failed to resolve it - processArchive swallows every" + + " per-item exception)." + diagnostics, + expectedEmbargoEnd, firstMetadataValue(item, "date", "embargoend")); + } + + // ----------------------------------------------------------------------------------------------- + // policy inspection helpers + // ----------------------------------------------------------------------------------------------- + + /** + * Answers the only question that matters: may a not-logged-in visitor download the file? + * + *

The authorisation state is a stack, not a flag, and the builders as well as {@code processArchive} push + * and pop around themselves, so the depth at assertion time is not guaranteed to be zero. The stack is + * therefore drained (otherwise {@code AuthorizeServiceImpl.authorize} short-circuits and every read looks + * allowed) and restored afterwards. The current user is cleared as well, because {@code setUp} leaves the + * test EPerson logged in.

+ */ + private boolean anonymousCanRead(Bitstream bitstream) throws Exception { + EPerson savedUser = context.getCurrentUser(); + int popped = 0; + while (context.ignoreAuthorization()) { + context.restoreAuthSystemState(); + popped++; + } + context.setCurrentUser(null); + try { + return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); + } finally { + context.setCurrentUser(savedUser); + for (int i = 0; i < popped; i++) { + context.turnOffAuthorisationSystem(); + } + } + } + + private List anonymousReadPolicies(Bitstream bitstream) throws Exception { + return resourcePolicyService.find(context, bitstream, Constants.READ).stream() + .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) + .collect(Collectors.toList()); + } + + private void dump(String label, Bitstream bitstream) throws Exception { + StringBuilder sb = new StringBuilder(); + sb.append(System.lineSeparator()) + .append(" === ").append(label).append(" ===").append(System.lineSeparator()) + .append(" bitstream=").append(bitstream.getID()).append(System.lineSeparator()) + .append(" anonymousCanRead=").append(anonymousCanRead(bitstream)).append(System.lineSeparator()); + + List policies = resourcePolicyService.find(context, bitstream, Constants.READ); + if (policies.isEmpty()) { + sb.append(" ").append(System.lineSeparator()); + } + for (ResourcePolicy policy : policies) { + sb.append(String.format(" id=%s group=%s action=%s rpType=%s rpName=%s start=%s end=%s valid=%s", + policy.getID(), + policy.getGroup() == null ? "" : policy.getGroup().getName(), + Constants.actionText[policy.getAction()], + policy.getRpType(), + policy.getRpName(), + policy.getStartDate(), + policy.getEndDate(), + resourcePolicyService.isDateValid(policy))) + .append(System.lineSeparator()); + } + + diagnostics.append(sb); + System.out.print(sb); + } + + /** + * {@code ResourcePolicy.startDate} is mapped as {@code @Temporal(DATE)}, so once it has been round-tripped + * through the database it comes back as a day-granular {@code java.sql.Date}. Compare calendar days, never + * {@code Date} instances, across the UTC/Europe-Dublin boundary. + */ + private LocalDate toLocalDate(Date date) { + if (date instanceof java.sql.Date) { + return ((java.sql.Date) date).toLocalDate(); + } + return date.toInstant().atZone(ZoneOffset.UTC).toLocalDate(); + } + + // ----------------------------------------------------------------------------------------------- + // fixture helpers + // ----------------------------------------------------------------------------------------------- + + /** Calendar "today" in UTC - the specification does all embargo arithmetic in UTC calendar days. */ + private LocalDate utcToday() { + return LocalDate.now(ZoneOffset.UTC); + } + + /** + * The next 1 July strictly after today, computed dynamically. Ireland observes Irish Summer Time (UTC+1) on + * that date every year, which is what makes midnight UTC and midnight in the server zone distinguishable. + */ + private LocalDate nextIrishSummerTimeDay() { + LocalDate today = utcToday(); + LocalDate candidate = LocalDate.of(today.getYear(), 7, 1); + if (!candidate.isAfter(today)) { + candidate = candidate.plusYears(1); + } + return candidate; + } + + private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { + MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); + MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); + if (existingField == null) { + MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); + } + } + + private Item createItem(String title, String... metadataTriples) throws Exception { + context.turnOffAuthorisationSystem(); + ItemBuilder builder = ItemBuilder.createItem(context, collection).withTitle(title); + for (int i = 0; i + 2 < metadataTriples.length; i += 3) { + builder.withMetadata("dc", metadataTriples[i], metadataTriples[i + 1], metadataTriples[i + 2]); + } + Item item = builder.build(); + context.restoreAuthSystemState(); + return item; + } + + private Bitstream createOriginalBitstream(Item item, String name) throws Exception { + context.turnOffAuthorisationSystem(); + Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, + new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8))) + .withName(name) + .withMimeType("text/plain") + .build(); + context.restoreAuthSystemState(); + return bitstream; + } + + private List metadataValues(Item item, String element, String qualifier) { + return itemService.getMetadata(item, "dc", element, qualifier, Item.ANY).stream() + .map(MetadataValue::getValue) + .collect(Collectors.toList()); + } + + private String firstMetadataValue(Item item, String element, String qualifier) { + List values = metadataValues(item, element, qualifier); + return values.isEmpty() ? null : values.get(0); + } + + /** + * Equivalent of {@code ItemUpdate -s SAFDIR -d dc.rights.access -d dc.date.embargoend -a dc.rights.access + * -a dc.date.embargoend}: an update whose target fields contain an embargo field, which is exactly what + * makes {@code processArchive} call {@code syncEmbargoPolicies}. + * + *

{@code ItemUpdate.main} is deliberately not used - it ends in {@code System.exit} and would kill the + * failsafe JVM.

+ * + * @return everything {@code ItemUpdate.pr()} printed during the run; the stream is teed, so the output still + * reaches the failsafe output file as well. + */ + private String runItemUpdate(Item item, String dublinCoreContent) throws Exception { + Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); + // Without suppress_undo, processArchive writes an undo archive as a SIBLING of the source directory. + Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); + + Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); + Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); + + ItemUpdate itemUpdate = new ItemUpdate(); + DeleteMetadataAction deleteAction = + (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); + deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + AddMetadataAction addAction = + (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); + addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + ByteArrayOutputStream captured = new ByteArrayOutputStream(); + PrintStream originalOut = System.out; + System.setOut(new PrintStream(new TeeOutputStream(originalOut, captured), true, + StandardCharsets.UTF_8.name())); + context.turnOffAuthorisationSystem(); + try { + itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); + } finally { + context.restoreAuthSystemState(); + System.out.flush(); + System.setOut(originalOut); + } + + context.uncacheEntity(item); + return captured.toString(StandardCharsets.UTF_8.name()); + } + + /** + * Builds a SAF {@code dublin_core.xml}. {@code ItemArchive.create} resolves the item by + * {@code dc.identifier.uri == ItemUpdate.HANDLE_PREFIX + handle}. + * + * @param rightsAccess value for {@code dc.rights.access}, or {@code null} to omit the element entirely + * @param embargoEndDates zero or more {@code dc.date.embargoend} values, emitted in the given order + */ + private String dublinCore(Item item, String rightsAccess, String... embargoEndDates) { + StringBuilder sb = new StringBuilder(); + sb.append("\n") + .append("\n") + .append(" ") + .append(ItemUpdate.HANDLE_PREFIX).append(item.getHandle()) + .append("\n"); + + if (rightsAccess != null) { + sb.append(" ") + .append(rightsAccess) + .append("\n"); + } + + for (String embargoEndDate : embargoEndDates) { + if (embargoEndDate == null) { + continue; + } + sb.append(" ") + // an empty XML element is dropped by the parser, a single space survives as a blank value + .append(embargoEndDate.isEmpty() ? " " : embargoEndDate) + .append("\n"); + } + + sb.append(""); + return sb.toString(); + } +} diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java new file mode 100644 index 000000000000..df99d0137383 --- /dev/null +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java @@ -0,0 +1,884 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.app.itemupdate; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.ByteArrayInputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.LocalDate; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.Set; +import java.util.TreeSet; +import java.util.UUID; +import java.util.stream.Collectors; + +import org.apache.commons.io.file.PathUtils; +import org.dspace.AbstractIntegrationTestWithDatabase; +import org.dspace.authorize.ResourcePolicy; +import org.dspace.authorize.factory.AuthorizeServiceFactory; +import org.dspace.authorize.service.AuthorizeService; +import org.dspace.authorize.service.ResourcePolicyService; +import org.dspace.builder.BitstreamBuilder; +import org.dspace.builder.CollectionBuilder; +import org.dspace.builder.CommunityBuilder; +import org.dspace.builder.ItemBuilder; +import org.dspace.builder.MetadataFieldBuilder; +import org.dspace.builder.ResourcePolicyBuilder; +import org.dspace.content.Bitstream; +import org.dspace.content.Bundle; +import org.dspace.content.Collection; +import org.dspace.content.DSpaceObject; +import org.dspace.content.Item; +import org.dspace.content.MetadataField; +import org.dspace.content.MetadataSchema; +import org.dspace.content.MetadataValue; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.BundleService; +import org.dspace.content.service.ItemService; +import org.dspace.content.service.MetadataFieldService; +import org.dspace.content.service.MetadataSchemaService; +import org.dspace.core.Constants; +import org.dspace.eperson.EPerson; +import org.dspace.eperson.Group; +import org.dspace.eperson.factory.EPersonServiceFactory; +import org.dspace.eperson.service.GroupService; +import org.dspace.handle.factory.HandleServiceFactory; +import org.dspace.handle.service.HandleService; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +/** + * Life cycle, legacy data migration and invariant tests for the VSB-TUO embargo synchronisation in + * {@link ItemUpdate}. + * + *

Where {@code EmbargoPastDateIT} reproduces the single incident reported by the customer, this class + * covers the whole life cycle of an embargo as it is really operated: setting it, re-running the very same + * SAF archive, lifting it again by dropping {@code dc.date.embargoend}, and doing all of that on bitstreams + * whose resource policies were written by the previous (PR #1313 / #1315) implementation and therefore still + * carry the legacy {@code rpName} values {@code "Standard Embargo"} and {@code "Special Case Embargo"}.

+ * + *

The binding rules exercised here are:

+ *
    + *
  • the survivor policy is located by {@code (group = Anonymous, action = READ)} and never by + * {@code rpName}, so policies written by the old code are picked up and normalised;
    • + *
  • the survivor is mutated, never deleted and recreated, so its {@code policy_id} is stable;
    • + *
  • an ORIGINAL bitstream that started with at least one READ policy always ends with at least one - + * whatever {@code dc.date.embargoend} contained;
    • + *
  • exactly one {@code Anonymous}/{@code READ} policy remains, so no immediate policy can survive next to + * a dated one and quietly defeat the embargo;
    • + *
  • bitstreams outside the ORIGINAL bundle are never touched - their policies belong to filter-media.
    • + *
+ */ +public class EmbargoLifecycleIT extends AbstractIntegrationTestWithDatabase { + + /** Target policy name of the fix. Must stay within the 30 char {@code resourcepolicy.rpname} column. */ + private static final String EMBARGO_POLICY_NAME = "embargo"; + + /** Policy names written by the previous implementation and still present in the customer database. */ + private static final String LEGACY_STANDARD_EMBARGO = "Standard Embargo"; + private static final String LEGACY_SPECIAL_CASE_EMBARGO = "Special Case Embargo"; + + private static final String OPEN_ACCESS = "openAccess"; + private static final String EMBARGOED_ACCESS = "embargoedAccess"; + + private static final String TEXT_BUNDLE = "TEXT"; + private static final String THUMBNAIL_BUNDLE = "THUMBNAIL"; + + private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); + private final BundleService bundleService = ContentServiceFactory.getInstance().getBundleService(); + private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); + private final ResourcePolicyService resourcePolicyService = + AuthorizeServiceFactory.getInstance().getResourcePolicyService(); + private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); + private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); + private final MetadataSchemaService metadataSchemaService = + ContentServiceFactory.getInstance().getMetadataSchemaService(); + private final MetadataFieldService metadataFieldService = + ContentServiceFactory.getInstance().getMetadataFieldService(); + + private Collection collection; + private Group anonymousGroup; + private Path tempDir; + private String previousHandlePrefix; + + @Before + @Override + public void setUp() throws Exception { + super.setUp(); + context.turnOffAuthorisationSystem(); + + parentCommunity = CommunityBuilder.createCommunity(context) + .withName("Parent Community") + .build(); + collection = CollectionBuilder.createCollection(context, parentCommunity) + .withName("Collection") + .build(); + + ensureMetadataFieldExists("rights", "access"); + ensureMetadataFieldExists("date", "embargoend"); + + anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); + previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; + ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); + + context.restoreAuthSystemState(); + + tempDir = Files.createTempDirectory("embargoLifecycleIT"); + } + + @After + @Override + public void destroy() throws Exception { + ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; + if (tempDir != null) { + PathUtils.deleteDirectory(tempDir); + } + super.destroy(); + } + + /** + * Removing {@code dc.date.embargoend} is the only way an operator lifts an embargo, and it is the second + * most frequent embargo operation after setting one. It must clear the start date of the surviving policy + * (spec row #5) instead of deleting the policy and leaving the file unreachable. + */ + @Test + public void removingEmbargoMetadataLiftsEmbargo() throws Exception { + String futureEmbargoEnd = LocalDate.now().plusMonths(6).toString(); + + Item item = createItem("Lift Embargo Thesis"); + Bitstream bitstream = createOriginalBitstream(item, "thesis.pdf"); + Integer importedPolicyId = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import").getID(); + + // (a) operator embargoes the item + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + assertEquals("itemupdate did not store the future embargo end date", + futureEmbargoEnd, singleMetadataValue(item, "date", "embargoend")); + ResourcePolicy embargoed = onlyAnonymousReadPolicy(bitstream, "the embargo run"); + assertNotNull("the surviving Anonymous READ policy must be dated while the embargo runs", + embargoed.getStartDate()); + assertEquals("the imported Anonymous READ policy must be mutated in place, not deleted and recreated", + importedPolicyId, embargoed.getID()); + assertFalse("while embargoed the file must not be publicly readable" + describePolicies(bitstream), + anonymousCanRead(bitstream)); + + // (b) operator lifts the embargo: the SAF no longer carries dc.date.embargoend + runItemUpdate(item, dublinCore(item, OPEN_ACCESS, null)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + assertTrue("itemupdate did not remove dc.date.embargoend from the item", + itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY).isEmpty()); + + ResourcePolicy lifted = onlyAnonymousReadPolicy(bitstream, "the embargo lift run"); + assertEquals("lifting an embargo must mutate the surviving policy, not delete and recreate it", + importedPolicyId, lifted.getID()); + assertNull("lifting an embargo must clear startDate on the surviving Anonymous READ policy" + + describePolicies(bitstream), + lifted.getStartDate()); + assertTrue("after the embargo was lifted the file must be publicly readable again" + + describePolicies(bitstream), + anonymousCanRead(bitstream)); + } + + /** + * Re-running the identical SAF archive - which happens every time an import job is retried - must be a + * no-op. The same {@code policy_id} has to come back, which is only possible if the policy is mutated + * rather than deleted and recreated. + */ + @Test + public void syncIsIdempotent() throws Exception { + String futureEmbargoEnd = LocalDate.now().plusMonths(3).toString(); + LocalDate expectedStart = LocalDate.parse(futureEmbargoEnd).plusDays(1); + + Item item = createItem("Idempotent Thesis"); + Bitstream bitstream = createOriginalBitstream(item, "idempotent.pdf"); + Integer importedPolicyId = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import").getID(); + + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + ResourcePolicy first = onlyAnonymousReadPolicy(bitstream, "the first run"); + Integer firstId = first.getID(); + assertEquals("the imported Anonymous READ policy must be mutated in place, not deleted and recreated", + importedPolicyId, firstId); + assertNotNull("the surviving policy must be dated after the first run", first.getStartDate()); + assertEquals("embargo must start the day after dc.date.embargoend", + expectedStart, toLocalDate(first.getStartDate())); + assertEquals("the surviving policy must be renamed to the canonical access condition", + EMBARGO_POLICY_NAME, first.getRpName()); + assertEquals("the surviving policy must be TYPE_CUSTOM", ResourcePolicy.TYPE_CUSTOM, first.getRpType()); + + // exactly the same archive again + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + ResourcePolicy second = onlyAnonymousReadPolicy(bitstream, "the second, identical run"); + assertEquals("a repeated identical run must mutate the same policy row - a changed policy_id proves" + + " the policy was deleted and recreated, which is what loses the file on the way" + + describePolicies(bitstream), + firstId, second.getID()); + assertEquals("a repeated identical run must not move the embargo start date", + expectedStart, toLocalDate(second.getStartDate())); + assertEquals("a repeated identical run must not change the policy name", + EMBARGO_POLICY_NAME, second.getRpName()); + assertEquals("a repeated identical run must not change the policy type", + ResourcePolicy.TYPE_CUSTOM, second.getRpType()); + assertFalse("the file must still be embargoed after the second run" + describePolicies(bitstream), + anonymousCanRead(bitstream)); + } + + /** + * The customer database is full of policies named {@code "Standard Embargo"} whose start date has already + * passed. Re-embargoing such a bitstream must find that policy by {@code (Anonymous, READ)}, reuse it and + * normalise its name. An implementation that looks the survivor up by {@code rpName == "embargo"} would + * create a second policy and leave the expired one in place, so the file would stay downloadable. + */ + @Test + public void legacyRpNameThenFutureEmbargoIsEnforced() throws Exception { + assertLegacyPolicyIsAdoptedAndEnforced(LEGACY_STANDARD_EMBARGO, EMBARGOED_ACCESS, "legacy-standard.pdf"); + } + + /** + * Same as {@link #legacyRpNameThenFutureEmbargoIsEnforced()} for the second legacy name, written by the + * previous implementation whenever {@code dc.rights.access} was not {@code embargoedAccess}. + */ + @Test + public void legacySpecialCaseRpNameIsAlsoPickedUp() throws Exception { + assertLegacyPolicyIsAdoptedAndEnforced(LEGACY_SPECIAL_CASE_EMBARGO, null, "legacy-special-case.pdf"); + } + + /** + * An item that never had an embargo carries a single immediate ({@code startDate == null}) policy. Putting + * it under embargo must consume that policy: leaving it next to a dated one would make the embargo a no-op + * because the immediate policy alone already grants anonymous READ (spec row #13). + */ + @Test + public void bornOpenItemThenFutureEmbargoIsEnforced() throws Exception { + String futureEmbargoEnd = LocalDate.now().plusMonths(4).toString(); + + Item item = createItem("Born Open Thesis"); + Bitstream bitstream = createOriginalBitstream(item, "born-open.pdf"); + + ResourcePolicy imported = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import"); + Integer importedPolicyId = imported.getID(); + assertNull("fixture precondition: a born open bitstream carries an immediate policy", + imported.getStartDate()); + assertTrue("fixture precondition: a born open bitstream is publicly readable", anonymousCanRead(bitstream)); + + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + List after = anonymousReadPolicies(bitstream); + assertEquals("the immediate Anonymous READ policy must be consumed by the embargo, not left beside a" + + " dated one" + describePolicies(bitstream), + 1, after.size()); + + ResourcePolicy survivor = after.get(0); + assertEquals("the immediate policy must be mutated in place, not deleted and recreated", + importedPolicyId, survivor.getID()); + assertNotNull("the surviving policy must be dated", survivor.getStartDate()); + assertEquals("embargo must start the day after dc.date.embargoend", + LocalDate.parse(futureEmbargoEnd).plusDays(1), toLocalDate(survivor.getStartDate())); + assertEquals("the surviving policy must be renamed to the canonical access condition", + EMBARGO_POLICY_NAME, survivor.getRpName()); + assertEquals("the surviving policy must be TYPE_CUSTOM", + ResourcePolicy.TYPE_CUSTOM, survivor.getRpType()); + assertFalse("an immediate Anonymous READ policy left next to the embargo makes the embargo a no-op" + + describePolicies(bitstream), + anonymousCanRead(bitstream)); + } + + /** + * A bitstream that accumulated several {@code Anonymous}/{@code READ} policies - the classic mix of one + * immediate policy and the leftovers of earlier embargo runs - must end up with exactly one policy, taken + * from the pre-existing ones (spec row #13). Any surviving second policy would either defeat the embargo + * or resurrect an obsolete date. + */ + @Test + public void duplicateAnonymousReadPoliciesCollapseToOne() throws Exception { + String futureEmbargoEnd = LocalDate.now().plusMonths(5).toString(); + + Item item = createItem("Duplicate Policies Thesis"); + Bitstream bitstream = createOriginalBitstream(item, "duplicates.pdf"); + + Integer immediateId = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import").getID(); + Integer oldestDatedId = addAnonymousReadPolicy(bitstream, startOfDayUtc(LocalDate.now().minusMonths(3)), + LEGACY_STANDARD_EMBARGO).getID(); + Integer newestDatedId = addAnonymousReadPolicy(bitstream, startOfDayUtc(LocalDate.now().plusMonths(2)), + LEGACY_SPECIAL_CASE_EMBARGO).getID(); + bitstream = context.reloadEntity(bitstream); + assertEquals("fixture precondition: three Anonymous READ policies" + describePolicies(bitstream), + 3, anonymousReadPolicies(bitstream).size()); + + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + List after = anonymousReadPolicies(bitstream); + assertEquals("duplicate Anonymous READ policies must collapse into exactly one" + + describePolicies(bitstream), + 1, after.size()); + + ResourcePolicy survivor = after.get(0); + Integer survivorId = survivor.getID(); + assertTrue("the survivor must be one of the three pre-existing policies - a brand new policy_id proves" + + " delete and recreate" + describePolicies(bitstream), + survivorId.equals(immediateId) || survivorId.equals(oldestDatedId) + || survivorId.equals(newestDatedId)); + assertTrue("the survivor must be the oldest policy: either the immediate one (in force since forever)" + + " or the one with the oldest start date - never the newest one" + + describePolicies(bitstream), + survivorId.equals(immediateId) || survivorId.equals(oldestDatedId)); + assertEquals("embargo must start the day after dc.date.embargoend", + LocalDate.parse(futureEmbargoEnd).plusDays(1), toLocalDate(survivor.getStartDate())); + assertEquals("the surviving policy must be renamed to the canonical access condition", + EMBARGO_POLICY_NAME, survivor.getRpName()); + assertFalse("a second Anonymous READ policy would defeat the embargo" + describePolicies(bitstream), + anonymousCanRead(bitstream)); + } + + /** + * The single invariant the whole fix exists for: an ORIGINAL bitstream that had at least one READ policy + * before an {@code itemupdate} run must still have one afterwards, no matter what + * {@code dc.date.embargoend} contained. Zero READ policies is what turns the file into an HTTP 401. + * + *

Every case first puts a real embargo in place, exactly like the customer did, because that is what + * replaces the collection default with a single dated policy - the one the second run then has the + * opportunity to delete.

+ */ + @Test + public void neverZeroReadPoliciesInvariant() throws Exception { + String initialEmbargoEnd = LocalDate.now().plusYears(1).toString(); + String impossibleCalendarDay = LocalDate.now().plusYears(1).getYear() + "-02-30"; + + List cases = new ArrayList<>(); + // { label, dc.rights.access, dc.date.embargoend (null = element absent from the SAF) } + cases.add(new String[] { "future date", OPEN_ACCESS, LocalDate.now().plusMonths(9).toString() }); + cases.add(new String[] { "today", OPEN_ACCESS, LocalDate.now().toString() }); + cases.add(new String[] { "past date", OPEN_ACCESS, LocalDate.now().minusMonths(1).toString() }); + cases.add(new String[] { "empty value", OPEN_ACCESS, "" }); + cases.add(new String[] { "unparseable value", OPEN_ACCESS, "not-a-date" }); + cases.add(new String[] { "impossible calendar day", OPEN_ACCESS, impossibleCalendarDay }); + cases.add(new String[] { "element removed", OPEN_ACCESS, null }); + + StringBuilder violations = new StringBuilder(); + + for (String[] testCase : cases) { + String label = testCase[0]; + + Item item = createItem("Invariant Thesis - " + label); + Bitstream bitstream = createOriginalBitstream(item, "invariant.pdf"); + + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, initialEmbargoEnd)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + int readPoliciesBefore = readPolicies(bitstream).size(); + int anonymousBefore = anonymousReadPolicies(bitstream).size(); + if (readPoliciesBefore < 1 || anonymousBefore < 1) { + violations.append(System.lineSeparator()) + .append(" [").append(label).append("] the fixture was already broken by the embargo run:") + .append(describePolicies(bitstream)); + continue; + } + + runItemUpdate(item, dublinCore(item, testCase[1], testCase[2])); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + int readPoliciesAfter = readPolicies(bitstream).size(); + int anonymousAfter = anonymousReadPolicies(bitstream).size(); + if (readPoliciesAfter < 1 || anonymousAfter < 1) { + violations.append(System.lineSeparator()) + .append(" [").append(label).append("] dc.date.embargoend=") + .append(testCase[2] == null ? "" : "'" + testCase[2] + "'") + .append(" reduced ").append(readPoliciesBefore).append(" READ policies (") + .append(anonymousBefore).append(" of them Anonymous) to ").append(readPoliciesAfter) + .append(" (").append(anonymousAfter).append(" Anonymous):") + .append(describePolicies(bitstream)); + } + } + + if (violations.length() > 0) { + fail("An itemupdate run must never leave an ORIGINAL bitstream without an Anonymous READ policy." + + " A bitstream with no READ policy answers HTTP 401 and no later run can publish it again." + + violations); + } + } + + /** + * A real VSB-TUO record carries several files in ORIGINAL, one of them flagged as the primary bitstream. + * All of them must reach exactly the same state - the primary bitstream is not special - and the primary + * flag itself must survive. + */ + @Test + public void multipleBitstreamsAllGetSameState() throws Exception { + String futureEmbargoEnd = LocalDate.now().plusMonths(7).toString(); + String pastEmbargoEnd = LocalDate.now().minusMonths(1).toString(); + + Item item = createItem("Six File Thesis"); + List bitstreams = new ArrayList<>(); + for (int i = 0; i < 6; i++) { + bitstreams.add(createOriginalBitstream(item, "file-" + i + ".pdf")); + } + Bundle originalBundle = bundleOf(item, Constants.CONTENT_BUNDLE_NAME); + setPrimaryBitstream(originalBundle, bitstreams.get(0)); + UUID primaryBitstreamId = bitstreams.get(0).getID(); + + // (a) embargo every file of the record + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + item = context.reloadEntity(item); + reloadAll(bitstreams); + originalBundle = context.reloadEntity(originalBundle); + + assertUniformState(bitstreams, "the embargo run"); + for (Bitstream bitstream : bitstreams) { + ResourcePolicy policy = onlyAnonymousReadPolicy(bitstream, "the embargo run"); + assertEquals("embargo must start the day after dc.date.embargoend", + LocalDate.parse(futureEmbargoEnd).plusDays(1), toLocalDate(policy.getStartDate())); + assertEquals("every ORIGINAL bitstream must carry the canonical policy name", + EMBARGO_POLICY_NAME, policy.getRpName()); + assertEquals("every ORIGINAL bitstream must carry a TYPE_CUSTOM policy", + ResourcePolicy.TYPE_CUSTOM, policy.getRpType()); + assertFalse("every embargoed file of the record must be closed" + describePolicies(bitstream), + anonymousCanRead(bitstream)); + } + assertNotNull("the ORIGINAL bundle lost its primary bitstream", originalBundle.getPrimaryBitstream()); + assertEquals("the primary bitstream flag must survive an embargo run", + primaryBitstreamId, originalBundle.getPrimaryBitstream().getID()); + + // (b) the embargo expires - the same archive is re-imported with a past date + runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd)); + item = context.reloadEntity(item); + reloadAll(bitstreams); + originalBundle = context.reloadEntity(originalBundle); + + assertUniformState(bitstreams, "the expired embargo run"); + for (Bitstream bitstream : bitstreams) { + ResourcePolicy policy = onlyAnonymousReadPolicy(bitstream, "the expired embargo run"); + assertEquals("an expired embargo still starts the day after dc.date.embargoend", + LocalDate.parse(pastEmbargoEnd).plusDays(1), toLocalDate(policy.getStartDate())); + assertTrue("an expired embargo must be effective immediately" + describePolicies(bitstream), + resourcePolicyService.isDateValid(policy)); + assertTrue("an expired embargo publishes every file of the record" + describePolicies(bitstream), + anonymousCanRead(bitstream)); + } + assertNotNull("the ORIGINAL bundle lost its primary bitstream", originalBundle.getPrimaryBitstream()); + assertEquals("the primary bitstream flag must survive an expired embargo run", + primaryBitstreamId, originalBundle.getPrimaryBitstream().getID()); + } + + /** + * The embargo synchronisation owns the bitstreams of the ORIGINAL bundle only. Derivatives (TEXT, + * THUMBNAIL) are produced and re-protected by filter-media, and the bundle objects themselves carry their + * own policies; touching either from here is how a record ends up with 18 unreachable bitstreams. + */ + @Test + public void derivativeBundlesAreNotTouchedDirectly() throws Exception { + String futureEmbargoEnd = LocalDate.now().plusMonths(2).toString(); + String pastEmbargoEnd = LocalDate.now().minusMonths(2).toString(); + + Item item = createItem("Derivatives Thesis"); + Bitstream original = createOriginalBitstream(item, "thesis.pdf"); + Bitstream extractedText = createBitstreamInBundle(item, "thesis.pdf.txt", TEXT_BUNDLE); + Bitstream thumbnail = createBitstreamInBundle(item, "thesis.pdf.jpg", THUMBNAIL_BUNDLE); + + Bundle originalBundle = bundleOf(item, Constants.CONTENT_BUNDLE_NAME); + Set textPolicies = policySignatures(extractedText); + Set thumbnailPolicies = policySignatures(thumbnail); + Set originalBundlePolicies = policySignatures(originalBundle); + assertFalse("fixture precondition: the TEXT bitstream must start with policies", textPolicies.isEmpty()); + assertFalse("fixture precondition: the THUMBNAIL bitstream must start with policies", + thumbnailPolicies.isEmpty()); + assertFalse("fixture precondition: the ORIGINAL bundle must start with policies", + originalBundlePolicies.isEmpty()); + + // (a) embargo run + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + item = context.reloadEntity(item); + original = context.reloadEntity(original); + extractedText = context.reloadEntity(extractedText); + thumbnail = context.reloadEntity(thumbnail); + originalBundle = context.reloadEntity(originalBundle); + + ResourcePolicy embargoed = onlyAnonymousReadPolicy(original, "the embargo run"); + assertEquals("sanity check: this run must have embargoed the ORIGINAL bitstream", + EMBARGO_POLICY_NAME, embargoed.getRpName()); + assertEquals("TEXT bitstream policies belong to filter-media and must not be rewritten here", + textPolicies, policySignatures(extractedText)); + assertEquals("THUMBNAIL bitstream policies belong to filter-media and must not be rewritten here", + thumbnailPolicies, policySignatures(thumbnail)); + assertEquals("the ORIGINAL bundle's own policies must not be touched", + originalBundlePolicies, policySignatures(originalBundle)); + + // (b) expired embargo run - the code path that wipes policies today + runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd)); + item = context.reloadEntity(item); + original = context.reloadEntity(original); + extractedText = context.reloadEntity(extractedText); + thumbnail = context.reloadEntity(thumbnail); + originalBundle = context.reloadEntity(originalBundle); + + assertTrue("sanity check: the expired embargo must publish the ORIGINAL bitstream" + + describePolicies(original), + anonymousCanRead(original)); + assertEquals("TEXT bitstream policies must survive an expired embargo untouched", + textPolicies, policySignatures(extractedText)); + assertEquals("THUMBNAIL bitstream policies must survive an expired embargo untouched", + thumbnailPolicies, policySignatures(thumbnail)); + assertEquals("the ORIGINAL bundle's own policies must survive an expired embargo untouched", + originalBundlePolicies, policySignatures(originalBundle)); + } + + /** + * Shared body of the two legacy {@code rpName} tests: a bitstream whose only {@code Anonymous}/{@code READ} + * policy was written by the previous implementation (legacy name, start date already passed, so the file is + * public) is put back under embargo. + */ + private void assertLegacyPolicyIsAdoptedAndEnforced(String legacyName, String rightsAccess, String fileName) + throws Exception { + String futureEmbargoEnd = LocalDate.now().plusYears(1).toString(); + + Item item = createItem("Legacy Policy Thesis - " + legacyName); + Bitstream bitstream = createOriginalBitstream(item, fileName); + Integer legacyPolicyId = + replaceAnonymousReadPolicies(bitstream, startOfDayUtc(LocalDate.now().minusMonths(2)), legacyName) + .getID(); + bitstream = context.reloadEntity(bitstream); + + ResourcePolicy legacy = onlyAnonymousReadPolicy(bitstream, "the legacy fixture"); + assertEquals("fixture precondition: the legacy policy must be the only Anonymous READ policy", + legacyPolicyId, legacy.getID()); + assertEquals("fixture precondition: the legacy policy keeps its old name", legacyName, legacy.getRpName()); + assertTrue("fixture precondition: an expired legacy embargo leaves the file public" + + describePolicies(bitstream), + anonymousCanRead(bitstream)); + + runItemUpdate(item, dublinCore(item, rightsAccess, futureEmbargoEnd)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + List after = anonymousReadPolicies(bitstream); + assertEquals("the legacy policy must be adopted, so exactly one Anonymous READ policy may remain." + + " Looking the survivor up by rpName instead of by (Anonymous, READ) leaves the expired" + + " legacy policy next to the new one and the embargo never takes effect." + + describePolicies(bitstream), + 1, after.size()); + + ResourcePolicy survivor = after.get(0); + assertEquals("the legacy policy must be mutated in place, not deleted and recreated", + legacyPolicyId, survivor.getID()); + assertEquals("the legacy policy name must be normalised", EMBARGO_POLICY_NAME, survivor.getRpName()); + assertEquals("the normalised policy must be TYPE_CUSTOM", + ResourcePolicy.TYPE_CUSTOM, survivor.getRpType()); + assertNotNull("the normalised policy must be dated", survivor.getStartDate()); + assertEquals("embargo must start the day after dc.date.embargoend", + LocalDate.parse(futureEmbargoEnd).plusDays(1), toLocalDate(survivor.getStartDate())); + assertFalse("re-embargoing a legacy bitstream must actually block anonymous download" + + describePolicies(bitstream), + anonymousCanRead(bitstream)); + } + + /** + * Asserts that every bitstream of the record ended in the same state - same policy count, same start date, + * same name, same type, same answer to "may an anonymous visitor download it". + */ + private void assertUniformState(List bitstreams, String what) throws Exception { + String expected = stateSignature(bitstreams.get(0)); + for (Bitstream bitstream : bitstreams) { + assertEquals("all ORIGINAL bitstreams of a record must share the same state after " + what + + " (bitstream " + bitstream.getID() + ")", + expected, stateSignature(bitstream)); + } + } + + /** + * Answers the only question that matters: may a not-logged-in visitor download the file? + */ + private boolean anonymousCanRead(Bitstream bitstream) throws Exception { + EPerson saved = context.getCurrentUser(); + int popped = 0; + while (context.ignoreAuthorization()) { + context.restoreAuthSystemState(); + popped++; + } + context.setCurrentUser(null); + try { + return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); + } finally { + context.setCurrentUser(saved); + for (int i = 0; i < popped; i++) { + context.turnOffAuthorisationSystem(); + } + } + } + + private List readPolicies(Bitstream bitstream) throws Exception { + return resourcePolicyService.find(context, bitstream, Constants.READ); + } + + private List anonymousReadPolicies(Bitstream bitstream) throws Exception { + return readPolicies(bitstream).stream() + .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) + .collect(Collectors.toList()); + } + + private ResourcePolicy onlyAnonymousReadPolicy(Bitstream bitstream, String what) throws Exception { + List policies = anonymousReadPolicies(bitstream); + assertEquals("exactly one Anonymous READ policy must remain after " + what + describePolicies(bitstream), + 1, policies.size()); + return policies.get(0); + } + + /** + * State of a bitstream with the policy identities left out, so two different bitstreams can be compared. + */ + private String stateSignature(Bitstream bitstream) throws Exception { + StringBuilder sb = new StringBuilder(); + List policies = anonymousReadPolicies(bitstream); + sb.append("anonymousReadPolicies=").append(policies.size()); + for (ResourcePolicy policy : policies) { + sb.append(" [start=") + .append(policy.getStartDate() == null ? "null" : toLocalDate(policy.getStartDate())) + .append(" end=").append(policy.getEndDate() == null ? "null" : toLocalDate(policy.getEndDate())) + .append(" rpName=").append(policy.getRpName()) + .append(" rpType=").append(policy.getRpType()) + .append(" valid=").append(resourcePolicyService.isDateValid(policy)) + .append(']'); + } + sb.append(" anonymousCanRead=").append(anonymousCanRead(bitstream)); + return sb.toString(); + } + + /** + * Full identity of every policy of a DSpace object, {@code policy_id} included - used to prove that a set + * of policies was not touched at all. + */ + private Set policySignatures(DSpaceObject dso) throws Exception { + Set signatures = new TreeSet<>(); + for (ResourcePolicy policy : resourcePolicyService.find(context, dso)) { + signatures.add(String.format("id=%s action=%s group=%s eperson=%s start=%s end=%s rpName=%s rpType=%s", + policy.getID(), + Constants.actionText[policy.getAction()], + policy.getGroup() == null ? "" : policy.getGroup().getName(), + policy.getEPerson() == null ? "" : policy.getEPerson().getEmail(), + policy.getStartDate(), + policy.getEndDate(), + policy.getRpName(), + policy.getRpType())); + } + return signatures; + } + + private String describePolicies(Bitstream bitstream) throws Exception { + StringBuilder sb = new StringBuilder(System.lineSeparator()); + sb.append(" bitstream=").append(bitstream.getID()).append(System.lineSeparator()) + .append(" anonymousCanRead=").append(anonymousCanRead(bitstream)) + .append(System.lineSeparator()); + + List policies = readPolicies(bitstream); + if (policies.isEmpty()) { + sb.append(" ").append(System.lineSeparator()); + } + for (ResourcePolicy policy : policies) { + sb.append(String.format(" id=%s group=%s action=%s rpType=%s rpName=%s start=%s end=%s valid=%s", + policy.getID(), + policy.getGroup() == null ? "" : policy.getGroup().getName(), + Constants.actionText[policy.getAction()], + policy.getRpType(), + policy.getRpName(), + policy.getStartDate(), + policy.getEndDate(), + resourcePolicyService.isDateValid(policy))) + .append(System.lineSeparator()); + } + return sb.toString(); + } + + private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { + MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); + MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); + if (existingField == null) { + MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); + } + } + + private Item createItem(String title) throws Exception { + context.turnOffAuthorisationSystem(); + Item item = ItemBuilder.createItem(context, collection) + .withTitle(title) + .build(); + context.restoreAuthSystemState(); + return item; + } + + /** + * Creates a bitstream in the ORIGINAL bundle. The collection grants DEFAULT_BITSTREAM_READ to Anonymous, + * so the new bitstream carries exactly one policy - Anonymous / READ / TYPE_INHERITED / startDate null - + * which is byte for byte the state of a freshly imported SAF package at the customer. + */ + private Bitstream createOriginalBitstream(Item item, String name) throws Exception { + context.turnOffAuthorisationSystem(); + Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, + new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8))) + .withName(name) + .withMimeType("text/plain") + .build(); + context.restoreAuthSystemState(); + return bitstream; + } + + private Bitstream createBitstreamInBundle(Item item, String name, String bundleName) throws Exception { + context.turnOffAuthorisationSystem(); + Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, + new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8)), bundleName) + .withName(name) + .withMimeType("text/plain") + .build(); + context.restoreAuthSystemState(); + return bitstream; + } + + private Bundle bundleOf(Item item, String bundleName) throws Exception { + List bundles = itemService.getBundles(item, bundleName); + assertFalse("fixture precondition: the item must have a " + bundleName + " bundle", bundles.isEmpty()); + return bundles.get(0); + } + + private void setPrimaryBitstream(Bundle bundle, Bitstream bitstream) throws Exception { + context.turnOffAuthorisationSystem(); + bundle.setPrimaryBitstreamID(bitstream); + bundleService.update(context, bundle); + context.restoreAuthSystemState(); + } + + private void reloadAll(List bitstreams) throws Exception { + for (int i = 0; i < bitstreams.size(); i++) { + bitstreams.set(i, context.reloadEntity(bitstreams.get(i))); + } + } + + private ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) + throws Exception { + context.turnOffAuthorisationSystem(); + ResourcePolicyBuilder builder = ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) + .withAction(Constants.READ) + .withDspaceObject(bitstream) + .withName(name); + if (startDate != null) { + builder.withStartDate(startDate); + } + ResourcePolicy policy = builder.build(); + context.restoreAuthSystemState(); + return policy; + } + + /** + * Replaces every READ policy of the bitstream with a single Anonymous READ policy - the state a bitstream + * is left in by the previous implementation. + */ + private ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name) + throws Exception { + context.turnOffAuthorisationSystem(); + authorizeService.removePoliciesActionFilter(context, bitstream, Constants.READ); + context.restoreAuthSystemState(); + return addAnonymousReadPolicy(bitstream, startDate, name); + } + + private String singleMetadataValue(Item item, String element, String qualifier) { + List values = itemService.getMetadata(item, "dc", element, qualifier, Item.ANY); + return values.isEmpty() ? null : values.get(0).getValue(); + } + + private Date startOfDayUtc(LocalDate day) { + return Date.from(day.atStartOfDay(ZoneOffset.UTC).toInstant()); + } + + private LocalDate toLocalDate(Date date) { + if (date instanceof java.sql.Date) { + return ((java.sql.Date) date).toLocalDate(); + } + return date.toInstant().atZone(ZoneId.systemDefault()).toLocalDate(); + } + + /** + * Equivalent of {@code dsrun ... ItemUpdate -s -d dc.rights.access -d dc.date.embargoend + * -a dc.rights.access -a dc.date.embargoend}, i.e. an update whose target fields contain an embargo field, + * which is what makes {@code processArchive} call {@code syncEmbargoPolicies}. + */ + private void runItemUpdate(Item item, String dublinCoreContent) throws Exception { + Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); + Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); + + Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); + Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); + + ItemUpdate itemUpdate = new ItemUpdate(); + DeleteMetadataAction deleteAction = + (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); + deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + AddMetadataAction addAction = + (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); + addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + context.turnOffAuthorisationSystem(); + itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); + context.restoreAuthSystemState(); + + context.uncacheEntity(item); + } + + /** + * Builds a SAF {@code dublin_core.xml}. A {@code null} value omits the element entirely (that is how an + * operator removes a field), an empty string is written as a single space because an empty XML element is + * dropped by the parser. + */ + private String dublinCore(Item item, String rightsAccess, String embargoEndDate) { + StringBuilder sb = new StringBuilder(); + sb.append("\n") + .append("\n") + .append(" ") + .append(ItemUpdate.HANDLE_PREFIX).append(item.getHandle()) + .append("\n"); + + if (rightsAccess != null) { + sb.append(" ") + .append(rightsAccess) + .append("\n"); + } + + if (embargoEndDate != null) { + sb.append(" ") + .append(embargoEndDate.isEmpty() ? " " : embargoEndDate) + .append("\n"); + } + + sb.append(""); + return sb.toString(); + } +} diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java new file mode 100644 index 000000000000..2240c2ca4771 --- /dev/null +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java @@ -0,0 +1,316 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.app.itemupdate; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.io.ByteArrayInputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.LocalDate; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; + +import org.apache.commons.io.file.PathUtils; +import org.dspace.AbstractIntegrationTestWithDatabase; +import org.dspace.authorize.ResourcePolicy; +import org.dspace.authorize.factory.AuthorizeServiceFactory; +import org.dspace.authorize.service.AuthorizeService; +import org.dspace.authorize.service.ResourcePolicyService; +import org.dspace.builder.BitstreamBuilder; +import org.dspace.builder.CollectionBuilder; +import org.dspace.builder.CommunityBuilder; +import org.dspace.builder.ItemBuilder; +import org.dspace.builder.MetadataFieldBuilder; +import org.dspace.content.Bitstream; +import org.dspace.content.Collection; +import org.dspace.content.Item; +import org.dspace.content.MetadataField; +import org.dspace.content.MetadataSchema; +import org.dspace.content.MetadataValue; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.ItemService; +import org.dspace.content.service.MetadataFieldService; +import org.dspace.content.service.MetadataSchemaService; +import org.dspace.core.Constants; +import org.dspace.eperson.EPerson; +import org.dspace.eperson.Group; +import org.dspace.eperson.factory.EPersonServiceFactory; +import org.dspace.eperson.service.GroupService; +import org.dspace.handle.factory.HandleServiceFactory; +import org.dspace.handle.service.HandleService; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +/** + * Regression test for the VSB-TUO embargo synchronisation in {@link ItemUpdate}. + * + *

Models the exact customer scenario observed on dspace7-test.vsb.cz: a thesis is first embargoed + * with a future {@code dc.date.embargoend} and later the very same SAF archive is re-imported with an + * {@code dc.date.embargoend} that already lies in the past. After the second run every + * {@code Anonymous}/{@code READ} policy is gone from the ORIGINAL bitstreams and the files answer + * HTTP 401 even though {@code dc.rights.access} says {@code openAccess}.

+ */ +public class EmbargoPastDateIT extends AbstractIntegrationTestWithDatabase { + + private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); + private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); + private final ResourcePolicyService resourcePolicyService = + AuthorizeServiceFactory.getInstance().getResourcePolicyService(); + private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); + private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); + private final MetadataSchemaService metadataSchemaService = + ContentServiceFactory.getInstance().getMetadataSchemaService(); + private final MetadataFieldService metadataFieldService = + ContentServiceFactory.getInstance().getMetadataFieldService(); + + private final StringBuilder diagnostics = new StringBuilder(); + + private Collection collection; + private Group anonymousGroup; + private Path tempDir; + private String previousHandlePrefix; + + @Before + @Override + public void setUp() throws Exception { + super.setUp(); + context.turnOffAuthorisationSystem(); + + parentCommunity = CommunityBuilder.createCommunity(context) + .withName("Parent Community") + .build(); + collection = CollectionBuilder.createCollection(context, parentCommunity) + .withName("Collection") + .build(); + + ensureMetadataFieldExists("rights", "access"); + ensureMetadataFieldExists("date", "embargoend"); + + anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); + previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; + ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); + + context.restoreAuthSystemState(); + + tempDir = Files.createTempDirectory("embargoPastDateIT"); + } + + @After + @Override + public void destroy() throws Exception { + ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; + if (tempDir != null) { + PathUtils.deleteDirectory(tempDir); + } + super.destroy(); + } + + /** + * A {@code dc.date.embargoend} in the past must never strip the ORIGINAL bitstreams of their last + * {@code Anonymous}/{@code READ} policy. An expired embargo means "publish", not "hide forever". + */ + @Test + public void pastEmbargoEndMustKeepFilesPublic() throws Exception { + String futureEmbargoEnd = LocalDate.now().plusYears(1).toString(); + String pastEmbargoEnd = LocalDate.now().minusMonths(1).toString(); + + // (a) item with an ORIGINAL bitstream in a collection granting DEFAULT_BITSTREAM_READ to Anonymous + List defaultBitstreamReadGroups = + authorizeService.getAuthorizedGroups(context, collection, Constants.DEFAULT_BITSTREAM_READ); + assertTrue("fixture precondition: collection must grant DEFAULT_BITSTREAM_READ to Anonymous", + defaultBitstreamReadGroups.contains(anonymousGroup)); + + Item item = createItem("VSB-TUO thesis"); + Bitstream bitstream = createOriginalBitstream(item, "thesis.pdf"); + + dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + assertFalse("fixture precondition: imported bitstream must carry an Anonymous READ policy", + anonymousReadPolicies(bitstream).isEmpty()); + assertTrue("fixture precondition: imported bitstream must be publicly readable", + anonymousCanRead(bitstream)); + + // (b) first itemupdate run - embargo end date in the FUTURE (state the customer confirmed as working) + runItemUpdate(item, dublinCore(item, "embargoedAccess", futureEmbargoEnd)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B - after itemupdate with FUTURE dc.date.embargoend=" + futureEmbargoEnd, bitstream); + + assertEquals("itemupdate did not store the future embargo end date", + futureEmbargoEnd, singleMetadataValue(item, "date", "embargoend")); + List embargoed = anonymousReadPolicies(bitstream); + assertEquals("future embargo must leave exactly one Anonymous READ policy", 1, embargoed.size()); + assertNotNull("the surviving Anonymous READ policy must be dated", embargoed.get(0).getStartDate()); + assertFalse("while embargoed the file must not be publicly readable", anonymousCanRead(bitstream)); + + // (c) second itemupdate run - embargo end date in the PAST, item declared openAccess + runItemUpdate(item, dublinCore(item, "openAccess", pastEmbargoEnd)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP C - after itemupdate with PAST dc.date.embargoend=" + pastEmbargoEnd + + " and dc.rights.access=openAccess", bitstream); + + // (d) an expired embargo publishes the file - it must never delete the last Anonymous READ policy + List afterExpiry = anonymousReadPolicies(bitstream); + assertFalse("Expired embargo wiped every Anonymous READ policy from the ORIGINAL bitstream." + + " The file is now unreachable (HTTP 401) although dc.rights.access=openAccess." + + diagnostics, + afterExpiry.isEmpty()); + assertTrue("Expired embargo left the ORIGINAL bitstream unreadable for anonymous users." + + diagnostics, + anonymousCanRead(bitstream)); + } + + /** + * Answers the only question that matters: may a not-logged-in visitor download the file? + */ + private boolean anonymousCanRead(Bitstream bs) throws Exception { + EPerson saved = context.getCurrentUser(); + int popped = 0; + while (context.ignoreAuthorization()) { + context.restoreAuthSystemState(); + popped++; + } + context.setCurrentUser(null); + try { + return authorizeService.authorizeActionBoolean(context, bs, Constants.READ); + } finally { + context.setCurrentUser(saved); + for (int i = 0; i < popped; i++) { + context.turnOffAuthorisationSystem(); + } + } + } + + private List anonymousReadPolicies(Bitstream bitstream) throws Exception { + return resourcePolicyService.find(context, bitstream, Constants.READ).stream() + .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) + .collect(Collectors.toList()); + } + + private void dump(String label, Bitstream bitstream) throws Exception { + List lines = new ArrayList<>(); + for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream, Constants.READ)) { + lines.add(String.format(" id=%s group=%s action=%s rpType=%s rpName=%s start=%s end=%s valid=%s", + policy.getID(), + policy.getGroup() == null ? "" : policy.getGroup().getName(), + Constants.actionText[policy.getAction()], + policy.getRpType(), + policy.getRpName(), + policy.getStartDate(), + policy.getEndDate(), + resourcePolicyService.isDateValid(policy))); + } + if (lines.isEmpty()) { + lines.add(" "); + } + + StringBuilder sb = new StringBuilder(); + sb.append(System.lineSeparator()) + .append(" === ").append(label).append(" ===").append(System.lineSeparator()) + .append(" bitstream=").append(bitstream.getID()).append(System.lineSeparator()) + .append(" anonymousCanRead=").append(anonymousCanRead(bitstream)).append(System.lineSeparator()); + for (String line : lines) { + sb.append(line).append(System.lineSeparator()); + } + diagnostics.append(sb); + System.out.print(sb); + } + + private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { + MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); + MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); + if (existingField == null) { + MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); + } + } + + private Item createItem(String title) throws Exception { + context.turnOffAuthorisationSystem(); + Item item = ItemBuilder.createItem(context, collection) + .withTitle(title) + .build(); + context.restoreAuthSystemState(); + return item; + } + + private Bitstream createOriginalBitstream(Item item, String name) throws Exception { + context.turnOffAuthorisationSystem(); + Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, + new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8))) + .withName(name) + .withMimeType("text/plain") + .build(); + context.restoreAuthSystemState(); + return bitstream; + } + + private String singleMetadataValue(Item item, String element, String qualifier) { + List values = itemService.getMetadata(item, "dc", element, qualifier, Item.ANY); + return values.isEmpty() ? null : values.get(0).getValue(); + } + + /** + * Equivalent of {@code dsrun ... ItemUpdate -s -d dc.rights.access -d dc.date.embargoend + * -a dc.rights.access -a dc.date.embargoend}, i.e. an update whose target fields contain an embargo + * field, which is what makes {@code processArchive} call {@code syncEmbargoPolicies}. + */ + private void runItemUpdate(Item item, String dublinCoreContent) throws Exception { + Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); + Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); + + Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); + Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); + + ItemUpdate itemUpdate = new ItemUpdate(); + DeleteMetadataAction deleteAction = + (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); + deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + AddMetadataAction addAction = + (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); + addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + context.turnOffAuthorisationSystem(); + itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); + context.restoreAuthSystemState(); + + context.uncacheEntity(item); + } + + private String dublinCore(Item item, String rightsAccess, String embargoEndDate) { + String identifierUri = ItemUpdate.HANDLE_PREFIX + item.getHandle(); + StringBuilder sb = new StringBuilder(); + sb.append("\n") + .append("\n") + .append(" ") + .append(identifierUri) + .append("\n"); + + if (rightsAccess != null) { + sb.append(" ") + .append(rightsAccess) + .append("\n"); + } + + if (embargoEndDate != null) { + sb.append(" ") + .append(embargoEndDate) + .append("\n"); + } + + sb.append(""); + return sb.toString(); + } +} diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java new file mode 100644 index 000000000000..e42efcc4e3bb --- /dev/null +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java @@ -0,0 +1,753 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.app.itemupdate; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.PrintStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.SQLException; +import java.time.LocalDate; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Date; +import java.util.List; +import java.util.Set; +import java.util.TreeSet; +import java.util.stream.Collectors; + +import org.apache.commons.io.file.PathUtils; +import org.dspace.AbstractIntegrationTestWithDatabase; +import org.dspace.authorize.ResourcePolicy; +import org.dspace.authorize.factory.AuthorizeServiceFactory; +import org.dspace.authorize.service.AuthorizeService; +import org.dspace.authorize.service.ResourcePolicyService; +import org.dspace.builder.BitstreamBuilder; +import org.dspace.builder.CollectionBuilder; +import org.dspace.builder.CommunityBuilder; +import org.dspace.builder.GroupBuilder; +import org.dspace.builder.ItemBuilder; +import org.dspace.builder.MetadataFieldBuilder; +import org.dspace.builder.ResourcePolicyBuilder; +import org.dspace.content.Bitstream; +import org.dspace.content.Collection; +import org.dspace.content.Item; +import org.dspace.content.MetadataField; +import org.dspace.content.MetadataSchema; +import org.dspace.content.MetadataValue; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.ItemService; +import org.dspace.content.service.MetadataFieldService; +import org.dspace.content.service.MetadataSchemaService; +import org.dspace.core.Constants; +import org.dspace.eperson.EPerson; +import org.dspace.eperson.Group; +import org.dspace.eperson.factory.EPersonServiceFactory; +import org.dspace.eperson.service.GroupService; +import org.dspace.handle.factory.HandleServiceFactory; +import org.dspace.handle.service.HandleService; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +/** + * Safety net for the VSB-TUO embargo synchronisation in {@link ItemUpdate}. + * + *

The bug being fixed is that an expired {@code dc.date.embargoend} strips the ORIGINAL bitstreams + * of their last {@code Anonymous}/{@code READ} policy. The obvious repair - "when the embargo has + * expired, just make the files public" - is far more dangerous than the bug itself, because it would + * publish material that has to stay closed. This class pins down everything the repair must NOT do.

+ * + *

Every "must not touch" assertion compares the full set of {@code policy_id} values plus a + * fingerprint of every policy (action, group, rpType, rpName, start and end date). Comparing counts + * would be useless: a policy deleted and immediately recreated keeps the count but loses its identity, + * and a policy mutated in place keeps its id but changes its meaning.

+ */ +public class EmbargoSafetyIT extends AbstractIntegrationTestWithDatabase { + + /** + * rpName written by the shipped (buggy) implementation. The repair has to recognise and normalise + * these legacy policies, so the fixtures use that name rather than a clean-room one. + */ + private static final String LEGACY_EMBARGO_POLICY_NAME = "Standard Embargo"; + + /** + * The only supported way of re-opening files whose Anonymous READ policy is already gone. + * ItemUpdate has to point the operator at it instead of inventing a public policy. + */ + private static final String BULK_ACCESS_CONTROL_HINT = "bulk-access-control"; + + /** + * Sentinel for {@link #deletePolicies(Bitstream, int)} meaning "every action", picked so it can never + * collide with a real value of {@link Constants#actionText}. + */ + private static final int ALL_ACTIONS = -1; + + private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); + private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); + private final ResourcePolicyService resourcePolicyService = + AuthorizeServiceFactory.getInstance().getResourcePolicyService(); + private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); + private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); + private final MetadataSchemaService metadataSchemaService = + ContentServiceFactory.getInstance().getMetadataSchemaService(); + private final MetadataFieldService metadataFieldService = + ContentServiceFactory.getInstance().getMetadataFieldService(); + + private Collection collection; + private Group anonymousGroup; + private Path tempDir; + private String previousHandlePrefix; + + @Before + @Override + public void setUp() throws Exception { + super.setUp(); + context.turnOffAuthorisationSystem(); + + parentCommunity = CommunityBuilder.createCommunity(context) + .withName("Parent Community") + .build(); + collection = CollectionBuilder.createCollection(context, parentCommunity) + .withName("Collection") + .build(); + + // neither field exists in the test metadata registry, AddMetadataAction needs both + ensureMetadataFieldExists("rights", "access"); + ensureMetadataFieldExists("date", "embargoend"); + + anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); + // ItemArchive resolves items through this mutable static, it is restored in destroy() + previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; + ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); + + context.restoreAuthSystemState(); + + tempDir = Files.createTempDirectory("embargoSafetyIT"); + } + + @After + @Override + public void destroy() throws Exception { + ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; + if (tempDir != null) { + PathUtils.deleteDirectory(tempDir); + } + super.destroy(); + } + + /** + * Specification row 9. A withdrawn item is hidden on purpose. Withdrawal converts every READ policy + * into WITHDRAWN_READ, so an embargo sync that "restores" access would silently undo a takedown. + */ + @Test + public void withdrawnItemIsNeverRepublished() throws Exception { + Item item = createItem("Withdrawn thesis"); + Bitstream bitstream = createEmbargoedBitstream(item, "withdrawn.pdf"); + + context.turnOffAuthorisationSystem(); + itemService.withdraw(context, item); + context.restoreAuthSystemState(); + + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + assertTrue("fixture precondition: the item must be withdrawn", item.isWithdrawn()); + assertFalse("fixture precondition: withdrawal must leave WITHDRAWN_READ policies behind" + + describe(bitstream), + policiesForAction(bitstream, Constants.WITHDRAWN_READ).isEmpty()); + assertTrue("fixture precondition: a withdrawn bitstream must carry no READ policy" + describe(bitstream), + policiesForAction(bitstream, Constants.READ).isEmpty()); + assertFalse("fixture precondition: a withdrawn file must not be publicly readable" + describe(bitstream), + anonymousCanRead(bitstream)); + + Set idsBefore = policyIds(bitstream); + List policiesBefore = policyFingerprints(bitstream); + + runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + assertTrue("An expired embargo on a WITHDRAWN item created an action=READ policy." + + " Withdrawal must never be undone by itemupdate, only WITHDRAWN_READ may remain." + + describe(bitstream), + policiesForAction(bitstream, Constants.READ).isEmpty()); + assertUntouched("withdrawn item", idsBefore, policiesBefore, bitstream); + assertFalse("A withdrawn file became publicly readable after an expired embargo was synchronised." + + describe(bitstream), + anonymousCanRead(bitstream)); + + // Row 9 says "any end date". The past-date run above only ever reaches the early return, so on its own + // it proves nothing about withdrawal; the future-date branch is the one that creates policies. + runItemUpdate(item, dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); + + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + assertTrue("A FUTURE dc.date.embargoend on a WITHDRAWN item created an action=READ policy. A withdrawn" + + " item must never gain one - only WITHDRAWN_READ may remain - otherwise the takedown" + + " undoes itself the moment the embargo lapses." + describe(bitstream), + policiesForAction(bitstream, Constants.READ).isEmpty()); + assertUntouched("withdrawn item with a future embargo end date", idsBefore, policiesBefore, bitstream); + assertFalse("A withdrawn file became publicly readable after a future embargo end date was synchronised." + + describe(bitstream), + anonymousCanRead(bitstream)); + } + + /** + * Specification row 4. {@code restrictedAccess} means the files stay closed no matter what the embargo + * end date says - a stale end date is not permission to publish. + */ + @Test + public void restrictedAccessWithStaleEmbargoEndIsUntouched() throws Exception { + assertEmbargoSyncIsANoOp("dc.rights.access=restrictedAccess with a past embargo end", + Collections.singletonList("restrictedAccess"), pastDate()); + } + + /** + * Specification row 4. {@code metadataOnlyAccess} means the bitstreams are never disclosed. + */ + @Test + public void metadataOnlyAccessIsUntouched() throws Exception { + assertEmbargoSyncIsANoOp("dc.rights.access=metadataOnlyAccess with a past embargo end", + Collections.singletonList("metadataOnlyAccess"), pastDate()); + } + + /** + * Specification row 4. An access right the tool does not understand is not an invitation to guess; + * an unknown value means "hands off", never "open". + */ + @Test + public void unknownAccessRightValueIsUntouched() throws Exception { + assertEmbargoSyncIsANoOp("dc.rights.access=someAccessRightWeDoNotKnow with a past embargo end", + Collections.singletonList("someAccessRightWeDoNotKnow"), pastDate()); + } + + /** + * Specification row 4. A single value outside the allowlist blocks the whole item, even when another + * value of the same field says {@code openAccess}. Contradictory metadata is never resolved in favour + * of disclosure. + */ + @Test + public void mixedAccessRightsWithOneDisallowedIsUntouched() throws Exception { + assertEmbargoSyncIsANoOp("dc.rights.access=openAccess + restrictedAccess with a past embargo end", + Arrays.asList("openAccess", "restrictedAccess"), pastDate()); + } + + /** + * Specification row 11. When READ is granted to a named group only there is no Anonymous policy to + * mutate. Creating one would hand the public a file that was deliberately limited to that group. + */ + @Test + public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { + Item item = createItem("Group restricted thesis"); + Bitstream bitstream = createOriginalBitstream(item, "group-restricted.pdf"); + + context.turnOffAuthorisationSystem(); + Group reviewers = GroupBuilder.createGroup(context) + .withName("Thesis reviewers " + System.nanoTime()) + .build(); + deletePolicies(bitstream, Constants.READ); + ResourcePolicyBuilder.createResourcePolicy(context, null, reviewers) + .withAction(Constants.READ) + .withDspaceObject(bitstream) + .withName("Reviewers only") + .build(); + context.restoreAuthSystemState(); + + bitstream = context.reloadEntity(bitstream); + + assertTrue("fixture precondition: no Anonymous READ policy may be left on the bitstream" + + describe(bitstream), anonymousReadPolicies(bitstream).isEmpty()); + assertFalse("fixture precondition: a group restricted file must not be publicly readable" + + describe(bitstream), anonymousCanRead(bitstream)); + + Set idsBefore = policyIds(bitstream); + List policiesBefore = policyFingerprints(bitstream); + + String pastRunOutput = + runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + + bitstream = context.reloadEntity(bitstream); + + assertTrue("An expired embargo created an Anonymous READ policy on a bitstream that only ever granted" + + " READ to a named group. Access may be re-dated, never widened." + + describe(bitstream), + anonymousReadPolicies(bitstream).isEmpty()); + assertUntouched("bitstream readable by a named group only", idsBefore, policiesBefore, bitstream); + assertFalse("A group restricted file became publicly readable after an expired embargo was synchronised." + + describe(bitstream), anonymousCanRead(bitstream)); + + // Row 11 says "any end date". A past date only reaches the early return; the future-date branch is the + // one that creates policies, so it is where a group-restricted file can silently gain an Anonymous one. + String futureRunOutput = runItemUpdate(item, + dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); + + bitstream = context.reloadEntity(bitstream); + + assertTrue("A FUTURE dc.date.embargoend created an Anonymous READ policy on a bitstream that only ever" + + " granted READ to a named group. The embargo would lapse into public access nobody" + + " ever granted." + describe(bitstream), + anonymousReadPolicies(bitstream).isEmpty()); + assertUntouched("bitstream readable by a named group only, future embargo end date", + idsBefore, policiesBefore, bitstream); + assertFalse("A group restricted file became publicly readable after a future embargo end date was" + + " synchronised." + describe(bitstream), anonymousCanRead(bitstream)); + + // Reporting is asserted last, after both legs have proved that no policy was invented: a missing log + // line must never be the reason a reader stops reading before the policy damage is on screen. + assertTrue("There is no Anonymous READ policy to re-date here, so ItemUpdate has to report the bitstream" + + " it could not synchronise and name '" + BULK_ACCESS_CONTROL_HINT + "' as the supported" + + " way to change access. Console output of the expired-embargo run was:" + + System.lineSeparator() + pastRunOutput, + pastRunOutput.contains(BULK_ACCESS_CONTROL_HINT)); + assertTrue("ItemUpdate stayed silent about a bitstream it could not synchronise. Console output of the" + + " future-embargo run was:" + System.lineSeparator() + futureRunOutput, + futureRunOutput.contains(BULK_ACCESS_CONTROL_HINT)); + } + + /** + * Specification row 11. This is the exact state of the customer record damaged by the shipped code: + * zero resource policies, HTTP 401 on every download. The repair must refuse to guess what those + * policies used to be - it may only report the damage and name the tool that can undo it. + */ + @Test + public void alreadyBrokenBitstreamWithZeroPoliciesStaysZero() throws Exception { + Item item = createItem("Already broken thesis"); + Bitstream bitstream = createOriginalBitstream(item, "already-broken.pdf"); + + context.turnOffAuthorisationSystem(); + deletePolicies(bitstream, ALL_ACTIONS); + context.restoreAuthSystemState(); + + bitstream = context.reloadEntity(bitstream); + + assertTrue("fixture precondition: the bitstream must carry no resource policy at all" + + describe(bitstream), allPolicies(bitstream).isEmpty()); + assertFalse("fixture precondition: a bitstream without policies must not be readable" + + describe(bitstream), anonymousCanRead(bitstream)); + + String consoleOutput = + runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + + bitstream = context.reloadEntity(bitstream); + + assertTrue("ItemUpdate invented resource policies for a bitstream that had none. A bitstream with zero" + + " policies carries no evidence of who was allowed to read it, so re-creating a policy is" + + " a guess - and the only wrong guess leaks the file." + + describe(bitstream), + allPolicies(bitstream).isEmpty()); + assertFalse("A bitstream with zero policies became publicly readable." + describe(bitstream), + anonymousCanRead(bitstream)); + assertTrue("ItemUpdate stayed silent about a bitstream it could not synchronise. It has to report the" + + " failure and name '" + BULK_ACCESS_CONTROL_HINT + "' as the supported way to restore" + + " access. Console output was:\n" + consoleOutput, + consoleOutput.contains(BULK_ACCESS_CONTROL_HINT)); + } + + /** + * Specification row 7. A blank end date is a broken export, not an instruction. Validation has to run + * before anything is mutated, so the existing policies survive untouched. + */ + @Test + public void blankEmbargoEndLeavesPoliciesUntouched() throws Exception { + Item item = assertEmbargoSyncIsANoOp("blank dc.date.embargoend", + Collections.singletonList("openAccess"), ""); + + List storedEndDates = itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY); + assertEquals("fixture precondition: the blank dc.date.embargoend has to be stored as an empty value." + + " If it were dropped the item would look like 'the operator removed the embargo' and" + + " this test would silently stop covering the blank value case.", + 1, storedEndDates.size()); + assertTrue("fixture precondition: the stored dc.date.embargoend has to be blank, not a real date", + storedEndDates.get(0).getValue().trim().isEmpty()); + } + + /** + * Specification row 8. Parsing has to be strict. {@code DCDate} silently rolls 30 February over into + * 2 March, which would turn an unparseable value into a real - possibly future - embargo date. + */ + @Test + public void invalidEmbargoEndLeavesPoliciesUntouched() throws Exception { + // dynamic year, but 30 February and month 13 do not exist in any year + int year = LocalDate.now().plusYears(1).getYear(); + List invalidEndDates = Arrays.asList("abc", year + "-02-30", year + "-13-01"); + + for (String invalidEndDate : invalidEndDates) { + assertEmbargoSyncIsANoOp("unparseable dc.date.embargoend=" + invalidEndDate, + Collections.singletonList("openAccess"), invalidEndDate); + } + } + + /** + * Specification row 6. {@code embargoedAccess} without an end date is self-contradictory metadata. + * The tool has to refuse it rather than pick one half of the contradiction. + */ + @Test + public void embargoedAccessWithoutEndDateLeavesPoliciesUntouched() throws Exception { + assertEmbargoSyncIsANoOp("dc.rights.access=embargoedAccess without dc.date.embargoend", + Collections.singletonList("embargoedAccess"), null); + } + + /** + * Specification row 10. An item outside the archive (workspace or workflow) is not published yet; its + * bitstream policies are the submission's business, not itemupdate's. + */ + @Test + public void notArchivedItemIsUntouched() throws Exception { + Item item = createItem("Not archived thesis"); + Bitstream bitstream = createEmbargoedBitstream(item, "not-archived.pdf"); + + // take the item back out of the archive; the handle stays, so the SAF update still resolves it + context.turnOffAuthorisationSystem(); + item.setArchived(false); + itemService.update(context, item); + context.restoreAuthSystemState(); + + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + assertFalse("fixture precondition: the item must not be archived", item.isArchived()); + assertFalse("fixture precondition: the embargoed file must not be publicly readable" + + describe(bitstream), anonymousCanRead(bitstream)); + + Set idsBefore = policyIds(bitstream); + List policiesBefore = policyFingerprints(bitstream); + + runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + + bitstream = context.reloadEntity(bitstream); + + assertUntouched("item outside the archive", idsBefore, policiesBefore, bitstream); + assertFalse("A file of an item outside the archive became publicly readable." + describe(bitstream), + anonymousCanRead(bitstream)); + } + + /** + * Runs one "ItemUpdate has to keep its hands off" scenario end to end and returns the reloaded item. + * + *

The fixture is the state the customer repository is in after an earlier itemupdate run with a + * future end date: exactly one Anonymous READ policy, dated, currently blocking access. Any repair + * that publishes, deletes or re-creates that policy is caught here.

+ */ + private Item assertEmbargoSyncIsANoOp(String scenario, List accessRights, String embargoEndDate) + throws Exception { + Item item = createItem("Safety scenario: " + scenario); + Bitstream bitstream = createEmbargoedBitstream(item, "thesis.pdf"); + + assertFalse("fixture precondition [" + scenario + "]: the embargoed file must not be publicly readable" + + describe(bitstream), anonymousCanRead(bitstream)); + + Set idsBefore = policyIds(bitstream); + List policiesBefore = policyFingerprints(bitstream); + + runItemUpdate(item, dublinCore(item, accessRights, embargoEndDate)); + + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + assertUntouched(scenario, idsBefore, policiesBefore, bitstream); + assertFalse("[" + scenario + "] the embargoed file became publicly readable, which is a data leak" + + describe(bitstream), anonymousCanRead(bitstream)); + + return item; + } + + /** + * Compares policy identity first (ids), then policy content (fingerprints). The id comparison catches + * delete-and-recreate, the fingerprint comparison catches in-place mutation of a surviving policy. + */ + private void assertUntouched(String scenario, Set idsBefore, List policiesBefore, + Bitstream bitstream) throws SQLException { + assertEquals("[" + scenario + "] the set of resource policy ids changed: policies were deleted and/or" + + " re-created although nothing at all should have happened." + describe(bitstream), + idsBefore, policyIds(bitstream)); + assertEquals("[" + scenario + "] a surviving resource policy was modified in place although nothing at" + + " all should have happened." + describe(bitstream), + policiesBefore, policyFingerprints(bitstream)); + } + + /** + * Equivalent of {@code dsrun ... ItemUpdate -s -d dc.rights.access -d dc.date.embargoend + * -a dc.rights.access -a dc.date.embargoend}, i.e. an update whose target fields contain an embargo + * field, which is what makes {@code processArchive} call {@code syncEmbargoPolicies}. + * + * @return everything ItemUpdate printed on the operator console during the run + */ + private String runItemUpdate(Item item, String dublinCoreContent) throws Exception { + Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); + // without this marker processArchive writes an undo archive next to the source directory + Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); + + Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); + Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); + + ItemUpdate itemUpdate = new ItemUpdate(); + DeleteMetadataAction deleteAction = + (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); + deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + AddMetadataAction addAction = + (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); + addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + // ItemUpdate reports to System.out only, so the operator console has to be captured to assert on it + ByteArrayOutputStream consoleBuffer = new ByteArrayOutputStream(); + PrintStream originalOut = System.out; + PrintStream originalErr = System.err; + PrintStream captureStream = new PrintStream(consoleBuffer, true, StandardCharsets.UTF_8); + + context.turnOffAuthorisationSystem(); + System.setOut(captureStream); + System.setErr(captureStream); + try { + itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); + } finally { + captureStream.flush(); + System.setOut(originalOut); + System.setErr(originalErr); + context.restoreAuthSystemState(); + } + + context.uncacheEntity(item); + + String consoleOutput = consoleBuffer.toString(StandardCharsets.UTF_8); + // replay it so the failsafe -output.txt still holds the full ItemUpdate log + System.out.println(consoleOutput); + return consoleOutput; + } + + /** + * Builds a {@code dublin_core.xml} carrying zero or more {@code dc.rights.access} values. + * + * @param embargoEndDate {@code null} omits {@code dc.date.embargoend} entirely (the operator lifted the + * embargo), the empty string writes a blank value (an empty XML element is dropped + * by the parser, so a single space is written instead) + */ + private String dublinCore(Item item, List accessRights, String embargoEndDate) { + StringBuilder sb = new StringBuilder(); + sb.append("\n") + .append("\n") + .append(" ") + .append(ItemUpdate.HANDLE_PREFIX) + .append(item.getHandle()) + .append("\n"); + + for (String accessRight : accessRights) { + sb.append(" ") + .append(accessRight) + .append("\n"); + } + + if (embargoEndDate != null) { + sb.append(" ") + .append(embargoEndDate.isEmpty() ? " " : embargoEndDate) + .append("\n"); + } + + sb.append(""); + return sb.toString(); + } + + private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { + MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); + MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); + if (existingField == null) { + MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); + } + } + + private Item createItem(String title) throws Exception { + context.turnOffAuthorisationSystem(); + Item item = ItemBuilder.createItem(context, collection) + .withTitle(title) + .build(); + context.restoreAuthSystemState(); + return item; + } + + /** + * A bitstream in the ORIGINAL bundle. The collection grants DEFAULT_BITSTREAM_READ to Anonymous, so + * BundleServiceImpl gives the new bitstream exactly one policy: Anonymous / READ / TYPE_INHERITED / + * rpName=null / startDate=null - byte for byte the state of a freshly imported SAF item. + */ + private Bitstream createOriginalBitstream(Item item, String name) throws Exception { + context.turnOffAuthorisationSystem(); + Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, + new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8))) + .withName(name) + .withMimeType("text/plain") + .build(); + context.restoreAuthSystemState(); + return bitstream; + } + + /** + * The state the customer repository is left in by an itemupdate run with a future end date: the + * immediate Anonymous READ policy is gone and a single dated legacy "Standard Embargo" policy blocks + * access. That policy is all that stands between the public and the file. + */ + private Bitstream createEmbargoedBitstream(Item item, String name) throws Exception { + Bitstream bitstream = createOriginalBitstream(item, name); + + context.turnOffAuthorisationSystem(); + deletePolicies(bitstream, Constants.READ); + ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) + .withAction(Constants.READ) + .withDspaceObject(bitstream) + .withName(LEGACY_EMBARGO_POLICY_NAME) + .withStartDate(startOfDayUtc(LocalDate.now().plusYears(1))) + .build(); + context.restoreAuthSystemState(); + + return context.reloadEntity(bitstream); + } + + /** + * Deletes policies one by one through {@code ResourcePolicyService#delete}, the same call the production + * code uses, so the Hibernate session stays consistent (the bulk removal helpers issue an HQL delete and + * leave the in-memory collection stale). + * + * @param actionId action to delete, or {@link #ALL_ACTIONS} for every policy regardless of action + */ + private void deletePolicies(Bitstream bitstream, int actionId) throws Exception { + List doomed = actionId == ALL_ACTIONS + ? allPolicies(bitstream) + : policiesForAction(bitstream, actionId); + for (ResourcePolicy policy : doomed) { + resourcePolicyService.delete(context, policy); + } + } + + private List allPolicies(Bitstream bitstream) throws SQLException { + return new ArrayList<>(resourcePolicyService.find(context, bitstream)); + } + + private List policiesForAction(Bitstream bitstream, int actionId) throws SQLException { + return new ArrayList<>(resourcePolicyService.find(context, bitstream, actionId)); + } + + private List anonymousReadPolicies(Bitstream bitstream) throws SQLException { + return policiesForAction(bitstream, Constants.READ).stream() + .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) + .collect(Collectors.toList()); + } + + private Set policyIds(Bitstream bitstream) throws SQLException { + Set ids = new TreeSet<>(); + for (ResourcePolicy policy : allPolicies(bitstream)) { + ids.add(policy.getID()); + } + return ids; + } + + private List policyFingerprints(Bitstream bitstream) throws SQLException { + List fingerprints = new ArrayList<>(); + for (ResourcePolicy policy : allPolicies(bitstream)) { + fingerprints.add(fingerprint(policy)); + } + Collections.sort(fingerprints); + return fingerprints; + } + + private String fingerprint(ResourcePolicy policy) { + return String.format("id=%s action=%s group=%s eperson=%s rpType=%s rpName=%s start=%s end=%s", + policy.getID(), + Constants.actionText[policy.getAction()], + policy.getGroup() == null ? "" : policy.getGroup().getName(), + policy.getEPerson() == null ? "" : policy.getEPerson().getEmail(), + policy.getRpType(), + policy.getRpName(), + day(policy.getStartDate()), + day(policy.getEndDate())); + } + + /** + * Renders the current policies of the bitstream for failure messages. Whoever reads a red build has to + * see which policy moved without re-running anything. + */ + private String describe(Bitstream bitstream) throws SQLException { + StringBuilder sb = new StringBuilder(System.lineSeparator()) + .append(" bitstream=").append(bitstream.getID()).append(System.lineSeparator()); + List fingerprints = policyFingerprints(bitstream); + if (fingerprints.isEmpty()) { + sb.append(" ").append(System.lineSeparator()); + } + for (String fingerprint : fingerprints) { + sb.append(" ").append(fingerprint).append(System.lineSeparator()); + } + return sb.toString(); + } + + /** + * Answers the only question that matters: may a visitor who is not logged in download the file? + * + *

The authorisation state is a stack, not a flag: the builders and processArchive push and pop + * around themselves, so the depth is not guaranteed to be zero here. It has to be drained, otherwise + * {@code authorize()} short circuits on {@code ignoreAuthorization()} and every read looks allowed.

+ */ + private boolean anonymousCanRead(Bitstream bitstream) throws SQLException { + EPerson savedUser = context.getCurrentUser(); + int popped = 0; + while (context.ignoreAuthorization()) { + context.restoreAuthSystemState(); + popped++; + } + context.setCurrentUser(null); + try { + return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); + } finally { + context.setCurrentUser(savedUser); + for (int i = 0; i < popped; i++) { + context.turnOffAuthorisationSystem(); + } + } + } + + private String pastDate() { + return LocalDate.now().minusMonths(1).toString(); + } + + /** + * A future end date is the dangerous half of every "hands off" rule: the past-date branch of the shipped + * code returns early and therefore looks harmless, while the future-date branch is the one that actually + * writes resource policies. + */ + private String futureDate() { + return LocalDate.now().plusYears(1).toString(); + } + + private Date startOfDayUtc(LocalDate day) { + return Date.from(day.atStartOfDay(ZoneOffset.UTC).toInstant()); + } + + /** + * Compares dates at calendar day granularity. The harness forces TZ Europe/Dublin while resource policy + * start dates come back from the database as {@code java.sql.Date}; comparing instants across that + * boundary would be flaky, comparing days is not. + */ + private String day(Date date) { + if (date == null) { + return ""; + } + if (date instanceof java.sql.Date) { + return ((java.sql.Date) date).toLocalDate().toString(); + } + return date.toInstant().atZone(ZoneId.systemDefault()).toLocalDate().toString(); + } +} diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java index 929cb0f06bd6..0eebc7dde7b0 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java @@ -10,6 +10,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -18,15 +19,18 @@ import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; +import java.sql.SQLException; import java.time.LocalDate; import java.time.ZoneId; import java.util.Date; import java.util.List; +import java.util.stream.Collectors; import org.apache.commons.io.file.PathUtils; import org.dspace.AbstractIntegrationTestWithDatabase; import org.dspace.authorize.ResourcePolicy; import org.dspace.authorize.factory.AuthorizeServiceFactory; +import org.dspace.authorize.service.AuthorizeService; import org.dspace.authorize.service.ResourcePolicyService; import org.dspace.builder.BitstreamBuilder; import org.dspace.builder.CollectionBuilder; @@ -45,6 +49,7 @@ import org.dspace.content.service.MetadataFieldService; import org.dspace.content.service.MetadataSchemaService; import org.dspace.core.Constants; +import org.dspace.eperson.EPerson; import org.dspace.eperson.Group; import org.dspace.eperson.factory.EPersonServiceFactory; import org.dspace.eperson.service.GroupService; @@ -59,13 +64,18 @@ */ public class ItemUpdateIT extends AbstractIntegrationTestWithDatabase { + /** rpNames written by the shipped implementation; the fix has to adopt and normalise them. */ private static final String STANDARD_EMBARGO = "Standard Embargo"; private static final String SPECIAL_CASE_EMBARGO = "Special Case Embargo"; + /** The single normalised rpName, matching the access condition name in access-conditions.xml. */ + private static final String EMBARGO_POLICY_NAME = "embargo"; + private ItemService itemService = ContentServiceFactory.getInstance().getItemService(); private HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); private ResourcePolicyService resourcePolicyService = AuthorizeServiceFactory.getInstance().getResourcePolicyService(); + private AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); private GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); private MetadataSchemaService metadataSchemaService = ContentServiceFactory.getInstance().getMetadataSchemaService(); @@ -174,7 +184,7 @@ public void itemArchiveCreateFailsForAmbiguousIdentifierUri() throws Exception { } @Test - public void syncEmbargoPoliciesCreatesStandardEmbargoAndRemovesImmediateAnonymousRead() throws Exception { + public void syncEmbargoPoliciesDatesTheAnonymousReadPolicyAndBlocksAccess() throws Exception { String futureDate = LocalDate.now().plusDays(14).toString(); Item item = createItem("Standard Embargo Item", "rights", "access", "embargoedAccess", @@ -186,24 +196,21 @@ public void syncEmbargoPoliciesCreatesStandardEmbargoAndRemovesImmediateAnonymou ItemUpdate itemUpdate = new ItemUpdate(); itemUpdate.syncEmbargoPolicies(context, item); - List readPolicies = resourcePolicyService.find(context, bitstream, Constants.READ); - - boolean hasImmediateAnonymousRead = readPolicies.stream() - .anyMatch(policy -> isAnonymousPolicy(policy) && policy.getStartDate() == null); - assertFalse(hasImmediateAnonymousRead); - - ResourcePolicy embargoPolicy = readPolicies.stream() - .filter(policy -> isAnonymousPolicy(policy) - && STANDARD_EMBARGO.equals(policy.getRpName()) - && policy.getStartDate() != null) - .findFirst() - .orElse(null); - - assertNotNull(embargoPolicy); + // Exactly one Anonymous READ policy has to be left behind. A second, undated one would silently + // defeat the embargo, so counting is part of the assertion, not a detail. + List anonymousRead = anonymousReadPolicies(bitstream); + assertEquals(1, anonymousRead.size()); + + ResourcePolicy embargoPolicy = anonymousRead.get(0); + assertEquals(EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); + assertEquals(ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); + assertNotNull(embargoPolicy.getStartDate()); + assertEquals(LocalDate.parse(futureDate).plusDays(1), toLocalDate(embargoPolicy.getStartDate())); + assertFalse(anonymousCanRead(bitstream)); } @Test - public void syncEmbargoPoliciesCreatesSpecialCasePolicyWithoutAccessRightMetadata() throws Exception { + public void syncEmbargoPoliciesAppliesEmbargoWithoutAccessRightMetadata() throws Exception { String futureDate = LocalDate.now().plusDays(21).toString(); Item item = createItem("Special Case Embargo Item", "date", "embargoend", futureDate); Bitstream bitstream = createBitstream(item, "special.txt"); @@ -211,35 +218,48 @@ public void syncEmbargoPoliciesCreatesSpecialCasePolicyWithoutAccessRightMetadat ItemUpdate itemUpdate = new ItemUpdate(); itemUpdate.syncEmbargoPolicies(context, item); - List readPolicies = resourcePolicyService.find(context, bitstream, Constants.READ); - - ResourcePolicy embargoPolicy = readPolicies.stream() - .filter(policy -> isAnonymousPolicy(policy) - && SPECIAL_CASE_EMBARGO.equals(policy.getRpName()) - && policy.getStartDate() != null) - .findFirst() - .orElse(null); - - assertNotNull(embargoPolicy); + List anonymousRead = anonymousReadPolicies(bitstream); + assertEquals(1, anonymousRead.size()); + + // The distinction between "standard" and "special case" embargo only ever existed in the rpName. + // Both are now written as the single access condition name from access-conditions.xml, which also + // keeps the value inside the 30 character resourcepolicy.rpname column. + ResourcePolicy embargoPolicy = anonymousRead.get(0); + assertEquals(EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); + assertEquals(ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); + assertNotNull(embargoPolicy.getStartDate()); + assertEquals(LocalDate.parse(futureDate).plusDays(1), toLocalDate(embargoPolicy.getStartDate())); + assertFalse(anonymousCanRead(bitstream)); } + /** + * A blank {@code dc.date.embargoend} is a broken export, not an instruction to change anything. + * + *

This test used to assert only {@code assertFalse(hasSafEmbargoPolicy)}, which an empty policy table + * satisfies just as well as a correct one - and an empty policy table is exactly the customer bug (HTTP 401 + * on every download). It now asserts what the operator actually cares about: not a single resource policy + * was touched.

+ */ @Test - public void syncEmbargoPoliciesClearsSafEmbargoPoliciesWhenEmbargoDateInvalid() throws Exception { + public void syncEmbargoPoliciesLeavesPoliciesUntouchedWhenEmbargoDateInvalid() throws Exception { Item item = createItem("Invalid Date Item", "date", "embargoend", ""); Bitstream bitstream = createBitstream(item, "invalid.txt"); - createAnonymousReadPolicy(bitstream, new Date(System.currentTimeMillis() + 86_400_000L), STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, + new Date(System.currentTimeMillis() + 86_400_000L), STANDARD_EMBARGO); + + List idsBefore = policyIds(bitstream); + boolean readableBefore = anonymousCanRead(bitstream); ItemUpdate itemUpdate = new ItemUpdate(); itemUpdate.syncEmbargoPolicies(context, item); - List readPolicies = resourcePolicyService.find(context, bitstream, Constants.READ); - - boolean hasSafEmbargoPolicy = readPolicies.stream() - .anyMatch(policy -> isAnonymousPolicy(policy) - && (STANDARD_EMBARGO.equals(policy.getRpName()) - || SPECIAL_CASE_EMBARGO.equals(policy.getRpName()))); + assertEquals(idsBefore, policyIds(bitstream)); + assertEquals(readableBefore, anonymousCanRead(bitstream)); - assertFalse(hasSafEmbargoPolicy); + // The dated policy the run could not validate is still there, unchanged, under its legacy name. + ResourcePolicy reloadedLegacy = resourcePolicyService.find(context, legacyPolicy.getID()); + assertNotNull(reloadedLegacy); + assertEquals(STANDARD_EMBARGO, reloadedLegacy.getRpName()); } @Test @@ -254,7 +274,8 @@ public void processArchiveUpdatesEmbargoMetadataAndResyncsEmbargoPolicy() throws LocalDate oldPolicyDate = LocalDate.parse(oldEmbargoDate).plusDays(1); Date oldPolicyStart = Date.from(oldPolicyDate.atStartOfDay(ZoneId.systemDefault()).toInstant()); - createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + Integer legacyPolicyId = legacyPolicy.getID(); runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, "embargoedAccess", newEmbargoDate)); @@ -266,23 +287,28 @@ public void processArchiveUpdatesEmbargoMetadataAndResyncsEmbargoPolicy() throws assertEquals(newEmbargoDate, embargoDates.get(0).getValue()); LocalDate expectedPolicyDate = LocalDate.parse(newEmbargoDate).plusDays(1); - List readPolicies = resourcePolicyService.find(context, reloadedBitstream, Constants.READ); - - boolean hasOldEmbargoPolicy = readPolicies.stream().anyMatch(policy -> isAnonymousPolicy(policy) - && STANDARD_EMBARGO.equals(policy.getRpName()) - && policy.getStartDate() != null - && toLocalDate(policy.getStartDate()).equals(oldPolicyDate)); - assertFalse(hasOldEmbargoPolicy); - - boolean hasNewEmbargoPolicy = readPolicies.stream().anyMatch(policy -> isAnonymousPolicy(policy) - && STANDARD_EMBARGO.equals(policy.getRpName()) - && policy.getStartDate() != null - && toLocalDate(policy.getStartDate()).equals(expectedPolicyDate)); - assertTrue(hasNewEmbargoPolicy); + List anonymousRead = anonymousReadPolicies(reloadedBitstream); + assertEquals(1, anonymousRead.size()); + + // The pre-existing policy is re-dated in place instead of being deleted and re-created. Between a + // delete and a create the file has no policy at all, which is the state the customer report was about. + ResourcePolicy embargoPolicy = anonymousRead.get(0); + assertEquals(legacyPolicyId, embargoPolicy.getID()); + assertEquals(EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); + assertEquals(ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); + assertNotNull(embargoPolicy.getStartDate()); + assertFalse(toLocalDate(embargoPolicy.getStartDate()).equals(oldPolicyDate)); + assertEquals(expectedPolicyDate, toLocalDate(embargoPolicy.getStartDate())); + assertFalse(anonymousCanRead(reloadedBitstream)); } + /** + * Blanking {@code dc.date.embargoend} in the SAF archive is a broken export. Same reasoning as + * {@link #syncEmbargoPoliciesLeavesPoliciesUntouchedWhenEmbargoDateInvalid()}: the old + * {@code assertFalse(hasSafEmbargoPolicy)} was also satisfied by a bitstream stripped of every policy. + */ @Test - public void processArchiveUpdateWithBlankEmbargoDateClearsSafEmbargoPolicies() throws Exception { + public void processArchiveUpdateWithBlankEmbargoDateLeavesPoliciesUntouched() throws Exception { String oldEmbargoDate = LocalDate.now().plusDays(12).toString(); Item item = createItem("Blank Embargo Date Update", @@ -292,21 +318,30 @@ public void processArchiveUpdateWithBlankEmbargoDateClearsSafEmbargoPolicies() t Date oldPolicyStart = Date.from(LocalDate.parse(oldEmbargoDate).plusDays(1) .atStartOfDay(ZoneId.systemDefault()).toInstant()); - createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + + List idsBefore = policyIds(bitstream); + boolean readableBefore = anonymousCanRead(bitstream); runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, "embargoedAccess", "")); Bitstream reloadedBitstream = context.reloadEntity(bitstream); - List readPolicies = resourcePolicyService.find(context, reloadedBitstream, Constants.READ); - boolean hasSafEmbargoPolicy = readPolicies.stream().anyMatch(policy -> isAnonymousPolicy(policy) - && (STANDARD_EMBARGO.equals(policy.getRpName()) - || SPECIAL_CASE_EMBARGO.equals(policy.getRpName()))); - assertFalse(hasSafEmbargoPolicy); + assertEquals(idsBefore, policyIds(reloadedBitstream)); + assertEquals(readableBefore, anonymousCanRead(reloadedBitstream)); + + ResourcePolicy reloadedLegacy = resourcePolicyService.find(context, legacyPolicy.getID()); + assertNotNull(reloadedLegacy); + assertEquals(STANDARD_EMBARGO, reloadedLegacy.getRpName()); } + /** + * Removing {@code dc.date.embargoend} is the only way an operator lifts an embargo, so the files have to + * become readable. The old assertion looked for the absence of a policy named "Standard Embargo", which + * a bitstream with zero policies - i.e. an unreadable one - passes just as well. + */ @Test - public void processArchiveUpdateRemovingEmbargoMetadataClearsPoliciesAndMetadata() throws Exception { + public void processArchiveUpdateRemovingEmbargoMetadataLiftsEmbargo() throws Exception { String oldEmbargoDate = LocalDate.now().plusDays(10).toString(); Item item = createItem("Remove Embargo Metadata Update", @@ -316,7 +351,8 @@ public void processArchiveUpdateRemovingEmbargoMetadataClearsPoliciesAndMetadata Date oldPolicyStart = Date.from(LocalDate.parse(oldEmbargoDate).plusDays(1) .atStartOfDay(ZoneId.systemDefault()).toInstant()); - createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + Integer legacyPolicyId = legacyPolicy.getID(); runEmbargoMetadataUpdate(item, dublinCore(item)); @@ -328,15 +364,20 @@ public void processArchiveUpdateRemovingEmbargoMetadataClearsPoliciesAndMetadata assertTrue(rightsAccess.isEmpty()); assertTrue(embargoDates.isEmpty()); - List readPolicies = resourcePolicyService.find(context, reloadedBitstream, Constants.READ); - boolean hasSafEmbargoPolicy = readPolicies.stream().anyMatch(policy -> isAnonymousPolicy(policy) - && (STANDARD_EMBARGO.equals(policy.getRpName()) - || SPECIAL_CASE_EMBARGO.equals(policy.getRpName()))); - assertFalse(hasSafEmbargoPolicy); + List anonymousRead = anonymousReadPolicies(reloadedBitstream); + assertEquals(1, anonymousRead.size()); + + ResourcePolicy liftedPolicy = anonymousRead.get(0); + assertEquals(legacyPolicyId, liftedPolicy.getID()); + assertNull(liftedPolicy.getStartDate()); + assertFalse(STANDARD_EMBARGO.equals(liftedPolicy.getRpName()) + || SPECIAL_CASE_EMBARGO.equals(liftedPolicy.getRpName())); + assertEquals(EMBARGO_POLICY_NAME, liftedPolicy.getRpName()); + assertTrue(anonymousCanRead(reloadedBitstream)); } @Test - public void processArchiveUpdateWithEmbargoDateAndNoRightsCreatesSpecialCasePolicy() throws Exception { + public void processArchiveUpdateWithEmbargoDateAndNoRightsAppliesEmbargo() throws Exception { String oldEmbargoDate = LocalDate.now().plusDays(8).toString(); String newEmbargoDate = LocalDate.now().plusDays(25).toString(); @@ -347,7 +388,8 @@ public void processArchiveUpdateWithEmbargoDateAndNoRightsCreatesSpecialCasePoli Date oldPolicyStart = Date.from(LocalDate.parse(oldEmbargoDate).plusDays(1) .atStartOfDay(ZoneId.systemDefault()).toInstant()); - createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + Integer legacyPolicyId = legacyPolicy.getID(); runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, null, newEmbargoDate)); @@ -358,17 +400,16 @@ public void processArchiveUpdateWithEmbargoDateAndNoRightsCreatesSpecialCasePoli assertTrue(rightsAccess.isEmpty()); LocalDate expectedPolicyDate = LocalDate.parse(newEmbargoDate).plusDays(1); - List readPolicies = resourcePolicyService.find(context, reloadedBitstream, Constants.READ); - - boolean hasSpecialCasePolicy = readPolicies.stream().anyMatch(policy -> isAnonymousPolicy(policy) - && SPECIAL_CASE_EMBARGO.equals(policy.getRpName()) - && policy.getStartDate() != null - && toLocalDate(policy.getStartDate()).equals(expectedPolicyDate)); - assertTrue(hasSpecialCasePolicy); - - boolean hasStandardPolicy = readPolicies.stream().anyMatch(policy -> isAnonymousPolicy(policy) - && STANDARD_EMBARGO.equals(policy.getRpName())); - assertFalse(hasStandardPolicy); + List anonymousRead = anonymousReadPolicies(reloadedBitstream); + assertEquals(1, anonymousRead.size()); + + ResourcePolicy embargoPolicy = anonymousRead.get(0); + assertEquals(legacyPolicyId, embargoPolicy.getID()); + assertEquals(EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); + assertEquals(ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); + assertNotNull(embargoPolicy.getStartDate()); + assertEquals(expectedPolicyDate, toLocalDate(embargoPolicy.getStartDate())); + assertFalse(anonymousCanRead(reloadedBitstream)); } private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { @@ -405,7 +446,8 @@ private Bitstream createBitstream(Item item, String name) throws Exception { return bitstream; } - private void createAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) throws Exception { + private ResourcePolicy createAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) + throws Exception { context.turnOffAuthorisationSystem(); ResourcePolicyBuilder builder = ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) .withAction(Constants.READ) @@ -415,14 +457,54 @@ private void createAnonymousReadPolicy(Bitstream bitstream, Date startDate, Stri if (startDate != null) { builder.withStartDate(startDate); } - builder.build(); + ResourcePolicy policy = builder.build(); context.restoreAuthSystemState(); + return policy; } private boolean isAnonymousPolicy(ResourcePolicy policy) { return policy.getGroup() != null && policy.getGroup().equals(anonymousGroup); } + private List anonymousReadPolicies(Bitstream bitstream) throws Exception { + return resourcePolicyService.find(context, bitstream, Constants.READ).stream() + .filter(this::isAnonymousPolicy) + .collect(Collectors.toList()); + } + + /** + * Identity of every resource policy on the bitstream. A policy deleted and immediately re-created keeps + * the count but loses its id, so ids are what "untouched" has to be measured with. + */ + private List policyIds(Bitstream bitstream) throws Exception { + return authorizeService.getPolicies(context, bitstream).stream() + .map(ResourcePolicy::getID) + .sorted() + .collect(Collectors.toList()); + } + + /** + * What an anonymous visitor of the REST API gets: the authorisation system asked as nobody, with the + * test's own turnOffAuthorisationSystem calls temporarily unwound. + */ + private boolean anonymousCanRead(Bitstream bitstream) throws SQLException { + EPerson savedUser = context.getCurrentUser(); + int popped = 0; + while (context.ignoreAuthorization()) { + context.restoreAuthSystemState(); + popped++; + } + context.setCurrentUser(null); + try { + return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); + } finally { + context.setCurrentUser(savedUser); + for (int i = 0; i < popped; i++) { + context.turnOffAuthorisationSystem(); + } + } + } + private Path createSafItemDirectory(String dublinCoreContent) throws IOException { Path safDir = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); Path itemDir = Files.createDirectory(safDir.resolve("item_000")); From f26f648c6c6b3c4d3af42bd58f3a89a0aa4845aa Mon Sep 17 00:00:00 2001 From: milanmajchrak Date: Tue, 18 Aug 2026 14:52:01 +0200 Subject: [PATCH 02/10] VSB-TUO/Fix review findings: keep workflow embargo, never lift foreign embargoes The review of f0c93aba found two ways this branch could itself publish an embargoed file. Both are closed here. Leak 1 - approving a workflow import published the files -------------------------------------------------------- f0c93aba moved `processEmbargoMetadata()` out of the common path of `ItemImportServiceImpl.addItem()` into the `!useWorkflow` branch, so `dspace import -a -w` created no embargo policy at all. On approval, `XmlWorkflowServiceImpl.archive` -> `installItem` -> `inheritCollectionDefaultPolicies(.., false)` -> `adjustBundleBitstreamPolicies` -> `removeAllPoliciesAndAddDefault` -> `addDefaultPoliciesNotInPlace` finds a bitstream with no Anonymous READ policy and clones the collection's *undated* `DEFAULT_BITSTREAM_READ` onto it. An item carrying `dc.rights.access=embargoedAccess` and a future `dc.date.embargoend` was therefore public from the second it was approved. The call is back on the common path. What kept `installItem` from cloning that default before the move was never the `rpType`, as the comment claimed, but `addDefaultPoliciesNotInPlace` -> `AuthorizeServiceImpl.isAnIdenticalPolicyAlreadyInPlace`, which matches on `(dso, group, action)` alone and therefore already sees the embargo policy; the comment says so now. `TYPE_CUSTOM` - the other half of f0c93aba - is what makes creating the policy early safe: `AuthorizeServiceImpl` skips custom policies on a bitstream that belongs to no installed item (DS-2614), so an item waiting for approval discloses nothing. `EmbargoImportIT.testWorkflowEmbargoSurvivesApproval` imports with `-w`, claims and approves the review task, then asserts that the file is not anonymously readable and that exactly one dated Anonymous READ policy remains. Moving the call back into the `!useWorkflow` branch fails it on precisely that assertion. Leak 2 - a missing dc.date.embargoend lifted every embargo in the batch ----------------------------------------------------------------------- Row 5 of the specification made an absent `dc.date.embargoend` clear the start date of the surviving policy. Since the survivor is located by `(group=Anonymous, action=READ)` and not by `rpName`, that also cleared embargoes `itemupdate` never set: the submission access condition and `dspace bulk-access-control` write precisely the same policy (Anonymous/READ, `TYPE_CUSTOM`, rpName `embargo`, future start date). `syncEmbargoPolicies` runs for every item of a batch as soon as `-a`/`-d` names an embargo field - the documented VSB command - so one SAF package without the field would have published every embargoed ORIGINAL bitstream in that batch. An absent `dc.date.embargoend` is now "no instruction": nothing is touched, the set of policy ids stays identical, the exit code stays 0, and an INFO line says so. The only way to open a file is a `dc.date.embargoend` in the past, which is the path that was already implemented and tested. The two "lift" tests are rewritten and renamed (`EmbargoLifecycleIT.removingEmbargoMetadataLeavesPoliciesUntouched`, `ItemUpdateIT.processArchiveUpdateRemovingEmbargoMetadataLeavesPoliciesUntouched`), and `EmbargoLifecycleIT.foreignEmbargoIsNeverLifted` covers the finding directly: it puts the submission's own policy on a bitstream and then runs a batch that does not mention the embargo. Exit code --------- Rows 6, 7, 8 and 11 require a non-zero exit code, but every test threw its `ItemUpdate` instance away, so `embargoSyncFailures` and its use in `main()` were completely uncovered. The mapping now lives in `ItemUpdate.exitStatus(status, failures)`, which `main()` calls and `ItemUpdateIT.embargoSyncFailuresDecideTheExitCode` asserts directly, and every `runItemUpdate` helper keeps its instance: 1 reported problem for rows 6/7/8/11, 0 for every run the tool is supposed to carry out. Also ---- * `EmbargoSafetyIT.withdrawnItemIsNeverRepublished` was vacuous with respect to its own subject: `ItemServiceImpl.withdraw()` also clears `archived`, so the `!isArchived` guard satisfied every assertion and the test passed with the withdrawal guard deleted. It now asserts that the console says "is withdrawn". * `EmbargoImportIT.testStandardEmbargoImport` and `testMultipleBitstreamsEmbargo` picked the embargo policy with `findFirst()`, so a second, undated policy next to it - exactly what leak 1 produces - passed unnoticed. They now count the Anonymous READ policies and ask the authorisation system whether an anonymous visitor can download the file. * Three `ItemUpdateIT` fixtures left the collection's undated default policy next to the embargo policy, so the file was readable throughout and their "nothing changed" assertions could not have detected a leak. The default is removed now and the fixtures assert that they really are embargoed. * The import-path "special case" branch (`dc.date.embargoend` without `dc.rights.access=embargoedAccess`) had no test at all - which is why nobody noticed it wrote a 48 character rpName into a `varchar(30)` column. `EmbargoImportIT.testEmbargoEndWithoutAccessRightsStillEmbargoes` covers it. * `EMBARGO_POLICY_NAME` was declared twice. Both tools now read `org.dspace.app.util.SafEmbargoConstants.EMBARGO_POLICY_NAME`: import writes the rpName, itemupdate later adopts it, and the two must not drift apart. 49 integration tests, 0 failures; checkstyle 0 violations. Co-Authored-By: Claude Opus 5 (1M context) --- .../app/itemimport/ItemImportServiceImpl.java | 36 +-- .../org/dspace/app/itemupdate/ItemUpdate.java | 121 ++++--- .../dspace/app/util/SafEmbargoConstants.java | 32 ++ .../app/itemimport/EmbargoImportIT.java | 296 +++++++++++++++++- .../app/itemupdate/EmbargoDateBoundaryIT.java | 15 +- .../app/itemupdate/EmbargoLifecycleIT.java | 153 +++++++-- .../app/itemupdate/EmbargoPastDateIT.java | 7 + .../app/itemupdate/EmbargoSafetyIT.java | 102 ++++-- .../dspace/app/itemupdate/ItemUpdateIT.java | 120 +++++-- 9 files changed, 714 insertions(+), 168 deletions(-) create mode 100644 dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java index f18917ae5f02..2aa074a25a43 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java @@ -73,6 +73,7 @@ import org.dspace.app.itemimport.service.ItemImportService; import org.dspace.app.util.LocalSchemaFilenameFilter; import org.dspace.app.util.RelationshipUtils; +import org.dspace.app.util.SafEmbargoConstants; import org.dspace.app.util.XMLUtils; import org.dspace.authorize.AuthorizeException; import org.dspace.authorize.ResourcePolicy; @@ -149,14 +150,6 @@ public class ItemImportServiceImpl implements ItemImportService, InitializingBean { private final Logger log = LogManager.getLogger(); - /** - * Name written on the embargo resource policies created during import. It is the {@code name} of the - * {@code embargoed} access condition in access-conditions.xml and has to fit the resourcepolicy.rpname - * column (varchar(30)): the previous "Special Case Embargo - No access rights metadata" was 48 characters - * and aborted the whole import on PostgreSQL with "value too long for type character varying(30)". - */ - private static final String EMBARGO_POLICY_NAME = "embargo"; - private DSpaceRunnableHandler handler; @Autowired(required = true) @@ -819,6 +812,14 @@ protected Item addItem(Context c, List mycollections, String path, // non-standard permissions List options = processContentsFile(c, myitem, itemPathDir, "contents"); + // Check for embargo metadata and set up embargo terms if needed. This has to happen on the common + // path, before the workflow is started as well as before installItem: an embargoed submission that + // reaches the workflow without its policy is given the collection's undated default READ policy the + // moment it is approved (ItemServiceImpl.addDefaultPoliciesNotInPlace) and is public from then on. + // The policy grants nothing while the item waits in the workflow - AuthorizeServiceImpl ignores + // TYPE_CUSTOM policies on a bitstream that belongs to no installed item (DS-2614). + processEmbargoMetadata(c, myitem); + if (useWorkflow) { // don't process handle file // start up a workflow @@ -834,13 +835,6 @@ protected Item addItem(Context c, List mycollections, String path, mapOutputString = itemname + " " + myitem.getID(); } } else { - // Check for embargo metadata and set up embargo terms if needed. - // Only on this branch, and before installItem: a workflow item must not be given an Anonymous READ - // policy before it has been approved, and the TYPE_CUSTOM embargo policy written here is what stops - // installItem from cloning the collection's undated default READ policy next to it (see - // ItemServiceImpl.addDefaultPoliciesNotInPlace), which would defeat the embargo outright. - processEmbargoMetadata(c, myitem); - // only process handle file if not using workflow system String myhandle = processHandleFile(c, myitem, itemPathDir, "handle"); @@ -2637,7 +2631,7 @@ protected void processEmbargoMetadata(Context c, Item item) throws SQLException, } // Both scenarios produce the same policy. They used to differ only in the rpName, and one of // those two names did not fit the 30 character rpname column at all. - applyEmbargoToItemBitstreams(c, item, accessStartDate, EMBARGO_POLICY_NAME); + applyEmbargoToItemBitstreams(c, item, accessStartDate, SafEmbargoConstants.EMBARGO_POLICY_NAME); } catch (Exception e) { logError("ERROR: Failed to apply embargo to bitstreams", e); } @@ -2688,9 +2682,13 @@ protected void applyEmbargoToItemBitstreams(Context c, Item item, Date accessSta policy.setAction(Constants.READ); policy.setStartDate(accessStartDate); policy.setRpName(policyReason); - // TYPE_CUSTOM is load bearing twice: AuthorizeServiceImpl only skips policies on a - // not-yet-installed item when they are custom, and installItem only clones the - // collection default READ policy onto a bitstream that has no custom one yet. + // TYPE_CUSTOM keeps the policy inert until the item is installed: AuthorizeServiceImpl + // skips custom policies on a bitstream that belongs to no installed item (DS-2614), so + // an item waiting in the workflow discloses nothing. What stops installItem from + // cloning the collection's undated default READ policy next to this one is not the + // type but ItemServiceImpl.addDefaultPoliciesNotInPlace -> + // AuthorizeServiceImpl.isAnIdenticalPolicyAlreadyInPlace, which matches on + // (dso, group, action) alone and therefore already sees this policy. policy.setRpType(ResourcePolicy.TYPE_CUSTOM); // Add policy to bitstream's existing policies diff --git a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java index 42e7ae71dafb..9726e45fa23a 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java +++ b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java @@ -35,6 +35,7 @@ import org.apache.commons.lang3.StringUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import org.dspace.app.util.SafEmbargoConstants; import org.dspace.authorize.AuthorizeException; import org.dspace.authorize.ResourcePolicy; import org.dspace.authorize.factory.AuthorizeServiceFactory; @@ -114,12 +115,6 @@ public class ItemUpdate { private static final String OPEN_ACCESS = "openAccess"; private static final String EMBARGOED_ACCESS = "embargoedAccess"; - /** - * Name written on every embargo policy. It is the {@code name} of the {@code embargoed} access condition in - * access-conditions.xml and has to fit the resourcepolicy.rpname column (varchar(30)). - */ - private static final String EMBARGO_POLICY_NAME = "embargo"; - /** The only supported way of changing access to a bitstream this tool refuses to touch. */ private static final String BULK_ACCESS_CONTROL_HINT = "Use the 'dspace bulk-access-control' script to grant or restore access to it."; @@ -398,12 +393,7 @@ public static void main(String[] argv) { context.restoreAuthSystemState(); } - // Embargo problems are reported per item and never abort the run, but they must not be reported as - // success either: an operator scripting itemupdate has to see them in the exit code. - if (iu.embargoSyncFailures > 0) { - prErr(iu.embargoSyncFailures + " embargo synchronisation problem(s) reported above."); - status = 1; - } + status = exitStatus(status, iu.embargoSyncFailures); if (isTest) { pr("***End of Test Run***"); @@ -414,6 +404,22 @@ public static void main(String[] argv) { System.exit(status); } + /** + * Exit code of a run. Embargo problems are reported per item and never abort the run, but they must not be + * reported as success either: an operator scripting {@code itemupdate} only ever sees the exit code. + * + * @param status exit code the run has produced so far + * @param embargoSyncFailures number of bitstreams/items whose embargo could not be synchronised + * @return {@code 1} when anything went wrong, the unchanged status otherwise + */ + protected static int exitStatus(int status, int embargoSyncFailures) { + if (embargoSyncFailures > 0) { + prErr(embargoSyncFailures + " embargo synchronisation problem(s) reported above."); + return 1; + } + return status; + } + /** * process an archive * @@ -691,6 +697,11 @@ protected static boolean containsEmbargoField(String[] targetFields) { * before its replacement has been stored, which is why the existing policy is mutated rather than replaced: * a failure between a delete and a create would leave the file with no policy at all, i.e. HTTP 401.

* + *

Only a {@code dc.date.embargoend} that is actually present is an instruction. Its absence means the + * SAF package says nothing about the embargo of this item, and the policies are left exactly as they are - + * an embargo this tool never set is never lifted by it. A file is opened by writing a + * {@code dc.date.embargoend} that lies in the past.

+ * * @param context DSpace context * @param item item that has just been updated from the SAF archive * @throws SQLException if a database error occurs @@ -730,8 +741,6 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti } // --- phase 2: validate and compute the target state ---------------------------------------------- - // null means "no embargo", i.e. the file is readable immediately - Date accessStartDate; List embargoEndDates = itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY); if (embargoEndDates.isEmpty()) { @@ -742,46 +751,53 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti embargoSyncFailures++; return; } - // Removing dc.date.embargoend is how an operator lifts an embargo. - pr("Item " + itemLabel(item) + " has no " + EMBARGO_FIELD_DATE_END + ", lifting the embargo."); - accessStartDate = null; - } else { - if (embargoEndDates.size() > 1) { - prWarn("Multiple " + EMBARGO_FIELD_DATE_END + " values found. Using first value only."); - } + // A missing dc.date.embargoend is no instruction at all, and it is never read as "lift the + // embargo". The embargo of an item may well have been set outside this tool - the submission + // access condition and dspace bulk-access-control both write exactly the policy that would be + // reopened here (Anonymous/READ, TYPE_CUSTOM, rpName "embargo") - while syncEmbargoPolicies runs + // for every item of a batch whose -a/-d fields mention an embargo field. A single SAF package + // without the field would therefore publish every embargoed file of that batch. + // The supported way to open a file is a dc.date.embargoend that lies in the past. + pr("Item " + itemLabel(item) + " has no " + EMBARGO_FIELD_DATE_END + ", resource policies left" + + " untouched. To end an embargo, set " + EMBARGO_FIELD_DATE_END + " to a date in the past."); + return; + } - String embargoEndDateStr = embargoEndDates.get(0).getValue(); - if (StringUtils.isBlank(embargoEndDateStr)) { - prErr(EMBARGO_FIELD_DATE_END + " is empty on item " + itemLabel(item) + ", its bitstream policies" - + " are left untouched."); - embargoSyncFailures++; - return; - } + if (embargoEndDates.size() > 1) { + prWarn("Multiple " + EMBARGO_FIELD_DATE_END + " values found. Using first value only."); + } - LocalDate embargoEndDay; - try { - // Strict ISO parsing on purpose: DCDate rolls 2026-02-30 over into 2026-03-02 and would turn a - // typo into a real embargo date. - embargoEndDay = LocalDate.parse(embargoEndDateStr.trim()); - } catch (DateTimeParseException e) { - prErr("Invalid " + EMBARGO_FIELD_DATE_END + " '" + embargoEndDateStr + "' on item " - + itemLabel(item) + ", expected a strict ISO date (yyyy-MM-dd). Its bitstream policies" - + " are left untouched."); - embargoSyncFailures++; - return; - } + String embargoEndDateStr = embargoEndDates.get(0).getValue(); + if (StringUtils.isBlank(embargoEndDateStr)) { + prErr(EMBARGO_FIELD_DATE_END + " is empty on item " + itemLabel(item) + ", its bitstream policies" + + " are left untouched."); + embargoSyncFailures++; + return; + } + + LocalDate embargoEndDay; + try { + // Strict ISO parsing on purpose: DCDate rolls 2026-02-30 over into 2026-03-02 and would turn a + // typo into a real embargo date. + embargoEndDay = LocalDate.parse(embargoEndDateStr.trim()); + } catch (DateTimeParseException e) { + prErr("Invalid " + EMBARGO_FIELD_DATE_END + " '" + embargoEndDateStr + "' on item " + + itemLabel(item) + ", expected a strict ISO date (yyyy-MM-dd). Its bitstream policies" + + " are left untouched."); + embargoSyncFailures++; + return; + } - // dc.date.embargoend is the inclusive last day of the embargo, so access starts the day after, at - // midnight UTC. Calendar.getInstance() would use the server time zone and shift that boundary. - LocalDate accessStartDay = embargoEndDay.plusDays(1); - accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); + // dc.date.embargoend is the inclusive last day of the embargo, so access starts the day after, at + // midnight UTC. Calendar.getInstance() would use the server time zone and shift that boundary. + LocalDate accessStartDay = embargoEndDay.plusDays(1); + Date accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); - if (!accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { - // An expired embargo is a publication, not a deletion. The real - already passed - start date is - // written, which makes the policy effective immediately. - pr("Embargo of item " + itemLabel(item) + " already expired on " + embargoEndDay - + ", its ORIGINAL bitstreams are public since " + accessStartDay + "."); - } + if (!accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { + // An expired embargo is a publication, not a deletion. The real - already passed - start date is + // written, which makes the policy effective immediately. + pr("Embargo of item " + itemLabel(item) + " already expired on " + embargoEndDay + + ", its ORIGINAL bitstreams are public since " + accessStartDay + "."); } // --- phase 3: mutate ----------------------------------------------------------------------------- @@ -798,7 +814,8 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti * * @param context DSpace context * @param item item whose ORIGINAL bitstreams are synchronised - * @param startDate day the files become publicly readable, or {@code null} to lift the embargo + * @param startDate day the files become publicly readable, never {@code null}; a day in the past makes the + * policy effective immediately * @throws SQLException if a database error occurs * @throws AuthorizeException if the policy update is not permitted */ @@ -837,7 +854,7 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta ResourcePolicy survivor = selectSurvivorPolicy(anonymousReadPolicies); survivor.setStartDate(startDate); survivor.setRpType(ResourcePolicy.TYPE_CUSTOM); - survivor.setRpName(EMBARGO_POLICY_NAME); + survivor.setRpName(SafEmbargoConstants.EMBARGO_POLICY_NAME); resourcePolicyService.update(context, survivor); // Only now, with the replacement safely stored, may the duplicates go. Reference identity is diff --git a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java new file mode 100644 index 000000000000..43d404c38bf0 --- /dev/null +++ b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java @@ -0,0 +1,32 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.app.util; + +/** + * Constants shared by the two SAF batch tools that write embargo resource policies: + * {@code dspace import} ({@link org.dspace.app.itemimport.ItemImportServiceImpl}) creates the policy on a + * freshly imported item, {@code dspace itemupdate} ({@link org.dspace.app.itemupdate.ItemUpdate}) later + * re-dates and normalises it. + * + *

The two tools have to agree on the value, so it is declared once. When they drift apart the operator + * sees two different names for the same thing in the policy list of a bitstream.

+ */ +public final class SafEmbargoConstants { + + /** + * Value written to {@code resourcepolicy.rpname} on every embargo policy created or adopted by the SAF + * tools. It is the {@code name} of the {@code embargoed} access condition in access-conditions.xml, which + * is what the submission UI and {@code dspace bulk-access-control} write, and it fits the 30 character + * {@code rpname} column - the previous "Special Case Embargo - No access rights metadata" was 48 + * characters and aborted the whole import on PostgreSQL. + */ + public static final String EMBARGO_POLICY_NAME = "embargo"; + + private SafEmbargoConstants() { + } +} diff --git a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java index 14501a1d8a02..0a34c0ab9406 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java @@ -8,6 +8,7 @@ package org.dspace.app.itemimport; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; @@ -17,7 +18,9 @@ import java.text.SimpleDateFormat; import java.time.LocalDate; import java.util.List; +import java.util.UUID; import java.util.stream.Collectors; +import javax.servlet.http.HttpServletRequest; import org.apache.commons.io.file.PathUtils; import org.dspace.AbstractIntegrationTestWithDatabase; @@ -42,9 +45,15 @@ import org.dspace.eperson.service.GroupService; import org.dspace.services.ConfigurationService; import org.dspace.services.factory.DSpaceServicesFactory; +import org.dspace.xmlworkflow.factory.XmlWorkflowServiceFactory; +import org.dspace.xmlworkflow.service.XmlWorkflowService; +import org.dspace.xmlworkflow.state.Workflow; +import org.dspace.xmlworkflow.storedcomponents.XmlWorkflowItem; +import org.dspace.xmlworkflow.storedcomponents.service.XmlWorkflowItemService; import org.junit.After; import org.junit.Before; import org.junit.Test; +import org.springframework.mock.web.MockHttpServletRequest; /** * Integration tests for embargo functionality in SAF Import feature. @@ -62,6 +71,12 @@ public class EmbargoImportIT extends AbstractIntegrationTestWithDatabase { private static final String EXPECTED_POLICY_START_DATE = EMBARGO_END_FUTURE.plusDays(1).toString(); private static final String EMBARGOEND_DATE_PAST = "2020-01-01"; private static final String ITEM_TITLE = "Test Embargo Item"; + /** + * The single rpName both SAF tools write. Deliberately repeated here instead of referencing + * {@code SafEmbargoConstants}: the value ends up in the database and must not change silently, and it has + * to fit the 30 character {@code resourcepolicy.rpname} column. + */ + private static final String EMBARGO_POLICY_NAME = "embargo"; private ItemService itemService = ContentServiceFactory.getInstance().getItemService(); private ResourcePolicyService resourcePolicyService = @@ -72,6 +87,10 @@ public class EmbargoImportIT extends AbstractIntegrationTestWithDatabase { DSpaceServicesFactory.getInstance().getConfigurationService(); private MetadataSchemaService metadataSchemaService = ContentServiceFactory.getInstance().getMetadataSchemaService(); + private XmlWorkflowService xmlWorkflowService = + XmlWorkflowServiceFactory.getInstance().getXmlWorkflowService(); + private XmlWorkflowItemService xmlWorkflowItemService = + XmlWorkflowServiceFactory.getInstance().getXmlWorkflowItemService(); private Collection collection; private Path tempDir; @@ -122,6 +141,7 @@ public void setUp() throws Exception { @After @Override public void destroy() throws Exception { + deleteRemainingWorkflowItems(); PathUtils.deleteDirectory(tempDir); for (Path path : Files.list(workDir).collect(Collectors.toList())) { PathUtils.delete(path); @@ -168,21 +188,29 @@ public void testStandardEmbargoImport() throws Exception { assertEquals("Should have one bitstream", 1, bitstreams.size()); Bitstream bitstream = bitstreams.get(0); - List policies = resourcePolicyService.find(context, bitstream, Constants.READ); - // Should have embargo policy for Anonymous group - ResourcePolicy embargoPolicy = policies.stream() - .filter(p -> p.getGroup() != null && p.getGroup().equals(anonymousGroup)) - .findFirst() - .orElse(null); + // Counting is part of the assertion, not a detail: installItem clones the collection's undated + // DEFAULT_BITSTREAM_READ onto a bitstream that has no Anonymous READ policy yet, and such a second, + // undated policy would make the file downloadable throughout the embargo. Picking the first matching + // policy with findFirst() cannot see that. + List anonymousRead = anonymousReadPolicies(bitstream); + assertEquals("exactly one Anonymous READ policy may remain: " + describe(bitstream), + 1, anonymousRead.size()); - assertNotNull("Should have embargo policy for Anonymous group", embargoPolicy); + ResourcePolicy embargoPolicy = anonymousRead.get(0); assertNotNull("Embargo policy should have start date", embargoPolicy.getStartDate()); + assertEquals("the embargo policy has to be TYPE_CUSTOM", + ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); + assertEquals("the embargo policy has to carry the access condition name", + EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); // Verify start date is embargoend + 1 day (file becomes accessible day after embargo ends) SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); assertEquals("Embargo policy start date should be embargoend + 1 day", EXPECTED_POLICY_START_DATE, sdf.format(embargoPolicy.getStartDate())); + + assertFalse("an embargoed file must not be downloadable by an anonymous visitor: " + describe(bitstream), + anonymousCanRead(bitstream)); } /** @@ -240,6 +268,242 @@ public void testPastEmbargoDateNoPolicy() throws Exception { anonymousCanRead(bitstream)); } + /** + * The regression this test exists for: {@code dspace import -a -w} puts the item into the workflow, and + * approving it calls {@code installItem}, which applies the collection's default policies. A bitstream + * that carries no embargo policy at that moment has no Anonymous READ policy at all, so + * {@code ItemServiceImpl.addDefaultPoliciesNotInPlace} clones the collection's undated + * DEFAULT_BITSTREAM_READ onto it - and the file is public from the second it is approved, while its + * metadata still says {@code embargoedAccess} with a future end date. + * + *

The embargo policy therefore has to be created on the common path, before the workflow starts. It + * discloses nothing while the item waits for approval, because {@code AuthorizeServiceImpl} ignores + * {@code TYPE_CUSTOM} policies on a bitstream that belongs to no installed item (DS-2614) - which the + * assertion on the workflow item below pins down.

+ */ + @Test + public void testWorkflowEmbargoSurvivesApproval() throws Exception { + context.turnOffAuthorisationSystem(); + Collection workflowCollection = CollectionBuilder.createCollection(context, parentCommunity) + .withName("Workflow Collection") + .withWorkflowGroup(1, admin) + .build(); + context.restoreAuthSystemState(); + + Path safDir = Files.createDirectory(Path.of(tempDir.toString() + "/test")); + Path itemDir = Files.createDirectory(Path.of(safDir.toString() + "/item_000")); + + String dublinCoreContent = "\n" + + "\n" + + " " + ITEM_TITLE + "\n" + + " embargoedAccess\n" + + " " + EMBARGOEND_DATE_FUTURE + "\n" + + ""; + Files.writeString(Path.of(itemDir.toString() + "/dublin_core.xml"), dublinCoreContent); + + Path contentsFile = Files.createFile(Path.of(itemDir.toString() + "/contents")); + Files.writeString(contentsFile, "test.txt"); + Path bitstreamFile = Files.createFile(Path.of(itemDir.toString() + "/test.txt")); + Files.writeString(bitstreamFile, "TEST CONTENT FOR WORKFLOW EMBARGO"); + + // -w: the submission goes through the workflow instead of straight into the archive + String[] args = new String[] { "import", "-a", "-w", "-e", admin.getEmail(), + "-c", workflowCollection.getID().toString(), + "-s", safDir.toString(), "-m", tempDir.toString() + "/mapfile.out" }; + runDSpaceScript(args); + + // itemService.findByMetadataField only returns archived items, and this one is deliberately not + // archived yet. The mapfile is what the operator gets instead: " ". + Item item = itemFromMapfile(tempDir.toString() + "/mapfile.out"); + assertFalse("fixture precondition: -w must leave the item in the workflow, not in the archive", + item.isArchived()); + + Bitstream bitstream = item.getBundles("ORIGINAL").get(0).getBitstreams().get(0); + assertFalse("a submission waiting for approval must not be downloadable: " + describe(bitstream), + anonymousCanRead(bitstream)); + + approveWorkflowItem(workflowCollection, item); + + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + assertTrue("fixture precondition: approving the workflow item must archive it", item.isArchived()); + + // The moment of the leak. Without the embargo policy the bitstream reaches installItem with no + // Anonymous READ policy, gets the collection default cloned onto it, and is public right here. + assertFalse("approving an embargoed submission published its files. dc.date.embargoend is " + + EMBARGOEND_DATE_FUTURE + ", so the file has to stay closed: " + describe(bitstream), + anonymousCanRead(bitstream)); + + List anonymousRead = anonymousReadPolicies(bitstream); + assertEquals("exactly one Anonymous READ policy may remain after approval - a second, undated one is" + + " the collection default and defeats the embargo: " + describe(bitstream), + 1, anonymousRead.size()); + + ResourcePolicy embargoPolicy = anonymousRead.get(0); + assertNotNull("the surviving Anonymous READ policy has to be dated", embargoPolicy.getStartDate()); + assertEquals("the embargo policy has to carry the access condition name", + EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); + assertEquals("the embargo policy has to be TYPE_CUSTOM", + ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); + + SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); + assertEquals("the embargo has to start the day after dc.date.embargoend", + EXPECTED_POLICY_START_DATE, sdf.format(embargoPolicy.getStartDate())); + } + + /** + * The "special case" branch: {@code dc.date.embargoend} without {@code dc.rights.access=embargoedAccess}. + * + *

It had no test at all, which is why nobody noticed that it wrote the 48 character rpName + * "Special Case Embargo - No access rights metadata" into a {@code varchar(30)} column - on PostgreSQL + * that aborts the whole import, and the SQL error names the column, not the branch that produced it.

+ */ + @Test + public void testEmbargoEndWithoutAccessRightsStillEmbargoes() throws Exception { + Path safDir = Files.createDirectory(Path.of(tempDir.toString() + "/test")); + Path itemDir = Files.createDirectory(Path.of(safDir.toString() + "/item_000")); + + // no dc.rights.access at all - this is the branch under test + String dublinCoreContent = "\n" + + "\n" + + " " + ITEM_TITLE + "\n" + + " " + EMBARGOEND_DATE_FUTURE + "\n" + + ""; + Files.writeString(Path.of(itemDir.toString() + "/dublin_core.xml"), dublinCoreContent); + + Path contentsFile = Files.createFile(Path.of(itemDir.toString() + "/contents")); + Files.writeString(contentsFile, "test.txt"); + Path bitstreamFile = Files.createFile(Path.of(itemDir.toString() + "/test.txt")); + Files.writeString(bitstreamFile, "TEST CONTENT FOR SPECIAL CASE EMBARGO"); + + String[] args = new String[] { "import", "-a", "-e", admin.getEmail(), "-c", collection.getID().toString(), + "-s", safDir.toString(), "-m", tempDir.toString() + "/mapfile.out" }; + runDSpaceScript(args); + + Item item = itemService.findByMetadataField(context, "dc", "title", null, ITEM_TITLE).next(); + assertNotNull("Item should be created", item); + assertTrue("the item has to be archived, i.e. the import must not have been aborted by an SQL error", + item.isArchived()); + + Bitstream bitstream = item.getBundles("ORIGINAL").get(0).getBitstreams().get(0); + + List anonymousRead = anonymousReadPolicies(bitstream); + assertEquals("exactly one Anonymous READ policy may remain: " + describe(bitstream), + 1, anonymousRead.size()); + + ResourcePolicy embargoPolicy = anonymousRead.get(0); + assertNotNull("the special case branch has to write a dated policy too", embargoPolicy.getStartDate()); + assertEquals("both branches write the same rpName", EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); + assertTrue("rpname is a varchar(30) column, so the value has to fit into it", + embargoPolicy.getRpName().length() <= 30); + assertEquals("the embargo policy has to be TYPE_CUSTOM", + ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); + + SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); + assertEquals("the embargo has to start the day after dc.date.embargoend", + EXPECTED_POLICY_START_DATE, sdf.format(embargoPolicy.getStartDate())); + + assertFalse("a dc.date.embargoend in the future closes the file even without dc.rights.access: " + + describe(bitstream), anonymousCanRead(bitstream)); + } + + /** + * The single item the import reported in its mapfile, looked up by id. Works for workflow imports too, + * where the item is not in the archive yet and therefore invisible to {@code findByMetadataField}. + */ + private Item itemFromMapfile(String mapfilePath) throws Exception { + List lines = Files.readAllLines(Path.of(mapfilePath)); + assertEquals("the import has to report exactly one item in its mapfile, got " + lines, + 1, lines.size()); + String[] columns = lines.get(0).trim().split("\\s+"); + assertEquals("a mapfile line is ' ', got '" + lines.get(0) + "'", + 2, columns.length); + + Item item = itemService.find(context, UUID.fromString(columns[1])); + assertNotNull("the item named in the mapfile has to exist", item); + return item; + } + + /** + * A workflow item that outlives its test keeps a {@code cwf_pooltask} row referencing the collection's + * workflow group, so {@code AbstractBuilder.cleanupObjects()} cannot delete that group - and every + * following test in this class then fails in cleanup instead of where the real problem is. + */ + private void deleteRemainingWorkflowItems() throws Exception { + if (context == null || !context.isValid()) { + return; + } + context.turnOffAuthorisationSystem(); + try { + for (XmlWorkflowItem workflowItem : xmlWorkflowItemService.findAll(context)) { + xmlWorkflowItemService.delete(context, workflowItem); + } + context.commit(); + } finally { + context.restoreAuthSystemState(); + } + } + + /** + * Claims and approves the single review task of a workflow item, which is what finally calls + * {@code installItem}. + */ + private void approveWorkflowItem(Collection workflowCollection, Item item) throws Exception { + XmlWorkflowItem workflowItem = xmlWorkflowItemService.findByItem(context, item); + assertNotNull("fixture precondition: the imported item has to be a workflow item", workflowItem); + + Workflow workflow = XmlWorkflowServiceFactory.getInstance().getWorkflowFactory() + .getWorkflow(workflowCollection); + MockHttpServletRequest request = new MockHttpServletRequest(); + request.setParameter("submit_approve", "submit_approve"); + HttpServletRequest servletRequest = request; + + EPerson previousUser = context.getCurrentUser(); + context.setCurrentUser(admin); + try { + Integer workflowItemId = workflowItem.getID(); + xmlWorkflowService.doState(context, admin, servletRequest, workflowItemId, workflow, + workflow.getStep("reviewstep").getActionConfig("claimaction")); + xmlWorkflowService.doState(context, admin, servletRequest, workflowItemId, workflow, + workflow.getStep("reviewstep").getActionConfig("reviewaction")); + } finally { + context.setCurrentUser(previousUser); + } + context.commit(); + } + + private List anonymousReadPolicies(Bitstream bitstream) throws Exception { + return resourcePolicyService.find(context, bitstream, Constants.READ).stream() + .filter(policy -> policy.getGroup() != null && policy.getGroup().equals(anonymousGroup)) + .collect(Collectors.toList()); + } + + /** + * Every resource policy of the bitstream, for failure messages - "the assertion failed" is not enough to + * tell an extra undated policy from a missing one. + */ + private String describe(Bitstream bitstream) throws Exception { + StringBuilder sb = new StringBuilder(System.lineSeparator()); + sb.append(" bitstream=").append(bitstream.getID()).append(System.lineSeparator()) + .append(" anonymousCanRead=").append(anonymousCanRead(bitstream)) + .append(System.lineSeparator()); + List policies = resourcePolicyService.find(context, bitstream, Constants.READ); + if (policies.isEmpty()) { + sb.append(" ").append(System.lineSeparator()); + } + for (ResourcePolicy policy : policies) { + sb.append(String.format(" id=%s group=%s rpType=%s rpName=%s start=%s end=%s", + policy.getID(), + policy.getGroup() == null ? "" : policy.getGroup().getName(), + policy.getRpType(), + policy.getRpName(), + policy.getStartDate(), + policy.getEndDate())) + .append(System.lineSeparator()); + } + return sb.toString(); + } + /** * What an anonymous visitor gets, with the test's own turnOffAuthorisationSystem calls temporarily unwound. */ @@ -387,19 +651,23 @@ public void testMultipleBitstreamsEmbargo() throws Exception { assertEquals("Should have two bitstreams", 2, bitstreams.size()); for (Bitstream bitstream : bitstreams) { - List policies = resourcePolicyService.find(context, bitstream, Constants.READ); - - ResourcePolicy embargoPolicy = policies.stream() - .filter(p -> p.getGroup() != null && p.getGroup().equals(anonymousGroup)) - .findFirst() - .orElse(null); + List anonymousRead = anonymousReadPolicies(bitstream); + assertEquals("exactly one Anonymous READ policy may remain on each bitstream: " + describe(bitstream), + 1, anonymousRead.size()); - assertNotNull("Each bitstream should have embargo policy", embargoPolicy); + ResourcePolicy embargoPolicy = anonymousRead.get(0); assertNotNull("Each embargo policy should have start date", embargoPolicy.getStartDate()); + assertEquals("the embargo policy has to be TYPE_CUSTOM", + ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); + assertEquals("the embargo policy has to carry the access condition name", + EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); assertEquals("Each embargo start date should be embargoend + 1 day", EXPECTED_POLICY_START_DATE, sdf.format(embargoPolicy.getStartDate())); + + assertFalse("an embargoed file must not be downloadable by an anonymous visitor: " + + describe(bitstream), anonymousCanRead(bitstream)); } } } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java index 4f58666ed0d1..5918c6dbd9ad 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java @@ -679,6 +679,11 @@ private String firstMetadataValue(Item item, String element, String qualifier) { *

{@code ItemUpdate.main} is deliberately not used - it ends in {@code System.exit} and would kill the * failsafe JVM.

* + *

Every scenario in this class is one {@code itemupdate} is supposed to carry out, so the helper also + * asserts the exit code the run would have produced: {@code embargoSyncFailures} is the only thing + * {@code main()} turns into a non-zero status, and a refusal that keeps the status at 0 is a silent + * failure for the operator's script.

+ * * @return everything {@code ItemUpdate.pr()} printed during the run; the stream is teed, so the output still * reaches the failsafe output file as well. */ @@ -713,7 +718,15 @@ private String runItemUpdate(Item item, String dublinCoreContent) throws Excepti } context.uncacheEntity(item); - return captured.toString(StandardCharsets.UTF_8.name()); + String consoleOutput = captured.toString(StandardCharsets.UTF_8.name()); + + assertEquals("this scenario is one itemupdate has to carry out, so it must not report an embargo" + + " synchronisation problem - ItemUpdate.main() would exit with " + + ItemUpdate.exitStatus(0, itemUpdate.embargoSyncFailures) + ". Console output was:\n" + + consoleOutput, + 0, itemUpdate.embargoSyncFailures); + + return consoleOutput; } /** diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java index df99d0137383..c0f542d90b10 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java @@ -155,12 +155,20 @@ public void destroy() throws Exception { } /** - * Removing {@code dc.date.embargoend} is the only way an operator lifts an embargo, and it is the second - * most frequent embargo operation after setting one. It must clear the start date of the surviving policy - * (spec row #5) instead of deleting the policy and leaving the file unreachable. + * A SAF package without {@code dc.date.embargoend} says nothing about the embargo of its item, and + * "nothing" is not "open the files". The policies have to come out of the run byte for byte identical - + * same {@code policy_id}, same start date, same name, same answer to "may an anonymous visitor download + * this". + * + *

Why this is not merely conservative: {@code syncEmbargoPolicies} runs for every item of a batch as + * soon as the {@code -a}/{@code -d} fields mention an embargo field, and the survivor is located by + * {@code (Anonymous, READ)}. One package whose {@code dublin_core.xml} happens to lack the field would + * otherwise publish that item's files - and a batch is exactly where nobody looks at the individual + * packages. Opening a file is done by writing a {@code dc.date.embargoend} that lies in the past, which + * is a deliberate, per-item statement.

*/ @Test - public void removingEmbargoMetadataLiftsEmbargo() throws Exception { + public void removingEmbargoMetadataLeavesPoliciesUntouched() throws Exception { String futureEmbargoEnd = LocalDate.now().plusMonths(6).toString(); Item item = createItem("Lift Embargo Thesis"); @@ -168,7 +176,8 @@ public void removingEmbargoMetadataLiftsEmbargo() throws Exception { Integer importedPolicyId = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import").getID(); // (a) operator embargoes the item - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + assertRunSucceeded("setting the embargo", runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, + futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -182,25 +191,83 @@ public void removingEmbargoMetadataLiftsEmbargo() throws Exception { assertFalse("while embargoed the file must not be publicly readable" + describePolicies(bitstream), anonymousCanRead(bitstream)); - // (b) operator lifts the embargo: the SAF no longer carries dc.date.embargoend - runItemUpdate(item, dublinCore(item, OPEN_ACCESS, null)); + Set policiesBefore = policySignatures(bitstream); + + // (b) the next SAF package simply does not carry dc.date.embargoend + assertRunSucceeded("running without dc.date.embargoend", + runItemUpdate(item, dublinCore(item, OPEN_ACCESS, null))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); - assertTrue("itemupdate did not remove dc.date.embargoend from the item", + assertTrue("fixture precondition: itemupdate has to remove dc.date.embargoend from the item, otherwise" + + " syncEmbargoPolicies still sees an end date and this test covers nothing", itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY).isEmpty()); - ResourcePolicy lifted = onlyAnonymousReadPolicy(bitstream, "the embargo lift run"); - assertEquals("lifting an embargo must mutate the surviving policy, not delete and recreate it", - importedPolicyId, lifted.getID()); - assertNull("lifting an embargo must clear startDate on the surviving Anonymous READ policy" + assertEquals("a SAF package without dc.date.embargoend must leave every resource policy exactly as it" + + " was - a changed policy set means an embargo was lifted by the absence of a field" + describePolicies(bitstream), - lifted.getStartDate()); - assertTrue("after the embargo was lifted the file must be publicly readable again" + policiesBefore, policySignatures(bitstream)); + assertFalse("removing dc.date.embargoend published an embargoed file. Absence of the field is not an" + + " instruction; an embargo is ended by a dc.date.embargoend in the past." + describePolicies(bitstream), anonymousCanRead(bitstream)); } + /** + * The same rule seen from the side that makes it a data leak rather than a matter of taste: an embargo + * that {@code itemupdate} never set. + * + *

The submission access condition and {@code dspace bulk-access-control} both write precisely this + * policy - {@code Anonymous}/{@code READ}, {@code TYPE_CUSTOM}, rpName {@code embargo}, future start date - + * and {@code syncEmbargoPolicies} finds its survivor by {@code (group, action)}, so it cannot tell that + * policy apart from one of its own. If the absence of {@code dc.date.embargoend} cleared the start date, + * a single {@code dspace itemupdate -a dc.date.embargoend} over a batch whose packages do not carry the + * field would publish every embargoed ORIGINAL bitstream in it.

+ */ + @Test + public void foreignEmbargoIsNeverLifted() throws Exception { + LocalDate foreignEmbargoStart = LocalDate.now().plusYears(2); + + Item item = createItem("Submission Embargo Thesis"); + Bitstream bitstream = createOriginalBitstream(item, "submission-embargo.pdf"); + + // exactly what the submission access condition / bulk-access-control leave behind + ResourcePolicy foreign = replaceAnonymousReadPolicies(bitstream, startOfDayUtc(foreignEmbargoStart), + EMBARGO_POLICY_NAME, ResourcePolicy.TYPE_CUSTOM); + bitstream = context.reloadEntity(bitstream); + + assertEquals("fixture precondition: the bitstream must carry exactly the policy the submission writes", + ResourcePolicy.TYPE_CUSTOM, + onlyAnonymousReadPolicy(bitstream, "the fixture").getRpType()); + assertFalse("fixture precondition: the foreign embargo must block anonymous access" + + describePolicies(bitstream), + anonymousCanRead(bitstream)); + + Set policiesBefore = policySignatures(bitstream); + Integer foreignPolicyId = foreign.getID(); + + // a routine metadata batch: the fields are targeted, the package carries neither of them + assertRunSucceeded("running a batch that does not mention the embargo", + runItemUpdate(item, dublinCore(item, null, null))); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + + assertEquals("itemupdate lifted an embargo it never set. The policy was written by the submission" + + " access condition or by bulk-access-control and is indistinguishable from one of" + + " itemupdate's own, so the absence of dc.date.embargoend must never touch it." + + describePolicies(bitstream), + policiesBefore, policySignatures(bitstream)); + + ResourcePolicy survivor = onlyAnonymousReadPolicy(bitstream, "a batch without dc.date.embargoend"); + assertEquals("the foreign policy must still be the very same row", foreignPolicyId, survivor.getID()); + assertNotNull("the foreign embargo start date must survive untouched" + describePolicies(bitstream), + survivor.getStartDate()); + assertEquals("the foreign embargo start date must not move", + foreignEmbargoStart, toLocalDate(survivor.getStartDate())); + assertFalse("a file embargoed outside itemupdate became publicly readable" + describePolicies(bitstream), + anonymousCanRead(bitstream)); + } + /** * Re-running the identical SAF archive - which happens every time an import job is retried - must be a * no-op. The same {@code policy_id} has to come back, which is only possible if the policy is mutated @@ -215,7 +282,8 @@ public void syncIsIdempotent() throws Exception { Bitstream bitstream = createOriginalBitstream(item, "idempotent.pdf"); Integer importedPolicyId = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import").getID(); - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + assertRunSucceeded("setting the embargo", + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -231,7 +299,8 @@ public void syncIsIdempotent() throws Exception { assertEquals("the surviving policy must be TYPE_CUSTOM", ResourcePolicy.TYPE_CUSTOM, first.getRpType()); // exactly the same archive again - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + assertRunSucceeded("re-running the identical archive", + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -288,7 +357,8 @@ public void bornOpenItemThenFutureEmbargoIsEnforced() throws Exception { imported.getStartDate()); assertTrue("fixture precondition: a born open bitstream is publicly readable", anonymousCanRead(bitstream)); - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + assertRunSucceeded("embargoing a born-open item", + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -334,7 +404,8 @@ public void duplicateAnonymousReadPoliciesCollapseToOne() throws Exception { assertEquals("fixture precondition: three Anonymous READ policies" + describePolicies(bitstream), 3, anonymousReadPolicies(bitstream).size()); - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + assertRunSucceeded("collapsing duplicate Anonymous READ policies", + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -450,7 +521,8 @@ public void multipleBitstreamsAllGetSameState() throws Exception { UUID primaryBitstreamId = bitstreams.get(0).getID(); // (a) embargo every file of the record - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + assertRunSucceeded("embargoing every ORIGINAL bitstream", + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); reloadAll(bitstreams); originalBundle = context.reloadEntity(originalBundle); @@ -472,7 +544,8 @@ public void multipleBitstreamsAllGetSameState() throws Exception { primaryBitstreamId, originalBundle.getPrimaryBitstream().getID()); // (b) the embargo expires - the same archive is re-imported with a past date - runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd)); + assertRunSucceeded("letting the embargo expire", + runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd))); item = context.reloadEntity(item); reloadAll(bitstreams); originalBundle = context.reloadEntity(originalBundle); @@ -518,7 +591,8 @@ public void derivativeBundlesAreNotTouchedDirectly() throws Exception { originalBundlePolicies.isEmpty()); // (a) embargo run - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd)); + assertRunSucceeded("embargoing the ORIGINAL bitstream", + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); original = context.reloadEntity(original); extractedText = context.reloadEntity(extractedText); @@ -536,7 +610,8 @@ public void derivativeBundlesAreNotTouchedDirectly() throws Exception { originalBundlePolicies, policySignatures(originalBundle)); // (b) expired embargo run - the code path that wipes policies today - runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd)); + assertRunSucceeded("letting the embargo expire", + runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd))); item = context.reloadEntity(item); original = context.reloadEntity(original); extractedText = context.reloadEntity(extractedText); @@ -578,7 +653,8 @@ private void assertLegacyPolicyIsAdoptedAndEnforced(String legacyName, String ri + describePolicies(bitstream), anonymousCanRead(bitstream)); - runItemUpdate(item, dublinCore(item, rightsAccess, futureEmbargoEnd)); + assertRunSucceeded("adopting a legacy embargo policy", + runItemUpdate(item, dublinCore(item, rightsAccess, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -784,6 +860,11 @@ private void reloadAll(List bitstreams) throws Exception { private ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) throws Exception { + return addAnonymousReadPolicy(bitstream, startDate, name, null); + } + + private ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name, + String policyType) throws Exception { context.turnOffAuthorisationSystem(); ResourcePolicyBuilder builder = ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) .withAction(Constants.READ) @@ -792,6 +873,9 @@ private ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDat if (startDate != null) { builder.withStartDate(startDate); } + if (policyType != null) { + builder.withPolicyType(policyType); + } ResourcePolicy policy = builder.build(); context.restoreAuthSystemState(); return policy; @@ -803,10 +887,15 @@ private ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDat */ private ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name) throws Exception { + return replaceAnonymousReadPolicies(bitstream, startDate, name, null); + } + + private ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name, + String policyType) throws Exception { context.turnOffAuthorisationSystem(); authorizeService.removePoliciesActionFilter(context, bitstream, Constants.READ); context.restoreAuthSystemState(); - return addAnonymousReadPolicy(bitstream, startDate, name); + return addAnonymousReadPolicy(bitstream, startDate, name, policyType); } private String singleMetadataValue(Item item, String element, String qualifier) { @@ -829,8 +918,11 @@ private LocalDate toLocalDate(Date date) { * Equivalent of {@code dsrun ... ItemUpdate -s -d dc.rights.access -d dc.date.embargoend * -a dc.rights.access -a dc.date.embargoend}, i.e. an update whose target fields contain an embargo field, * which is what makes {@code processArchive} call {@code syncEmbargoPolicies}. + * + * @return the number of embargo problems the run reported; this is the only thing {@code ItemUpdate.main()} + * turns into a non-zero exit code, so it is what an operator's script sees */ - private void runItemUpdate(Item item, String dublinCoreContent) throws Exception { + private int runItemUpdate(Item item, String dublinCoreContent) throws Exception { Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); @@ -851,6 +943,17 @@ private void runItemUpdate(Item item, String dublinCoreContent) throws Exception context.restoreAuthSystemState(); context.uncacheEntity(item); + return itemUpdate.embargoSyncFailures; + } + + /** + * A run the tool is supposed to carry out has to end with exit code 0. + */ + private void assertRunSucceeded(String what, int embargoSyncFailures) { + assertEquals("itemupdate reported an embargo synchronisation problem while " + what + ", so" + + " ItemUpdate.main() would exit with " + ItemUpdate.exitStatus(0, embargoSyncFailures) + + " although nothing was wrong with the input", + 0, embargoSyncFailures); } /** diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java index 2240c2ca4771..fd07a3793cba 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java @@ -287,6 +287,13 @@ private void runItemUpdate(Item item, String dublinCoreContent) throws Exception context.restoreAuthSystemState(); context.uncacheEntity(item); + + // Both runs of this test are runs itemupdate has to carry out. embargoSyncFailures is what + // ItemUpdate.main() turns into a non-zero exit code, so a refusal that leaves it at 0 would be + // invisible to the operator's script. + assertEquals("itemupdate reported an embargo synchronisation problem, so ItemUpdate.main() would exit" + + " with " + ItemUpdate.exitStatus(0, itemUpdate.embargoSyncFailures), + 0, itemUpdate.embargoSyncFailures); } private String dublinCore(Item item, String rightsAccess, String embargoEndDate) { diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java index e42efcc4e3bb..9391f24b6efb 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java @@ -178,11 +178,19 @@ public void withdrawnItemIsNeverRepublished() throws Exception { Set idsBefore = policyIds(bitstream); List policiesBefore = policyFingerprints(bitstream); - runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + Run pastRun = runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); + // Which guard stopped the run has to be nailed down. ItemServiceImpl.withdraw() also clears + // archived, so the !isArchived guard alone would satisfy every policy assertion below and this test + // would keep passing after the withdrawal guard was deleted. + assertTrue("ItemUpdate has to refuse a withdrawn item because it is withdrawn. Nothing in the console" + + " output says so, so some other guard stopped the run and the withdrawal guard is" + + " untested. Console output was:" + System.lineSeparator() + pastRun.console, + pastRun.console.contains("is withdrawn")); + assertExitCode("withdrawn item, expired embargo", 0, pastRun); assertTrue("An expired embargo on a WITHDRAWN item created an action=READ policy." + " Withdrawal must never be undone by itemupdate, only WITHDRAWN_READ may remain." + describe(bitstream), @@ -194,11 +202,16 @@ public void withdrawnItemIsNeverRepublished() throws Exception { // Row 9 says "any end date". The past-date run above only ever reaches the early return, so on its own // it proves nothing about withdrawal; the future-date branch is the one that creates policies. - runItemUpdate(item, dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); + Run futureRun = + runItemUpdate(item, dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); + assertTrue("ItemUpdate has to refuse a withdrawn item because it is withdrawn. Console output was:" + + System.lineSeparator() + futureRun.console, + futureRun.console.contains("is withdrawn")); + assertExitCode("withdrawn item, future embargo", 0, futureRun); assertTrue("A FUTURE dc.date.embargoend on a WITHDRAWN item created an action=READ policy. A withdrawn" + " item must never gain one - only WITHDRAWN_READ may remain - otherwise the takedown" + " undoes itself the moment the embargo lapses." + describe(bitstream), @@ -216,7 +229,7 @@ public void withdrawnItemIsNeverRepublished() throws Exception { @Test public void restrictedAccessWithStaleEmbargoEndIsUntouched() throws Exception { assertEmbargoSyncIsANoOp("dc.rights.access=restrictedAccess with a past embargo end", - Collections.singletonList("restrictedAccess"), pastDate()); + Collections.singletonList("restrictedAccess"), pastDate(), 0); } /** @@ -225,7 +238,7 @@ public void restrictedAccessWithStaleEmbargoEndIsUntouched() throws Exception { @Test public void metadataOnlyAccessIsUntouched() throws Exception { assertEmbargoSyncIsANoOp("dc.rights.access=metadataOnlyAccess with a past embargo end", - Collections.singletonList("metadataOnlyAccess"), pastDate()); + Collections.singletonList("metadataOnlyAccess"), pastDate(), 0); } /** @@ -235,7 +248,7 @@ public void metadataOnlyAccessIsUntouched() throws Exception { @Test public void unknownAccessRightValueIsUntouched() throws Exception { assertEmbargoSyncIsANoOp("dc.rights.access=someAccessRightWeDoNotKnow with a past embargo end", - Collections.singletonList("someAccessRightWeDoNotKnow"), pastDate()); + Collections.singletonList("someAccessRightWeDoNotKnow"), pastDate(), 0); } /** @@ -246,7 +259,7 @@ public void unknownAccessRightValueIsUntouched() throws Exception { @Test public void mixedAccessRightsWithOneDisallowedIsUntouched() throws Exception { assertEmbargoSyncIsANoOp("dc.rights.access=openAccess + restrictedAccess with a past embargo end", - Arrays.asList("openAccess", "restrictedAccess"), pastDate()); + Arrays.asList("openAccess", "restrictedAccess"), pastDate(), 0); } /** @@ -280,8 +293,7 @@ public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { Set idsBefore = policyIds(bitstream); List policiesBefore = policyFingerprints(bitstream); - String pastRunOutput = - runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + Run pastRun = runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); bitstream = context.reloadEntity(bitstream); @@ -295,7 +307,7 @@ public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { // Row 11 says "any end date". A past date only reaches the early return; the future-date branch is the // one that creates policies, so it is where a group-restricted file can silently gain an Anonymous one. - String futureRunOutput = runItemUpdate(item, + Run futureRun = runItemUpdate(item, dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); bitstream = context.reloadEntity(bitstream); @@ -314,11 +326,16 @@ public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { assertTrue("There is no Anonymous READ policy to re-date here, so ItemUpdate has to report the bitstream" + " it could not synchronise and name '" + BULK_ACCESS_CONTROL_HINT + "' as the supported" + " way to change access. Console output of the expired-embargo run was:" - + System.lineSeparator() + pastRunOutput, - pastRunOutput.contains(BULK_ACCESS_CONTROL_HINT)); + + System.lineSeparator() + pastRun.console, + pastRun.console.contains(BULK_ACCESS_CONTROL_HINT)); assertTrue("ItemUpdate stayed silent about a bitstream it could not synchronise. Console output of the" - + " future-embargo run was:" + System.lineSeparator() + futureRunOutput, - futureRunOutput.contains(BULK_ACCESS_CONTROL_HINT)); + + " future-embargo run was:" + System.lineSeparator() + futureRun.console, + futureRun.console.contains(BULK_ACCESS_CONTROL_HINT)); + + // Spec row 11 requires a non-zero exit code: an unsynchronised bitstream that leaves the exit code at + // 0 is invisible to the operator who started the batch. + assertExitCode("bitstream without Anonymous READ, expired embargo", 1, pastRun); + assertExitCode("bitstream without Anonymous READ, future embargo", 1, futureRun); } /** @@ -342,8 +359,7 @@ public void alreadyBrokenBitstreamWithZeroPoliciesStaysZero() throws Exception { assertFalse("fixture precondition: a bitstream without policies must not be readable" + describe(bitstream), anonymousCanRead(bitstream)); - String consoleOutput = - runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + Run run = runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); bitstream = context.reloadEntity(bitstream); @@ -356,8 +372,9 @@ public void alreadyBrokenBitstreamWithZeroPoliciesStaysZero() throws Exception { anonymousCanRead(bitstream)); assertTrue("ItemUpdate stayed silent about a bitstream it could not synchronise. It has to report the" + " failure and name '" + BULK_ACCESS_CONTROL_HINT + "' as the supported way to restore" - + " access. Console output was:\n" + consoleOutput, - consoleOutput.contains(BULK_ACCESS_CONTROL_HINT)); + + " access. Console output was:\n" + run.console, + run.console.contains(BULK_ACCESS_CONTROL_HINT)); + assertExitCode("bitstream with zero policies", 1, run); } /** @@ -367,7 +384,7 @@ public void alreadyBrokenBitstreamWithZeroPoliciesStaysZero() throws Exception { @Test public void blankEmbargoEndLeavesPoliciesUntouched() throws Exception { Item item = assertEmbargoSyncIsANoOp("blank dc.date.embargoend", - Collections.singletonList("openAccess"), ""); + Collections.singletonList("openAccess"), "", 1); List storedEndDates = itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY); assertEquals("fixture precondition: the blank dc.date.embargoend has to be stored as an empty value." @@ -390,7 +407,7 @@ public void invalidEmbargoEndLeavesPoliciesUntouched() throws Exception { for (String invalidEndDate : invalidEndDates) { assertEmbargoSyncIsANoOp("unparseable dc.date.embargoend=" + invalidEndDate, - Collections.singletonList("openAccess"), invalidEndDate); + Collections.singletonList("openAccess"), invalidEndDate, 1); } } @@ -401,7 +418,7 @@ public void invalidEmbargoEndLeavesPoliciesUntouched() throws Exception { @Test public void embargoedAccessWithoutEndDateLeavesPoliciesUntouched() throws Exception { assertEmbargoSyncIsANoOp("dc.rights.access=embargoedAccess without dc.date.embargoend", - Collections.singletonList("embargoedAccess"), null); + Collections.singletonList("embargoedAccess"), null, 1); } /** @@ -429,10 +446,11 @@ public void notArchivedItemIsUntouched() throws Exception { Set idsBefore = policyIds(bitstream); List policiesBefore = policyFingerprints(bitstream); - runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + Run run = runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); bitstream = context.reloadEntity(bitstream); + assertExitCode("item outside the archive", 0, run); assertUntouched("item outside the archive", idsBefore, policiesBefore, bitstream); assertFalse("A file of an item outside the archive became publicly readable." + describe(bitstream), anonymousCanRead(bitstream)); @@ -445,7 +463,8 @@ public void notArchivedItemIsUntouched() throws Exception { * future end date: exactly one Anonymous READ policy, dated, currently blocking access. Any repair * that publishes, deletes or re-creates that policy is caught here.

*/ - private Item assertEmbargoSyncIsANoOp(String scenario, List accessRights, String embargoEndDate) + private Item assertEmbargoSyncIsANoOp(String scenario, List accessRights, String embargoEndDate, + int expectedFailures) throws Exception { Item item = createItem("Safety scenario: " + scenario); Bitstream bitstream = createEmbargoedBitstream(item, "thesis.pdf"); @@ -456,11 +475,12 @@ private Item assertEmbargoSyncIsANoOp(String scenario, List accessRights Set idsBefore = policyIds(bitstream); List policiesBefore = policyFingerprints(bitstream); - runItemUpdate(item, dublinCore(item, accessRights, embargoEndDate)); + Run run = runItemUpdate(item, dublinCore(item, accessRights, embargoEndDate)); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); + assertExitCode(scenario, expectedFailures, run); assertUntouched(scenario, idsBefore, policiesBefore, bitstream); assertFalse("[" + scenario + "] the embargoed file became publicly readable, which is a data leak" + describe(bitstream), anonymousCanRead(bitstream)); @@ -487,9 +507,13 @@ private void assertUntouched(String scenario, Set idsBefore, ListThe {@link ItemUpdate} instance is kept, not thrown away: {@code embargoSyncFailures} is what + * {@code main()} turns into a non-zero exit code, and an operator scripting {@code itemupdate} sees + * nothing else. A refusal that leaves the exit code at 0 is a silent failure.

+ * + * @return the console output of the run and the number of embargo problems it counted */ - private String runItemUpdate(Item item, String dublinCoreContent) throws Exception { + private Run runItemUpdate(Item item, String dublinCoreContent) throws Exception { Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); // without this marker processArchive writes an undo archive next to the source directory Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); @@ -529,7 +553,33 @@ private String runItemUpdate(Item item, String dublinCoreContent) throws Excepti String consoleOutput = consoleBuffer.toString(StandardCharsets.UTF_8); // replay it so the failsafe -output.txt still holds the full ItemUpdate log System.out.println(consoleOutput); - return consoleOutput; + return new Run(consoleOutput, itemUpdate.embargoSyncFailures); + } + + /** + * Everything a finished {@code itemupdate} run is judged by: what it told the operator, and what it would + * have exited with. + */ + private static final class Run { + private final String console; + private final int embargoSyncFailures; + + private Run(String console, int embargoSyncFailures) { + this.console = console; + this.embargoSyncFailures = embargoSyncFailures; + } + } + + /** + * A run that refused to do something has to say so in its exit code, otherwise the operator's script + * treats a skipped item as a synchronised one. + */ + private void assertExitCode(String scenario, int expectedFailures, Run run) { + assertEquals("[" + scenario + "] wrong number of reported embargo problems, so ItemUpdate.main() would" + + " exit with " + ItemUpdate.exitStatus(0, run.embargoSyncFailures) + " instead of " + + ItemUpdate.exitStatus(0, expectedFailures) + ". Console output was:" + + System.lineSeparator() + run.console, + expectedFailures, run.embargoSyncFailures); } /** diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java index 0eebc7dde7b0..953e96629c74 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java @@ -10,7 +10,6 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; -import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -64,9 +63,8 @@ */ public class ItemUpdateIT extends AbstractIntegrationTestWithDatabase { - /** rpNames written by the shipped implementation; the fix has to adopt and normalise them. */ + /** rpName written by the shipped implementation; the fix has to adopt and normalise it. */ private static final String STANDARD_EMBARGO = "Standard Embargo"; - private static final String SPECIAL_CASE_EMBARGO = "Special Case Embargo"; /** The single normalised rpName, matching the access condition name in access-conditions.xml. */ private static final String EMBARGO_POLICY_NAME = "embargo"; @@ -123,6 +121,20 @@ public void destroy() throws Exception { super.destroy(); } + /** + * The embargo refusals are counted per item and never abort the run, so the exit code is the only place + * an operator's script can see them. Every test below asserts {@code embargoSyncFailures}; this one + * asserts the step that turns that counter into the process exit code, which is otherwise only reachable + * through {@code main()} and its {@code System.exit}. + */ + @Test + public void embargoSyncFailuresDecideTheExitCode() { + assertEquals("a clean run has to exit 0", 0, ItemUpdate.exitStatus(0, 0)); + assertEquals("a single unsynchronised bitstream has to fail the run", 1, ItemUpdate.exitStatus(0, 1)); + assertEquals("several problems still fail the run once", 1, ItemUpdate.exitStatus(0, 7)); + assertEquals("an already failed run stays failed", 1, ItemUpdate.exitStatus(1, 0)); + } + @Test public void containsEmbargoFieldHandlesNullsAndWhitespace() { assertFalse(ItemUpdate.containsEmbargoField(null)); @@ -195,6 +207,7 @@ public void syncEmbargoPoliciesDatesTheAnonymousReadPolicyAndBlocksAccess() thro ItemUpdate itemUpdate = new ItemUpdate(); itemUpdate.syncEmbargoPolicies(context, item); + assertEquals("setting a future embargo is not a failure", 0, itemUpdate.embargoSyncFailures); // Exactly one Anonymous READ policy has to be left behind. A second, undated one would silently // defeat the embargo, so counting is part of the assertion, not a detail. @@ -217,6 +230,7 @@ public void syncEmbargoPoliciesAppliesEmbargoWithoutAccessRightMetadata() throws ItemUpdate itemUpdate = new ItemUpdate(); itemUpdate.syncEmbargoPolicies(context, item); + assertEquals("setting a future embargo is not a failure", 0, itemUpdate.embargoSyncFailures); List anonymousRead = anonymousReadPolicies(bitstream); assertEquals(1, anonymousRead.size()); @@ -244,17 +258,24 @@ public void syncEmbargoPoliciesAppliesEmbargoWithoutAccessRightMetadata() throws public void syncEmbargoPoliciesLeavesPoliciesUntouchedWhenEmbargoDateInvalid() throws Exception { Item item = createItem("Invalid Date Item", "date", "embargoend", ""); Bitstream bitstream = createBitstream(item, "invalid.txt"); - ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, + ResourcePolicy legacyPolicy = replaceAnonymousReadPolicies(bitstream, new Date(System.currentTimeMillis() + 86_400_000L), STANDARD_EMBARGO); + bitstream = context.reloadEntity(bitstream); List idsBefore = policyIds(bitstream); - boolean readableBefore = anonymousCanRead(bitstream); + assertFalse("fixture precondition: the embargoed file must not be publicly readable, otherwise the" + + " 'nothing changed' assertions below say nothing about a leak", + anonymousCanRead(bitstream)); ItemUpdate itemUpdate = new ItemUpdate(); itemUpdate.syncEmbargoPolicies(context, item); + // Spec row 7: a blank end date is broken input, and the run has to exit non-zero because of it. + assertEquals("a blank dc.date.embargoend has to fail the run", 1, itemUpdate.embargoSyncFailures); + assertEquals(1, ItemUpdate.exitStatus(0, itemUpdate.embargoSyncFailures)); assertEquals(idsBefore, policyIds(bitstream)); - assertEquals(readableBefore, anonymousCanRead(bitstream)); + assertFalse("an unparseable dc.date.embargoend published an embargoed file", + anonymousCanRead(bitstream)); // The dated policy the run could not validate is still there, unchanged, under its legacy name. ResourcePolicy reloadedLegacy = resourcePolicyService.find(context, legacyPolicy.getID()); @@ -277,7 +298,8 @@ public void processArchiveUpdatesEmbargoMetadataAndResyncsEmbargoPolicy() throws ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); Integer legacyPolicyId = legacyPolicy.getID(); - runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, "embargoedAccess", newEmbargoDate)); + assertEquals("re-dating an embargo is not a failure", 0, + runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, "embargoedAccess", newEmbargoDate))); Item reloadedItem = context.reloadEntity(item); Bitstream reloadedBitstream = context.reloadEntity(bitstream); @@ -318,17 +340,22 @@ public void processArchiveUpdateWithBlankEmbargoDateLeavesPoliciesUntouched() th Date oldPolicyStart = Date.from(LocalDate.parse(oldEmbargoDate).plusDays(1) .atStartOfDay(ZoneId.systemDefault()).toInstant()); - ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = replaceAnonymousReadPolicies(bitstream, oldPolicyStart, STANDARD_EMBARGO); + bitstream = context.reloadEntity(bitstream); List idsBefore = policyIds(bitstream); - boolean readableBefore = anonymousCanRead(bitstream); + assertFalse("fixture precondition: the embargoed file must not be publicly readable, otherwise the" + + " 'nothing changed' assertions below say nothing about a leak", + anonymousCanRead(bitstream)); - runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, "embargoedAccess", "")); + assertEquals("a blank dc.date.embargoend has to fail the run", 1, + runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, "embargoedAccess", ""))); Bitstream reloadedBitstream = context.reloadEntity(bitstream); assertEquals(idsBefore, policyIds(reloadedBitstream)); - assertEquals(readableBefore, anonymousCanRead(reloadedBitstream)); + assertFalse("a blank dc.date.embargoend published an embargoed file", + anonymousCanRead(reloadedBitstream)); ResourcePolicy reloadedLegacy = resourcePolicyService.find(context, legacyPolicy.getID()); assertNotNull(reloadedLegacy); @@ -336,12 +363,14 @@ public void processArchiveUpdateWithBlankEmbargoDateLeavesPoliciesUntouched() th } /** - * Removing {@code dc.date.embargoend} is the only way an operator lifts an embargo, so the files have to - * become readable. The old assertion looked for the absence of a policy named "Standard Embargo", which - * a bitstream with zero policies - i.e. an unreadable one - passes just as well. + * A SAF package that does not carry {@code dc.date.embargoend} carries no instruction about the embargo, + * and an absent field must never open a file. {@code syncEmbargoPolicies} runs for every item of a batch + * whose target fields mention an embargo field, so reading "field missing" as "lift the embargo" would + * publish every embargoed item of a batch whose packages happen not to carry it. A file is opened by + * writing a {@code dc.date.embargoend} that lies in the past. */ @Test - public void processArchiveUpdateRemovingEmbargoMetadataLiftsEmbargo() throws Exception { + public void processArchiveUpdateRemovingEmbargoMetadataLeavesPoliciesUntouched() throws Exception { String oldEmbargoDate = LocalDate.now().plusDays(10).toString(); Item item = createItem("Remove Embargo Metadata Update", @@ -351,10 +380,17 @@ public void processArchiveUpdateRemovingEmbargoMetadataLiftsEmbargo() throws Exc Date oldPolicyStart = Date.from(LocalDate.parse(oldEmbargoDate).plusDays(1) .atStartOfDay(ZoneId.systemDefault()).toInstant()); - ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + // The collection default leaves an undated Anonymous READ policy on a new bitstream. It has to go, + // otherwise the file is readable throughout and the assertions below would prove nothing. + ResourcePolicy legacyPolicy = replaceAnonymousReadPolicies(bitstream, oldPolicyStart, STANDARD_EMBARGO); Integer legacyPolicyId = legacyPolicy.getID(); + bitstream = context.reloadEntity(bitstream); + + List idsBefore = policyIds(bitstream); + assertFalse("fixture precondition: the embargoed file must not be publicly readable", + anonymousCanRead(bitstream)); - runEmbargoMetadataUpdate(item, dublinCore(item)); + int failures = runEmbargoMetadataUpdate(item, dublinCore(item)); Item reloadedItem = context.reloadEntity(item); Bitstream reloadedBitstream = context.reloadEntity(bitstream); @@ -362,18 +398,22 @@ public void processArchiveUpdateRemovingEmbargoMetadataLiftsEmbargo() throws Exc List rightsAccess = itemService.getMetadata(reloadedItem, "dc", "rights", "access", Item.ANY); List embargoDates = itemService.getMetadata(reloadedItem, "dc", "date", "embargoend", Item.ANY); assertTrue(rightsAccess.isEmpty()); - assertTrue(embargoDates.isEmpty()); - - List anonymousRead = anonymousReadPolicies(reloadedBitstream); - assertEquals(1, anonymousRead.size()); - - ResourcePolicy liftedPolicy = anonymousRead.get(0); - assertEquals(legacyPolicyId, liftedPolicy.getID()); - assertNull(liftedPolicy.getStartDate()); - assertFalse(STANDARD_EMBARGO.equals(liftedPolicy.getRpName()) - || SPECIAL_CASE_EMBARGO.equals(liftedPolicy.getRpName())); - assertEquals(EMBARGO_POLICY_NAME, liftedPolicy.getRpName()); - assertTrue(anonymousCanRead(reloadedBitstream)); + assertTrue("fixture precondition: dc.date.embargoend has to be gone from the item", embargoDates.isEmpty()); + + // Nothing happened: same policy rows, same name, same start date, same answer to "can anyone read it". + assertEquals("removing dc.date.embargoend must not add or remove a single resource policy", + idsBefore, policyIds(reloadedBitstream)); + assertEquals(1, anonymousReadPolicies(reloadedBitstream).size()); + + ResourcePolicy untouchedPolicy = anonymousReadPolicies(reloadedBitstream).get(0); + assertEquals(legacyPolicyId, untouchedPolicy.getID()); + assertNotNull("removing dc.date.embargoend must not clear the embargo start date", + untouchedPolicy.getStartDate()); + assertEquals(STANDARD_EMBARGO, untouchedPolicy.getRpName()); + assertFalse("removing dc.date.embargoend published an embargoed file", anonymousCanRead(reloadedBitstream)); + + // "No instruction" is not a failure - the batch has to keep its exit code 0. + assertEquals("a SAF package without dc.date.embargoend is not an error", 0, failures); } @Test @@ -391,7 +431,8 @@ public void processArchiveUpdateWithEmbargoDateAndNoRightsAppliesEmbargo() throw ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); Integer legacyPolicyId = legacyPolicy.getID(); - runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, null, newEmbargoDate)); + assertEquals("an embargo end date without dc.rights.access is not a failure", 0, + runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, null, newEmbargoDate))); Item reloadedItem = context.reloadEntity(item); Bitstream reloadedBitstream = context.reloadEntity(bitstream); @@ -446,6 +487,18 @@ private Bitstream createBitstream(Item item, String name) throws Exception { return bitstream; } + /** + * Leaves the bitstream with exactly one Anonymous READ policy: the collection's undated default is + * removed first. Without that step a "the file is embargoed" fixture is not embargoed at all. + */ + private ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name) + throws Exception { + context.turnOffAuthorisationSystem(); + authorizeService.removePoliciesActionFilter(context, bitstream, Constants.READ); + context.restoreAuthSystemState(); + return createAnonymousReadPolicy(bitstream, startDate, name); + } + private ResourcePolicy createAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) throws Exception { context.turnOffAuthorisationSystem(); @@ -530,7 +583,11 @@ private String dublinCore(String identifierUri, String thesisIdentifier) { return sb.toString(); } - private void runEmbargoMetadataUpdate(Item item, String dublinCoreContent) throws Exception { + /** + * @return the number of embargo problems reported by the run; {@link ItemUpdate#exitStatus(int, int)} is + * what turns it into the exit code of {@code dspace itemupdate} + */ + private int runEmbargoMetadataUpdate(Item item, String dublinCoreContent) throws Exception { Path sourceRoot = Files.createDirectory(tempDir.resolve("update-source-" + System.nanoTime())); Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); @@ -552,6 +609,7 @@ private void runEmbargoMetadataUpdate(Item item, String dublinCoreContent) throw // Force entity reload in caller assertions after update transaction. context.uncacheEntity(item); + return itemUpdate.embargoSyncFailures; } private String dublinCore(Item item) { From 322a2486f6f88084e21c336ca518b96456f7bbc3 Mon Sep 17 00:00:00 2001 From: milanmajchrak Date: Tue, 18 Aug 2026 16:28:42 +0200 Subject: [PATCH 03/10] VSB-TUO/Fix: unparseable embargo end date must never publish an embargoed item The review of f26f648c found one more way this branch can publish a closed file, and it is the branch's own doing. Both commits replaced `new DCDate(str)` with a strict `LocalDate.parse(str)`. On the `itemupdate` path that is fail-closed - the item is refused, no policy is touched, exit code 1. On the import path the very same parse failure was a log line and a `return`: catch (DateTimeParseException e) { logError(...); return; } no embargo policy is created, `addItem` archives the item anyway, `installItem` -> `addDefaultPoliciesNotInPlace` clones the collection's *undated* DEFAULT_BITSTREAM_READ onto the bitstream, and the file is public although its own metadata says `dc.rights.access=embargoedAccess`. Exit code 0. That half was fail-open before this branch too, but the branch widened the set of inputs that fall into it: `DCDate` accepted `yyyy`, `yyyy-MM` and four full ISO timestamp shapes, `LocalDate.parse` accepts `yyyy-MM-dd` and nothing else. A package with `dc.date.embargoend=2099` used to produce a dated policy on `origin/customer/vsb-tuo` and produced none here. Reproduced on the branch code before touching it, with a SAF package carrying `embargoedAccess` + `dc.date.embargoend=invalid-date-format`: item=05f0fee9-37c7-4443-94ea-21caeda8553d archived=true bitstream=e8f96c7f-826d-4b08-9e4e-86a30b85f375 anonymousCanRead=true id=257 group=Anonymous rpType=TYPE_INHERITED rpName=null start=null Backwards compatible parsing ---------------------------- `org.dspace.app.util.SafEmbargoDateParser` reads the shapes `DCDate` read, and maps each one to the day `DCDate.toDate()` mapped it to - verified against `DCDate` itself, not from memory: * `yyyy-MM-dd`, and `yyyy-M-d` unpadded * `yyyy-MM-dd'T'HH[:mm[:ss[.fff]]]['Z']` - the UTC day of the instant, the time of day is dropped * `yyyy-MM` -> the *first* day of that month * `yyyy` -> *1 January* of that year, not 31 December. `DCDate` keeps a granularity but `toDate()` returns the first instant of the period, and the old code used that `Date` as the embargo end, so `2099` has always meant "closed until 1 January 2099, open on the 2nd". Widening it to the end of the year would extend embargoes the operators already live with. Deliberately not kept from `DCDate`: the lenient roll-over (`2026-02-30` became 2 March, i.e. a typo became a real embargo date), trailing garbage (`SimpleDateFormat` read `2099garbage` as the year 2099), and a numeric UTC offset, which `DCDate` mis-read as UTC anyway. All three now throw, and a throw means "refuse the package", never "no embargo". `ItemUpdate` uses the same parser, or the same SAF package would mean two different days in the two tools; on that path an unreadable date keeps its existing behaviour (policies untouched, `embargoSyncFailures`, exit 1). Fail closed on the import path ------------------------------ Every early return of `processEmbargoMetadata` and `applyEmbargoToItemBitstreams` was audited by asking one question: what happens to an item that says `dc.rights.access=embargoedAccess`? Five of them answered "archived and public", and those now throw `EmbargoMetadataException`, which `ItemImport.internalRun()` turns into `context.abort()` and exit 1: * `embargoedAccess` without any `dc.date.embargoend` * `dc.date.embargoend` present but empty * `dc.date.embargoend` that no format accepts * the `Anonymous` group not found * the two blanket `catch (Exception e) { logError(...) }` blocks and the per-bitstream one, which swallowed every failure of the policy write - the bitstream then reached `installItem` without an Anonymous READ policy, which is exactly the leak above The other three returns stay as they are and are documented: no embargo metadata at all (the collection defaults decide), an embargo that has already expired (a publication, not a failure - the branch's rule, covered by `EmbargoPastDateIT` and `testPastEmbargoDateNoPolicy`), and no ORIGINAL bundle (no file exists, so no file can be disclosed). Removing the outer `catch (Exception)` exposed what it had been hiding: `dspace import --test` creates no item, so `processEmbargoMetadata` was called with `null` and logged an NPE as "ERROR: Failed to process embargo metadata" on every package. It now returns before touching anything. Multiple `dc.date.embargoend` values keep the specified behaviour (first value wins) and now produce the same operator warning as `itemupdate`. Tests ----- `EmbargoImportIT` grows from 7 to 15 tests; the 9 changed or added ones all fail on the code of f26f648c: * `testInvalidEmbargoDateFormat` asserted only "no embargo policy", which a wide open bitstream satisfies just as well - it is the empty assertion that let the leak through. It now asserts that the operator is told *and* that no file of the package is anonymously readable. * `testLenientRollOverEmbargoDateIsRefused`, `testBlankEmbargoEndIsRefused`, `testEmbargoedAccessWithoutEndDateIsRefused` - the other refusal paths. * `testYearOnlyEmbargoEndIsFirstOfJanuary`, `testYearMonthEmbargoEndIsFirstOfMonth`, `testIsoTimestampEmbargoEndIsTruncatedToUtcDay` - the `DCDate` shapes produce a policy again, with the start date `DCDate` would have produced. * `testFailureToWriteThePolicyIsNotSwallowed`, `testMissingAnonymousGroupIsNotSwallowed` inject the failure into a hand-wired service instance instead of breaking the test database, and assert that it reaches the caller. 57 integration tests, 0 failures; checkstyle 0 violations. Co-Authored-By: Claude Opus 5 (1M context) --- .../itemimport/EmbargoMetadataException.java | 38 ++ .../app/itemimport/ItemImportServiceImpl.java | 287 ++++++++------- .../org/dspace/app/itemupdate/ItemUpdate.java | 13 +- .../dspace/app/util/SafEmbargoDateParser.java | 138 ++++++++ .../app/itemimport/EmbargoImportIT.java | 329 ++++++++++++++++-- 5 files changed, 642 insertions(+), 163 deletions(-) create mode 100644 dspace-api/src/main/java/org/dspace/app/itemimport/EmbargoMetadataException.java create mode 100644 dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/EmbargoMetadataException.java b/dspace-api/src/main/java/org/dspace/app/itemimport/EmbargoMetadataException.java new file mode 100644 index 000000000000..14ceccd988d2 --- /dev/null +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/EmbargoMetadataException.java @@ -0,0 +1,38 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.app.itemimport; + +/** + * Thrown when a SAF package claims an embargo ({@code dc.rights.access=embargoedAccess} or + * {@code dc.date.embargoend}) that the import cannot turn into a resource policy. + * + *

Checked on purpose. Every one of these conditions used to be a log line followed by a {@code return}, + * after which the item was archived anyway - and {@code installItem} then gave its bitstreams the collection's + * undated default READ policy, so the files were public although their own metadata says they are closed, with + * exit code 0. There is no correct way to swallow this exception: an embargo that cannot be written means the + * package has to be refused.

+ */ +public class EmbargoMetadataException extends Exception { + + private static final long serialVersionUID = 1L; + + /** + * @param message what the package asks for and why it cannot be done + */ + public EmbargoMetadataException(String message) { + super(message); + } + + /** + * @param message what the package asks for and why it cannot be done + * @param cause the underlying failure + */ + public EmbargoMetadataException(String message, Throwable cause) { + super(message, cause); + } +} diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java index 2aa074a25a43..2c97aea54467 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java @@ -74,6 +74,7 @@ import org.dspace.app.util.LocalSchemaFilenameFilter; import org.dspace.app.util.RelationshipUtils; import org.dspace.app.util.SafEmbargoConstants; +import org.dspace.app.util.SafEmbargoDateParser; import org.dspace.app.util.XMLUtils; import org.dspace.authorize.AuthorizeException; import org.dspace.authorize.ResourcePolicy; @@ -818,7 +819,13 @@ protected Item addItem(Context c, List mycollections, String path, // moment it is approved (ItemServiceImpl.addDefaultPoliciesNotInPlace) and is public from then on. // The policy grants nothing while the item waits in the workflow - AuthorizeServiceImpl ignores // TYPE_CUSTOM policies on a bitstream that belongs to no installed item (DS-2614). - processEmbargoMetadata(c, myitem); + try { + processEmbargoMetadata(c, myitem); + } catch (EmbargoMetadataException e) { + // The operator gets the package directory, which is the thing they have to fix; the item id + // means nothing to them and the import is rolled back anyway. + throw new EmbargoMetadataException("SAF package '" + itemname + "': " + e.getMessage(), e); + } if (useWorkflow) { // don't process handle file @@ -2552,93 +2559,114 @@ private void logError(String message, Exception e) { } /** - * Process embargo metadata and set up ResourcePolicy-based embargo. - * This method checks for embargo metadata fields and directly creates ResourcePolicy - * with embargo start date for Anonymous group READ access. - * - * Handles two scenarios: - * 1. dc.rights.access="embargoedAccess" + dc.date.embargoend (standard embargo) - * 2. Only dc.date.embargoend present (special case with warning logs) + * Set up the ResourcePolicy based embargo of an item being imported, from its own metadata + * ({@code dc.rights.access}, {@code dc.date.embargoend}). + * + *

Every path out of this method is either "the embargo policy is written" or "the import fails". What + * it must never do is return quietly on a package that claims an embargo: {@code addItem} archives the + * item a few lines later, {@code installItem} clones the collection undated default READ policy onto + * every bitstream that has none - and the files are public although their own metadata says + * {@code embargoedAccess}, with exit code 0. That is why the blanket {@code catch (Exception)} this method + * used to end with is gone, and why the failure cases below throw instead of logging and returning.

+ * + *

Two scenarios produce an embargo:

+ *
    + *
  1. {@code dc.rights.access=embargoedAccess} together with {@code dc.date.embargoend}
    2. + *
  2. {@code dc.date.embargoend} on its own - same policy, logged as a special case
    3. + *
+ * + * @param c DSpace context + * @param item item being imported, already carrying the metadata of the package + * @throws SQLException if a database error occurs + * @throws AuthorizeException if the policy may not be written + * @throws EmbargoMetadataException if the package claims an embargo that cannot be written. Never treat it + * as "then there is no embargo" - the item must not be archived. */ - protected void processEmbargoMetadata(Context c, Item item) throws SQLException, AuthorizeException { - try { - // Get embargo end date from dc.date.embargoend - List embargoEndDates = itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY); - - if (embargoEndDates.isEmpty()) { - // No embargo date found, check if there's embargoedAccess without date - List accessRights = itemService.getMetadata(item, "dc", "rights", "access", Item.ANY); - for (MetadataValue accessRight : accessRights) { - if ("embargoedAccess".equals(accessRight.getValue())) { - logError("WARNING: Item has dc.rights.access=embargoedAccess but no dc.date.embargoend. " + - "Cannot set embargo without end date."); - break; - } - } - return; // No embargo to process - } + protected void processEmbargoMetadata(Context c, Item item) + throws SQLException, AuthorizeException, EmbargoMetadataException { + if (isTest || item == null) { + // A test run creates no item and loads no metadata, so there is nothing to read and no file that + // could be disclosed. + return; + } - String embargoEndDateStr = embargoEndDates.get(0).getValue(); - if (StringUtils.isBlank(embargoEndDateStr)) { - logError("WARNING: dc.date.embargoend is empty. Cannot set embargo."); - return; - } + List embargoEndDates = itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY); + boolean hasEmbargoedAccess = hasEmbargoedAccess(item); - // Parse and validate embargo date. All arithmetic is done in UTC calendar days: neither - // Calendar.getInstance() (server time zone) nor DCDate (lenient, rolls 2026-02-30 over into - // 2026-03-02) can decide a day boundary reliably. - Date accessStartDate; - try { - LocalDate embargoEndDay = LocalDate.parse(embargoEndDateStr.trim()); - - // dc.date.embargoend is the inclusive last day of the embargo, so access starts the day after. - // The "already passed" test has to run on that start day and not on the end day, otherwise an - // embargo ending today would be dropped although the file must still be closed today. - LocalDate accessStartDay = embargoEndDay.plusDays(1); - if (!accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { - logInfo("Embargo: end date " + embargoEndDateStr + " has already passed, no embargo policy" - + " is created. installItem applies the collection default policies, so the files" - + " are as accessible as the collection says."); - return; - } - accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); - } catch (DateTimeParseException e) { - logError("ERROR: Invalid embargo end date format: " + embargoEndDateStr - + ". Expected a strict ISO date (yyyy-MM-dd)."); - return; + if (embargoEndDates.isEmpty()) { + if (hasEmbargoedAccess) { + throw new EmbargoMetadataException("dc.rights.access=embargoedAccess without a" + + " dc.date.embargoend. An embargo cannot be set without an end date, and archiving" + + " the item anyway would publish the files the package says are closed."); } + // No embargo metadata at all: the package says nothing about access, the collection defaults decide. + return; + } - // Check embargo scenario - List accessRights = itemService.getMetadata(item, "dc", "rights", "access", Item.ANY); - boolean hasEmbargoedAccess = false; + if (embargoEndDates.size() > 1) { + // Same rule as itemupdate: the first value wins and the operator is told, because two embargo + // end dates are a data error only they can resolve. + logError("WARNING: Multiple dc.date.embargoend values found. Using first value only."); + } - for (MetadataValue accessRight : accessRights) { - if ("embargoedAccess".equals(accessRight.getValue())) { - hasEmbargoedAccess = true; - break; - } - } + String embargoEndDateStr = embargoEndDates.get(0).getValue(); + if (StringUtils.isBlank(embargoEndDateStr)) { + throw new EmbargoMetadataException("dc.date.embargoend is present but empty. The field is an" + + " instruction to close the files, it cannot be carried out, and an item whose embargo" + + " could not be written must not be archived."); + } - try { - if (hasEmbargoedAccess) { - // Scenario 1: Standard embargo (embargoedAccess + embargoend) - logInfo("Embargo: Setting standard embargo on item until " + embargoEndDateStr); - } else { - // Scenario 2: Only embargo end date present (special case) - logInfo("Embargo: SPECIAL CASE - Found dc.date.embargoend without " + - "dc.rights.access=embargoedAccess"); - logInfo("Embargo: Applying embargo based on end date only until " + embargoEndDateStr); - } - // Both scenarios produce the same policy. They used to differ only in the rpName, and one of - // those two names did not fit the 30 character rpname column at all. - applyEmbargoToItemBitstreams(c, item, accessStartDate, SafEmbargoConstants.EMBARGO_POLICY_NAME); - } catch (Exception e) { - logError("ERROR: Failed to apply embargo to bitstreams", e); - } + // All arithmetic is done in UTC calendar days: neither Calendar.getInstance() (server time zone) nor + // DCDate (lenient, rolls 2026-02-30 over into 2026-03-02) can decide a day boundary reliably. The + // shapes DCDate did accept are still accepted, see SafEmbargoDateParser. + LocalDate embargoEndDay; + try { + embargoEndDay = SafEmbargoDateParser.parseEmbargoEndDay(embargoEndDateStr); + } catch (DateTimeParseException e) { + throw new EmbargoMetadataException("Invalid dc.date.embargoend '" + embargoEndDateStr + "'," + + " expected " + SafEmbargoDateParser.ACCEPTED_FORMATS + ". An end date nobody can read is" + + " not the same as no embargo, so the package is refused instead of archived.", e); + } + + // dc.date.embargoend is the inclusive last day of the embargo, so access starts the day after. The + // "already passed" test has to run on that start day and not on the end day, otherwise an embargo + // ending today would be dropped although the file must still be closed today. + LocalDate accessStartDay = embargoEndDay.plusDays(1); + if (!accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { + logInfo("Embargo: end date " + embargoEndDateStr + " has already passed, no embargo policy" + + " is created. installItem applies the collection default policies, so the files" + + " are as accessible as the collection says."); + return; + } + Date accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); - } catch (Exception e) { - logError("ERROR: Failed to process embargo metadata", e); + if (hasEmbargoedAccess) { + // Scenario 1: standard embargo (embargoedAccess + embargoend) + logInfo("Embargo: Setting standard embargo on item until " + embargoEndDateStr); + } else { + // Scenario 2: only the embargo end date is present + logInfo("Embargo: SPECIAL CASE - Found dc.date.embargoend without " + + "dc.rights.access=embargoedAccess"); + logInfo("Embargo: Applying embargo based on end date only until " + embargoEndDateStr); + } + // Both scenarios produce the same policy. They used to differ only in the rpName, and one of those two + // names did not fit the 30 character rpname column at all. + applyEmbargoToItemBitstreams(c, item, accessStartDate, SafEmbargoConstants.EMBARGO_POLICY_NAME); + } + + /** + * Whether the item carries {@code dc.rights.access=embargoedAccess}. + * + * @param item item being imported + * @return true if at least one value says the item is embargoed + */ + protected boolean hasEmbargoedAccess(Item item) { + for (MetadataValue accessRight : itemService.getMetadata(item, "dc", "rights", "access", Item.ANY)) { + if ("embargoedAccess".equals(StringUtils.trimToEmpty(accessRight.getValue()))) { + return true; + } } + return false; } /** @@ -2648,67 +2676,64 @@ protected void processEmbargoMetadata(Context c, Item item) throws SQLException, * get one from the collection default when installItem runs - so the policy is created here. ItemUpdate * works on already archived items and mutates the existing policy instead; the two must not be merged.

* + *

Nothing here is caught and logged away. A bitstream whose policy could not be written is exactly the + * bitstream {@code installItem} then hands the collection undated default READ policy to, so a swallowed + * failure here is a published file.

+ * * @param c DSpace context * @param item item being imported * @param accessStartDate day the files become publicly readable (dc.date.embargoend + 1 day, midnight UTC) * @param policyReason value for resourcepolicy.rpname, at most 30 characters + * @throws SQLException if a database error occurs + * @throws AuthorizeException if the policy may not be written + * @throws EmbargoMetadataException if the Anonymous group is missing, i.e. the policy cannot be created */ protected void applyEmbargoToItemBitstreams(Context c, Item item, Date accessStartDate, String policyReason) - throws SQLException, AuthorizeException { + throws SQLException, AuthorizeException, EmbargoMetadataException { - try { - // Get Anonymous group - Group anonymousGroup = groupService.findByName(c, Group.ANONYMOUS); - if (anonymousGroup == null) { - logError("ERROR: Cannot find Anonymous group for embargo policy"); - return; - } - - int bitstreamsProcessed = 0; - - // Only process ORIGINAL bundles to avoid affecting system bundles - List originalBundles = item.getBundles("ORIGINAL"); - if (originalBundles.isEmpty()) { - logInfo("Embargo: No ORIGINAL bundles found, no embargo applied"); - return; - } - - for (Bundle bundle : originalBundles) { - for (Bitstream bitstream : bundle.getBitstreams()) { - try { - // Create ResourcePolicy for READ access with start date = embargo end date - ResourcePolicy policy = resourcePolicyService.create(c, null, anonymousGroup); - policy.setdSpaceObject(bitstream); - policy.setAction(Constants.READ); - policy.setStartDate(accessStartDate); - policy.setRpName(policyReason); - // TYPE_CUSTOM keeps the policy inert until the item is installed: AuthorizeServiceImpl - // skips custom policies on a bitstream that belongs to no installed item (DS-2614), so - // an item waiting in the workflow discloses nothing. What stops installItem from - // cloning the collection's undated default READ policy next to this one is not the - // type but ItemServiceImpl.addDefaultPoliciesNotInPlace -> - // AuthorizeServiceImpl.isAnIdenticalPolicyAlreadyInPlace, which matches on - // (dso, group, action) alone and therefore already sees this policy. - policy.setRpType(ResourcePolicy.TYPE_CUSTOM); - - // Add policy to bitstream's existing policies - bitstream.getResourcePolicies().add(policy); - resourcePolicyService.update(c, policy); - bitstreamsProcessed++; - - } catch (Exception e) { - logError("ERROR: Failed to apply embargo policy to bitstream " + bitstream.getName(), e); - } - } - } - - logInfo("Embargo: Applied embargo policy to " + bitstreamsProcessed + - " bitstreams, readable from " + accessStartDate.toString()); + Group anonymousGroup = groupService.findByName(c, Group.ANONYMOUS); + if (anonymousGroup == null) { + throw new EmbargoMetadataException("Group '" + Group.ANONYMOUS + "' not found, the embargo policy" + + " cannot be created. Archiving the item anyway would leave it with the collection" + + " default policies and no embargo at all."); + } - } catch (Exception e) { - logError("ERROR: Failed to apply embargo to item bitstreams", e); - throw e; // Re-throw to maintain method signature contract + // Only process ORIGINAL bundles to avoid affecting system bundles + List originalBundles = item.getBundles("ORIGINAL"); + if (originalBundles.isEmpty()) { + // Nothing to close. A package without ORIGINAL bitstreams discloses no file, however loudly its + // metadata claims an embargo, so this one really is a no-op and not a silent failure. + logInfo("Embargo: No ORIGINAL bundles found, no embargo applied"); + return; } + + int bitstreamsProcessed = 0; + for (Bundle bundle : originalBundles) { + for (Bitstream bitstream : bundle.getBitstreams()) { + // Create ResourcePolicy for READ access with start date = embargo end date + 1 day + ResourcePolicy policy = resourcePolicyService.create(c, null, anonymousGroup); + policy.setdSpaceObject(bitstream); + policy.setAction(Constants.READ); + policy.setStartDate(accessStartDate); + policy.setRpName(policyReason); + // TYPE_CUSTOM keeps the policy inert until the item is installed: AuthorizeServiceImpl + // skips custom policies on a bitstream that belongs to no installed item (DS-2614), so + // an item waiting in the workflow discloses nothing. What stops installItem from + // cloning the collection undated default READ policy next to this one is not the + // type but ItemServiceImpl.addDefaultPoliciesNotInPlace -> + // AuthorizeServiceImpl.isAnIdenticalPolicyAlreadyInPlace, which matches on + // (dso, group, action) alone and therefore already sees this policy. + policy.setRpType(ResourcePolicy.TYPE_CUSTOM); + + // Add policy to bitstream existing policies + bitstream.getResourcePolicies().add(policy); + resourcePolicyService.update(c, policy); + bitstreamsProcessed++; + } + } + + logInfo("Embargo: Applied embargo policy to " + bitstreamsProcessed + + " bitstreams, readable from " + accessStartDate.toString()); } } diff --git a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java index 9726e45fa23a..6491040ab65b 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java +++ b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java @@ -36,6 +36,7 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.dspace.app.util.SafEmbargoConstants; +import org.dspace.app.util.SafEmbargoDateParser; import org.dspace.authorize.AuthorizeException; import org.dspace.authorize.ResourcePolicy; import org.dspace.authorize.factory.AuthorizeServiceFactory; @@ -777,13 +778,15 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti LocalDate embargoEndDay; try { - // Strict ISO parsing on purpose: DCDate rolls 2026-02-30 over into 2026-03-02 and would turn a - // typo into a real embargo date. - embargoEndDay = LocalDate.parse(embargoEndDateStr.trim()); + // Strict parsing on purpose: DCDate rolls 2026-02-30 over into 2026-03-02 and would turn a typo + // into a real embargo date. The shapes DCDate accepted are still read, and read as the same day, + // so the SAF packages of this repository keep working - see SafEmbargoDateParser. The import side + // uses the same parser, or the same package would mean two different days in the two tools. + embargoEndDay = SafEmbargoDateParser.parseEmbargoEndDay(embargoEndDateStr); } catch (DateTimeParseException e) { prErr("Invalid " + EMBARGO_FIELD_DATE_END + " '" + embargoEndDateStr + "' on item " - + itemLabel(item) + ", expected a strict ISO date (yyyy-MM-dd). Its bitstream policies" - + " are left untouched."); + + itemLabel(item) + ", expected " + SafEmbargoDateParser.ACCEPTED_FORMATS + ". Its" + + " bitstream policies are left untouched."); embargoSyncFailures++; return; } diff --git a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java new file mode 100644 index 000000000000..186ba4cca48a --- /dev/null +++ b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java @@ -0,0 +1,138 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.app.util; + +import java.time.LocalDate; +import java.time.Year; +import java.time.YearMonth; +import java.time.format.DateTimeFormatter; +import java.time.format.DateTimeFormatterBuilder; +import java.time.format.DateTimeParseException; +import java.time.format.ResolverStyle; +import java.time.format.SignStyle; +import java.time.temporal.ChronoField; + +import org.apache.commons.lang3.StringUtils; + +/** + * Turns the {@code dc.date.embargoend} value of a SAF package into the UTC calendar day on which the embargo + * ends, for {@code dspace import} and {@code dspace itemupdate} alike. + * + *

Both tools used to read that value with {@link org.dspace.content.DCDate}, which accepts seven shapes and + * is lenient: {@code 2026-02-30} silently becomes 2 March and {@code 2026-13-01} becomes 1 January 2027, so a + * typo turns into a real - possibly future - embargo date. Parsing is therefore strict here, but it still has + * to accept the shapes {@code DCDate} accepted, or SAF packages that used to import would stop working. Each + * shape is mapped to exactly the day {@code DCDate} mapped it to (verified against the class itself):

+ * + * + * + * + * + * + * + * + * + *
accepted values
{@code 2027-05-01}1 May 2027
{@code 2027-5-1}1 May 2027 - unpadded, {@code SimpleDateFormat} took it
{@code 2027-05-01T00:00:00Z}, {@code ...T00:00:00}, {@code ...T00:00}, {@code ...T00}1 May 2027, the UTC day of the instant; the time of day is dropped
{@code 2027-05}1 May 2027, not the end of the month
{@code 2027}1 January 2027, not the end of the year
+ * + *

The last two rows are the ones worth reading twice. {@code DCDate} keeps a granularity, but + * {@code toDate()} returns the first instant of that year or month, and the old code took that + * {@code Date} as the embargo end. So {@code 2027} has always meant "the embargo ends on 1 January 2027", the + * files open on 2 January 2027, and that reading is kept - widening it to 31 December would extend embargoes + * that operators have already been living with.

+ * + *

What is deliberately not kept from {@code DCDate}: the lenient roll-over of impossible dates, + * trailing garbage ({@code SimpleDateFormat} read {@code 2099garbage} as the year 2099), and a numeric UTC + * offset, which {@code DCDate} did not really support either - it ignored the offset and read + * {@code 2027-05-01T00:00:00+02:00} as if it were UTC. All of those now throw, and every caller has to treat a + * throw as "refuse the package", never as "no embargo".

+ */ +public final class SafEmbargoDateParser { + + /** + * {@code yyyy-MM-dd'T'HH[:mm[:ss[.fff]]]['Z']}, the four full ISO shapes of {@code DCDate} plus the + * fractional seconds its prefix matching used to swallow. Everything is UTC, which is what the trailing + * {@code Z} says and what {@code DCDate} assumed for the shapes without it. + */ + private static final DateTimeFormatter LEGACY_TIMESTAMP = new DateTimeFormatterBuilder() + .append(DateTimeFormatter.ISO_LOCAL_DATE) + .appendLiteral('T') + .appendValue(ChronoField.HOUR_OF_DAY, 2) + .optionalStart().appendLiteral(':').appendValue(ChronoField.MINUTE_OF_HOUR, 2) + .optionalStart().appendLiteral(':').appendValue(ChronoField.SECOND_OF_MINUTE, 2) + .optionalStart().appendFraction(ChronoField.NANO_OF_SECOND, 1, 9, true) + .optionalEnd().optionalEnd().optionalEnd() + .optionalStart().appendLiteral('Z').optionalEnd() + .toFormatter().withResolverStyle(ResolverStyle.STRICT); + + /** + * {@code yyyy-M-d} with unpadded month and day. {@code SimpleDateFormat} accepted {@code 2027-5-1} and + * meant 1 May 2027 by it, without any roll-over, so it is accepted here too - strictly, unlike + * {@code DCDate}: {@code 2027-2-30} is still rejected. + */ + private static final DateTimeFormatter UNPADDED_DATE = new DateTimeFormatterBuilder() + .appendValue(ChronoField.YEAR, 4, 10, SignStyle.EXCEEDS_PAD) + .appendLiteral('-') + .appendValue(ChronoField.MONTH_OF_YEAR) + .appendLiteral('-') + .appendValue(ChronoField.DAY_OF_MONTH) + .toFormatter().withResolverStyle(ResolverStyle.STRICT); + + /** Listed in operator messages, so that the two tools describe the same set of values. */ + public static final String ACCEPTED_FORMATS = + "yyyy-MM-dd, yyyy-MM (first of the month), yyyy (1 January) or yyyy-MM-dd'T'HH[:mm[:ss]][Z]"; + + private SafEmbargoDateParser() { + } + + /** + * The UTC calendar day on which the embargo ends, i.e. the last day the files stay closed. + * + * @param value raw {@code dc.date.embargoend}, surrounding whitespace is ignored + * @return the embargo end day, never {@code null} + * @throws DateTimeParseException if the value is none of the accepted shapes. It is never a licence to + * skip the embargo: a caller that cannot read the date does not know + * whether the item is embargoed, and has to refuse it. + */ + public static LocalDate parseEmbargoEndDay(String value) { + String trimmed = StringUtils.trimToEmpty(value); + + try { + // yyyy-MM-dd, the shape everything written by DSpace itself has + return LocalDate.parse(trimmed); + } catch (DateTimeParseException notAnIsoDay) { + // one of the older shapes, or garbage - decided below + } + + try { + return LocalDate.parse(trimmed, LEGACY_TIMESTAMP); + } catch (DateTimeParseException notAnIsoTimestamp) { + // ditto + } + + try { + // a month is a period; its embargo ends on its first day, as DCDate.toDate() reported it + return YearMonth.parse(trimmed).atDay(1); + } catch (DateTimeParseException notAYearMonth) { + // ditto + } + + try { + // and a year on 1 January, for the same reason + return Year.parse(trimmed).atDay(1); + } catch (DateTimeParseException notAYear) { + // ditto + } + + try { + return LocalDate.parse(trimmed, UNPADDED_DATE); + } catch (DateTimeParseException notAnUnpaddedDay) { + throw new DateTimeParseException("Unparseable embargo end date '" + value + "', expected " + + ACCEPTED_FORMATS, trimmed, notAnUnpaddedDay.getErrorIndex()); + } + } +} diff --git a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java index 0a34c0ab9406..1bed51716a2b 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java @@ -10,14 +10,25 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; import java.io.File; import java.nio.file.Files; import java.nio.file.Path; +import java.sql.SQLException; import java.text.SimpleDateFormat; import java.time.LocalDate; +import java.time.YearMonth; +import java.time.ZoneOffset; +import java.util.Date; +import java.util.Iterator; import java.util.List; +import java.util.TimeZone; import java.util.UUID; import java.util.stream.Collectors; import javax.servlet.http.HttpServletRequest; @@ -32,6 +43,7 @@ import org.dspace.builder.CommunityBuilder; import org.dspace.builder.MetadataFieldBuilder; import org.dspace.content.Bitstream; +import org.dspace.content.Bundle; import org.dspace.content.Collection; import org.dspace.content.Item; import org.dspace.content.MetadataSchema; @@ -570,48 +582,311 @@ public void testNoEmbargoMetadataNoPolicy() throws Exception { } /** - * Test embargo with invalid date format + * The leak of this round, and the reason the old version of this test did not catch it: it asserted + * "no embargo policy", which is satisfied just as well by a bitstream that is wide open. An unparseable + * {@code dc.date.embargoend} used to be a log line and a {@code return}, after which the item was + * archived, {@code installItem} cloned the collection undated default READ policy onto the bitstream, and + * the file was public although its own metadata says {@code embargoedAccess} - with exit code 0. + * + *

The value here is one no date parser accepts. The values that {@code DCDate} did accept are + * a different story and are still imported, see the three format tests below.

*/ @Test public void testInvalidEmbargoDateFormat() throws Exception { - // Create SAF with invalid embargo date format + assertBrokenEmbargoPackageIsRefused("invalid-date-format", "an unparseable dc.date.embargoend"); + } + + /** + * {@code DCDate} is lenient and reads 30 February as 2 March, i.e. it turns a typo into a real embargo + * date. Rejecting the value is right, but rejecting it and archiving the item anyway is the leak above, + * so this pins down both halves on the import path. + */ + @Test + public void testLenientRollOverEmbargoDateIsRefused() throws Exception { + int year = LocalDate.now(ZoneOffset.UTC).getYear() + 1; + assertBrokenEmbargoPackageIsRefused(year + "-02-30", "a dc.date.embargoend that does not exist"); + } + + /** + * {@code embargoedAccess} without an end date is self-contradictory metadata. Importing it archives an + * item that says its files are closed while the collection default policies make them public, which is + * exactly the contradiction resolved in the direction of disclosure. + */ + @Test + public void testEmbargoedAccessWithoutEndDateIsRefused() throws Exception { + Path itemDir = safPackage("embargoedAccess", null, "TEST CONTENT NO END DATE"); + + Exception reported = runImport(itemDir.getParent()); + assertNotNull("dc.rights.access=embargoedAccess without dc.date.embargoend has to be reported to the" + + " operator instead of being archived as a public item: " + describeArchived(ITEM_TITLE), + reported); + assertNoFileOfTheItemIsPublic("embargoedAccess without an end date"); + } + + /** + * A present but empty {@code dc.date.embargoend} is a broken export. The field is an instruction to close + * the files, and an instruction that cannot be carried out must not end as "no embargo". + */ + @Test + public void testBlankEmbargoEndIsRefused() throws Exception { + assertBrokenEmbargoPackageIsRefused("", "an empty dc.date.embargoend"); + } + + /** + * Backwards compatibility, first of three. {@code DCDate} accepted a bare year and + * {@code DCDate.toDate()} returned its first instant, so the old code has always read + * {@code 2099} as "the embargo ends on 1 January 2099" - not on 31 December. Widening it now would extend + * embargoes the operators have been living with, so the day is kept and only the fail-open half changed. + */ + @Test + public void testYearOnlyEmbargoEndIsFirstOfJanuary() throws Exception { + int year = LocalDate.now(ZoneOffset.UTC).getYear() + 1; + assertEmbargoIsAppliedFrom(String.valueOf(year), LocalDate.of(year, 1, 1).plusDays(1)); + } + + /** + * Backwards compatibility, second of three: {@code yyyy-MM} is the first day of that month, again because + * that is the instant {@code DCDate.toDate()} returned. + */ + @Test + public void testYearMonthEmbargoEndIsFirstOfMonth() throws Exception { + YearMonth yearMonth = YearMonth.from(LocalDate.now(ZoneOffset.UTC).plusYears(1)); + assertEmbargoIsAppliedFrom(yearMonth.toString(), yearMonth.atDay(1).plusDays(1)); + } + + /** + * Backwards compatibility, third of three: a full ISO timestamp. The time of day is dropped and the UTC + * day of the instant is the last closed day, which for the {@code T00:00:00Z} form every DSpace export + * writes is the same day and the same policy start date as before. + */ + @Test + public void testIsoTimestampEmbargoEndIsTruncatedToUtcDay() throws Exception { + LocalDate embargoEndDay = LocalDate.now(ZoneOffset.UTC).plusYears(1); + assertEmbargoIsAppliedFrom(embargoEndDay + "T00:00:00Z", embargoEndDay.plusDays(1)); + } + + /** + * The same class of bug as the parse failure, one layer down: writing the policy fails and the failure is + * caught, logged and forgotten. The bitstream then reaches {@code installItem} without an Anonymous READ + * policy and gets the collection undated default cloned onto it, i.e. the file is published by the very + * code path that was supposed to close it. + * + *

The failure is injected instead of provoked: breaking the resourcepolicy table of a running test + * database would take the rest of the suite with it. The test class sits in the same package as the + * service, so the Spring-injected collaborators can be replaced by hand.

+ */ + @Test + public void testFailureToWriteThePolicyIsNotSwallowed() throws Exception { + Item item = importEmbargoedItem(); + + ItemImportServiceImpl service = serviceWithTestDependencies(); + ResourcePolicyService failingPolicies = mock(ResourcePolicyService.class); + when(failingPolicies.create(any(), any(), any())) + .thenThrow(new SQLException("resource policy store is down")); + service.resourcePolicyService = failingPolicies; + + try { + service.processEmbargoMetadata(context, item); + fail("an embargo policy that could not be written has to stop the import. The item is archived a" + + " few lines later in addItem and installItem then hands its bitstreams the collection" + + " undated default READ policy, so a swallowed failure here publishes the files."); + } catch (Exception expected) { + // fail closed: the exception is the whole point, the import is rolled back by ItemImport + } + } + + /** + * Same again for the other collaborator: without the {@code Anonymous} group there is no embargo policy + * to create, and an item archived without one is public by collection default. + */ + @Test + public void testMissingAnonymousGroupIsNotSwallowed() throws Exception { + Item item = importEmbargoedItem(); + + ItemImportServiceImpl service = serviceWithTestDependencies(); + // a GroupService whose findByName answers null, which is the branch under test + service.groupService = mock(GroupService.class); + + try { + service.processEmbargoMetadata(context, item); + fail("without the Anonymous group the embargo policy cannot be created, and archiving the item" + + " anyway leaves it public under the collection default policies"); + } catch (Exception expected) { + // fail closed + } + } + + /** + * Writes a SAF package with the given access right and embargo end date into a fresh source directory. + * + * @param accessRight value of dc.rights.access, {@code null} to leave the field out + * @param embargoEnd value of dc.date.embargoend, {@code null} to leave the field out + * @param content payload of the single ORIGINAL bitstream + * @return the item directory; its parent is the source directory to hand to the import + */ + private Path safPackage(String accessRight, String embargoEnd, String content) throws Exception { Path safDir = Files.createDirectory(Path.of(tempDir.toString() + "/test")); Path itemDir = Files.createDirectory(Path.of(safDir.toString() + "/item_000")); - String dublinCoreContent = "\n" + - "\n" + - " " + ITEM_TITLE + "\n" + - " embargoedAccess\n" + - " invalid-date-format\n" + - ""; - Files.writeString(Path.of(itemDir.toString() + "/dublin_core.xml"), dublinCoreContent); + StringBuilder dublinCore = new StringBuilder("\n") + .append("\n") + .append(" ").append(ITEM_TITLE) + .append("\n"); + if (accessRight != null) { + dublinCore.append(" ").append(accessRight) + .append("\n"); + } + if (embargoEnd != null) { + dublinCore.append(" ").append(embargoEnd) + .append("\n"); + } + dublinCore.append(""); + Files.writeString(Path.of(itemDir.toString() + "/dublin_core.xml"), dublinCore.toString()); - // Add bitstream - Path contentsFile = Files.createFile(Path.of(itemDir.toString() + "/contents")); - Files.writeString(contentsFile, "test.txt"); - Path bitstreamFile = Files.createFile(Path.of(itemDir.toString() + "/test.txt")); - Files.writeString(bitstreamFile, "TEST CONTENT INVALID DATE"); + Files.writeString(Files.createFile(Path.of(itemDir.toString() + "/contents")), "test.txt"); + Files.writeString(Files.createFile(Path.of(itemDir.toString() + "/test.txt")), content); + return itemDir; + } - // Perform import - should not fail but should not apply embargo + /** + * Runs {@code dspace import -a} on the source directory. + * + * @param safDir source directory holding the item directories + * @return the exception the script reported, or {@code null} when it ran through + */ + private Exception runImport(Path safDir) throws Exception { String[] args = new String[] { "import", "-a", "-e", admin.getEmail(), "-c", collection.getID().toString(), "-s", safDir.toString(), "-m", tempDir.toString() + "/mapfile.out" }; - runDSpaceScript(args); + try { + runDSpaceScript(args); + return null; + } catch (Exception reported) { + return reported; + } + } + + /** + * Both halves of "fail closed" for a package whose {@code dc.date.embargoend} cannot be used: the + * operator is told (so the exit code is not 0), and no file of that package ends up readable. Asserting + * only the first half would pass on an import that leaves the files open, asserting only the second would + * pass on a silent import of nothing. + */ + private void assertBrokenEmbargoPackageIsRefused(String embargoEnd, String what) throws Exception { + Path itemDir = safPackage("embargoedAccess", embargoEnd, "TEST CONTENT " + what); + + Exception reported = runImport(itemDir.getParent()); + assertNotNull(what + " has to be reported to the operator instead of being archived as a public item: " + + describeArchived(ITEM_TITLE), reported); + assertNoFileOfTheItemIsPublic(what); + } + + /** + * No ORIGINAL bitstream of an archived item with this test title may be readable by an anonymous visitor. + * A refused import leaves no item at all, which is why the loop may legitimately find nothing. + */ + private void assertNoFileOfTheItemIsPublic(String what) throws Exception { + Iterator items = itemService.findByMetadataField(context, "dc", "title", null, ITEM_TITLE); + while (items.hasNext()) { + Item item = items.next(); + for (Bundle bundle : item.getBundles("ORIGINAL")) { + for (Bitstream bitstream : bundle.getBitstreams()) { + assertFalse("the package says dc.rights.access=embargoedAccess and " + what + ", so this" + + " file must not be readable by an anonymous visitor: " + describe(bitstream), + anonymousCanRead(bitstream)); + } + } + } + } + + /** + * A {@code dc.date.embargoend} in one of the shapes {@code DCDate} accepted still produces an embargo, on + * the day {@code DCDate} mapped it to. + * + * @param embargoEnd value written into dc.date.embargoend + * @param expectedStartDay UTC day the files are expected to open, i.e. embargo end day + 1 + */ + private void assertEmbargoIsAppliedFrom(String embargoEnd, LocalDate expectedStartDay) throws Exception { + Path itemDir = safPackage("embargoedAccess", embargoEnd, "TEST CONTENT " + embargoEnd); + + assertNull("dc.date.embargoend=" + embargoEnd + " was accepted by DCDate, so the packages of this" + + " repository use it and the import must not fail on it", runImport(itemDir.getParent())); - // Verify item was created (import should not fail) Item item = itemService.findByMetadataField(context, "dc", "title", null, ITEM_TITLE).next(); - assertNotNull("Item should be created even with invalid date format", item); + assertNotNull("Item should be created", item); + Bitstream bitstream = item.getBundles("ORIGINAL").get(0).getBitstreams().get(0); - // Verify no embargo policies due to invalid date - List bitstreams = item.getBundles("ORIGINAL").get(0).getBitstreams(); - Bitstream bitstream = bitstreams.get(0); - List policies = resourcePolicyService.find(context, bitstream, Constants.READ); + List anonymousRead = anonymousReadPolicies(bitstream); + assertEquals("exactly one Anonymous READ policy may remain: " + describe(bitstream), + 1, anonymousRead.size()); - boolean hasEmbargoPolicy = policies.stream() - .anyMatch(p -> p.getGroup() != null && - p.getGroup().equals(anonymousGroup) && - p.getStartDate() != null); + ResourcePolicy embargoPolicy = anonymousRead.get(0); + assertNotNull("the embargo policy has to be dated", embargoPolicy.getStartDate()); + assertEquals("dc.date.embargoend=" + embargoEnd + " has to mean the same day it meant with DCDate," + + " and the files open the day after it", + expectedStartDay.toString(), utcDay(embargoPolicy.getStartDate())); + assertEquals("the embargo policy has to carry the access condition name", + EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); + assertEquals("the embargo policy has to be TYPE_CUSTOM", + ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); + assertFalse("an embargoed file must not be downloadable by an anonymous visitor: " + describe(bitstream), + anonymousCanRead(bitstream)); + } - assertTrue("Should not have embargo policy with invalid date format", !hasEmbargoPolicy); + /** + * Imports one valid, still running embargo and returns the archived item. + */ + private Item importEmbargoedItem() throws Exception { + Path itemDir = safPackage("embargoedAccess", EMBARGOEND_DATE_FUTURE, "TEST CONTENT FOR INJECTION"); + assertNull("fixture precondition: the valid package has to import", runImport(itemDir.getParent())); + + Item item = itemService.findByMetadataField(context, "dc", "title", null, ITEM_TITLE).next(); + assertNotNull("fixture precondition: the item has to exist", item); + return item; + } + + /** + * An {@code ItemImportServiceImpl} whose collaborators the test can replace one by one. Only the three the + * embargo code uses are wired; the service is never asked to import anything through this instance. + */ + private ItemImportServiceImpl serviceWithTestDependencies() { + ItemImportServiceImpl service = new ItemImportServiceImpl(); + service.itemService = itemService; + service.groupService = groupService; + service.resourcePolicyService = resourcePolicyService; + return service; + } + + /** + * The policy start date as the UTC calendar day it is stored as. {@code SimpleDateFormat} would otherwise + * render midnight UTC in the time zone of the build machine and report the previous day west of Greenwich. + */ + private String utcDay(Date date) { + SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); + sdf.setTimeZone(TimeZone.getTimeZone("UTC")); + return sdf.format(date); + } + + /** + * Every archived item with this title and the policies of its ORIGINAL bitstreams, for failure messages. + * "The import did not fail" is not enough to tell a refused package from a published one. + */ + private String describeArchived(String title) throws Exception { + StringBuilder sb = new StringBuilder(System.lineSeparator()); + Iterator items = itemService.findByMetadataField(context, "dc", "title", null, title); + if (!items.hasNext()) { + sb.append(" ").append(System.lineSeparator()); + } + while (items.hasNext()) { + Item item = items.next(); + sb.append(" item=").append(item.getID()).append(" archived=").append(item.isArchived()) + .append(System.lineSeparator()); + for (Bundle bundle : item.getBundles("ORIGINAL")) { + for (Bitstream bitstream : bundle.getBitstreams()) { + sb.append(describe(bitstream)); + } + } + } + return sb.toString(); } /** From 57cd9727bb8b5168ae9811136246a8f90e068510 Mon Sep 17 00:00:00 2001 From: milanmajchrak Date: Tue, 18 Aug 2026 17:40:26 +0200 Subject: [PATCH 04/10] VSB-TUO/Test: cover legacy embargo date shapes on the itemupdate path The DCDate backwards compatibility of SafEmbargoDateParser was only covered on the import side. Reducing parseEmbargoEndDay to a bare LocalDate.parse failed 3 tests in EmbargoImportIT and none of the 42 itemupdate ones, although ItemUpdate.syncEmbargoPolicies uses the same parser and aligning the two tools was the point of the change. EmbargoDateBoundaryIT now pins the itemupdate path down as well: * legacyEmbargoEndShapesKeepTheirDcDateDay - a bare year, a bare month and an ISO timestamp close the file until exactly the day DCDate mapped them to (1 January, the 1st of the month, the UTC day of the instant). * legacyPastEmbargoEndPublishesOnPurpose - a legacy value whose day lies in the past publishes the file. That is a deliberate reading of the metadata and no longer an accident of strict parsing: "2020" says the embargo ended in 2020. Before the DCDate shapes were read again such a value threw, the item was refused and the run exited 1; that refusal was a side effect, not a decision, so the new behaviour is asserted rather than left to happen silently. * unparseableLegacyLookalikeLeavesPoliciesUntouched - trailing garbage after a year and a numeric UTC offset, both of which DCDate used to swallow, are refused: identical policy ids, unchanged start dates, one counted failure. A failing embargo synchronisation is no longer reported as a successful run. processArchive caught every per-item exception, printed it and left the exit code at 0, but syncEmbargoPolicies re-dates the surviving Anonymous READ policy of a bitstream before deleting the duplicates, so an exception in that last step left context.complete() committing a published file while the run claimed success. The call now counts an embargo failure before the exception propagates, covered by EmbargoSafetyIT.embargoSyncThatDiesHalfWayIsNotReportedAsSuccess. Known limitation, unchanged and now correctly commented in ItemImportServiceImpl.applyEmbargoToItemBitstreams: a package with no ORIGINAL bundle is a no-op for the embargo code, which is not the same as "no file is disclosed". A SAF contents file can route its files into another bundle with the bundle: marker, and those bitstreams are archived with the collection default READ policy however loudly dc.rights.access claims an embargo. Both SAF tools declare a scope of ORIGINAL bitstreams only; the comment used to claim a security property that scope does not give. --- .../app/itemimport/ItemImportServiceImpl.java | 8 +- .../org/dspace/app/itemupdate/ItemUpdate.java | 13 +- .../app/itemupdate/EmbargoDateBoundaryIT.java | 215 +++++++++++++++++- .../app/itemupdate/EmbargoSafetyIT.java | 52 ++++- 4 files changed, 280 insertions(+), 8 deletions(-) diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java index 2c97aea54467..a5663076d2a7 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java @@ -2701,8 +2701,12 @@ protected void applyEmbargoToItemBitstreams(Context c, Item item, Date accessSta // Only process ORIGINAL bundles to avoid affecting system bundles List originalBundles = item.getBundles("ORIGINAL"); if (originalBundles.isEmpty()) { - // Nothing to close. A package without ORIGINAL bitstreams discloses no file, however loudly its - // metadata claims an embargo, so this one really is a no-op and not a silent failure. + // Nothing this method could close: both SAF tools declare a scope of "ORIGINAL bitstreams only". + // Known limitation, deliberately left as it is: a contents file may route its files into another + // bundle with the "bundle:" marker, and such a package is archived with the collection + // default READ policy on those bitstreams however loudly its metadata claims an embargo. So this + // is not the guarantee that no file is disclosed, only the end of what an ORIGINAL-scoped tool + // has to say about it. logInfo("Embargo: No ORIGINAL bundles found, no embargo applied"); return; } diff --git a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java index 6491040ab65b..71d5203fd6c2 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java +++ b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java @@ -500,7 +500,18 @@ protected void processArchive(Context context, String sourceDirPath, String item if (!isTest) { Item item = itarch.getItem(); if (syncEmbargoPolicies) { - this.syncEmbargoPolicies(context, item); + try { + this.syncEmbargoPolicies(context, item); + } catch (Exception embargoFailure) { + // The catch below only prints the exception, and a printed exception is an exit + // code of 0. An embargo synchronisation that died half way has already re-dated + // the surviving Anonymous READ policy of a bitstream - the duplicate deletion + // that follows it is the last thing to run - and context.complete() commits that + // state. Counting the failure is what stops a published file from being reported + // as a successful run. + embargoSyncFailures++; + throw embargoFailure; + } } itemService.update(context, item); //need to update before commit context.uncacheEntity(item); diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java index 5918c6dbd9ad..2301d471c499 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java @@ -26,6 +26,8 @@ import java.util.Date; import java.util.List; import java.util.Locale; +import java.util.Set; +import java.util.TreeSet; import java.util.stream.Collectors; import org.apache.commons.io.file.PathUtils; @@ -459,6 +461,187 @@ public void multipleEmbargoEndValuesUsesFirst() throws Exception { anonymousCanRead(bitstream)); } + // ----------------------------------------------------------------------------------------------- + // legacy dc.date.embargoend shapes + // ----------------------------------------------------------------------------------------------- + + /** + * The value shapes {@code DCDate} used to accept still have to work here, and have to mean the same day + * here as on the import path. {@code DCDate.toDate()} reported the first instant of a year or a + * month, so {@code 2099} has always meant 1 January 2099 and {@code 2027-05} 1 May 2027, never the end of + * the period; a timestamp was truncated to its UTC day. + * + *

The same SAF package is first fed to {@code dspace import} and later re-fed to {@code dspace + * itemupdate}, so a disagreement between the two tools is a file one of them closes and the other opens. + * The import-side mirror of this test is {@code EmbargoImportIT#testYearOnlyEmbargoEndIsFirstOfJanuary} + * and its two neighbours.

+ */ + @Test + public void legacyEmbargoEndShapesKeepTheirDcDateDay() throws Exception { + int nextYear = utcToday().getYear() + 1; + LocalDate tomorrow = utcToday().plusDays(1); + + // a bare year is 1 January of it - not 31 December, which would extend embargoes operators live with + assertLegacyEmbargoEndClosesTheFileUntil(String.valueOf(nextYear), LocalDate.of(nextYear, 1, 1)); + // a bare month is the 1st of it, for the same reason + assertLegacyEmbargoEndClosesTheFileUntil(nextYear + "-05", LocalDate.of(nextYear, 5, 1)); + // the shape every DSpace export writes; the time of day is dropped, the UTC day is the last closed day + assertLegacyEmbargoEndClosesTheFileUntil(tomorrow + "T00:00:00Z", tomorrow); + } + + /** + * A legacy shape whose day lies in the past publishes the file, and that is a decision, not an + * accident. {@code 2020} says the embargo ended in 2020, so the files are public - exactly as a written + * out {@code 2020-01-01} would be. Until the {@code DCDate} shapes were read again, such a value threw and + * the item was refused; that refusal was a side effect of strict parsing and not what the metadata says. + * + *

This is the one test in the class that watches a file being opened by a legacy value, so it asserts + * the whole outcome: one immediately effective policy, an anonymous visitor who really gets the file, and + * a run that reports no problem ({@link #runItemUpdate} checks the last part).

+ */ + @Test + public void legacyPastEmbargoEndPublishesOnPurpose() throws Exception { + LocalDate primingEnd = utcToday().plusYears(1); + int legacyPastYear = utcToday().getYear() - 5; + String legacyValue = String.valueOf(legacyPastYear); + LocalDate expectedEmbargoEnd = LocalDate.of(legacyPastYear, 1, 1); + LocalDate expectedStartDay = expectedEmbargoEnd.plusDays(1); + String scenario = "legacy dc.date.embargoend=" + legacyValue + " (DCDate day " + expectedEmbargoEnd + ")"; + + Item item = createItem("Legacy year-only embargo end in the past"); + Bitstream bitstream = createOriginalBitstream(item, "legacy-past.pdf"); + dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + assertFreshImportBaseline(bitstream); + + // The priming run is what makes this a publication rather than a no-op: it leaves the single dated + // policy that keeps the file closed, so opening it afterwards is a state change that can be observed. + runItemUpdate(item, dublinCore(item, "embargoedAccess", primingEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B - after priming itemupdate with a FUTURE dc.date.embargoend=" + primingEnd, bitstream); + assertFalse("fixture precondition: after the priming run with dc.date.embargoend=" + primingEnd + + " the file has to be closed, otherwise the legacy value below opens nothing." + + diagnostics, + anonymousCanRead(bitstream)); + + runItemUpdate(item, dublinCore(item, "openAccess", legacyValue)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP C - after itemupdate with " + scenario, bitstream); + + assertEmbargoEndStored(item, legacyValue); + + ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy(scenario, bitstream); + assertNormalisedEmbargoPolicy(scenario, policy, expectedStartDay); + + assertTrue("dc.date.embargoend=" + legacyValue + " is the year " + legacyPastYear + ", i.e. an embargo" + + " that ended on " + expectedEmbargoEnd + ", so resource policy #" + policy.getID() + + " (start=" + policy.getStartDate() + ") has to be date-valid already." + diagnostics, + resourcePolicyService.isDateValid(policy)); + assertTrue("An embargo that ended in " + legacyPastYear + " publishes the file. This is the intended" + + " reading of a bare year and not a parsing accident: the ORIGINAL bitstream has to be" + + " readable by anonymous visitors." + diagnostics, + anonymousCanRead(bitstream)); + } + + /** + * Reading the {@code DCDate} shapes is not the same as swallowing what {@code DCDate} swallowed. + * {@code SimpleDateFormat} took {@code 2099garbage} for the year 2099, and {@code DCDate} ignored a numeric + * UTC offset instead of applying it - both move an embargo boundary silently. They are refused, and a + * refusal has to leave every policy exactly where it was and fail the run: an operator scripting + * {@code itemupdate} sees nothing but the exit code. + */ + @Test + public void unparseableLegacyLookalikeLeavesPoliciesUntouched() throws Exception { + int nextYear = utcToday().getYear() + 1; + + // trailing garbage after a year that SimpleDateFormat used to ignore + assertEmbargoEndIsRefused(nextYear + "garbage"); + // a numeric UTC offset: DCDate read this as midnight UTC, i.e. two hours off, and never said so + assertEmbargoEndIsRefused(nextYear + "-05-01T00:00:00+02:00"); + } + + /** + * One legacy value that has to close the file until exactly the day {@code DCDate} mapped it to. + * + * @param legacyValue raw {@code dc.date.embargoend} as a legacy SAF package writes it + * @param expectedEmbargoEnd last closed day {@code DCDate} mapped that value to; has to be in the future + */ + private void assertLegacyEmbargoEndClosesTheFileUntil(String legacyValue, LocalDate expectedEmbargoEnd) + throws Exception { + LocalDate expectedStartDay = expectedEmbargoEnd.plusDays(1); + String scenario = "legacy dc.date.embargoend=" + legacyValue + " (DCDate day " + expectedEmbargoEnd + ")"; + + assertTrue("test bug [" + scenario + "]: the expected embargo end day has to lie in the future, or the" + + " scenario silently turns into the expired-embargo one.", + expectedStartDay.isAfter(utcToday())); + + Item item = createItem("Legacy embargo end " + legacyValue); + Bitstream bitstream = createOriginalBitstream(item, "legacy.pdf"); + dump("STEP A [" + legacyValue + "] - fresh SAF import, before any itemupdate", bitstream); + assertFreshImportBaseline(bitstream); + + runItemUpdate(item, dublinCore(item, "embargoedAccess", legacyValue)); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B [" + legacyValue + "] - after itemupdate", bitstream); + + assertEmbargoEndStored(item, legacyValue); + + ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy(scenario, bitstream); + assertNormalisedEmbargoPolicy(scenario, policy, expectedStartDay); + + assertFalse("[" + scenario + "] resource policy #" + policy.getID() + " must not be date-valid yet." + + diagnostics, + resourcePolicyService.isDateValid(policy)); + assertFalse("[" + scenario + "] the embargo ends on " + expectedEmbargoEnd + ", which is in the future," + + " so an anonymous visitor must NOT be able to download the ORIGINAL bitstream." + + diagnostics, + anonymousCanRead(bitstream)); + } + + /** + * One value that has to be refused: the policies of an embargoed file stay byte for byte what they were and + * the run counts a failure, so {@code ItemUpdate.main()} exits non-zero. + * + * @param rejectedValue raw {@code dc.date.embargoend} that no accepted shape matches + */ + private void assertEmbargoEndIsRefused(String rejectedValue) throws Exception { + LocalDate primingEnd = utcToday().plusYears(1); + LocalDate primingStartDay = primingEnd.plusDays(1); + String scenario = "unparseable dc.date.embargoend=" + rejectedValue; + + Item item = createItem("Unparseable embargo end " + rejectedValue); + Bitstream bitstream = createOriginalBitstream(item, "unparseable.pdf"); + assertFreshImportBaseline(bitstream); + + runItemUpdate(item, dublinCore(item, "embargoedAccess", primingEnd.toString())); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP A [" + rejectedValue + "] - embargoed until " + primingEnd, bitstream); + assertFalse("fixture precondition [" + scenario + "]: the file has to be closed before the unparseable" + + " value is fed in." + diagnostics, anonymousCanRead(bitstream)); + + Set idsBefore = allPolicyIds(bitstream); + + String consoleOutput = runItemUpdate(item, dublinCore(item, "embargoedAccess", rejectedValue), 1); + item = context.reloadEntity(item); + bitstream = context.reloadEntity(bitstream); + dump("STEP B [" + rejectedValue + "] - after itemupdate with the unparseable value", bitstream); + + assertEquals("[" + scenario + "] the set of resource policy ids changed, so policies were deleted and/or" + + " re-created although an unreadable date is no instruction at all. Console output" + + " was:\n" + consoleOutput + diagnostics, + idsBefore, allPolicyIds(bitstream)); + + ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy(scenario, bitstream); + assertEquals("[" + scenario + "] the surviving policy was re-dated although the value could not be read." + + " Whatever the tool cannot parse it must not act on." + diagnostics, + primingStartDay, toLocalDate(policy.getStartDate())); + assertFalse("[" + scenario + "] the embargoed file became publicly readable after an unreadable" + + " dc.date.embargoend." + diagnostics, anonymousCanRead(bitstream)); + } + // ----------------------------------------------------------------------------------------------- // assertions // ----------------------------------------------------------------------------------------------- @@ -562,6 +745,18 @@ private boolean anonymousCanRead(Bitstream bitstream) throws Exception { } } + /** + * Every resource policy id of the bitstream, whatever the action. Comparing ids and not counts is the point: + * a policy deleted and immediately re-created keeps the count but loses its identity. + */ + private Set allPolicyIds(Bitstream bitstream) throws Exception { + Set ids = new TreeSet<>(); + for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream)) { + ids.add(policy.getID()); + } + return ids; + } + private List anonymousReadPolicies(Bitstream bitstream) throws Exception { return resourcePolicyService.find(context, bitstream, Constants.READ).stream() .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) @@ -688,6 +883,17 @@ private String firstMetadataValue(Item item, String element, String qualifier) { * reaches the failsafe output file as well. */ private String runItemUpdate(Item item, String dublinCoreContent) throws Exception { + return runItemUpdate(item, dublinCoreContent, 0); + } + + /** + * Same run, for the scenarios {@code itemupdate} has to refuse. + * + * @param expectedEmbargoSyncFailures number of embargo problems the run has to count; anything but 0 means + * {@code ItemUpdate.main()} would exit with 1 + */ + private String runItemUpdate(Item item, String dublinCoreContent, int expectedEmbargoSyncFailures) + throws Exception { Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); // Without suppress_undo, processArchive writes an undo archive as a SIBLING of the source directory. Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); @@ -720,11 +926,12 @@ private String runItemUpdate(Item item, String dublinCoreContent) throws Excepti context.uncacheEntity(item); String consoleOutput = captured.toString(StandardCharsets.UTF_8.name()); - assertEquals("this scenario is one itemupdate has to carry out, so it must not report an embargo" - + " synchronisation problem - ItemUpdate.main() would exit with " - + ItemUpdate.exitStatus(0, itemUpdate.embargoSyncFailures) + ". Console output was:\n" + assertEquals("wrong number of reported embargo synchronisation problems - ItemUpdate.main() would exit" + + " with " + ItemUpdate.exitStatus(0, itemUpdate.embargoSyncFailures) + " instead of " + + ItemUpdate.exitStatus(0, expectedEmbargoSyncFailures) + ", and the exit code is the" + + " only thing an operator scripting itemupdate ever sees. Console output was:\n" + consoleOutput, - 0, itemUpdate.embargoSyncFailures); + expectedEmbargoSyncFailures, itemUpdate.embargoSyncFailures); return consoleOutput; } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java index 9391f24b6efb..06063560960e 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java @@ -32,6 +32,7 @@ import org.apache.commons.io.file.PathUtils; import org.dspace.AbstractIntegrationTestWithDatabase; +import org.dspace.authorize.AuthorizeException; import org.dspace.authorize.ResourcePolicy; import org.dspace.authorize.factory.AuthorizeServiceFactory; import org.dspace.authorize.service.AuthorizeService; @@ -54,6 +55,7 @@ import org.dspace.content.service.MetadataFieldService; import org.dspace.content.service.MetadataSchemaService; import org.dspace.core.Constants; +import org.dspace.core.Context; import org.dspace.eperson.EPerson; import org.dspace.eperson.Group; import org.dspace.eperson.factory.EPersonServiceFactory; @@ -456,6 +458,47 @@ public void notArchivedItemIsUntouched() throws Exception { anonymousCanRead(bitstream)); } + /** + * The per-item {@code catch} of {@code processArchive} prints the exception and carries on, and a printed + * exception is an exit code of 0. That is harmless for an action that failed before it changed anything. + * It is not harmless for the embargo synchronisation, whose last step is deleting the duplicate + * Anonymous READ policies of a bitstream whose surviving policy has just been re-dated: if that step + * throws, {@code context.complete()} commits the re-dated policy anyway, so the file is open and the run + * reports success. The exit code is all the operator's script gets to see. + * + *

The failure is provoked by letting {@code applyEmbargoToItemBitstreams} throw after doing its + * work, which is exactly the state the duplicate deletion loop runs in.

+ */ + @Test + public void embargoSyncThatDiesHalfWayIsNotReportedAsSuccess() throws Exception { + Item item = createItem("Embargo sync dies half way"); + Bitstream bitstream = createEmbargoedBitstream(item, "half-way.pdf"); + + assertFalse("fixture precondition: the embargoed file must not be publicly readable" + + describe(bitstream), anonymousCanRead(bitstream)); + + ItemUpdate itemUpdate = new ItemUpdate() { + @Override + protected void applyEmbargoToItemBitstreams(Context context, Item item, Date startDate) + throws SQLException, AuthorizeException { + super.applyEmbargoToItemBitstreams(context, item, startDate); + throw new IllegalStateException("simulated failure while deleting the duplicate policies"); + } + }; + + Run run = runItemUpdate(itemUpdate, item, + dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + + bitstream = context.reloadEntity(bitstream); + + assertTrue("fixture precondition: the simulated failure has to strike AFTER the surviving policy was" + + " re-dated, otherwise this test does not describe the dangerous case at all. The file" + + " is readable now, and that is the state context.complete() would commit." + + describe(bitstream), + anonymousCanRead(bitstream)); + assertExitCode("embargo synchronisation that threw after re-dating a policy", 1, run); + } + /** * Runs one "ItemUpdate has to keep its hands off" scenario end to end and returns the reloaded item. * @@ -514,6 +557,14 @@ private void assertUntouched(String scenario, Set idsBefore, List Date: Wed, 19 Aug 2026 09:43:12 +0200 Subject: [PATCH 05/10] VSB-TUO/Review: address Copilot findings on PR #1415 ACCEPTED * An Anonymous READ policy that carries an END date is no longer synchronised (ItemUpdate.applyEmbargoToItemBitstreams). Such a policy is a "lease" - the access condition of access-conditions.xml with groupName=Anonymous and hasEndDate=true, written by the submission UI, by a REST access condition patch and by dspace bulk-access-control - and it means "public now, closed again on that day". dc.date.embargoend says nothing about that day, so both ways of synchronising it decide access on the operator's behalf: keeping the end date next to a fresh start date can close the file for good (measured on the mutant: start=2027-08-20 end=2027-02-19), and losing it publishes for ever a file that was to close itself. The bitstream is therefore left untouched, the offending policy is named on the console together with the bulk-access-control hint, and embargoSyncFailures is incremented so the run exits 1. Adjustment to the proposal: EVERY Anonymous READ policy of the bitstream is examined, not only the survivor. A lease has no start date, so selectSurvivorPolicy prefers a dated embargo policy next to it and the lease falls into the deletion loop - deleting it is exactly what drops the end date. The mutant proves it: policy ids went from [202, 203] to [202]. Copilot's own suggestion (normalise the end date, i.e. set it to null) was NOT implemented: clearing an end date widens access, which this branch forbids. * EmbargoLifecycleIT class javadoc corrected. It claimed the life cycle ends by "lifting it again by dropping dc.date.embargoend", the exact opposite of the contract and of what removingEmbargoMetadataLeavesPoliciesUntouched() asserts. The absence of the field is no instruction; an embargo is ended by a dc.date.embargoend in the past. Grepped the other embargo ITs and ItemUpdate itself for the same stale claim - no other occurrence. ADJUSTED * @Temporal(DATE) / policy start date (ItemUpdate:808, ItemImportServiceImpl:2641) - production code deliberately unchanged, documentation and test added instead. Copilot is right that resourcepolicy.start_date is a DATE column behind a @Temporal(DATE) field and that only the calendar day survives; on PostgreSQL in a JVM zone behind UTC the stored day even shifts back one day. But the value written here is byte-for-byte what core DSpace writes for the same day (DCDate.toDate() parses date-only values in UTC; the REST and submission layer goes through TimeHelpers.toMidnightUTC), so the defect belongs to the mapping and hits every DSpace embargo path identically. Changing only these two lines to local midnight would desynchronise itemupdate/itemimport from REST, from bulk-access-control and from the core embargo lifter on the same repository, and it would be measurably worse here: with atStartOfDay(ZoneId.systemDefault()) the IT suite stores the day BEFORE the intended one (expected 2026-08-20, stored 2026-08-19 - eight failures in EmbargoDateBoundaryIT), i.e. every embargo would open a day early. The customer instance runs Europe/Prague, where the stored day is correct either way. hibernate.jdbc.time_zone cannot help: it is ignored for @Temporal(DATE). Both call sites now carry a comment recording why midnight UTC is used, that only the calendar day is stored, and that the negative-offset day shift is an upstream limitation - so this is not re-litigated in a sixth review round. startDateIsUtcMidnightNotServerZone was rewritten as startDateSurvivesTheDatabaseAsTheExpectedCalendarDay. It no longer stops at the in-session instant (which no reload ever sees): it commits, drops the Hibernate session, reads the policy back out of the DATE column and asserts the stored calendar day, that the policy is not date-valid while that day is ahead, and that it IS date-valid - and the file readable - once the day has arrived. The in-session midnight-UTC assertion is kept as the encoding contract. Its javadoc records the blind spot: the harness pins H2 to TIME ZONE=UTC, so no test in this class can reproduce the PostgreSQL day shift. * Atomicity of syncEmbargoPolicies (ItemUpdate:513) - behaviour unchanged, the misleading comment that invited the finding corrected. The premise does not hold. A DB error cannot commit a half-synchronised bitstream: Hibernate marks the transaction rollback-only for every RuntimeException that passes through a session call (SessionImpl.fireDelete / doFlush -> ExceptionConverterImpl.markForRollbackOnly), and HibernateDBConnection.commit() refuses to commit a MARKED_ROLLBACK transaction, so context.complete() writes the whole batch or nothing. What remains is a non-DB exception between the survivor re-dating and the duplicate deletion, and that state is never more open than the state the tool found - the only widening step is the survivor re-date, it runs first, and it writes what the item's own metadata instructs. embargoSyncFailures + exit 1 reports it, and the synchronisation is idempotent, so re-running the same SAF package is the repair. The old comment asserted unconditionally that "context.complete() commits that state"; it now says what actually happens. REJECTED * Aborting the whole run on a per-item embargo failure. There is no per-item rollback primitive in this DSpace (no savepoint support anywhere), Context is one transaction, and an abort would discard 499 correct items to protect against a state that is not a disclosure. It would also create a footgun: the undo archive and undo_*_command.sh are written to disk BEFORE the sync, and because the undo command carries -a dc.date.embargoend it re-enables embargo syncing - replaying it after an abort would push stale embargo dates onto items that were never touched. * Clearing the survivor's end date (see ACCEPTED, first item). IMPORT PATH Checked and deliberately not changed. ItemImportServiceImpl creates a brand new policy and never sets an end date; at import time the bitstream has no Anonymous READ policy to adopt, and installItem's addDefaultPoliciesNotInPlace skips the collection default because isAnIdenticalPolicyAlreadyInPlace matches on (dso, group, action) alone and already sees the created policy. So no lease can be adopted or deleted there. TESTS * EmbargoSafetyIT.leasedAnonymousReadPolicyIsUntouched - a bitstream whose only Anonymous READ policy has an end date, run with a future and with an expired dc.date.embargoend: policy ids and every policy value unchanged, embargoSyncFailures == 1 per run, console names bulk-access-control. * EmbargoSafetyIT.leaseNextToADatedEmbargoPolicyIsNotDeleted - lease next to a dated embargo policy: nothing deleted, nothing mutated, embargoSyncFailures == 1. * EmbargoDateBoundaryIT.startDateSurvivesTheDatabaseAsTheExpectedCalendarDay replaces startDateIsUtcMidnightNotServerZone (see above). Every new assertion was mutation-verified: with the end-date guard disabled both EmbargoSafetyIT tests fail, and with atStartOfDay(ZoneId.systemDefault()) the rewritten boundary test fails. Co-Authored-By: Claude Opus 5 (1M context) --- .../app/itemimport/ItemImportServiceImpl.java | 6 + .../org/dspace/app/itemupdate/ItemUpdate.java | 58 +++++- .../app/itemupdate/EmbargoDateBoundaryIT.java | 170 +++++++++++++----- .../app/itemupdate/EmbargoLifecycleIT.java | 11 +- .../app/itemupdate/EmbargoSafetyIT.java | 110 ++++++++++++ 5 files changed, 298 insertions(+), 57 deletions(-) diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java index a5663076d2a7..a0157a5debee 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java @@ -2638,6 +2638,12 @@ protected void processEmbargoMetadata(Context c, Item item) + " are as accessible as the collection says."); return; } + // ResourcePolicy.startDate is mapped @Temporal(DATE), so only the calendar day survives the + // database. Midnight UTC is what core DSpace writes for the same day (DCDate.toDate() parses + // date-only values in UTC, the REST and submission layer uses TimeHelpers.toMidnightUTC), so every + // embargo path of one repository stores the same day. Upstream limitation, deliberately not patched + // here: on a JVM whose zone is behind UTC the driver stores the previous day - equally true of all + // those paths, and hibernate.jdbc.time_zone has no effect on @Temporal(DATE). Date accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); if (hasEmbargoedAccess) { diff --git a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java index 71d5203fd6c2..386a8c925c8e 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java +++ b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java @@ -504,11 +504,14 @@ protected void processArchive(Context context, String sourceDirPath, String item this.syncEmbargoPolicies(context, item); } catch (Exception embargoFailure) { // The catch below only prints the exception, and a printed exception is an exit - // code of 0. An embargo synchronisation that died half way has already re-dated - // the surviving Anonymous READ policy of a bitstream - the duplicate deletion - // that follows it is the last thing to run - and context.complete() commits that - // state. Counting the failure is what stops a published file from being reported - // as a successful run. + // code of 0. A database error cannot leave half of the synchronisation behind: + // Hibernate marks the transaction rollback-only and HibernateDBConnection.commit() + // then commits nothing at all, so context.complete() writes the whole batch or + // none of it. What does get committed is a non-database failure raised between the + // re-dating of the surviving Anonymous READ policy and the deletion of the + // duplicates that follow it. Counting the failure is what stops such an item from + // being reported as a successful run; the synchronisation is idempotent, so + // re-running the same SAF package is the repair. embargoSyncFailures++; throw embargoFailure; } @@ -804,6 +807,12 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti // dc.date.embargoend is the inclusive last day of the embargo, so access starts the day after, at // midnight UTC. Calendar.getInstance() would use the server time zone and shift that boundary. + // ResourcePolicy.startDate is mapped @Temporal(DATE), so only the calendar day below survives the + // database. Midnight UTC is what core DSpace writes for the same day (DCDate.toDate() parses + // date-only values in UTC, the REST and submission layer uses TimeHelpers.toMidnightUTC), so every + // embargo path of one repository stores the same day. Upstream limitation, deliberately not patched + // here: on a JVM whose zone is behind UTC the driver stores the previous day - equally true of all + // those paths, and hibernate.jdbc.time_zone has no effect on @Temporal(DATE). LocalDate accessStartDay = embargoEndDay.plusDays(1); Date accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); @@ -824,7 +833,10 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti * *

Exactly one such policy is left behind per bitstream: a second, undated one would silently defeat the * embargo. A policy is never created - a bitstream without an {@code Anonymous}/{@code READ} policy was not - * public, and inventing one would widen access instead of re-dating it.

+ * public, and inventing one would widen access instead of re-dating it. A bitstream whose + * {@code Anonymous}/{@code READ} access expires by itself (a policy with an end date, as written by the + * {@code lease} access condition) is skipped: this tool has no instruction for that end date and both + * keeping and dropping it would decide access on the operator's behalf.

* * @param context DSpace context * @param item item whose ORIGINAL bitstreams are synchronised @@ -865,6 +877,25 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta continue; } + // A policy that carries an end date is a lease (public now, closed again on that day), not + // an embargo, and this tool is given no instruction about end dates. Re-dating one would + // either close the file for good (an end date before the new start date) or, once the other + // Anonymous READ policies are deleted below, drop the end date that was to close it again + // and publish the file for ever. Both are the operator's decision, so the bitstream is left + // exactly as it is. Every Anonymous READ policy is examined and not only the survivor: the + // ones that are not selected are the ones the deletion loop removes. + ResourcePolicy leasePolicy = firstPolicyWithEndDate(anonymousReadPolicies); + if (leasePolicy != null) { + prErr("Bitstream '" + bitstream.getName() + "' (" + bitstream.getID() + ") of item " + + itemLabel(item) + " has an " + Group.ANONYMOUS + " READ policy with an end" + + " date (policy #" + leasePolicy.getID() + ", rpName='" + + leasePolicy.getRpName() + "'), which this tool does not manage, so its" + + " embargo could not be synchronised and the bitstream is left untouched. " + + BULK_ACCESS_CONTROL_HINT); + embargoSyncFailures++; + continue; + } + ResourcePolicy survivor = selectSurvivorPolicy(anonymousReadPolicies); survivor.setStartDate(startDate); survivor.setRpType(ResourcePolicy.TYPE_CUSTOM); @@ -882,6 +913,21 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta } } + /** + * First policy of the list that expires by itself. + * + * @param anonymousReadPolicies the Anonymous READ policies of a single bitstream + * @return a policy with a non-null end date, or {@code null} when none of them has one + */ + protected ResourcePolicy firstPolicyWithEndDate(List anonymousReadPolicies) { + for (ResourcePolicy policy : anonymousReadPolicies) { + if (policy.getEndDate() != null) { + return policy; + } + } + return null; + } + /** * Pick the policy that has been in force the longest: the dated one with the oldest start date or, when none * of them is dated, the first one. Adopting the newest would resurrect an obsolete embargo date. diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java index 2301d471c499..924e6e8f6a73 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java @@ -11,6 +11,7 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotEquals; import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNotSame; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; @@ -345,38 +346,72 @@ public void pastEmbargoEndWithEmbargoedAccessOpensAccess() throws Exception { } /** - * The policy start date must be built as {@code Date.from(day.atStartOfDay(ZoneOffset.UTC).toInstant())}, - * i.e. midnight UTC of a calendar day, and never midnight in the JVM time zone as produced by - * {@code Calendar.getInstance()} or {@code java.sql.Date.valueOf(LocalDate)}. + * The only thing {@code resourcepolicy.start_date} can hold is a calendar day - it is a DATE column + * behind a {@code @Temporal(DATE)} field - and that stored day is what decides access from the next + * request on: {@code ResourcePolicyServiceImpl.isDateValid} compares "now" against the value that came + * back from the database, never against the instant this tool computed. * - *

The harness pins the JVM to Europe/Dublin, which is UTC+1 during Irish Summer Time, so the two - * candidates differ by exactly one hour. The embargo end date is therefore anchored to the next 1 July - - * still computed from today, so fully dynamic - which is guaranteed to fall inside Irish Summer Time and - * makes the difference observable all year round.

+ *

So the assertion that matters is a round trip. Synchronise, commit, drop the Hibernate session, read + * the policy back: the stored day has to be {@code dc.date.embargoend + 1}, the policy has to keep the + * file closed while that day is still ahead, and it has to be date-valid - and the file readable - once + * that day has arrived.

* - *

{@code syncEmbargoPolicies} is invoked directly rather than through {@code processArchive} on purpose: - * {@code processArchive} ends with {@code context.uncacheEntity(item)}, which evicts the bitstream policies, - * so they would come back from the day-granular {@code @Temporal(DATE)} column as a {@code java.sql.Date} - * and the exact instant could no longer be inspected.

+ *

The in-session instant is asserted too, before the commit: it must be midnight UTC and never + * midnight in the JVM time zone as produced by {@code Calendar.getInstance()} or + * {@code java.sql.Date.valueOf(LocalDate)}. Midnight UTC is what core DSpace writes for the same day + * ({@code DCDate.toDate()} for the embargo lifter, {@code TimeHelpers.toMidnightUTC} for REST and the + * submission UI), so every embargo path of one repository stores the same day. The closed leg anchors its + * end date to the next 1 July - still computed from today, so fully dynamic - which always falls inside + * Irish Summer Time (UTC+1), the zone the harness pins, so the two candidate instants are one hour apart + * all year round.

+ * + *

Known blind spot, for whoever extends this: the harness pins H2 to {@code TIME ZONE=UTC} + * ({@code local.cfg}), so no test in this class can detect that a JVM zone behind UTC makes + * PostgreSQL store the previous day. That is a property of {@code @Temporal(DATE)} shared by every DSpace + * embargo path - see the comment next to the start date computation in {@link ItemUpdate} - and guarding + * it needs a PostgreSQL backed test.

*/ @Test - public void startDateIsUtcMidnightNotServerZone() throws Exception { - LocalDate embargoEnd = nextIrishSummerTimeDay(); - LocalDate expectedStartDay = embargoEnd.plusDays(1); + public void startDateSurvivesTheDatabaseAsTheExpectedCalendarDay() throws Exception { + LocalDate futureEnd = nextIrishSummerTimeDay(); + LocalDate futureStartDay = futureEnd.plusDays(1); + assertNotEquals("fixture precondition: the JVM time zone (" + ZoneId.systemDefault() + ") must differ" + + " from UTC on " + futureStartDay + ", otherwise this test cannot tell midnight UTC" + + " apart from midnight in the server zone. The harness pins Europe/Dublin in" + + " AbstractDSpaceIntegrationTest.", + Date.from(futureStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()), + Date.from(futureStartDay.atStartOfDay(ZoneId.systemDefault()).toInstant())); + + // (a) the stored day is still ahead, so the file has to stay closed after the reload + assertStoredStartDaySurvivesRoundTrip(futureEnd, false); + // (b) the embargo ended yesterday, so the stored day is today and the policy has to be in force + assertStoredStartDaySurvivesRoundTrip(utcToday().minusDays(1), true); + } + + /** + * One leg of {@link #startDateSurvivesTheDatabaseAsTheExpectedCalendarDay()}. + * + *

{@code syncEmbargoPolicies} is invoked directly rather than through {@code processArchive} because + * the in-session instant is asserted before the commit, and {@code processArchive} ends with + * {@code context.uncacheEntity(item)}.

+ * + * @param embargoEnd value of {@code dc.date.embargoend} + * @param expectedReadable whether an anonymous visitor must be able to download the file once the policy + * has been read back from the database + */ + private void assertStoredStartDaySurvivesRoundTrip(LocalDate embargoEnd, boolean expectedReadable) + throws Exception { + LocalDate expectedStartDay = embargoEnd.plusDays(1); + String leg = "dc.date.embargoend=" + embargoEnd; Date expectedUtcMidnight = Date.from(expectedStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); Date serverZoneMidnight = Date.from(expectedStartDay.atStartOfDay(ZoneId.systemDefault()).toInstant()); - assertNotEquals("fixture precondition: the JVM time zone (" + ZoneId.systemDefault() + ") must differ from" - + " UTC on " + expectedStartDay + ", otherwise this test cannot tell midnight UTC apart" - + " from midnight in the server zone. The harness pins Europe/Dublin in" - + " AbstractDSpaceIntegrationTest.", - expectedUtcMidnight, serverZoneMidnight); - Item item = createItem("UTC midnight embargo", + Item item = createItem("Round trip embargo " + embargoEnd, "rights", "access", "embargoedAccess", "date", "embargoend", embargoEnd.toString()); - Bitstream bitstream = createOriginalBitstream(item, "utc-midnight.pdf"); - dump("STEP A - fresh SAF import, before any itemupdate", bitstream); + Bitstream bitstream = createOriginalBitstream(item, "round-trip-" + embargoEnd + ".pdf"); + dump("STEP A [" + leg + "] - fresh SAF import, before any itemupdate", bitstream); assertFreshImportBaseline(bitstream); Integer importedPolicyId = anonymousReadPolicies(bitstream).get(0).getID(); @@ -386,34 +421,73 @@ public void startDateIsUtcMidnightNotServerZone() throws Exception { } finally { context.restoreAuthSystemState(); } - dump("STEP B - after syncEmbargoPolicies with dc.date.embargoend=" + embargoEnd, bitstream); + dump("STEP B [" + leg + "] - after syncEmbargoPolicies, still inside the Hibernate session", bitstream); - ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy("dc.date.embargoend=" + embargoEnd, bitstream); - assertNotNull("resource policy #" + policy.getID() + " must carry a start date." + diagnostics, - policy.getStartDate()); - assertEquals("the inherited Anonymous/READ policy must be MUTATED in place, not deleted and recreated:" - + " a policy may never be removed before its replacement is stored, otherwise a failure" - + " between the two leaves the file with zero policies (HTTP 401)." + diagnostics, - importedPolicyId, policy.getID()); - - long actualMillis = policy.getStartDate().getTime(); - assertEquals("dc.date.embargoend=" + embargoEnd + " must yield a start date of exactly midnight UTC on " - + expectedStartDay + " (epochMillis=" + expectedUtcMidnight.getTime() + "). Midnight in" - + " the server time zone " + ZoneId.systemDefault() + " would be epochMillis=" + ResourcePolicy inSession = assertExactlyOneAnonymousReadPolicy(leg, bitstream); + assertEquals("[" + leg + "] the inherited Anonymous/READ policy must be MUTATED in place, not deleted" + + " and recreated: a policy may never be removed before its replacement is stored," + + " otherwise a failure between the two leaves the file with zero policies (HTTP 401)." + + diagnostics, + importedPolicyId, inSession.getID()); + assertNotNull("[" + leg + "] resource policy #" + inSession.getID() + " must carry a start date." + + diagnostics, + inSession.getStartDate()); + assertEquals(leg + " must yield a start date of exactly midnight UTC on " + expectedStartDay + + " (epochMillis=" + expectedUtcMidnight.getTime() + "). Midnight in the server time" + + " zone " + ZoneId.systemDefault() + " would be epochMillis=" + serverZoneMidnight.getTime() + ", which is what Calendar.getInstance() or" - + " java.sql.Date.valueOf(LocalDate) produce. Actual epochMillis=" + actualMillis + " (" - + new Date(actualMillis).toInstant().atZone(ZoneOffset.UTC) + ")." + diagnostics, - expectedUtcMidnight.getTime(), actualMillis); - - assertFalse("dc.date.embargoend=" + embargoEnd + " lies in the future, so an anonymous visitor must NOT" - + " be able to download the ORIGINAL bitstream." + diagnostics, - anonymousCanRead(bitstream)); - assertEquals("resource policy #" + policy.getID() + " must be normalised to rpType=" - + ResourcePolicy.TYPE_CUSTOM + "." + diagnostics, - ResourcePolicy.TYPE_CUSTOM, policy.getRpType()); - assertEquals("resource policy #" + policy.getID() + " must be normalised to rpName=\"" - + EMBARGO_POLICY_NAME + "\"." + diagnostics, - EMBARGO_POLICY_NAME, policy.getRpName()); + + " java.sql.Date.valueOf(LocalDate) produce. Actual epochMillis=" + + inSession.getStartDate().getTime() + " (" + + inSession.getStartDate().toInstant().atZone(ZoneOffset.UTC) + ")." + diagnostics, + expectedUtcMidnight.getTime(), inSession.getStartDate().getTime()); + + // Leave the session: commit what the synchronisation wrote and evict every cached entity, so the + // start date has to come back out of the DATE column instead of out of Hibernate's memory. That is + // the value the next request authorises against. + context.commit(); + context.uncacheEntities(); + // Everything the test itself holds is detached by now, the fixture fields of the class included, and + // the next leg creates its item in this very collection. + collection = context.reloadEntity(collection); + anonymousGroup = context.reloadEntity(anonymousGroup); + bitstream = context.reloadEntity(bitstream); + dump("STEP C [" + leg + "] - after commit + uncacheEntities, read back from the database", bitstream); + + ResourcePolicy stored = assertExactlyOneAnonymousReadPolicy(leg + ", read back from the database", + bitstream); + assertNotSame("[" + leg + "] fixture precondition: the policy has to be read back from the database," + + " but the identical instance came out of the Hibernate session, so this leg would" + + " prove nothing about the stored value." + diagnostics, + inSession, stored); + assertEquals("[" + leg + "] the policy id must survive the round trip." + diagnostics, + importedPolicyId, stored.getID()); + assertEquals("[" + leg + "] resourcepolicy.start_date is a DATE column, so the calendar day is the" + + " only part of the start date that survives - and it is the part that decides" + + " access. Expected " + expectedStartDay + " (dc.date.embargoend + 1 day), stored " + + stored.getStartDate() + "." + diagnostics, + expectedStartDay, toLocalDate(stored.getStartDate())); + assertNull("[" + leg + "] this tool must never write an end date: a policy that expires by itself" + + " would close the file again on that day." + diagnostics, + stored.getEndDate()); + assertNormalisedEmbargoPolicy(leg + ", read back from the database", stored, expectedStartDay); + + if (expectedReadable) { + assertTrue("[" + leg + "] access starts on " + expectedStartDay + ", which is not after " + + utcToday() + ", so the policy read back from the database has to be date-valid." + + diagnostics, + resourcePolicyService.isDateValid(stored)); + assertTrue("[" + leg + "] the embargo has expired, so an anonymous visitor has to be able to" + + " download the ORIGINAL bitstream after the round trip." + diagnostics, + anonymousCanRead(bitstream)); + } else { + assertFalse("[" + leg + "] access starts on " + expectedStartDay + ", which is still ahead of " + + utcToday() + ", so the policy read back from the database must not be date-valid." + + diagnostics, + resourcePolicyService.isDateValid(stored)); + assertFalse("[" + leg + "] the embargo is still running, so an anonymous visitor must NOT be able" + + " to download the ORIGINAL bitstream after the round trip." + diagnostics, + anonymousCanRead(bitstream)); + } } /** diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java index c0f542d90b10..a6f4da9764be 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java @@ -71,9 +71,14 @@ * *

Where {@code EmbargoPastDateIT} reproduces the single incident reported by the customer, this class * covers the whole life cycle of an embargo as it is really operated: setting it, re-running the very same - * SAF archive, lifting it again by dropping {@code dc.date.embargoend}, and doing all of that on bitstreams - * whose resource policies were written by the previous (PR #1313 / #1315) implementation and therefore still - * carry the legacy {@code rpName} values {@code "Standard Embargo"} and {@code "Special Case Embargo"}.

+ * SAF archive, ending it with a {@code dc.date.embargoend} that lies in the past, and doing all of that on + * bitstreams whose resource policies were written by the previous (PR #1313 / #1315) implementation and + * therefore still carry the legacy {@code rpName} values {@code "Standard Embargo"} and + * {@code "Special Case Embargo"}.

+ * + *

The absence of {@code dc.date.embargoend} is the opposite of an instruction to open the files, and + * {@code removingEmbargoMetadataLeavesPoliciesUntouched()} is what pins that down: a SAF package that simply + * does not carry the field leaves every resource policy exactly as it was.

* *

The binding rules exercised here are:

*
    diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java index 06063560960e..00ca020d299c 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java @@ -93,6 +93,13 @@ public class EmbargoSafetyIT extends AbstractIntegrationTestWithDatabase { */ private static final String BULK_ACCESS_CONTROL_HINT = "bulk-access-control"; + /** + * Access condition of access-conditions.xml that writes an {@code Anonymous}/{@code READ} policy with an + * END date ({@code groupName=Anonymous}, {@code hasEndDate=true}, {@code endDateLimit=+6MONTHS}): public + * now, closed again on that day. It is the one shape of Anonymous READ policy this tool refuses to touch. + */ + private static final String LEASE_POLICY_NAME = "lease"; + /** * Sentinel for {@link #deletePolicies(Bitstream, int)} meaning "every action", picked so it can never * collide with a real value of {@link Constants#actionText}. @@ -379,6 +386,84 @@ public void alreadyBrokenBitstreamWithZeroPoliciesStaysZero() throws Exception { assertExitCode("bitstream with zero policies", 1, run); } + /** + * An {@code Anonymous}/{@code READ} policy that carries an END date is a {@code lease}, not an embargo: + * access-conditions.xml declares it with {@code groupName=Anonymous} and {@code hasEndDate=true}, and it + * is written by the submission UI, by a REST access condition patch and by + * {@code dspace bulk-access-control}. It means "public now, closed again on that day", which is not + * something {@code dc.date.embargoend} says anything about. + * + *

    Both ways of synchronising it would decide access on the operator's behalf: keeping the end date + * next to a fresh start date can close the file for good (an end date that lies before the start date), + * and losing the end date publishes for ever a file that was supposed to close itself. So the bitstream + * is left exactly as it is, the operator is told which policy stopped the synchronisation, and the run + * reports a failure instead of a success.

    + */ + @Test + public void leasedAnonymousReadPolicyIsUntouched() throws Exception { + Item item = createItem("Leased thesis"); + Bitstream bitstream = createLeasedBitstream(item, "leased.pdf"); + + assertTrue("fixture precondition: a lease that has not expired yet makes the file publicly readable" + + describe(bitstream), anonymousCanRead(bitstream)); + + Set idsBefore = policyIds(bitstream); + List policiesBefore = policyFingerprints(bitstream); + + // The future-date branch is the one that writes policies, so it is where the end date would be lost. + Run futureRun = runItemUpdate(item, + dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); + + bitstream = context.reloadEntity(bitstream); + + assertUntouched("leased Anonymous READ policy, future embargo end", idsBefore, policiesBefore, bitstream); + assertTrue("ItemUpdate has to name '" + BULK_ACCESS_CONTROL_HINT + "' as the supported way to change an" + + " access condition it does not manage. Console output was:" + System.lineSeparator() + + futureRun.console, + futureRun.console.contains(BULK_ACCESS_CONTROL_HINT)); + assertExitCode("leased Anonymous READ policy, future embargo end", 1, futureRun); + + // An expired end date reaches the same mutation, only with a start date that has already passed. + Run pastRun = runItemUpdate(item, + dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + + bitstream = context.reloadEntity(bitstream); + + assertUntouched("leased Anonymous READ policy, expired embargo end", idsBefore, policiesBefore, bitstream); + assertExitCode("leased Anonymous READ policy, expired embargo end", 1, pastRun); + } + + /** + * The refusal has to consider every {@code Anonymous}/{@code READ} policy of the bitstream and not only + * the one that would be mutated. A leased policy has no start date, so {@code selectSurvivorPolicy} + * prefers the dated embargo policy next to it and the lease falls into the deletion loop - and deleting + * the lease is precisely what removes the end date that was to close the file again. + */ + @Test + public void leaseNextToADatedEmbargoPolicyIsNotDeleted() throws Exception { + Item item = createItem("Leased and embargoed thesis"); + Bitstream bitstream = createEmbargoedBitstream(item, "leased-and-embargoed.pdf"); + + context.turnOffAuthorisationSystem(); + addLeasePolicy(bitstream); + context.restoreAuthSystemState(); + bitstream = context.reloadEntity(bitstream); + + assertEquals("fixture precondition: the bitstream has to carry the dated embargo policy AND the lease" + + describe(bitstream), 2, anonymousReadPolicies(bitstream).size()); + + Set idsBefore = policyIds(bitstream); + List policiesBefore = policyFingerprints(bitstream); + + Run run = runItemUpdate(item, + dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); + + bitstream = context.reloadEntity(bitstream); + + assertUntouched("lease next to a dated embargo policy", idsBefore, policiesBefore, bitstream); + assertExitCode("lease next to a dated embargo policy", 1, run); + } + /** * Specification row 7. A blank end date is a broken export, not an instruction. Validation has to run * before anything is mutated, so the existing policies survive untouched. @@ -718,6 +803,31 @@ private Bitstream createEmbargoedBitstream(Item item, String name) throws Except return context.reloadEntity(bitstream); } + /** + * A bitstream whose {@code Anonymous}/{@code READ} access expires by itself, i.e. exactly what the + * {@code lease} access condition writes: no start date and an end date at most six months out. + */ + private Bitstream createLeasedBitstream(Item item, String name) throws Exception { + Bitstream bitstream = createOriginalBitstream(item, name); + + context.turnOffAuthorisationSystem(); + deletePolicies(bitstream, Constants.READ); + addLeasePolicy(bitstream); + context.restoreAuthSystemState(); + + return context.reloadEntity(bitstream); + } + + private void addLeasePolicy(Bitstream bitstream) throws Exception { + ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) + .withAction(Constants.READ) + .withDspaceObject(bitstream) + .withName(LEASE_POLICY_NAME) + .withPolicyType(ResourcePolicy.TYPE_CUSTOM) + .withEndDate(startOfDayUtc(LocalDate.now().plusMonths(6))) + .build(); + } + /** * Deletes policies one by one through {@code ResourcePolicyService#delete}, the same call the production * code uses, so the Hibernate session stays consistent (the bulk removal helpers issue an HQL delete and From 4193f13bc5715178b79685d99b633c1191147681 Mon Sep 17 00:00:00 2001 From: milanmajchrak Date: Wed, 19 Aug 2026 15:30:42 +0200 Subject: [PATCH 06/10] VSB-TUO/Docs: shorten the embargo code comments Cut the design essays and the review narration the previous commits left in the embargo code down to short notes about why the code does what it does. Javadoc is shortened in place, never removed. Phase banners, internal call chain traces and issue references are gone. Assertion messages are untouched: they are the diagnostics a failing test prints. No executable code changed. Verified with a Java lexer that strips comments and compares the token streams: all eleven files are token-identical to the previous commit. Co-Authored-By: Claude Opus 5 (1M context) --- .../itemimport/EmbargoMetadataException.java | 10 +- .../app/itemimport/ItemImportServiceImpl.java | 90 +++----- .../org/dspace/app/itemupdate/ItemUpdate.java | 100 +++----- .../dspace/app/util/SafEmbargoConstants.java | 17 +- .../dspace/app/util/SafEmbargoDateParser.java | 55 ++--- .../app/itemimport/EmbargoImportIT.java | 125 ++++------ .../app/itemupdate/EmbargoDateBoundaryIT.java | 214 +++++------------- .../app/itemupdate/EmbargoLifecycleIT.java | 138 ++++------- .../app/itemupdate/EmbargoPastDateIT.java | 32 +-- .../app/itemupdate/EmbargoSafetyIT.java | 182 ++++++--------- .../dspace/app/itemupdate/ItemUpdateIT.java | 63 +++--- 11 files changed, 322 insertions(+), 704 deletions(-) diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/EmbargoMetadataException.java b/dspace-api/src/main/java/org/dspace/app/itemimport/EmbargoMetadataException.java index 14ceccd988d2..7316eaaec81c 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/EmbargoMetadataException.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/EmbargoMetadataException.java @@ -9,13 +9,9 @@ /** * Thrown when a SAF package claims an embargo ({@code dc.rights.access=embargoedAccess} or - * {@code dc.date.embargoend}) that the import cannot turn into a resource policy. - * - *

    Checked on purpose. Every one of these conditions used to be a log line followed by a {@code return}, - * after which the item was archived anyway - and {@code installItem} then gave its bitstreams the collection's - * undated default READ policy, so the files were public although their own metadata says they are closed, with - * exit code 0. There is no correct way to swallow this exception: an embargo that cannot be written means the - * package has to be refused.

    + * {@code dc.date.embargoend}) that the import cannot turn into a resource policy. It is checked because the + * package has to be refused: an item archived without its embargo policy gets the collection default READ + * policy from {@code installItem} and its files become public. */ public class EmbargoMetadataException extends Exception { diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java index a0157a5debee..0a125852f78c 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java @@ -813,17 +813,13 @@ protected Item addItem(Context c, List mycollections, String path, // non-standard permissions List options = processContentsFile(c, myitem, itemPathDir, "contents"); - // Check for embargo metadata and set up embargo terms if needed. This has to happen on the common - // path, before the workflow is started as well as before installItem: an embargoed submission that - // reaches the workflow without its policy is given the collection's undated default READ policy the - // moment it is approved (ItemServiceImpl.addDefaultPoliciesNotInPlace) and is public from then on. - // The policy grants nothing while the item waits in the workflow - AuthorizeServiceImpl ignores - // TYPE_CUSTOM policies on a bitstream that belongs to no installed item (DS-2614). + // Check for embargo metadata and set up embargo terms if needed + // Runs before the workflow is started as well as before installItem: a submission that reaches the + // workflow without its policy is given the collection default READ policy the moment it is approved. try { processEmbargoMetadata(c, myitem); } catch (EmbargoMetadataException e) { - // The operator gets the package directory, which is the thing they have to fix; the item id - // means nothing to them and the import is rolled back anyway. + // The operator needs the package directory, not the item id: the package is what they fix. throw new EmbargoMetadataException("SAF package '" + itemname + "': " + e.getMessage(), e); } @@ -2560,33 +2556,21 @@ private void logError(String message, Exception e) { /** * Set up the ResourcePolicy based embargo of an item being imported, from its own metadata - * ({@code dc.rights.access}, {@code dc.date.embargoend}). - * - *

    Every path out of this method is either "the embargo policy is written" or "the import fails". What - * it must never do is return quietly on a package that claims an embargo: {@code addItem} archives the - * item a few lines later, {@code installItem} clones the collection undated default READ policy onto - * every bitstream that has none - and the files are public although their own metadata says - * {@code embargoedAccess}, with exit code 0. That is why the blanket {@code catch (Exception)} this method - * used to end with is gone, and why the failure cases below throw instead of logging and returning.

    - * - *

    Two scenarios produce an embargo:

    - *
      - *
    1. {@code dc.rights.access=embargoedAccess} together with {@code dc.date.embargoend}
      2. - *
    2. {@code dc.date.embargoend} on its own - same policy, logged as a special case
      3. - *
    + * ({@code dc.rights.access}, {@code dc.date.embargoend}). A package that claims an embargo either gets its + * policy or fails the import: an item archived without one is given the collection default READ policy by + * {@code installItem}, which publishes files whose own metadata says they are closed. * * @param c DSpace context * @param item item being imported, already carrying the metadata of the package * @throws SQLException if a database error occurs * @throws AuthorizeException if the policy may not be written - * @throws EmbargoMetadataException if the package claims an embargo that cannot be written. Never treat it - * as "then there is no embargo" - the item must not be archived. + * @throws EmbargoMetadataException if the package claims an embargo that cannot be written; the item must + * not be archived then */ protected void processEmbargoMetadata(Context c, Item item) throws SQLException, AuthorizeException, EmbargoMetadataException { if (isTest || item == null) { - // A test run creates no item and loads no metadata, so there is nothing to read and no file that - // could be disclosed. + // A test run creates no item, so there is no metadata to read and no file to disclose. return; } @@ -2604,8 +2588,7 @@ protected void processEmbargoMetadata(Context c, Item item) } if (embargoEndDates.size() > 1) { - // Same rule as itemupdate: the first value wins and the operator is told, because two embargo - // end dates are a data error only they can resolve. + // Two embargo end dates are a data error only the operator can resolve. logError("WARNING: Multiple dc.date.embargoend values found. Using first value only."); } @@ -2616,9 +2599,8 @@ protected void processEmbargoMetadata(Context c, Item item) + " could not be written must not be archived."); } - // All arithmetic is done in UTC calendar days: neither Calendar.getInstance() (server time zone) nor - // DCDate (lenient, rolls 2026-02-30 over into 2026-03-02) can decide a day boundary reliably. The - // shapes DCDate did accept are still accepted, see SafEmbargoDateParser. + // UTC calendar days throughout: the server time zone must not decide a day boundary, and DCDate is + // lenient enough to roll 2026-02-30 over into a real date. Same parser as itemupdate, so one day. LocalDate embargoEndDay; try { embargoEndDay = SafEmbargoDateParser.parseEmbargoEndDay(embargoEndDateStr); @@ -2628,9 +2610,8 @@ protected void processEmbargoMetadata(Context c, Item item) + " not the same as no embargo, so the package is refused instead of archived.", e); } - // dc.date.embargoend is the inclusive last day of the embargo, so access starts the day after. The - // "already passed" test has to run on that start day and not on the end day, otherwise an embargo - // ending today would be dropped although the file must still be closed today. + // dc.date.embargoend is the last day of the embargo, so the test below runs on the start day: an + // embargo ending today has to keep the file closed for the rest of today. LocalDate accessStartDay = embargoEndDay.plusDays(1); if (!accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { logInfo("Embargo: end date " + embargoEndDateStr + " has already passed, no embargo policy" @@ -2638,12 +2619,8 @@ protected void processEmbargoMetadata(Context c, Item item) + " are as accessible as the collection says."); return; } - // ResourcePolicy.startDate is mapped @Temporal(DATE), so only the calendar day survives the - // database. Midnight UTC is what core DSpace writes for the same day (DCDate.toDate() parses - // date-only values in UTC, the REST and submission layer uses TimeHelpers.toMidnightUTC), so every - // embargo path of one repository stores the same day. Upstream limitation, deliberately not patched - // here: on a JVM whose zone is behind UTC the driver stores the previous day - equally true of all - // those paths, and hibernate.jdbc.time_zone has no effect on @Temporal(DATE). + // ResourcePolicy.startDate is mapped @Temporal(DATE), so only the calendar day survives; midnight + // UTC is what the other DSpace embargo paths store for that same day. Date accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); if (hasEmbargoedAccess) { @@ -2655,8 +2632,8 @@ protected void processEmbargoMetadata(Context c, Item item) + "dc.rights.access=embargoedAccess"); logInfo("Embargo: Applying embargo based on end date only until " + embargoEndDateStr); } - // Both scenarios produce the same policy. They used to differ only in the rpName, and one of those two - // names did not fit the 30 character rpname column at all. + // Both scenarios produce the same policy; they used to differ only in an rpName, one of which did + // not fit the 30 character rpname column. applyEmbargoToItemBitstreams(c, item, accessStartDate, SafEmbargoConstants.EMBARGO_POLICY_NAME); } @@ -2676,15 +2653,9 @@ protected boolean hasEmbargoedAccess(Item item) { } /** - * Apply an embargo ResourcePolicy to all ORIGINAL bitstreams of the item. - * - *

    This is the import path, where the bitstreams do not have an Anonymous READ policy yet - they only - * get one from the collection default when installItem runs - so the policy is created here. ItemUpdate - * works on already archived items and mutates the existing policy instead; the two must not be merged.

    - * - *

    Nothing here is caught and logged away. A bitstream whose policy could not be written is exactly the - * bitstream {@code installItem} then hands the collection undated default READ policy to, so a swallowed - * failure here is a published file.

    + * Apply an embargo ResourcePolicy to all ORIGINAL bitstreams of the item. The import path creates the + * policy because the bitstreams have none until {@code installItem} copies the collection defaults, while + * {@code ItemUpdate} works on archived items and re-dates the policy that is already there. * * @param c DSpace context * @param item item being imported @@ -2707,12 +2678,8 @@ protected void applyEmbargoToItemBitstreams(Context c, Item item, Date accessSta // Only process ORIGINAL bundles to avoid affecting system bundles List originalBundles = item.getBundles("ORIGINAL"); if (originalBundles.isEmpty()) { - // Nothing this method could close: both SAF tools declare a scope of "ORIGINAL bitstreams only". - // Known limitation, deliberately left as it is: a contents file may route its files into another - // bundle with the "bundle:" marker, and such a package is archived with the collection - // default READ policy on those bitstreams however loudly its metadata claims an embargo. So this - // is not the guarantee that no file is disclosed, only the end of what an ORIGINAL-scoped tool - // has to say about it. + // Known limitation: a contents file can route its files into another bundle with the + // "bundle:" marker, and those bitstreams are outside the ORIGINAL scope of both SAF tools. logInfo("Embargo: No ORIGINAL bundles found, no embargo applied"); return; } @@ -2726,13 +2693,8 @@ protected void applyEmbargoToItemBitstreams(Context c, Item item, Date accessSta policy.setAction(Constants.READ); policy.setStartDate(accessStartDate); policy.setRpName(policyReason); - // TYPE_CUSTOM keeps the policy inert until the item is installed: AuthorizeServiceImpl - // skips custom policies on a bitstream that belongs to no installed item (DS-2614), so - // an item waiting in the workflow discloses nothing. What stops installItem from - // cloning the collection undated default READ policy next to this one is not the - // type but ItemServiceImpl.addDefaultPoliciesNotInPlace -> - // AuthorizeServiceImpl.isAnIdenticalPolicyAlreadyInPlace, which matches on - // (dso, group, action) alone and therefore already sees this policy. + // TYPE_CUSTOM grants nothing while the item is still in the workflow, since custom policies + // on a bitstream of an item that is not installed yet are skipped when access is checked. policy.setRpType(ResourcePolicy.TYPE_CUSTOM); // Add policy to bitstream existing policies diff --git a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java index 386a8c925c8e..fa50acaafcd6 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java +++ b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java @@ -406,8 +406,8 @@ public static void main(String[] argv) { } /** - * Exit code of a run. Embargo problems are reported per item and never abort the run, but they must not be - * reported as success either: an operator scripting {@code itemupdate} only ever sees the exit code. + * Exit code of a run. Embargo problems are reported per item without aborting the run, but a run that + * reported them must not exit as a success: a script around this tool sees the exit code only. * * @param status exit code the run has produced so far * @param embargoSyncFailures number of bitstreams/items whose embargo could not be synchronised @@ -503,15 +503,8 @@ protected void processArchive(Context context, String sourceDirPath, String item try { this.syncEmbargoPolicies(context, item); } catch (Exception embargoFailure) { - // The catch below only prints the exception, and a printed exception is an exit - // code of 0. A database error cannot leave half of the synchronisation behind: - // Hibernate marks the transaction rollback-only and HibernateDBConnection.commit() - // then commits nothing at all, so context.complete() writes the whole batch or - // none of it. What does get committed is a non-database failure raised between the - // re-dating of the surviving Anonymous READ policy and the deletion of the - // duplicates that follow it. Counting the failure is what stops such an item from - // being reported as a successful run; the synchronisation is idempotent, so - // re-running the same SAF package is the repair. + // The catch below only prints the exception, which would leave the run exiting 0. + // Synchronisation is idempotent, so re-running the SAF package repairs the item. embargoSyncFailures++; throw embargoFailure; } @@ -703,19 +696,9 @@ protected static boolean containsEmbargoField(String[] targetFields) { /** * Bring the {@code Anonymous}/{@code READ} resource policies of the ORIGINAL bitstreams in line with the - * embargo metadata of the item ({@code dc.rights.access}, {@code dc.date.embargoend}). - * - *

    The order of the three phases below is the whole point of this method. A policy whose start date lies - * in the future only postpones access and is therefore harmless, but deleting a policy - or writing a start - * date that has already passed - is a publishing operation. So every objection is raised first, the target - * state is computed second, and only a run that got that far may touch a single policy. Nothing is deleted - * before its replacement has been stored, which is why the existing policy is mutated rather than replaced: - * a failure between a delete and a create would leave the file with no policy at all, i.e. HTTP 401.

    - * - *

    Only a {@code dc.date.embargoend} that is actually present is an instruction. Its absence means the - * SAF package says nothing about the embargo of this item, and the policies are left exactly as they are - - * an embargo this tool never set is never lifted by it. A file is opened by writing a - * {@code dc.date.embargoend} that lies in the past.

    + * embargo metadata of the item ({@code dc.rights.access}, {@code dc.date.embargoend}). The metadata is + * validated before any policy is touched, and the surviving policy is re-dated instead of being replaced, + * so that a failure cannot leave a bitstream without any policy. * * @param context DSpace context * @param item item that has just been updated from the SAF archive @@ -723,7 +706,6 @@ protected static boolean containsEmbargoField(String[] targetFields) { * @throws AuthorizeException if the policy update is not permitted */ protected void syncEmbargoPolicies(Context context, Item item) throws SQLException, AuthorizeException { - // --- phase 1: objections ------------------------------------------------------------------------- if (item.isWithdrawn()) { prWarn("Item " + itemLabel(item) + " is withdrawn, its bitstream policies are left untouched." + " A withdrawn item must never regain a READ policy, or the takedown would undo itself" @@ -744,9 +726,8 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti if (EMBARGOED_ACCESS.equals(value)) { hasEmbargoedAccess = true; } else if (!OPEN_ACCESS.equals(value)) { - // restrictedAccess, metadataOnlyAccess or a value we do not know. A single such value blocks the - // whole item even next to an openAccess one: contradictory metadata is never resolved towards - // disclosure, and an unknown access right is not an invitation to guess. + // restrictedAccess, metadataOnlyAccess or an unknown value blocks the whole item, even next + // to an openAccess one: contradictory metadata is not resolved towards disclosure. prWarn("Item " + itemLabel(item) + " carries " + EMBARGO_FIELD_RIGHTS_ACCESS + "='" + accessRight.getValue() + "', which is not an embargo access right (" + OPEN_ACCESS + ", " + EMBARGOED_ACCESS + "), its bitstream policies are left" @@ -755,7 +736,6 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti } } - // --- phase 2: validate and compute the target state ---------------------------------------------- List embargoEndDates = itemService.getMetadata(item, "dc", "date", "embargoend", Item.ANY); if (embargoEndDates.isEmpty()) { @@ -766,13 +746,8 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti embargoSyncFailures++; return; } - // A missing dc.date.embargoend is no instruction at all, and it is never read as "lift the - // embargo". The embargo of an item may well have been set outside this tool - the submission - // access condition and dspace bulk-access-control both write exactly the policy that would be - // reopened here (Anonymous/READ, TYPE_CUSTOM, rpName "embargo") - while syncEmbargoPolicies runs - // for every item of a batch whose -a/-d fields mention an embargo field. A single SAF package - // without the field would therefore publish every embargoed file of that batch. - // The supported way to open a file is a dc.date.embargoend that lies in the past. + // A missing dc.date.embargoend is no instruction, not a request to lift the embargo: it may have + // been set outside this tool, and this method runs for every item of the batch. pr("Item " + itemLabel(item) + " has no " + EMBARGO_FIELD_DATE_END + ", resource policies left" + " untouched. To end an embargo, set " + EMBARGO_FIELD_DATE_END + " to a date in the past."); return; @@ -792,10 +767,8 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti LocalDate embargoEndDay; try { - // Strict parsing on purpose: DCDate rolls 2026-02-30 over into 2026-03-02 and would turn a typo - // into a real embargo date. The shapes DCDate accepted are still read, and read as the same day, - // so the SAF packages of this repository keep working - see SafEmbargoDateParser. The import side - // uses the same parser, or the same package would mean two different days in the two tools. + // Strict parsing: DCDate rolls 2026-02-30 over into 2026-03-02 and would turn a typo into a real + // embargo date. The import path uses the same parser, so a package means one day in both tools. embargoEndDay = SafEmbargoDateParser.parseEmbargoEndDay(embargoEndDateStr); } catch (DateTimeParseException e) { prErr("Invalid " + EMBARGO_FIELD_DATE_END + " '" + embargoEndDateStr + "' on item " @@ -805,43 +778,30 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti return; } - // dc.date.embargoend is the inclusive last day of the embargo, so access starts the day after, at - // midnight UTC. Calendar.getInstance() would use the server time zone and shift that boundary. - // ResourcePolicy.startDate is mapped @Temporal(DATE), so only the calendar day below survives the - // database. Midnight UTC is what core DSpace writes for the same day (DCDate.toDate() parses - // date-only values in UTC, the REST and submission layer uses TimeHelpers.toMidnightUTC), so every - // embargo path of one repository stores the same day. Upstream limitation, deliberately not patched - // here: on a JVM whose zone is behind UTC the driver stores the previous day - equally true of all - // those paths, and hibernate.jdbc.time_zone has no effect on @Temporal(DATE). + // dc.date.embargoend is the last day of the embargo, so access starts the day after at midnight UTC: + // startDate is mapped @Temporal(DATE) and the other DSpace embargo paths store that same day. LocalDate accessStartDay = embargoEndDay.plusDays(1); Date accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); if (!accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { - // An expired embargo is a publication, not a deletion. The real - already passed - start date is - // written, which makes the policy effective immediately. + // An expired embargo is a publication, not a deletion: the start date that has already passed + // makes the policy effective immediately. pr("Embargo of item " + itemLabel(item) + " already expired on " + embargoEndDay + ", its ORIGINAL bitstreams are public since " + accessStartDay + "."); } - // --- phase 3: mutate ----------------------------------------------------------------------------- applyEmbargoToItemBitstreams(context, item, accessStartDate); } /** * Write the target embargo state onto the {@code Anonymous}/{@code READ} policy of every ORIGINAL bitstream - * of the item. - * - *

    Exactly one such policy is left behind per bitstream: a second, undated one would silently defeat the - * embargo. A policy is never created - a bitstream without an {@code Anonymous}/{@code READ} policy was not - * public, and inventing one would widen access instead of re-dating it. A bitstream whose - * {@code Anonymous}/{@code READ} access expires by itself (a policy with an end date, as written by the - * {@code lease} access condition) is skipped: this tool has no instruction for that end date and both - * keeping and dropping it would decide access on the operator's behalf.

    + * of the item, leaving exactly one such policy per bitstream, as a second undated one would defeat the + * embargo. Where there is no such policy none is created: the bitstream was not public, and creating one + * would widen access instead of re-dating it. * * @param context DSpace context * @param item item whose ORIGINAL bitstreams are synchronised - * @param startDate day the files become publicly readable, never {@code null}; a day in the past makes the - * policy effective immediately + * @param startDate day the files become publicly readable; a day in the past takes effect immediately * @throws SQLException if a database error occurs * @throws AuthorizeException if the policy update is not permitted */ @@ -857,9 +817,8 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta for (Bundle bundle : item.getBundles(Constants.CONTENT_BUNDLE_NAME)) { for (Bitstream bitstream : bundle.getBitstreams()) { - // Deliberately located by (group, action) and not by rpName: policies written by earlier - // versions of this tool carry "Standard Embargo" or "Special Case Embargo" and have to be - // adopted and normalised instead of being left behind next to a new one. + // Located by (group, action) rather than by rpName: policies written under earlier names + // have to be adopted and normalised instead of being left behind next to a new one. List anonymousReadPolicies = new ArrayList<>(); for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream, Constants.READ)) { if (anonymousGroup.equals(policy.getGroup())) { @@ -877,13 +836,8 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta continue; } - // A policy that carries an end date is a lease (public now, closed again on that day), not - // an embargo, and this tool is given no instruction about end dates. Re-dating one would - // either close the file for good (an end date before the new start date) or, once the other - // Anonymous READ policies are deleted below, drop the end date that was to close it again - // and publish the file for ever. Both are the operator's decision, so the bitstream is left - // exactly as it is. Every Anonymous READ policy is examined and not only the survivor: the - // ones that are not selected are the ones the deletion loop removes. + // A policy with an end date is a lease, not an embargo: re-dating it would either close the + // file for good or drop the end date, and either way decide access for the operator. ResourcePolicy leasePolicy = firstPolicyWithEndDate(anonymousReadPolicies); if (leasePolicy != null) { prErr("Bitstream '" + bitstream.getName() + "' (" + bitstream.getID() + ") of item " @@ -902,8 +856,8 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta survivor.setRpName(SafEmbargoConstants.EMBARGO_POLICY_NAME); resourcePolicyService.update(context, survivor); - // Only now, with the replacement safely stored, may the duplicates go. Reference identity is - // used on purpose: ResourcePolicy.equals compares values, which the lines above just changed. + // The duplicates go only once the survivor is stored. Reference identity rather than + // equals(): ResourcePolicy.equals compares values, which the lines above just changed. for (ResourcePolicy policy : anonymousReadPolicies) { if (policy != survivor) { resourcePolicyService.delete(context, policy); diff --git a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java index 43d404c38bf0..212e6777da21 100644 --- a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java +++ b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java @@ -8,22 +8,15 @@ package org.dspace.app.util; /** - * Constants shared by the two SAF batch tools that write embargo resource policies: - * {@code dspace import} ({@link org.dspace.app.itemimport.ItemImportServiceImpl}) creates the policy on a - * freshly imported item, {@code dspace itemupdate} ({@link org.dspace.app.itemupdate.ItemUpdate}) later - * re-dates and normalises it. - * - *

    The two tools have to agree on the value, so it is declared once. When they drift apart the operator - * sees two different names for the same thing in the policy list of a bitstream.

    + * Constants of the embargo resource policies written by the SAF batch tools, declared once so that + * {@code dspace import} and {@code dspace itemupdate} cannot drift apart. */ public final class SafEmbargoConstants { /** - * Value written to {@code resourcepolicy.rpname} on every embargo policy created or adopted by the SAF - * tools. It is the {@code name} of the {@code embargoed} access condition in access-conditions.xml, which - * is what the submission UI and {@code dspace bulk-access-control} write, and it fits the 30 character - * {@code rpname} column - the previous "Special Case Embargo - No access rights metadata" was 48 - * characters and aborted the whole import on PostgreSQL. + * Value written to {@code resourcepolicy.rpname} on embargo policies. It is the name of the + * {@code embargoed} access condition, as written by the submission UI and {@code bulk-access-control}, and + * it fits the 30 character {@code rpname} column, which a longer name would overflow. */ public static final String EMBARGO_POLICY_NAME = "embargo"; diff --git a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java index 186ba4cca48a..4ef48bc8b576 100644 --- a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java +++ b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java @@ -23,40 +23,15 @@ * Turns the {@code dc.date.embargoend} value of a SAF package into the UTC calendar day on which the embargo * ends, for {@code dspace import} and {@code dspace itemupdate} alike. * - *

    Both tools used to read that value with {@link org.dspace.content.DCDate}, which accepts seven shapes and - * is lenient: {@code 2026-02-30} silently becomes 2 March and {@code 2026-13-01} becomes 1 January 2027, so a - * typo turns into a real - possibly future - embargo date. Parsing is therefore strict here, but it still has - * to accept the shapes {@code DCDate} accepted, or SAF packages that used to import would stop working. Each - * shape is mapped to exactly the day {@code DCDate} mapped it to (verified against the class itself):

    - * - * - * - * - * - * - * - * - * - *
    accepted values
    {@code 2027-05-01}1 May 2027
    {@code 2027-5-1}1 May 2027 - unpadded, {@code SimpleDateFormat} took it
    {@code 2027-05-01T00:00:00Z}, {@code ...T00:00:00}, {@code ...T00:00}, {@code ...T00}1 May 2027, the UTC day of the instant; the time of day is dropped
    {@code 2027-05}1 May 2027, not the end of the month
    {@code 2027}1 January 2027, not the end of the year
    - * - *

    The last two rows are the ones worth reading twice. {@code DCDate} keeps a granularity, but - * {@code toDate()} returns the first instant of that year or month, and the old code took that - * {@code Date} as the embargo end. So {@code 2027} has always meant "the embargo ends on 1 January 2027", the - * files open on 2 January 2027, and that reading is kept - widening it to 31 December would extend embargoes - * that operators have already been living with.

    - * - *

    What is deliberately not kept from {@code DCDate}: the lenient roll-over of impossible dates, - * trailing garbage ({@code SimpleDateFormat} read {@code 2099garbage} as the year 2099), and a numeric UTC - * offset, which {@code DCDate} did not really support either - it ignored the offset and read - * {@code 2027-05-01T00:00:00+02:00} as if it were UTC. All of those now throw, and every caller has to treat a - * throw as "refuse the package", never as "no embargo".

    + *

    Parsing is strict, unlike the {@link org.dspace.content.DCDate} both tools used before, which rolls + * {@code 2026-02-30} over into 2 March and so turns a typo into a real embargo date. The shapes + * {@code DCDate} accepted are still read, and read as the same day, so existing SAF packages keep working.

    */ public final class SafEmbargoDateParser { /** - * {@code yyyy-MM-dd'T'HH[:mm[:ss[.fff]]]['Z']}, the four full ISO shapes of {@code DCDate} plus the - * fractional seconds its prefix matching used to swallow. Everything is UTC, which is what the trailing - * {@code Z} says and what {@code DCDate} assumed for the shapes without it. + * {@code yyyy-MM-dd'T'HH[:mm[:ss[.fff]]]['Z']}, the ISO shapes {@code DCDate} accepted. Always read as + * UTC, which is what {@code DCDate} assumed for the shapes without a trailing {@code Z}. */ private static final DateTimeFormatter LEGACY_TIMESTAMP = new DateTimeFormatterBuilder() .append(DateTimeFormatter.ISO_LOCAL_DATE) @@ -70,9 +45,8 @@ public final class SafEmbargoDateParser { .toFormatter().withResolverStyle(ResolverStyle.STRICT); /** - * {@code yyyy-M-d} with unpadded month and day. {@code SimpleDateFormat} accepted {@code 2027-5-1} and - * meant 1 May 2027 by it, without any roll-over, so it is accepted here too - strictly, unlike - * {@code DCDate}: {@code 2027-2-30} is still rejected. + * {@code yyyy-M-d} with unpadded month and day, which {@code SimpleDateFormat} accepted and older SAF + * packages therefore contain. Strict all the same: {@code 2027-2-30} is rejected. */ private static final DateTimeFormatter UNPADDED_DATE = new DateTimeFormatterBuilder() .appendValue(ChronoField.YEAR, 4, 10, SignStyle.EXCEEDS_PAD) @@ -93,19 +67,18 @@ private SafEmbargoDateParser() { * The UTC calendar day on which the embargo ends, i.e. the last day the files stay closed. * * @param value raw {@code dc.date.embargoend}, surrounding whitespace is ignored - * @return the embargo end day, never {@code null} - * @throws DateTimeParseException if the value is none of the accepted shapes. It is never a licence to - * skip the embargo: a caller that cannot read the date does not know - * whether the item is embargoed, and has to refuse it. + * @return the embargo end day + * @throws DateTimeParseException if the value is none of the accepted shapes; a caller that cannot read + * the date has to refuse the package instead of assuming no embargo */ public static LocalDate parseEmbargoEndDay(String value) { String trimmed = StringUtils.trimToEmpty(value); try { - // yyyy-MM-dd, the shape everything written by DSpace itself has + // yyyy-MM-dd, the shape DSpace itself writes return LocalDate.parse(trimmed); } catch (DateTimeParseException notAnIsoDay) { - // one of the older shapes, or garbage - decided below + // an older shape or garbage, decided below } try { @@ -115,14 +88,14 @@ public static LocalDate parseEmbargoEndDay(String value) { } try { - // a month is a period; its embargo ends on its first day, as DCDate.toDate() reported it + // DCDate.toDate() reported a bare month as its first day, so it keeps meaning that day return YearMonth.parse(trimmed).atDay(1); } catch (DateTimeParseException notAYearMonth) { // ditto } try { - // and a year on 1 January, for the same reason + // and a bare year as 1 January, for the same reason return Year.parse(trimmed).atDay(1); } catch (DateTimeParseException notAYear) { // ditto diff --git a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java index 1bed51716a2b..b31e2f9d9395 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java @@ -84,9 +84,8 @@ public class EmbargoImportIT extends AbstractIntegrationTestWithDatabase { private static final String EMBARGOEND_DATE_PAST = "2020-01-01"; private static final String ITEM_TITLE = "Test Embargo Item"; /** - * The single rpName both SAF tools write. Deliberately repeated here instead of referencing - * {@code SafEmbargoConstants}: the value ends up in the database and must not change silently, and it has - * to fit the 30 character {@code resourcepolicy.rpname} column. + * The single rpName both SAF tools write. Repeated here rather than referenced, because the value ends up + * in the database and has to fit the 30 character {@code resourcepolicy.rpname} column. */ private static final String EMBARGO_POLICY_NAME = "embargo"; @@ -201,10 +200,8 @@ public void testStandardEmbargoImport() throws Exception { Bitstream bitstream = bitstreams.get(0); - // Counting is part of the assertion, not a detail: installItem clones the collection's undated - // DEFAULT_BITSTREAM_READ onto a bitstream that has no Anonymous READ policy yet, and such a second, - // undated policy would make the file downloadable throughout the embargo. Picking the first matching - // policy with findFirst() cannot see that. + // installItem clones the collection's undated DEFAULT_BITSTREAM_READ onto a bitstream without an + // Anonymous READ policy, and such a second policy would open the file throughout the embargo. List anonymousRead = anonymousReadPolicies(bitstream); assertEquals("exactly one Anonymous READ policy may remain: " + describe(bitstream), 1, anonymousRead.size()); @@ -226,11 +223,8 @@ public void testStandardEmbargoImport() throws Exception { } /** - * An embargo that has already expired must not be turned into a policy - but "no embargo policy" is only - * half the requirement. The original assertion ("no Anonymous policy carries a start date") is satisfied - * just as well by a bitstream that has no policy at all and answers HTTP 401, which is the failure mode - * this branch is fixing. The test therefore also asserts that the file really is publicly readable, which - * on the import path means the collection default policies installItem applies. + * Verifies that an embargo which has already expired produces no policy and leaves the file readable, + * which on the import path means the collection default policies applied by installItem. */ @Test public void testPastEmbargoDateNoPolicy() throws Exception { @@ -274,24 +268,15 @@ public void testPastEmbargoDateNoPolicy() throws Exception { assertTrue("Should not have embargo policy for past dates", !hasEmbargoPolicy); - // The point of not writing an expired embargo policy is that the file stays available. Zero policies - // would satisfy the assertion above and leave every download at HTTP 401. + // The assertion above is also satisfied by a bitstream with no policy at all, which is unreadable. assertTrue("An expired embargo end date must leave the bitstream readable, not policy-less", anonymousCanRead(bitstream)); } /** - * The regression this test exists for: {@code dspace import -a -w} puts the item into the workflow, and - * approving it calls {@code installItem}, which applies the collection's default policies. A bitstream - * that carries no embargo policy at that moment has no Anonymous READ policy at all, so - * {@code ItemServiceImpl.addDefaultPoliciesNotInPlace} clones the collection's undated - * DEFAULT_BITSTREAM_READ onto it - and the file is public from the second it is approved, while its - * metadata still says {@code embargoedAccess} with a future end date. - * - *

    The embargo policy therefore has to be created on the common path, before the workflow starts. It - * discloses nothing while the item waits for approval, because {@code AuthorizeServiceImpl} ignores - * {@code TYPE_CUSTOM} policies on a bitstream that belongs to no installed item (DS-2614) - which the - * assertion on the workflow item below pins down.

    + * Verifies that an embargo imported with {@code -w} survives approval. Approval calls {@code installItem}, + * which clones the collection's undated default onto any bitstream without an Anonymous READ policy, so + * the embargo policy has to exist before the workflow starts. */ @Test public void testWorkflowEmbargoSurvivesApproval() throws Exception { @@ -324,8 +309,8 @@ public void testWorkflowEmbargoSurvivesApproval() throws Exception { "-s", safDir.toString(), "-m", tempDir.toString() + "/mapfile.out" }; runDSpaceScript(args); - // itemService.findByMetadataField only returns archived items, and this one is deliberately not - // archived yet. The mapfile is what the operator gets instead: " ". + // findByMetadataField only returns archived items and this one is still in the workflow, so the + // mapfile is what identifies it: " ". Item item = itemFromMapfile(tempDir.toString() + "/mapfile.out"); assertFalse("fixture precondition: -w must leave the item in the workflow, not in the archive", item.isArchived()); @@ -340,8 +325,8 @@ public void testWorkflowEmbargoSurvivesApproval() throws Exception { bitstream = context.reloadEntity(bitstream); assertTrue("fixture precondition: approving the workflow item must archive it", item.isArchived()); - // The moment of the leak. Without the embargo policy the bitstream reaches installItem with no - // Anonymous READ policy, gets the collection default cloned onto it, and is public right here. + // Without the embargo policy the bitstream would reach installItem with no Anonymous READ policy and + // get the collection default cloned onto it. assertFalse("approving an embargoed submission published its files. dc.date.embargoend is " + EMBARGOEND_DATE_FUTURE + ", so the file has to stay closed: " + describe(bitstream), anonymousCanRead(bitstream)); @@ -364,11 +349,8 @@ public void testWorkflowEmbargoSurvivesApproval() throws Exception { } /** - * The "special case" branch: {@code dc.date.embargoend} without {@code dc.rights.access=embargoedAccess}. - * - *

    It had no test at all, which is why nobody noticed that it wrote the 48 character rpName - * "Special Case Embargo - No access rights metadata" into a {@code varchar(30)} column - on PostgreSQL - * that aborts the whole import, and the SQL error names the column, not the branch that produced it.

    + * Verifies the branch where {@code dc.date.embargoend} arrives without + * {@code dc.rights.access=embargoedAccess}: it embargoes the files under the same, short enough rpName. */ @Test public void testEmbargoEndWithoutAccessRightsStillEmbargoes() throws Exception { @@ -437,9 +419,8 @@ private Item itemFromMapfile(String mapfilePath) throws Exception { } /** - * A workflow item that outlives its test keeps a {@code cwf_pooltask} row referencing the collection's - * workflow group, so {@code AbstractBuilder.cleanupObjects()} cannot delete that group - and every - * following test in this class then fails in cleanup instead of where the real problem is. + * A workflow item that outlives its test keeps a {@code cwf_pooltask} row referencing the workflow group, + * which then blocks {@code AbstractBuilder.cleanupObjects()} from deleting that group. */ private void deleteRemainingWorkflowItems() throws Exception { if (context == null || !context.isValid()) { @@ -491,8 +472,8 @@ private List anonymousReadPolicies(Bitstream bitstream) throws E } /** - * Every resource policy of the bitstream, for failure messages - "the assertion failed" is not enough to - * tell an extra undated policy from a missing one. + * Every resource policy of the bitstream, for failure messages: an extra undated policy and a missing one + * are otherwise indistinguishable. */ private String describe(Bitstream bitstream) throws Exception { StringBuilder sb = new StringBuilder(System.lineSeparator()); @@ -517,7 +498,8 @@ private String describe(Bitstream bitstream) throws Exception { } /** - * What an anonymous visitor gets, with the test's own turnOffAuthorisationSystem calls temporarily unwound. + * Tells whether a visitor who is not logged in may read the bitstream, with the test's own + * turnOffAuthorisationSystem calls temporarily unwound. */ private boolean anonymousCanRead(Bitstream bitstream) throws Exception { EPerson savedUser = context.getCurrentUser(); @@ -582,14 +564,8 @@ public void testNoEmbargoMetadataNoPolicy() throws Exception { } /** - * The leak of this round, and the reason the old version of this test did not catch it: it asserted - * "no embargo policy", which is satisfied just as well by a bitstream that is wide open. An unparseable - * {@code dc.date.embargoend} used to be a log line and a {@code return}, after which the item was - * archived, {@code installItem} cloned the collection undated default READ policy onto the bitstream, and - * the file was public although its own metadata says {@code embargoedAccess} - with exit code 0. - * - *

    The value here is one no date parser accepts. The values that {@code DCDate} did accept are - * a different story and are still imported, see the three format tests below.

    + * Verifies that a {@code dc.date.embargoend} no date parser accepts is reported and leaves no readable + * file behind. The shapes {@code DCDate} did accept are still imported, see the format tests below. */ @Test public void testInvalidEmbargoDateFormat() throws Exception { @@ -597,9 +573,8 @@ public void testInvalidEmbargoDateFormat() throws Exception { } /** - * {@code DCDate} is lenient and reads 30 February as 2 March, i.e. it turns a typo into a real embargo - * date. Rejecting the value is right, but rejecting it and archiving the item anyway is the leak above, - * so this pins down both halves on the import path. + * Verifies that a date only a lenient parser would accept is refused; reading 30 February as 2 March turns + * a typo into a real embargo date. */ @Test public void testLenientRollOverEmbargoDateIsRefused() throws Exception { @@ -608,9 +583,8 @@ public void testLenientRollOverEmbargoDateIsRefused() throws Exception { } /** - * {@code embargoedAccess} without an end date is self-contradictory metadata. Importing it archives an - * item that says its files are closed while the collection default policies make them public, which is - * exactly the contradiction resolved in the direction of disclosure. + * Verifies that {@code embargoedAccess} without an end date is refused: archiving it would leave an item + * whose metadata says closed while the collection default policies make the files public. */ @Test public void testEmbargoedAccessWithoutEndDateIsRefused() throws Exception { @@ -624,8 +598,8 @@ public void testEmbargoedAccessWithoutEndDateIsRefused() throws Exception { } /** - * A present but empty {@code dc.date.embargoend} is a broken export. The field is an instruction to close - * the files, and an instruction that cannot be carried out must not end as "no embargo". + * Verifies that a present but empty {@code dc.date.embargoend} is refused rather than read as "no + * embargo"; it is a broken export. */ @Test public void testBlankEmbargoEndIsRefused() throws Exception { @@ -633,10 +607,8 @@ public void testBlankEmbargoEndIsRefused() throws Exception { } /** - * Backwards compatibility, first of three. {@code DCDate} accepted a bare year and - * {@code DCDate.toDate()} returned its first instant, so the old code has always read - * {@code 2099} as "the embargo ends on 1 January 2099" - not on 31 December. Widening it now would extend - * embargoes the operators have been living with, so the day is kept and only the fail-open half changed. + * Verifies that a bare year keeps the day {@code DCDate} mapped it to, 1 January; reading it as + * 31 December would extend embargoes that repositories already live with. */ @Test public void testYearOnlyEmbargoEndIsFirstOfJanuary() throws Exception { @@ -645,8 +617,7 @@ public void testYearOnlyEmbargoEndIsFirstOfJanuary() throws Exception { } /** - * Backwards compatibility, second of three: {@code yyyy-MM} is the first day of that month, again because - * that is the instant {@code DCDate.toDate()} returned. + * Verifies that {@code yyyy-MM} keeps the day {@code DCDate} mapped it to, the first of that month. */ @Test public void testYearMonthEmbargoEndIsFirstOfMonth() throws Exception { @@ -655,9 +626,7 @@ public void testYearMonthEmbargoEndIsFirstOfMonth() throws Exception { } /** - * Backwards compatibility, third of three: a full ISO timestamp. The time of day is dropped and the UTC - * day of the instant is the last closed day, which for the {@code T00:00:00Z} form every DSpace export - * writes is the same day and the same policy start date as before. + * Verifies that a full ISO timestamp is truncated to its UTC day, which is then the last closed day. */ @Test public void testIsoTimestampEmbargoEndIsTruncatedToUtcDay() throws Exception { @@ -666,14 +635,8 @@ public void testIsoTimestampEmbargoEndIsTruncatedToUtcDay() throws Exception { } /** - * The same class of bug as the parse failure, one layer down: writing the policy fails and the failure is - * caught, logged and forgotten. The bitstream then reaches {@code installItem} without an Anonymous READ - * policy and gets the collection undated default cloned onto it, i.e. the file is published by the very - * code path that was supposed to close it. - * - *

    The failure is injected instead of provoked: breaking the resourcepolicy table of a running test - * database would take the rest of the suite with it. The test class sits in the same package as the - * service, so the Spring-injected collaborators can be replaced by hand.

    + * Verifies that a policy which cannot be written stops the import instead of being logged and forgotten. + * The failure is injected because breaking the resourcepolicy table would take the rest of the suite down. */ @Test public void testFailureToWriteThePolicyIsNotSwallowed() throws Exception { @@ -691,13 +654,13 @@ public void testFailureToWriteThePolicyIsNotSwallowed() throws Exception { + " few lines later in addItem and installItem then hands its bitstreams the collection" + " undated default READ policy, so a swallowed failure here publishes the files."); } catch (Exception expected) { - // fail closed: the exception is the whole point, the import is rolled back by ItemImport + // fail closed: ItemImport rolls the import back } } /** - * Same again for the other collaborator: without the {@code Anonymous} group there is no embargo policy - * to create, and an item archived without one is public by collection default. + * Same for the other collaborator: without the {@code Anonymous} group no embargo policy can be created, + * and an item archived without one is public by collection default. */ @Test public void testMissingAnonymousGroupIsNotSwallowed() throws Exception { @@ -766,10 +729,8 @@ private Exception runImport(Path safDir) throws Exception { } /** - * Both halves of "fail closed" for a package whose {@code dc.date.embargoend} cannot be used: the - * operator is told (so the exit code is not 0), and no file of that package ends up readable. Asserting - * only the first half would pass on an import that leaves the files open, asserting only the second would - * pass on a silent import of nothing. + * Both halves of "fail closed" for a package whose {@code dc.date.embargoend} cannot be used: the failure + * is reported, and no file of that package ends up readable. */ private void assertBrokenEmbargoPackageIsRefused(String embargoEnd, String what) throws Exception { Path itemDir = safPackage("embargoedAccess", embargoEnd, "TEST CONTENT " + what); @@ -867,8 +828,8 @@ private String utcDay(Date date) { } /** - * Every archived item with this title and the policies of its ORIGINAL bitstreams, for failure messages. - * "The import did not fail" is not enough to tell a refused package from a published one. + * Every archived item with this title and the policies of its ORIGINAL bitstreams, for failure messages: + * a refused package and a published one are otherwise indistinguishable. */ private String describeArchived(String title) throws Exception { StringBuilder sb = new StringBuilder(System.lineSeparator()); diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java index 924e6e8f6a73..0b0d3ed068e1 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java @@ -65,25 +65,10 @@ import org.junit.Test; /** - * Date-boundary and baseline behaviour of the VSB-TUO embargo synchronisation in {@link ItemUpdate}. - * - *

    Every scenario models the customer workflow literally: a SAF archive is re-imported with - * {@code ItemUpdate -s SAFDIR -d dc.rights.access -d dc.date.embargoend -a dc.rights.access - * -a dc.date.embargoend}, which is what makes {@code processArchive} call {@code syncEmbargoPolicies}.

    - * - *

    The binding rules under test (VSB-TUO embargo specification):

    - *
      - *
    • the resulting {@code Anonymous}/{@code READ} policy on every ORIGINAL bitstream starts at - * {@code dc.date.embargoend + 1 day} at midnight UTC, because {@code dc.date.embargoend} - * is the inclusive last day of the embargo;
      • - *
    • there is always exactly one such policy - never zero (the file would answer HTTP 401) and never - * two (an undated one would silently neutralise the embargo);
      • - *
    • the policy is normalised to {@code rpType=TYPE_CUSTOM} and {@code rpName="embargo"};
      • - *
    • an embargo end date that already lies in the past is a publication, not a deletion.
      • - *
    - * - *

    All dates are derived from {@code LocalDate.now(ZoneOffset.UTC)}, never hard-coded, so the suite cannot - * become a time bomb (lesson of PR #1359) and cannot straddle the UTC/Europe-Dublin day boundary.

    + * Date-boundary behaviour of the embargo synchronisation in {@link ItemUpdate}: which calendar day the + * resulting {@code Anonymous}/{@code READ} policy starts on, that exactly one such policy is left behind, and + * that an embargo end date already in the past opens the file. All dates are derived from + * {@code LocalDate.now(ZoneOffset.UTC)} so the suite cannot expire. */ public class EmbargoDateBoundaryIT extends AbstractIntegrationTestWithDatabase { @@ -147,8 +132,8 @@ public void destroy() throws Exception { } /** - * Row 1 of the specification: a future {@code dc.date.embargoend} closes the file and leaves exactly one - * normalised {@code Anonymous}/{@code READ} policy starting the day after the embargo end date. + * Verifies that a future {@code dc.date.embargoend} closes the file and leaves one normalised + * {@code Anonymous}/{@code READ} policy starting the day after the embargo end date. */ @Test public void futureEmbargoEndBlocksAccess() throws Exception { @@ -179,9 +164,8 @@ public void futureEmbargoEndBlocksAccess() throws Exception { } /** - * Row 2 of the specification: {@code dc.date.embargoend} is the inclusive last day of the embargo. - * When it equals today the file must still be closed today and open only tomorrow, so the policy start date - * is tomorrow - the "+1 day" has to be applied before, not after, any past/future comparison. + * Verifies that {@code dc.date.embargoend} is the inclusive last day: an end date of today keeps the file + * closed today and opens it tomorrow. */ @Test public void embargoEndTodayStillBlocksToday() throws Exception { @@ -214,8 +198,8 @@ public void embargoEndTodayStillBlocksToday() throws Exception { } /** - * Row 3 of the specification: an embargo that ended yesterday is expired, which means the file is published. - * The policy must survive with a start date of today, be date-valid and let anonymous visitors in. + * Verifies that an embargo which ended yesterday publishes the file, with the policy surviving and starting + * today. */ @Test public void embargoEndYesterdayOpensAccess() throws Exception { @@ -248,12 +232,8 @@ public void embargoEndYesterdayOpensAccess() throws Exception { } /** - * Main regression test - the exact record observed on dspace7-test.vsb.cz (item - * 4dde91f3-7078-4241-9938-9b8488623bb1: {@code dc.date.embargoend} in the past, - * {@code dc.rights.access=openAccess}, yet HTTP 401 on all 18 bitstreams). - * - *

    The priming run with a future date is mandatory: it is what replaces the inherited collection default - * with a single dated policy, so that the following past-date run has exactly one policy left to destroy.

    + * Verifies that an expired {@code dc.date.embargoend} with {@code dc.rights.access=openAccess} publishes the + * file. The priming run replaces the inherited collection default with a single dated policy. */ @Test public void pastEmbargoEndWithOpenAccessOpensAccess() throws Exception { @@ -266,7 +246,7 @@ public void pastEmbargoEndWithOpenAccessOpensAccess() throws Exception { dump("STEP A - fresh SAF import, before any itemupdate", bitstream); assertFreshImportBaseline(bitstream); - // (b) the state the customer confirmed as working: an embargo with a future end date + // priming run: an embargo with a future end date runItemUpdate(item, dublinCore(item, "embargoedAccess", futureEnd.toString())); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -275,7 +255,7 @@ public void pastEmbargoEndWithOpenAccessOpensAccess() throws Exception { + " not be publicly readable." + diagnostics, anonymousCanRead(bitstream)); - // (c) the operator lets the embargo expire: past end date, item declared openAccess + // the embargo expires: past end date, item declared openAccess runItemUpdate(item, dublinCore(item, "openAccess", pastEnd.toString())); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -301,9 +281,8 @@ public void pastEmbargoEndWithOpenAccessOpensAccess() throws Exception { } /** - * Same as {@link #pastEmbargoEndWithOpenAccessOpensAccess()} but the item keeps - * {@code dc.rights.access=embargoedAccess}. An expired embargo publishes the file regardless: the access - * right only names the licence regime, the end date decides when it lapses. + * Same as {@link #pastEmbargoEndWithOpenAccessOpensAccess()} with {@code dc.rights.access=embargoedAccess}: + * the access right names the licence regime, the end date decides when the embargo lapses. */ @Test public void pastEmbargoEndWithEmbargoedAccessOpensAccess() throws Exception { @@ -346,30 +325,8 @@ public void pastEmbargoEndWithEmbargoedAccessOpensAccess() throws Exception { } /** - * The only thing {@code resourcepolicy.start_date} can hold is a calendar day - it is a DATE column - * behind a {@code @Temporal(DATE)} field - and that stored day is what decides access from the next - * request on: {@code ResourcePolicyServiceImpl.isDateValid} compares "now" against the value that came - * back from the database, never against the instant this tool computed. - * - *

    So the assertion that matters is a round trip. Synchronise, commit, drop the Hibernate session, read - * the policy back: the stored day has to be {@code dc.date.embargoend + 1}, the policy has to keep the - * file closed while that day is still ahead, and it has to be date-valid - and the file readable - once - * that day has arrived.

    - * - *

    The in-session instant is asserted too, before the commit: it must be midnight UTC and never - * midnight in the JVM time zone as produced by {@code Calendar.getInstance()} or - * {@code java.sql.Date.valueOf(LocalDate)}. Midnight UTC is what core DSpace writes for the same day - * ({@code DCDate.toDate()} for the embargo lifter, {@code TimeHelpers.toMidnightUTC} for REST and the - * submission UI), so every embargo path of one repository stores the same day. The closed leg anchors its - * end date to the next 1 July - still computed from today, so fully dynamic - which always falls inside - * Irish Summer Time (UTC+1), the zone the harness pins, so the two candidate instants are one hour apart - * all year round.

    - * - *

    Known blind spot, for whoever extends this: the harness pins H2 to {@code TIME ZONE=UTC} - * ({@code local.cfg}), so no test in this class can detect that a JVM zone behind UTC makes - * PostgreSQL store the previous day. That is a property of {@code @Temporal(DATE)} shared by every DSpace - * embargo path - see the comment next to the start date computation in {@link ItemUpdate} - and guarding - * it needs a PostgreSQL backed test.

    + * Verifies that the start date is midnight UTC in session and comes back from the DATE column as the + * calendar day {@code dc.date.embargoend + 1}, which is the day later requests are authorised against. */ @Test public void startDateSurvivesTheDatabaseAsTheExpectedCalendarDay() throws Exception { @@ -382,19 +339,17 @@ public void startDateSurvivesTheDatabaseAsTheExpectedCalendarDay() throws Except Date.from(futureStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()), Date.from(futureStartDay.atStartOfDay(ZoneId.systemDefault()).toInstant())); - // (a) the stored day is still ahead, so the file has to stay closed after the reload + // stored day still ahead: the file stays closed after the reload assertStoredStartDaySurvivesRoundTrip(futureEnd, false); - // (b) the embargo ended yesterday, so the stored day is today and the policy has to be in force + // embargo ended yesterday: the stored day is today and the policy is in force assertStoredStartDaySurvivesRoundTrip(utcToday().minusDays(1), true); } /** - * One leg of {@link #startDateSurvivesTheDatabaseAsTheExpectedCalendarDay()}. - * - *

    {@code syncEmbargoPolicies} is invoked directly rather than through {@code processArchive} because - * the in-session instant is asserted before the commit, and {@code processArchive} ends with - * {@code context.uncacheEntity(item)}.

    + * One leg of {@link #startDateSurvivesTheDatabaseAsTheExpectedCalendarDay()}. Calls + * {@code syncEmbargoPolicies} directly because the in-session instant is asserted before the commit, and + * {@code processArchive} ends with {@code context.uncacheEntity(item)}. * * @param embargoEnd value of {@code dc.date.embargoend} * @param expectedReadable whether an anonymous visitor must be able to download the file once the policy @@ -441,13 +396,11 @@ private void assertStoredStartDaySurvivesRoundTrip(LocalDate embargoEnd, boolean + inSession.getStartDate().toInstant().atZone(ZoneOffset.UTC) + ")." + diagnostics, expectedUtcMidnight.getTime(), inSession.getStartDate().getTime()); - // Leave the session: commit what the synchronisation wrote and evict every cached entity, so the - // start date has to come back out of the DATE column instead of out of Hibernate's memory. That is - // the value the next request authorises against. + // Read the start date back out of the DATE column instead of out of Hibernate's memory - that is the + // value the next request authorises against. context.commit(); context.uncacheEntities(); - // Everything the test itself holds is detached by now, the fixture fields of the class included, and - // the next leg creates its item in this very collection. + // Everything the test holds is detached by now, the fixture fields included. collection = context.reloadEntity(collection); anonymousGroup = context.reloadEntity(anonymousGroup); bitstream = context.reloadEntity(bitstream); @@ -491,8 +444,8 @@ private void assertStoredStartDaySurvivesRoundTrip(LocalDate embargoEnd, boolean } /** - * Row 12 of the specification: several {@code dc.date.embargoend} values are a data error the operator has - * to see, but the run still completes and uses the first value. + * Verifies that several {@code dc.date.embargoend} values are reported to the operator while the run + * completes and follows the first value. */ @Test public void multipleEmbargoEndValuesUsesFirst() throws Exception { @@ -535,43 +488,26 @@ public void multipleEmbargoEndValuesUsesFirst() throws Exception { anonymousCanRead(bitstream)); } - // ----------------------------------------------------------------------------------------------- - // legacy dc.date.embargoend shapes - // ----------------------------------------------------------------------------------------------- - /** - * The value shapes {@code DCDate} used to accept still have to work here, and have to mean the same day - * here as on the import path. {@code DCDate.toDate()} reported the first instant of a year or a - * month, so {@code 2099} has always meant 1 January 2099 and {@code 2027-05} 1 May 2027, never the end of - * the period; a timestamp was truncated to its UTC day. - * - *

    The same SAF package is first fed to {@code dspace import} and later re-fed to {@code dspace - * itemupdate}, so a disagreement between the two tools is a file one of them closes and the other opens. - * The import-side mirror of this test is {@code EmbargoImportIT#testYearOnlyEmbargoEndIsFirstOfJanuary} - * and its two neighbours.

    + * Verifies that the value shapes {@code DCDate} accepted mean the same day here as on the import path: a + * bare year or month is its first day, a timestamp is truncated to its UTC day. */ @Test public void legacyEmbargoEndShapesKeepTheirDcDateDay() throws Exception { int nextYear = utcToday().getYear() + 1; LocalDate tomorrow = utcToday().plusDays(1); - // a bare year is 1 January of it - not 31 December, which would extend embargoes operators live with + // a bare year is 1 January of it, not 31 December, which would extend the embargo assertLegacyEmbargoEndClosesTheFileUntil(String.valueOf(nextYear), LocalDate.of(nextYear, 1, 1)); // a bare month is the 1st of it, for the same reason assertLegacyEmbargoEndClosesTheFileUntil(nextYear + "-05", LocalDate.of(nextYear, 5, 1)); - // the shape every DSpace export writes; the time of day is dropped, the UTC day is the last closed day + // the shape DSpace exports write; the time of day is dropped, the UTC day is the last closed day assertLegacyEmbargoEndClosesTheFileUntil(tomorrow + "T00:00:00Z", tomorrow); } /** - * A legacy shape whose day lies in the past publishes the file, and that is a decision, not an - * accident. {@code 2020} says the embargo ended in 2020, so the files are public - exactly as a written - * out {@code 2020-01-01} would be. Until the {@code DCDate} shapes were read again, such a value threw and - * the item was refused; that refusal was a side effect of strict parsing and not what the metadata says. - * - *

    This is the one test in the class that watches a file being opened by a legacy value, so it asserts - * the whole outcome: one immediately effective policy, an anonymous visitor who really gets the file, and - * a run that reports no problem ({@link #runItemUpdate} checks the last part).

    + * Verifies that a legacy shape whose day lies in the past publishes the file: one immediately effective + * policy, an anonymous visitor who gets the file, and a run that reports no problem. */ @Test public void legacyPastEmbargoEndPublishesOnPurpose() throws Exception { @@ -619,11 +555,8 @@ public void legacyPastEmbargoEndPublishesOnPurpose() throws Exception { } /** - * Reading the {@code DCDate} shapes is not the same as swallowing what {@code DCDate} swallowed. - * {@code SimpleDateFormat} took {@code 2099garbage} for the year 2099, and {@code DCDate} ignored a numeric - * UTC offset instead of applying it - both move an embargo boundary silently. They are refused, and a - * refusal has to leave every policy exactly where it was and fail the run: an operator scripting - * {@code itemupdate} sees nothing but the exit code. + * Verifies that values only a lenient parser would accept are refused, leaving every policy where it was + * and failing the run - both shapes would move an embargo boundary without saying so. */ @Test public void unparseableLegacyLookalikeLeavesPoliciesUntouched() throws Exception { @@ -636,10 +569,10 @@ public void unparseableLegacyLookalikeLeavesPoliciesUntouched() throws Exception } /** - * One legacy value that has to close the file until exactly the day {@code DCDate} mapped it to. + * One legacy value that has to close the file until the day {@code DCDate} mapped it to. * * @param legacyValue raw {@code dc.date.embargoend} as a legacy SAF package writes it - * @param expectedEmbargoEnd last closed day {@code DCDate} mapped that value to; has to be in the future + * @param expectedEmbargoEnd last closed day that value maps to; has to be in the future */ private void assertLegacyEmbargoEndClosesTheFileUntil(String legacyValue, LocalDate expectedEmbargoEnd) throws Exception { @@ -675,8 +608,8 @@ private void assertLegacyEmbargoEndClosesTheFileUntil(String legacyValue, LocalD } /** - * One value that has to be refused: the policies of an embargoed file stay byte for byte what they were and - * the run counts a failure, so {@code ItemUpdate.main()} exits non-zero. + * One value that has to be refused: the policies of an embargoed file stay as they were and the run counts + * a failure. * * @param rejectedValue raw {@code dc.date.embargoend} that no accepted shape matches */ @@ -716,13 +649,9 @@ private void assertEmbargoEndIsRefused(String rejectedValue) throws Exception { + " dc.date.embargoend." + diagnostics, anonymousCanRead(bitstream)); } - // ----------------------------------------------------------------------------------------------- - // assertions - // ----------------------------------------------------------------------------------------------- - /** - * A bitstream created by {@code BitstreamBuilder} inherits the collection DEFAULT_BITSTREAM_READ, which is - * byte-for-byte the state a fresh SAF import leaves behind: one Anonymous/READ policy without a start date. + * Asserts the state a fresh SAF import leaves behind: one Anonymous/READ policy without a start date, + * inherited from the collection DEFAULT_BITSTREAM_READ. */ private void assertFreshImportBaseline(Bitstream bitstream) throws Exception { List defaultBitstreamReadGroups = @@ -744,8 +673,8 @@ private void assertFreshImportBaseline(Bitstream bitstream) throws Exception { } /** - * Zero policies means HTTP 401 on the file; more than one means an undated policy can coexist with the dated - * one and silently neutralise the embargo. Exactly one is the only acceptable outcome. + * Asserts a single Anonymous/READ policy: zero leaves the file unreachable, more than one lets an undated + * policy coexist with the dated one and neutralise the embargo. */ private ResourcePolicy assertExactlyOneAnonymousReadPolicy(String scenario, Bitstream bitstream) throws Exception { @@ -758,10 +687,8 @@ private ResourcePolicy assertExactlyOneAnonymousReadPolicy(String scenario, Bits } /** - * The surviving policy has to be normalised: {@code TYPE_CUSTOM} (AuthorizeServiceImpl only honours custom - * policies on not-yet-installed items), {@code rpName="embargo"} (the access condition name used by - * access-conditions.xml, short enough for the 30 character column) and a start date of - * {@code dc.date.embargoend + 1 day}. + * Asserts the normalised shape of the surviving policy: {@code TYPE_CUSTOM}, {@code rpName="embargo"} and a + * start date of {@code dc.date.embargoend + 1 day}. */ private void assertNormalisedEmbargoPolicy(String scenario, ResourcePolicy policy, LocalDate expectedStartDay) { assertNotNull("After " + scenario + " resource policy #" + policy.getID() + " must carry a start date." @@ -788,18 +715,9 @@ private void assertEmbargoEndStored(Item item, String expectedEmbargoEnd) { expectedEmbargoEnd, firstMetadataValue(item, "date", "embargoend")); } - // ----------------------------------------------------------------------------------------------- - // policy inspection helpers - // ----------------------------------------------------------------------------------------------- - /** - * Answers the only question that matters: may a not-logged-in visitor download the file? - * - *

    The authorisation state is a stack, not a flag, and the builders as well as {@code processArchive} push - * and pop around themselves, so the depth at assertion time is not guaranteed to be zero. The stack is - * therefore drained (otherwise {@code AuthorizeServiceImpl.authorize} short-circuits and every read looks - * allowed) and restored afterwards. The current user is cleared as well, because {@code setUp} leaves the - * test EPerson logged in.

    + * Tells whether a visitor who is not logged in may read the bitstream. The authorisation state is a stack + * the builders push and pop, so it is drained first - otherwise every read looks allowed. */ private boolean anonymousCanRead(Bitstream bitstream) throws Exception { EPerson savedUser = context.getCurrentUser(); @@ -820,8 +738,8 @@ private boolean anonymousCanRead(Bitstream bitstream) throws Exception { } /** - * Every resource policy id of the bitstream, whatever the action. Comparing ids and not counts is the point: - * a policy deleted and immediately re-created keeps the count but loses its identity. + * Every resource policy id of the bitstream. Ids rather than counts, so a policy that was deleted and + * re-created is visible. */ private Set allPolicyIds(Bitstream bitstream) throws Exception { Set ids = new TreeSet<>(); @@ -866,9 +784,8 @@ private void dump(String label, Bitstream bitstream) throws Exception { } /** - * {@code ResourcePolicy.startDate} is mapped as {@code @Temporal(DATE)}, so once it has been round-tripped - * through the database it comes back as a day-granular {@code java.sql.Date}. Compare calendar days, never - * {@code Date} instances, across the UTC/Europe-Dublin boundary. + * Calendar day of a start date. {@code ResourcePolicy.startDate} is mapped as {@code @Temporal(DATE)}, so + * after a round trip through the database it comes back as a day-granular {@code java.sql.Date}. */ private LocalDate toLocalDate(Date date) { if (date instanceof java.sql.Date) { @@ -877,18 +794,14 @@ private LocalDate toLocalDate(Date date) { return date.toInstant().atZone(ZoneOffset.UTC).toLocalDate(); } - // ----------------------------------------------------------------------------------------------- - // fixture helpers - // ----------------------------------------------------------------------------------------------- - - /** Calendar "today" in UTC - the specification does all embargo arithmetic in UTC calendar days. */ + /** Calendar "today" in UTC - all embargo arithmetic is done in UTC calendar days. */ private LocalDate utcToday() { return LocalDate.now(ZoneOffset.UTC); } /** - * The next 1 July strictly after today, computed dynamically. Ireland observes Irish Summer Time (UTC+1) on - * that date every year, which is what makes midnight UTC and midnight in the server zone distinguishable. + * The next 1 July after today. Ireland is on UTC+1 then, which makes midnight UTC and midnight in the + * harness time zone distinguishable. */ private LocalDate nextIrishSummerTimeDay() { LocalDate today = utcToday(); @@ -941,17 +854,8 @@ private String firstMetadataValue(Item item, String element, String qualifier) { } /** - * Equivalent of {@code ItemUpdate -s SAFDIR -d dc.rights.access -d dc.date.embargoend -a dc.rights.access - * -a dc.date.embargoend}: an update whose target fields contain an embargo field, which is exactly what - * makes {@code processArchive} call {@code syncEmbargoPolicies}. - * - *

    {@code ItemUpdate.main} is deliberately not used - it ends in {@code System.exit} and would kill the - * failsafe JVM.

    - * - *

    Every scenario in this class is one {@code itemupdate} is supposed to carry out, so the helper also - * asserts the exit code the run would have produced: {@code embargoSyncFailures} is the only thing - * {@code main()} turns into a non-zero status, and a refusal that keeps the status at 0 is a silent - * failure for the operator's script.

    + * Runs itemupdate with both embargo fields as targets, the combination that triggers embargo + * synchronisation. {@code main()} is not used because it ends in {@code System.exit}. * * @return everything {@code ItemUpdate.pr()} printed during the run; the stream is teed, so the output still * reaches the failsafe output file as well. @@ -963,8 +867,8 @@ private String runItemUpdate(Item item, String dublinCoreContent) throws Excepti /** * Same run, for the scenarios {@code itemupdate} has to refuse. * - * @param expectedEmbargoSyncFailures number of embargo problems the run has to count; anything but 0 means - * {@code ItemUpdate.main()} would exit with 1 + * @param expectedEmbargoSyncFailures number of embargo problems the run has to count; anything but 0 makes + * {@code ItemUpdate.main()} exit with 1 */ private String runItemUpdate(Item item, String dublinCoreContent, int expectedEmbargoSyncFailures) throws Exception { diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java index a6f4da9764be..e78f8ae22abd 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java @@ -66,38 +66,17 @@ import org.junit.Test; /** - * Life cycle, legacy data migration and invariant tests for the VSB-TUO embargo synchronisation in - * {@link ItemUpdate}. - * - *

    Where {@code EmbargoPastDateIT} reproduces the single incident reported by the customer, this class - * covers the whole life cycle of an embargo as it is really operated: setting it, re-running the very same - * SAF archive, ending it with a {@code dc.date.embargoend} that lies in the past, and doing all of that on - * bitstreams whose resource policies were written by the previous (PR #1313 / #1315) implementation and - * therefore still carry the legacy {@code rpName} values {@code "Standard Embargo"} and - * {@code "Special Case Embargo"}.

    - * - *

    The absence of {@code dc.date.embargoend} is the opposite of an instruction to open the files, and - * {@code removingEmbargoMetadataLeavesPoliciesUntouched()} is what pins that down: a SAF package that simply - * does not carry the field leaves every resource policy exactly as it was.

    - * - *

    The binding rules exercised here are:

    - *
      - *
    • the survivor policy is located by {@code (group = Anonymous, action = READ)} and never by - * {@code rpName}, so policies written by the old code are picked up and normalised;
      • - *
    • the survivor is mutated, never deleted and recreated, so its {@code policy_id} is stable;
      • - *
    • an ORIGINAL bitstream that started with at least one READ policy always ends with at least one - - * whatever {@code dc.date.embargoend} contained;
      • - *
    • exactly one {@code Anonymous}/{@code READ} policy remains, so no immediate policy can survive next to - * a dated one and quietly defeat the embargo;
      • - *
    • bitstreams outside the ORIGINAL bundle are never touched - their policies belong to filter-media.
      • - *
    + * Life cycle of an embargo driven by {@link ItemUpdate}: setting it, re-running the same SAF archive, and + * ending it with a {@code dc.date.embargoend} in the past, including on policies written by earlier versions. + * Covers the invariants that one {@code Anonymous}/{@code READ} policy survives every run, that it is mutated + * rather than recreated, and that bitstreams outside the ORIGINAL bundle are left alone. */ public class EmbargoLifecycleIT extends AbstractIntegrationTestWithDatabase { /** Target policy name of the fix. Must stay within the 30 char {@code resourcepolicy.rpname} column. */ private static final String EMBARGO_POLICY_NAME = "embargo"; - /** Policy names written by the previous implementation and still present in the customer database. */ + /** Policy names written by earlier versions and still present in existing repositories. */ private static final String LEGACY_STANDARD_EMBARGO = "Standard Embargo"; private static final String LEGACY_SPECIAL_CASE_EMBARGO = "Special Case Embargo"; @@ -160,17 +139,8 @@ public void destroy() throws Exception { } /** - * A SAF package without {@code dc.date.embargoend} says nothing about the embargo of its item, and - * "nothing" is not "open the files". The policies have to come out of the run byte for byte identical - - * same {@code policy_id}, same start date, same name, same answer to "may an anonymous visitor download - * this". - * - *

    Why this is not merely conservative: {@code syncEmbargoPolicies} runs for every item of a batch as - * soon as the {@code -a}/{@code -d} fields mention an embargo field, and the survivor is located by - * {@code (Anonymous, READ)}. One package whose {@code dublin_core.xml} happens to lack the field would - * otherwise publish that item's files - and a batch is exactly where nobody looks at the individual - * packages. Opening a file is done by writing a {@code dc.date.embargoend} that lies in the past, which - * is a deliberate, per-item statement.

    + * Verifies that a SAF package without {@code dc.date.embargoend} leaves every policy as it was. A missing + * field says nothing about the embargo; an embargo is ended by an end date that lies in the past. */ @Test public void removingEmbargoMetadataLeavesPoliciesUntouched() throws Exception { @@ -180,7 +150,7 @@ public void removingEmbargoMetadataLeavesPoliciesUntouched() throws Exception { Bitstream bitstream = createOriginalBitstream(item, "thesis.pdf"); Integer importedPolicyId = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import").getID(); - // (a) operator embargoes the item + // the operator embargoes the item assertRunSucceeded("setting the embargo", runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); @@ -198,7 +168,7 @@ public void removingEmbargoMetadataLeavesPoliciesUntouched() throws Exception { Set policiesBefore = policySignatures(bitstream); - // (b) the next SAF package simply does not carry dc.date.embargoend + // the next SAF package does not carry dc.date.embargoend assertRunSucceeded("running without dc.date.embargoend", runItemUpdate(item, dublinCore(item, OPEN_ACCESS, null))); item = context.reloadEntity(item); @@ -219,15 +189,9 @@ public void removingEmbargoMetadataLeavesPoliciesUntouched() throws Exception { } /** - * The same rule seen from the side that makes it a data leak rather than a matter of taste: an embargo - * that {@code itemupdate} never set. - * - *

    The submission access condition and {@code dspace bulk-access-control} both write precisely this - * policy - {@code Anonymous}/{@code READ}, {@code TYPE_CUSTOM}, rpName {@code embargo}, future start date - - * and {@code syncEmbargoPolicies} finds its survivor by {@code (group, action)}, so it cannot tell that - * policy apart from one of its own. If the absence of {@code dc.date.embargoend} cleared the start date, - * a single {@code dspace itemupdate -a dc.date.embargoend} over a batch whose packages do not carry the - * field would publish every embargoed ORIGINAL bitstream in it.

    + * Verifies that an embargo itemupdate never set survives a run without {@code dc.date.embargoend}. The + * survivor is found by {@code (group, action)}, so a policy from the submission UI or from + * {@code bulk-access-control} is indistinguishable from one of itemupdate's own. */ @Test public void foreignEmbargoIsNeverLifted() throws Exception { @@ -274,9 +238,8 @@ public void foreignEmbargoIsNeverLifted() throws Exception { } /** - * Re-running the identical SAF archive - which happens every time an import job is retried - must be a - * no-op. The same {@code policy_id} has to come back, which is only possible if the policy is mutated - * rather than deleted and recreated. + * Verifies that re-running the identical SAF archive is a no-op: the same {@code policy_id} comes back, + * which only holds if the policy is mutated rather than deleted and recreated. */ @Test public void syncIsIdempotent() throws Exception { @@ -325,10 +288,9 @@ public void syncIsIdempotent() throws Exception { } /** - * The customer database is full of policies named {@code "Standard Embargo"} whose start date has already - * passed. Re-embargoing such a bitstream must find that policy by {@code (Anonymous, READ)}, reuse it and - * normalise its name. An implementation that looks the survivor up by {@code rpName == "embargo"} would - * create a second policy and leave the expired one in place, so the file would stay downloadable. + * Verifies that a legacy policy whose start date has passed is found by {@code (Anonymous, READ)}, reused + * and normalised. Looking the survivor up by {@code rpName} would leave the expired policy in place next + * to a new one, and the file would stay downloadable. */ @Test public void legacyRpNameThenFutureEmbargoIsEnforced() throws Exception { @@ -336,8 +298,8 @@ public void legacyRpNameThenFutureEmbargoIsEnforced() throws Exception { } /** - * Same as {@link #legacyRpNameThenFutureEmbargoIsEnforced()} for the second legacy name, written by the - * previous implementation whenever {@code dc.rights.access} was not {@code embargoedAccess}. + * Same as {@link #legacyRpNameThenFutureEmbargoIsEnforced()} for the second legacy name, written whenever + * {@code dc.rights.access} was not {@code embargoedAccess}. */ @Test public void legacySpecialCaseRpNameIsAlsoPickedUp() throws Exception { @@ -345,9 +307,8 @@ public void legacySpecialCaseRpNameIsAlsoPickedUp() throws Exception { } /** - * An item that never had an embargo carries a single immediate ({@code startDate == null}) policy. Putting - * it under embargo must consume that policy: leaving it next to a dated one would make the embargo a no-op - * because the immediate policy alone already grants anonymous READ (spec row #13). + * Verifies that embargoing a born-open item consumes its immediate ({@code startDate == null}) policy; + * left next to a dated one it would keep granting anonymous READ and the embargo would be a no-op. */ @Test public void bornOpenItemThenFutureEmbargoIsEnforced() throws Exception { @@ -388,10 +349,8 @@ public void bornOpenItemThenFutureEmbargoIsEnforced() throws Exception { } /** - * A bitstream that accumulated several {@code Anonymous}/{@code READ} policies - the classic mix of one - * immediate policy and the leftovers of earlier embargo runs - must end up with exactly one policy, taken - * from the pre-existing ones (spec row #13). Any surviving second policy would either defeat the embargo - * or resurrect an obsolete date. + * Verifies that accumulated {@code Anonymous}/{@code READ} policies collapse into one of the pre-existing + * ones; a second survivor would either defeat the embargo or resurrect an obsolete date. */ @Test public void duplicateAnonymousReadPoliciesCollapseToOne() throws Exception { @@ -438,13 +397,9 @@ public void duplicateAnonymousReadPoliciesCollapseToOne() throws Exception { } /** - * The single invariant the whole fix exists for: an ORIGINAL bitstream that had at least one READ policy - * before an {@code itemupdate} run must still have one afterwards, no matter what - * {@code dc.date.embargoend} contained. Zero READ policies is what turns the file into an HTTP 401. - * - *

    Every case first puts a real embargo in place, exactly like the customer did, because that is what - * replaces the collection default with a single dated policy - the one the second run then has the - * opportunity to delete.

    + * Verifies that an ORIGINAL bitstream which had a READ policy before a run still has one afterwards, + * whatever {@code dc.date.embargoend} contained. Each case first puts a real embargo in place, which is + * what replaces the collection default with a single dated policy. */ @Test public void neverZeroReadPoliciesInvariant() throws Exception { @@ -507,9 +462,8 @@ public void neverZeroReadPoliciesInvariant() throws Exception { } /** - * A real VSB-TUO record carries several files in ORIGINAL, one of them flagged as the primary bitstream. - * All of them must reach exactly the same state - the primary bitstream is not special - and the primary - * flag itself must survive. + * Verifies that every ORIGINAL bitstream of a record reaches the same state, the primary one included, and + * that the primary bitstream flag survives. */ @Test public void multipleBitstreamsAllGetSameState() throws Exception { @@ -525,7 +479,7 @@ public void multipleBitstreamsAllGetSameState() throws Exception { setPrimaryBitstream(originalBundle, bitstreams.get(0)); UUID primaryBitstreamId = bitstreams.get(0).getID(); - // (a) embargo every file of the record + // embargo every file of the record assertRunSucceeded("embargoing every ORIGINAL bitstream", runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); @@ -548,7 +502,7 @@ public void multipleBitstreamsAllGetSameState() throws Exception { assertEquals("the primary bitstream flag must survive an embargo run", primaryBitstreamId, originalBundle.getPrimaryBitstream().getID()); - // (b) the embargo expires - the same archive is re-imported with a past date + // the embargo expires: the same archive is re-imported with a past date assertRunSucceeded("letting the embargo expire", runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd))); item = context.reloadEntity(item); @@ -571,9 +525,8 @@ public void multipleBitstreamsAllGetSameState() throws Exception { } /** - * The embargo synchronisation owns the bitstreams of the ORIGINAL bundle only. Derivatives (TEXT, - * THUMBNAIL) are produced and re-protected by filter-media, and the bundle objects themselves carry their - * own policies; touching either from here is how a record ends up with 18 unreachable bitstreams. + * Verifies that only ORIGINAL bitstreams are synchronised. Derivatives are produced and re-protected by + * filter-media, and the bundle objects carry policies of their own. */ @Test public void derivativeBundlesAreNotTouchedDirectly() throws Exception { @@ -595,7 +548,7 @@ public void derivativeBundlesAreNotTouchedDirectly() throws Exception { assertFalse("fixture precondition: the ORIGINAL bundle must start with policies", originalBundlePolicies.isEmpty()); - // (a) embargo run + // embargo run assertRunSucceeded("embargoing the ORIGINAL bitstream", runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); @@ -614,7 +567,7 @@ public void derivativeBundlesAreNotTouchedDirectly() throws Exception { assertEquals("the ORIGINAL bundle's own policies must not be touched", originalBundlePolicies, policySignatures(originalBundle)); - // (b) expired embargo run - the code path that wipes policies today + // expired embargo run assertRunSucceeded("letting the embargo expire", runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd))); item = context.reloadEntity(item); @@ -636,8 +589,7 @@ public void derivativeBundlesAreNotTouchedDirectly() throws Exception { /** * Shared body of the two legacy {@code rpName} tests: a bitstream whose only {@code Anonymous}/{@code READ} - * policy was written by the previous implementation (legacy name, start date already passed, so the file is - * public) is put back under embargo. + * policy carries a legacy name and a start date that has passed is put back under embargo. */ private void assertLegacyPolicyIsAdoptedAndEnforced(String legacyName, String rightsAccess, String fileName) throws Exception { @@ -698,7 +650,7 @@ private void assertUniformState(List bitstreams, String what) throws } /** - * Answers the only question that matters: may a not-logged-in visitor download the file? + * Tells whether a visitor who is not logged in may read the bitstream. */ private boolean anonymousCanRead(Bitstream bitstream) throws Exception { EPerson saved = context.getCurrentUser(); @@ -818,9 +770,8 @@ private Item createItem(String title) throws Exception { } /** - * Creates a bitstream in the ORIGINAL bundle. The collection grants DEFAULT_BITSTREAM_READ to Anonymous, - * so the new bitstream carries exactly one policy - Anonymous / READ / TYPE_INHERITED / startDate null - - * which is byte for byte the state of a freshly imported SAF package at the customer. + * Creates a bitstream in the ORIGINAL bundle. It inherits the collection DEFAULT_BITSTREAM_READ and so + * carries one undated Anonymous READ policy, the state a freshly imported SAF package is in. */ private Bitstream createOriginalBitstream(Item item, String name) throws Exception { context.turnOffAuthorisationSystem(); @@ -887,8 +838,8 @@ private ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDat } /** - * Replaces every READ policy of the bitstream with a single Anonymous READ policy - the state a bitstream - * is left in by the previous implementation. + * Replaces every READ policy of the bitstream with a single Anonymous READ policy, the state earlier + * versions left behind. */ private ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name) throws Exception { @@ -920,12 +871,11 @@ private LocalDate toLocalDate(Date date) { } /** - * Equivalent of {@code dsrun ... ItemUpdate -s -d dc.rights.access -d dc.date.embargoend - * -a dc.rights.access -a dc.date.embargoend}, i.e. an update whose target fields contain an embargo field, - * which is what makes {@code processArchive} call {@code syncEmbargoPolicies}. + * Runs itemupdate with both embargo fields as targets, the combination that triggers embargo + * synchronisation. * - * @return the number of embargo problems the run reported; this is the only thing {@code ItemUpdate.main()} - * turns into a non-zero exit code, so it is what an operator's script sees + * @return the number of embargo problems the run reported, which is what {@code ItemUpdate.main()} turns + * into a non-zero exit code */ private int runItemUpdate(Item item, String dublinCoreContent) throws Exception { Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java index fd07a3793cba..a0d99faf3139 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java @@ -54,13 +54,9 @@ import org.junit.Test; /** - * Regression test for the VSB-TUO embargo synchronisation in {@link ItemUpdate}. - * - *

    Models the exact customer scenario observed on dspace7-test.vsb.cz: a thesis is first embargoed - * with a future {@code dc.date.embargoend} and later the very same SAF archive is re-imported with an - * {@code dc.date.embargoend} that already lies in the past. After the second run every - * {@code Anonymous}/{@code READ} policy is gone from the ORIGINAL bitstreams and the files answer - * HTTP 401 even though {@code dc.rights.access} says {@code openAccess}.

    + * Covers embargo synchronisation in {@link ItemUpdate} when the same archive is re-imported with a + * {@code dc.date.embargoend} that has already passed: the ORIGINAL bitstreams have to stay readable + * for anonymous users instead of losing their last {@code READ} policy. */ public class EmbargoPastDateIT extends AbstractIntegrationTestWithDatabase { @@ -118,15 +114,14 @@ public void destroy() throws Exception { } /** - * A {@code dc.date.embargoend} in the past must never strip the ORIGINAL bitstreams of their last - * {@code Anonymous}/{@code READ} policy. An expired embargo means "publish", not "hide forever". + * Verifies that a future embargo followed by an expired one leaves the ORIGINAL bitstream publicly + * readable. */ @Test public void pastEmbargoEndMustKeepFilesPublic() throws Exception { String futureEmbargoEnd = LocalDate.now().plusYears(1).toString(); String pastEmbargoEnd = LocalDate.now().minusMonths(1).toString(); - // (a) item with an ORIGINAL bitstream in a collection granting DEFAULT_BITSTREAM_READ to Anonymous List defaultBitstreamReadGroups = authorizeService.getAuthorizedGroups(context, collection, Constants.DEFAULT_BITSTREAM_READ); assertTrue("fixture precondition: collection must grant DEFAULT_BITSTREAM_READ to Anonymous", @@ -141,7 +136,7 @@ public void pastEmbargoEndMustKeepFilesPublic() throws Exception { assertTrue("fixture precondition: imported bitstream must be publicly readable", anonymousCanRead(bitstream)); - // (b) first itemupdate run - embargo end date in the FUTURE (state the customer confirmed as working) + // first run: embargo end date in the future runItemUpdate(item, dublinCore(item, "embargoedAccess", futureEmbargoEnd)); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -154,14 +149,13 @@ public void pastEmbargoEndMustKeepFilesPublic() throws Exception { assertNotNull("the surviving Anonymous READ policy must be dated", embargoed.get(0).getStartDate()); assertFalse("while embargoed the file must not be publicly readable", anonymousCanRead(bitstream)); - // (c) second itemupdate run - embargo end date in the PAST, item declared openAccess + // second run: embargo end date in the past, item declared openAccess runItemUpdate(item, dublinCore(item, "openAccess", pastEmbargoEnd)); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP C - after itemupdate with PAST dc.date.embargoend=" + pastEmbargoEnd + " and dc.rights.access=openAccess", bitstream); - // (d) an expired embargo publishes the file - it must never delete the last Anonymous READ policy List afterExpiry = anonymousReadPolicies(bitstream); assertFalse("Expired embargo wiped every Anonymous READ policy from the ORIGINAL bitstream." + " The file is now unreachable (HTTP 401) although dc.rights.access=openAccess." @@ -173,7 +167,7 @@ public void pastEmbargoEndMustKeepFilesPublic() throws Exception { } /** - * Answers the only question that matters: may a not-logged-in visitor download the file? + * Tells whether a visitor who is not logged in may read the bitstream. */ private boolean anonymousCanRead(Bitstream bs) throws Exception { EPerson saved = context.getCurrentUser(); @@ -262,9 +256,8 @@ private String singleMetadataValue(Item item, String element, String qualifier) } /** - * Equivalent of {@code dsrun ... ItemUpdate -s -d dc.rights.access -d dc.date.embargoend - * -a dc.rights.access -a dc.date.embargoend}, i.e. an update whose target fields contain an embargo - * field, which is what makes {@code processArchive} call {@code syncEmbargoPolicies}. + * Runs itemupdate with both embargo fields as targets, the combination that triggers embargo + * synchronisation. */ private void runItemUpdate(Item item, String dublinCoreContent) throws Exception { Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); @@ -288,9 +281,8 @@ private void runItemUpdate(Item item, String dublinCoreContent) throws Exception context.uncacheEntity(item); - // Both runs of this test are runs itemupdate has to carry out. embargoSyncFailures is what - // ItemUpdate.main() turns into a non-zero exit code, so a refusal that leaves it at 0 would be - // invisible to the operator's script. + // embargoSyncFailures drives the exit code of ItemUpdate.main(), so a refusal that left it at + // zero would be invisible to the calling script. assertEquals("itemupdate reported an embargo synchronisation problem, so ItemUpdate.main() would exit" + " with " + ItemUpdate.exitStatus(0, itemUpdate.embargoSyncFailures), 0, itemUpdate.embargoSyncFailures); diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java index 00ca020d299c..db2178557d65 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java @@ -67,36 +67,27 @@ import org.junit.Test; /** - * Safety net for the VSB-TUO embargo synchronisation in {@link ItemUpdate}. - * - *

    The bug being fixed is that an expired {@code dc.date.embargoend} strips the ORIGINAL bitstreams - * of their last {@code Anonymous}/{@code READ} policy. The obvious repair - "when the embargo has - * expired, just make the files public" - is far more dangerous than the bug itself, because it would - * publish material that has to stay closed. This class pins down everything the repair must NOT do.

    - * - *

    Every "must not touch" assertion compares the full set of {@code policy_id} values plus a - * fingerprint of every policy (action, group, rpType, rpName, start and end date). Comparing counts - * would be useless: a policy deleted and immediately recreated keeps the count but loses its identity, - * and a policy mutated in place keeps its id but changes its meaning.

    + * Pins down the cases in which embargo synchronisation in {@link ItemUpdate} has to leave a bitstream alone + * rather than publish it. Every "must not touch" assertion compares policy ids and a fingerprint of every + * policy, because counts hide both delete-and-recreate and in-place mutation. */ public class EmbargoSafetyIT extends AbstractIntegrationTestWithDatabase { /** - * rpName written by the shipped (buggy) implementation. The repair has to recognise and normalise - * these legacy policies, so the fixtures use that name rather than a clean-room one. + * rpName written by earlier versions; the fixtures use it so that normalisation of legacy policies is + * exercised. */ private static final String LEGACY_EMBARGO_POLICY_NAME = "Standard Embargo"; /** - * The only supported way of re-opening files whose Anonymous READ policy is already gone. - * ItemUpdate has to point the operator at it instead of inventing a public policy. + * The supported way of re-opening files whose Anonymous READ policy is already gone; ItemUpdate points + * the operator at it instead of inventing a public policy. */ private static final String BULK_ACCESS_CONTROL_HINT = "bulk-access-control"; /** - * Access condition of access-conditions.xml that writes an {@code Anonymous}/{@code READ} policy with an - * END date ({@code groupName=Anonymous}, {@code hasEndDate=true}, {@code endDateLimit=+6MONTHS}): public - * now, closed again on that day. It is the one shape of Anonymous READ policy this tool refuses to touch. + * Access condition writing an {@code Anonymous}/{@code READ} policy with an END date: public now, closed + * again on that day. */ private static final String LEASE_POLICY_NAME = "lease"; @@ -160,8 +151,8 @@ public void destroy() throws Exception { } /** - * Specification row 9. A withdrawn item is hidden on purpose. Withdrawal converts every READ policy - * into WITHDRAWN_READ, so an embargo sync that "restores" access would silently undo a takedown. + * Verifies that a withdrawn item is left alone: withdrawal turns every READ policy into WITHDRAWN_READ, + * and synchronising an embargo must not undo a takedown. */ @Test public void withdrawnItemIsNeverRepublished() throws Exception { @@ -192,9 +183,8 @@ public void withdrawnItemIsNeverRepublished() throws Exception { item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); - // Which guard stopped the run has to be nailed down. ItemServiceImpl.withdraw() also clears - // archived, so the !isArchived guard alone would satisfy every policy assertion below and this test - // would keep passing after the withdrawal guard was deleted. + // withdraw() also clears archived, so without this check the !isArchived guard alone would satisfy + // every policy assertion below. assertTrue("ItemUpdate has to refuse a withdrawn item because it is withdrawn. Nothing in the console" + " output says so, so some other guard stopped the run and the withdrawal guard is" + " untested. Console output was:" + System.lineSeparator() + pastRun.console, @@ -209,8 +199,8 @@ public void withdrawnItemIsNeverRepublished() throws Exception { + describe(bitstream), anonymousCanRead(bitstream)); - // Row 9 says "any end date". The past-date run above only ever reaches the early return, so on its own - // it proves nothing about withdrawal; the future-date branch is the one that creates policies. + // The past-date run above only reaches the early return; the future-date branch is the one that + // creates policies. Run futureRun = runItemUpdate(item, dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); @@ -232,8 +222,7 @@ public void withdrawnItemIsNeverRepublished() throws Exception { } /** - * Specification row 4. {@code restrictedAccess} means the files stay closed no matter what the embargo - * end date says - a stale end date is not permission to publish. + * Verifies that {@code restrictedAccess} keeps the files closed whatever the embargo end date says. */ @Test public void restrictedAccessWithStaleEmbargoEndIsUntouched() throws Exception { @@ -242,7 +231,7 @@ public void restrictedAccessWithStaleEmbargoEndIsUntouched() throws Exception { } /** - * Specification row 4. {@code metadataOnlyAccess} means the bitstreams are never disclosed. + * Verifies that {@code metadataOnlyAccess} keeps the bitstreams undisclosed. */ @Test public void metadataOnlyAccessIsUntouched() throws Exception { @@ -251,8 +240,7 @@ public void metadataOnlyAccessIsUntouched() throws Exception { } /** - * Specification row 4. An access right the tool does not understand is not an invitation to guess; - * an unknown value means "hands off", never "open". + * Verifies that an access right the tool does not understand means "hands off" rather than "open". */ @Test public void unknownAccessRightValueIsUntouched() throws Exception { @@ -261,9 +249,8 @@ public void unknownAccessRightValueIsUntouched() throws Exception { } /** - * Specification row 4. A single value outside the allowlist blocks the whole item, even when another - * value of the same field says {@code openAccess}. Contradictory metadata is never resolved in favour - * of disclosure. + * Verifies that one value outside the allowlist blocks the whole item, even next to {@code openAccess}: + * contradictory metadata is not resolved in favour of disclosure. */ @Test public void mixedAccessRightsWithOneDisallowedIsUntouched() throws Exception { @@ -272,8 +259,8 @@ public void mixedAccessRightsWithOneDisallowedIsUntouched() throws Exception { } /** - * Specification row 11. When READ is granted to a named group only there is no Anonymous policy to - * mutate. Creating one would hand the public a file that was deliberately limited to that group. + * Verifies that a bitstream readable by a named group only gains no Anonymous policy: with nothing to + * mutate, creating one would widen access nobody granted. */ @Test public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { @@ -314,8 +301,8 @@ public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { assertFalse("A group restricted file became publicly readable after an expired embargo was synchronised." + describe(bitstream), anonymousCanRead(bitstream)); - // Row 11 says "any end date". A past date only reaches the early return; the future-date branch is the - // one that creates policies, so it is where a group-restricted file can silently gain an Anonymous one. + // A past date only reaches the early return; the future-date branch is where a group-restricted file + // could gain an Anonymous policy. Run futureRun = runItemUpdate(item, dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); @@ -330,8 +317,6 @@ public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { assertFalse("A group restricted file became publicly readable after a future embargo end date was" + " synchronised." + describe(bitstream), anonymousCanRead(bitstream)); - // Reporting is asserted last, after both legs have proved that no policy was invented: a missing log - // line must never be the reason a reader stops reading before the policy damage is on screen. assertTrue("There is no Anonymous READ policy to re-date here, so ItemUpdate has to report the bitstream" + " it could not synchronise and name '" + BULK_ACCESS_CONTROL_HINT + "' as the supported" + " way to change access. Console output of the expired-embargo run was:" @@ -341,16 +326,14 @@ public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { + " future-embargo run was:" + System.lineSeparator() + futureRun.console, futureRun.console.contains(BULK_ACCESS_CONTROL_HINT)); - // Spec row 11 requires a non-zero exit code: an unsynchronised bitstream that leaves the exit code at - // 0 is invisible to the operator who started the batch. + // An unsynchronised bitstream that leaves the exit code at 0 is invisible to the caller. assertExitCode("bitstream without Anonymous READ, expired embargo", 1, pastRun); assertExitCode("bitstream without Anonymous READ, future embargo", 1, futureRun); } /** - * Specification row 11. This is the exact state of the customer record damaged by the shipped code: - * zero resource policies, HTTP 401 on every download. The repair must refuse to guess what those - * policies used to be - it may only report the damage and name the tool that can undo it. + * Verifies that a bitstream left without any resource policy stays that way: what it used to grant cannot + * be reconstructed, so the run reports the damage instead of guessing. */ @Test public void alreadyBrokenBitstreamWithZeroPoliciesStaysZero() throws Exception { @@ -387,17 +370,8 @@ public void alreadyBrokenBitstreamWithZeroPoliciesStaysZero() throws Exception { } /** - * An {@code Anonymous}/{@code READ} policy that carries an END date is a {@code lease}, not an embargo: - * access-conditions.xml declares it with {@code groupName=Anonymous} and {@code hasEndDate=true}, and it - * is written by the submission UI, by a REST access condition patch and by - * {@code dspace bulk-access-control}. It means "public now, closed again on that day", which is not - * something {@code dc.date.embargoend} says anything about. - * - *

    Both ways of synchronising it would decide access on the operator's behalf: keeping the end date - * next to a fresh start date can close the file for good (an end date that lies before the start date), - * and losing the end date publishes for ever a file that was supposed to close itself. So the bitstream - * is left exactly as it is, the operator is told which policy stopped the synchronisation, and the run - * reports a failure instead of a success.

    + * Verifies that an {@code Anonymous}/{@code READ} policy with an END date is left alone. It is a lease + * ("public now, closed again on that day"), which {@code dc.date.embargoend} says nothing about. */ @Test public void leasedAnonymousReadPolicyIsUntouched() throws Exception { @@ -434,10 +408,8 @@ public void leasedAnonymousReadPolicyIsUntouched() throws Exception { } /** - * The refusal has to consider every {@code Anonymous}/{@code READ} policy of the bitstream and not only - * the one that would be mutated. A leased policy has no start date, so {@code selectSurvivorPolicy} - * prefers the dated embargo policy next to it and the lease falls into the deletion loop - and deleting - * the lease is precisely what removes the end date that was to close the file again. + * Verifies that a lease sitting next to a dated embargo policy is not deleted: the refusal has to consider + * every {@code Anonymous}/{@code READ} policy, not only the one that would be mutated. */ @Test public void leaseNextToADatedEmbargoPolicyIsNotDeleted() throws Exception { @@ -465,8 +437,7 @@ public void leaseNextToADatedEmbargoPolicyIsNotDeleted() throws Exception { } /** - * Specification row 7. A blank end date is a broken export, not an instruction. Validation has to run - * before anything is mutated, so the existing policies survive untouched. + * Verifies that a blank end date changes nothing: validation runs before anything is mutated. */ @Test public void blankEmbargoEndLeavesPoliciesUntouched() throws Exception { @@ -483,8 +454,8 @@ public void blankEmbargoEndLeavesPoliciesUntouched() throws Exception { } /** - * Specification row 8. Parsing has to be strict. {@code DCDate} silently rolls 30 February over into - * 2 March, which would turn an unparseable value into a real - possibly future - embargo date. + * Verifies that parsing is strict: a lenient parser rolls 30 February over into 2 March and turns an + * unreadable value into a real embargo date. */ @Test public void invalidEmbargoEndLeavesPoliciesUntouched() throws Exception { @@ -499,8 +470,7 @@ public void invalidEmbargoEndLeavesPoliciesUntouched() throws Exception { } /** - * Specification row 6. {@code embargoedAccess} without an end date is self-contradictory metadata. - * The tool has to refuse it rather than pick one half of the contradiction. + * Verifies that {@code embargoedAccess} without an end date is refused rather than half-applied. */ @Test public void embargoedAccessWithoutEndDateLeavesPoliciesUntouched() throws Exception { @@ -509,8 +479,8 @@ public void embargoedAccessWithoutEndDateLeavesPoliciesUntouched() throws Except } /** - * Specification row 10. An item outside the archive (workspace or workflow) is not published yet; its - * bitstream policies are the submission's business, not itemupdate's. + * Verifies that an item outside the archive is left alone; its bitstream policies belong to the + * submission, not to itemupdate. */ @Test public void notArchivedItemIsUntouched() throws Exception { @@ -544,15 +514,8 @@ public void notArchivedItemIsUntouched() throws Exception { } /** - * The per-item {@code catch} of {@code processArchive} prints the exception and carries on, and a printed - * exception is an exit code of 0. That is harmless for an action that failed before it changed anything. - * It is not harmless for the embargo synchronisation, whose last step is deleting the duplicate - * Anonymous READ policies of a bitstream whose surviving policy has just been re-dated: if that step - * throws, {@code context.complete()} commits the re-dated policy anyway, so the file is open and the run - * reports success. The exit code is all the operator's script gets to see. - * - *

    The failure is provoked by letting {@code applyEmbargoToItemBitstreams} throw after doing its - * work, which is exactly the state the duplicate deletion loop runs in.

    + * Verifies that a synchronisation which throws after re-dating a policy is not reported as success: the + * per-item catch of {@code processArchive} prints the exception and commits what was written so far. */ @Test public void embargoSyncThatDiesHalfWayIsNotReportedAsSuccess() throws Exception { @@ -585,11 +548,8 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta } /** - * Runs one "ItemUpdate has to keep its hands off" scenario end to end and returns the reloaded item. - * - *

    The fixture is the state the customer repository is in after an earlier itemupdate run with a - * future end date: exactly one Anonymous READ policy, dated, currently blocking access. Any repair - * that publishes, deletes or re-creates that policy is caught here.

    + * Runs one "hands off" scenario end to end and returns the reloaded item. The fixture is the state an + * earlier run with a future end date leaves: one dated Anonymous READ policy currently blocking access. */ private Item assertEmbargoSyncIsANoOp(String scenario, List accessRights, String embargoEndDate, int expectedFailures) @@ -631,13 +591,8 @@ private void assertUntouched(String scenario, Set idsBefore, List -d dc.rights.access -d dc.date.embargoend - * -a dc.rights.access -a dc.date.embargoend}, i.e. an update whose target fields contain an embargo - * field, which is what makes {@code processArchive} call {@code syncEmbargoPolicies}. - * - *

    The {@link ItemUpdate} instance is kept, not thrown away: {@code embargoSyncFailures} is what - * {@code main()} turns into a non-zero exit code, and an operator scripting {@code itemupdate} sees - * nothing else. A refusal that leaves the exit code at 0 is a silent failure.

    + * Runs itemupdate with both embargo fields as targets, the combination that triggers embargo + * synchronisation. The {@link ItemUpdate} instance is kept because it carries the failure count. * * @return the console output of the run and the number of embargo problems it counted */ @@ -692,8 +647,7 @@ private Run runItemUpdate(ItemUpdate itemUpdate, Item item, String dublinCoreCon } /** - * Everything a finished {@code itemupdate} run is judged by: what it told the operator, and what it would - * have exited with. + * What a finished {@code itemupdate} run is judged by: its console output and its failure count. */ private static final class Run { private final String console; @@ -706,8 +660,7 @@ private Run(String console, int embargoSyncFailures) { } /** - * A run that refused to do something has to say so in its exit code, otherwise the operator's script - * treats a skipped item as a synchronised one. + * Asserts the exit code the run would have produced; without it a skipped item looks synchronised. */ private void assertExitCode(String scenario, int expectedFailures, Run run) { assertEquals("[" + scenario + "] wrong number of reported embargo problems, so ItemUpdate.main() would" @@ -720,9 +673,9 @@ private void assertExitCode(String scenario, int expectedFailures, Run run) { /** * Builds a {@code dublin_core.xml} carrying zero or more {@code dc.rights.access} values. * - * @param embargoEndDate {@code null} omits {@code dc.date.embargoend} entirely (the operator lifted the - * embargo), the empty string writes a blank value (an empty XML element is dropped - * by the parser, so a single space is written instead) + * @param embargoEndDate {@code null} omits {@code dc.date.embargoend} entirely, the empty string writes a + * blank value (an empty XML element is dropped by the parser, so a single space is + * written instead) */ private String dublinCore(Item item, List accessRights, String embargoEndDate) { StringBuilder sb = new StringBuilder(); @@ -767,9 +720,8 @@ private Item createItem(String title) throws Exception { } /** - * A bitstream in the ORIGINAL bundle. The collection grants DEFAULT_BITSTREAM_READ to Anonymous, so - * BundleServiceImpl gives the new bitstream exactly one policy: Anonymous / READ / TYPE_INHERITED / - * rpName=null / startDate=null - byte for byte the state of a freshly imported SAF item. + * A bitstream in the ORIGINAL bundle. It inherits the collection DEFAULT_BITSTREAM_READ and so carries one + * undated Anonymous READ policy, the state a freshly imported SAF item is in. */ private Bitstream createOriginalBitstream(Item item, String name) throws Exception { context.turnOffAuthorisationSystem(); @@ -783,9 +735,8 @@ private Bitstream createOriginalBitstream(Item item, String name) throws Excepti } /** - * The state the customer repository is left in by an itemupdate run with a future end date: the - * immediate Anonymous READ policy is gone and a single dated legacy "Standard Embargo" policy blocks - * access. That policy is all that stands between the public and the file. + * The state an itemupdate run with a future end date leaves: a single dated legacy policy blocking access, + * and nothing else between the public and the file. */ private Bitstream createEmbargoedBitstream(Item item, String name) throws Exception { Bitstream bitstream = createOriginalBitstream(item, name); @@ -804,8 +755,8 @@ private Bitstream createEmbargoedBitstream(Item item, String name) throws Except } /** - * A bitstream whose {@code Anonymous}/{@code READ} access expires by itself, i.e. exactly what the - * {@code lease} access condition writes: no start date and an end date at most six months out. + * A bitstream whose {@code Anonymous}/{@code READ} access expires by itself, as the {@code lease} access + * condition writes it: no start date and an end date six months out. */ private Bitstream createLeasedBitstream(Item item, String name) throws Exception { Bitstream bitstream = createOriginalBitstream(item, name); @@ -829,9 +780,8 @@ private void addLeasePolicy(Bitstream bitstream) throws Exception { } /** - * Deletes policies one by one through {@code ResourcePolicyService#delete}, the same call the production - * code uses, so the Hibernate session stays consistent (the bulk removal helpers issue an HQL delete and - * leave the in-memory collection stale). + * Deletes policies one by one, because the bulk removal helpers issue an HQL delete and leave the + * in-memory collection stale. * * @param actionId action to delete, or {@link #ALL_ACTIONS} for every policy regardless of action */ @@ -888,8 +838,8 @@ private String fingerprint(ResourcePolicy policy) { } /** - * Renders the current policies of the bitstream for failure messages. Whoever reads a red build has to - * see which policy moved without re-running anything. + * Renders the current policies of the bitstream for failure messages, so a red build shows which policy + * moved. */ private String describe(Bitstream bitstream) throws SQLException { StringBuilder sb = new StringBuilder(System.lineSeparator()) @@ -905,11 +855,8 @@ private String describe(Bitstream bitstream) throws SQLException { } /** - * Answers the only question that matters: may a visitor who is not logged in download the file? - * - *

    The authorisation state is a stack, not a flag: the builders and processArchive push and pop - * around themselves, so the depth is not guaranteed to be zero here. It has to be drained, otherwise - * {@code authorize()} short circuits on {@code ignoreAuthorization()} and every read looks allowed.

    + * Tells whether a visitor who is not logged in may read the bitstream. The authorisation state is a stack + * the builders push and pop, so it is drained first - otherwise every read looks allowed. */ private boolean anonymousCanRead(Bitstream bitstream) throws SQLException { EPerson savedUser = context.getCurrentUser(); @@ -934,9 +881,7 @@ private String pastDate() { } /** - * A future end date is the dangerous half of every "hands off" rule: the past-date branch of the shipped - * code returns early and therefore looks harmless, while the future-date branch is the one that actually - * writes resource policies. + * A future end date, the branch that writes resource policies; the past-date branch returns early. */ private String futureDate() { return LocalDate.now().plusYears(1).toString(); @@ -947,9 +892,8 @@ private Date startOfDayUtc(LocalDate day) { } /** - * Compares dates at calendar day granularity. The harness forces TZ Europe/Dublin while resource policy - * start dates come back from the database as {@code java.sql.Date}; comparing instants across that - * boundary would be flaky, comparing days is not. + * Renders a date at calendar day granularity: start dates come back from the database as + * {@code java.sql.Date}, so comparing instants across the harness time zone would be flaky. */ private String day(Date date) { if (date == null) { diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java index 953e96629c74..04b579c48218 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java @@ -63,7 +63,7 @@ */ public class ItemUpdateIT extends AbstractIntegrationTestWithDatabase { - /** rpName written by the shipped implementation; the fix has to adopt and normalise it. */ + /** rpName written by earlier versions; it has to be adopted and normalised. */ private static final String STANDARD_EMBARGO = "Standard Embargo"; /** The single normalised rpName, matching the access condition name in access-conditions.xml. */ @@ -122,10 +122,8 @@ public void destroy() throws Exception { } /** - * The embargo refusals are counted per item and never abort the run, so the exit code is the only place - * an operator's script can see them. Every test below asserts {@code embargoSyncFailures}; this one - * asserts the step that turns that counter into the process exit code, which is otherwise only reachable - * through {@code main()} and its {@code System.exit}. + * Verifies the step that turns {@code embargoSyncFailures} into the process exit code, which is otherwise + * only reachable through {@code main()} and its {@code System.exit}. */ @Test public void embargoSyncFailuresDecideTheExitCode() { @@ -209,8 +207,7 @@ public void syncEmbargoPoliciesDatesTheAnonymousReadPolicyAndBlocksAccess() thro itemUpdate.syncEmbargoPolicies(context, item); assertEquals("setting a future embargo is not a failure", 0, itemUpdate.embargoSyncFailures); - // Exactly one Anonymous READ policy has to be left behind. A second, undated one would silently - // defeat the embargo, so counting is part of the assertion, not a detail. + // A second, undated policy would defeat the embargo, so the count is part of the assertion. List anonymousRead = anonymousReadPolicies(bitstream); assertEquals(1, anonymousRead.size()); @@ -235,9 +232,8 @@ public void syncEmbargoPoliciesAppliesEmbargoWithoutAccessRightMetadata() throws List anonymousRead = anonymousReadPolicies(bitstream); assertEquals(1, anonymousRead.size()); - // The distinction between "standard" and "special case" embargo only ever existed in the rpName. - // Both are now written as the single access condition name from access-conditions.xml, which also - // keeps the value inside the 30 character resourcepolicy.rpname column. + // Both cases share one access condition name, which also fits the 30 character + // resourcepolicy.rpname column. ResourcePolicy embargoPolicy = anonymousRead.get(0); assertEquals(EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); assertEquals(ResourcePolicy.TYPE_CUSTOM, embargoPolicy.getRpType()); @@ -247,12 +243,8 @@ public void syncEmbargoPoliciesAppliesEmbargoWithoutAccessRightMetadata() throws } /** - * A blank {@code dc.date.embargoend} is a broken export, not an instruction to change anything. - * - *

    This test used to assert only {@code assertFalse(hasSafEmbargoPolicy)}, which an empty policy table - * satisfies just as well as a correct one - and an empty policy table is exactly the customer bug (HTTP 401 - * on every download). It now asserts what the operator actually cares about: not a single resource policy - * was touched.

    + * Verifies that a blank {@code dc.date.embargoend} leaves every resource policy untouched and fails the + * run; it is a broken export, not an instruction to change anything. */ @Test public void syncEmbargoPoliciesLeavesPoliciesUntouchedWhenEmbargoDateInvalid() throws Exception { @@ -270,7 +262,7 @@ public void syncEmbargoPoliciesLeavesPoliciesUntouchedWhenEmbargoDateInvalid() t ItemUpdate itemUpdate = new ItemUpdate(); itemUpdate.syncEmbargoPolicies(context, item); - // Spec row 7: a blank end date is broken input, and the run has to exit non-zero because of it. + // A blank end date is broken input, so the run has to exit non-zero. assertEquals("a blank dc.date.embargoend has to fail the run", 1, itemUpdate.embargoSyncFailures); assertEquals(1, ItemUpdate.exitStatus(0, itemUpdate.embargoSyncFailures)); assertEquals(idsBefore, policyIds(bitstream)); @@ -312,8 +304,7 @@ public void processArchiveUpdatesEmbargoMetadataAndResyncsEmbargoPolicy() throws List anonymousRead = anonymousReadPolicies(reloadedBitstream); assertEquals(1, anonymousRead.size()); - // The pre-existing policy is re-dated in place instead of being deleted and re-created. Between a - // delete and a create the file has no policy at all, which is the state the customer report was about. + // The pre-existing policy is re-dated in place, so the file is covered by a policy at every moment. ResourcePolicy embargoPolicy = anonymousRead.get(0); assertEquals(legacyPolicyId, embargoPolicy.getID()); assertEquals(EMBARGO_POLICY_NAME, embargoPolicy.getRpName()); @@ -325,9 +316,8 @@ public void processArchiveUpdatesEmbargoMetadataAndResyncsEmbargoPolicy() throws } /** - * Blanking {@code dc.date.embargoend} in the SAF archive is a broken export. Same reasoning as - * {@link #syncEmbargoPoliciesLeavesPoliciesUntouchedWhenEmbargoDateInvalid()}: the old - * {@code assertFalse(hasSafEmbargoPolicy)} was also satisfied by a bitstream stripped of every policy. + * Same as {@link #syncEmbargoPoliciesLeavesPoliciesUntouchedWhenEmbargoDateInvalid()} driven through a SAF + * archive whose {@code dc.date.embargoend} is blank. */ @Test public void processArchiveUpdateWithBlankEmbargoDateLeavesPoliciesUntouched() throws Exception { @@ -363,11 +353,8 @@ public void processArchiveUpdateWithBlankEmbargoDateLeavesPoliciesUntouched() th } /** - * A SAF package that does not carry {@code dc.date.embargoend} carries no instruction about the embargo, - * and an absent field must never open a file. {@code syncEmbargoPolicies} runs for every item of a batch - * whose target fields mention an embargo field, so reading "field missing" as "lift the embargo" would - * publish every embargoed item of a batch whose packages happen not to carry it. A file is opened by - * writing a {@code dc.date.embargoend} that lies in the past. + * Verifies that a SAF package without {@code dc.date.embargoend} leaves every policy untouched. The field + * carries no instruction about the embargo; a file is opened by an end date that lies in the past. */ @Test public void processArchiveUpdateRemovingEmbargoMetadataLeavesPoliciesUntouched() throws Exception { @@ -380,8 +367,8 @@ public void processArchiveUpdateRemovingEmbargoMetadataLeavesPoliciesUntouched() Date oldPolicyStart = Date.from(LocalDate.parse(oldEmbargoDate).plusDays(1) .atStartOfDay(ZoneId.systemDefault()).toInstant()); - // The collection default leaves an undated Anonymous READ policy on a new bitstream. It has to go, - // otherwise the file is readable throughout and the assertions below would prove nothing. + // The undated Anonymous READ policy from the collection default has to go, otherwise the file is + // readable throughout and the assertions below prove nothing. ResourcePolicy legacyPolicy = replaceAnonymousReadPolicies(bitstream, oldPolicyStart, STANDARD_EMBARGO); Integer legacyPolicyId = legacyPolicy.getID(); bitstream = context.reloadEntity(bitstream); @@ -488,8 +475,8 @@ private Bitstream createBitstream(Item item, String name) throws Exception { } /** - * Leaves the bitstream with exactly one Anonymous READ policy: the collection's undated default is - * removed first. Without that step a "the file is embargoed" fixture is not embargoed at all. + * Leaves the bitstream with one Anonymous READ policy, removing the collection's undated default first; + * without that an "embargoed" fixture is not embargoed at all. */ private ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name) throws Exception { @@ -526,8 +513,8 @@ private List anonymousReadPolicies(Bitstream bitstream) throws E } /** - * Identity of every resource policy on the bitstream. A policy deleted and immediately re-created keeps - * the count but loses its id, so ids are what "untouched" has to be measured with. + * Identity of every resource policy on the bitstream. Ids rather than counts, so a policy that was deleted + * and re-created is visible. */ private List policyIds(Bitstream bitstream) throws Exception { return authorizeService.getPolicies(context, bitstream).stream() @@ -537,8 +524,8 @@ private List policyIds(Bitstream bitstream) throws Exception { } /** - * What an anonymous visitor of the REST API gets: the authorisation system asked as nobody, with the - * test's own turnOffAuthorisationSystem calls temporarily unwound. + * Tells whether a visitor who is not logged in may read the bitstream, with the test's own + * turnOffAuthorisationSystem calls temporarily unwound. */ private boolean anonymousCanRead(Bitstream bitstream) throws SQLException { EPerson savedUser = context.getCurrentUser(); @@ -584,8 +571,10 @@ private String dublinCore(String identifierUri, String thesisIdentifier) { } /** - * @return the number of embargo problems reported by the run; {@link ItemUpdate#exitStatus(int, int)} is - * what turns it into the exit code of {@code dspace itemupdate} + * Runs one SAF metadata update over the item. + * + * @return the number of embargo problems reported by the run, which {@link ItemUpdate#exitStatus(int, int)} + * turns into the exit code of {@code dspace itemupdate} */ private int runEmbargoMetadataUpdate(Item item, String dublinCoreContent) throws Exception { Path sourceRoot = Files.createDirectory(tempDir.resolve("update-source-" + System.nanoTime())); From c4889320c3a8ce4f44d8f373dbf8174f66003434 Mon Sep 17 00:00:00 2001 From: milanmajchrak Date: Wed, 19 Aug 2026 16:11:31 +0200 Subject: [PATCH 07/10] VSB-TUO/Fix: embargo now also covers the TEXT and THUMBNAIL bundles itemupdate synchronised the Anonymous READ policy of the ORIGINAL bundle only, so an embargoed item kept a public thumbnail and, worse, a public TEXT bitstream holding the full text extracted from the embargoed file. The whole content of an embargoed thesis was therefore downloadable for the entire embargo period. applyEmbargoToItemBitstreams now walks ORIGINAL, TEXT and THUMBNAIL. The embargo is a property of the item, so all three get the same start date, and the per-bitstream rules are unchanged: mutate the surviving Anonymous READ policy, never create one where none exists, delete duplicates only after the survivor is stored, leave a policy with an end date alone, count every skipped bitstream in embargoSyncFailures. The bundle list lives in SafEmbargoConstants so the two SAF tools cannot drift apart. LICENSE, CC-LICENSE and METADATA stay out of scope, the licence text has to remain readable. Reports now name the bundle a bitstream belongs to, and the hint at the end of a failure report depends on it: bulk-access-control walks the ORIGINAL bundle only, so recommending it for a TEXT or THUMBNAIL bitstream would send the operator down a dead end. MediaFilterService.updatePoliciesOfDerivativeBitstreams was deliberately not reused. It returns immediately unless setFilterClasses() was called, which only MediaFilterScript does, so it is a no-op for every other caller; and when primed it re-publishes thumbnails listed in filter.*.publicPermission, which is the opposite of what an embargo needs. EmbargoLifecycleIT.derivativeBundlesAreNotTouchedDirectly asserted the old behaviour - that TEXT and THUMBNAIL policies are byte-identical before and after a run - which is exactly the leak. It is rewritten as derivativeBundlesFollowTheEmbargo and now pins the new rule, keeping its assertions on the ORIGINAL bundle object: the fix works on bitstreams, bundle-level policies stay untouched. EmbargoDerivativesIT covers the closing and re-opening of derivatives, in-place mutation, the untouched licence bundles, a derivative without an Anonymous READ policy, and the withdrawn/restrictedAccess guards. The import path needs no change: at import time the derivatives do not exist yet, they are created later by filter-media, which derives their policies itself. Co-Authored-By: Claude Opus 5 (1M context) --- .../app/itemimport/ItemImportServiceImpl.java | 2 +- .../org/dspace/app/itemupdate/ItemUpdate.java | 151 ++-- .../dspace/app/util/SafEmbargoConstants.java | 19 + .../app/itemupdate/EmbargoDerivativesIT.java | 707 ++++++++++++++++++ .../app/itemupdate/EmbargoLifecycleIT.java | 78 +- 5 files changed, 870 insertions(+), 87 deletions(-) create mode 100644 dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java index 0a125852f78c..943a97e6a2c0 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java @@ -2679,7 +2679,7 @@ protected void applyEmbargoToItemBitstreams(Context c, Item item, Date accessSta List originalBundles = item.getBundles("ORIGINAL"); if (originalBundles.isEmpty()) { // Known limitation: a contents file can route its files into another bundle with the - // "bundle:" marker, and those bitstreams are outside the ORIGINAL scope of both SAF tools. + // "bundle:" marker; only the ORIGINAL bundle is embargoed at import time. logInfo("Embargo: No ORIGINAL bundles found, no embargo applied"); return; } diff --git a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java index fa50acaafcd6..19922d650c07 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java +++ b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java @@ -116,10 +116,15 @@ public class ItemUpdate { private static final String OPEN_ACCESS = "openAccess"; private static final String EMBARGOED_ACCESS = "embargoedAccess"; - /** The only supported way of changing access to a bitstream this tool refuses to touch. */ + /** The only supported way of changing access to an ORIGINAL bitstream this tool refuses to touch. */ private static final String BULK_ACCESS_CONTROL_HINT = "Use the 'dspace bulk-access-control' script to grant or restore access to it."; + /** Derived bundles are out of reach of bulk-access-control, which walks the ORIGINAL bundle only. */ + private static final String DERIVATIVE_ACCESS_HINT = + "Change its access in the administrative UI, or let 'dspace filter-media' recreate it from the" + + " source file."; + static { filterAliases.put("ORIGINAL", "org.dspace.app.itemupdate.OriginalBitstreamFilter"); filterAliases @@ -695,7 +700,7 @@ protected static boolean containsEmbargoField(String[] targetFields) { } /** - * Bring the {@code Anonymous}/{@code READ} resource policies of the ORIGINAL bitstreams in line with the + * Bring the {@code Anonymous}/{@code READ} resource policies of the item bitstreams in line with the * embargo metadata of the item ({@code dc.rights.access}, {@code dc.date.embargoend}). The metadata is * validated before any policy is touched, and the surviving policy is re-dated instead of being replaced, * so that a failure cannot leave a bitstream without any policy. @@ -787,20 +792,21 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti // An expired embargo is a publication, not a deletion: the start date that has already passed // makes the policy effective immediately. pr("Embargo of item " + itemLabel(item) + " already expired on " + embargoEndDay - + ", its ORIGINAL bitstreams are public since " + accessStartDay + "."); + + ", its bitstreams are public since " + accessStartDay + "."); } applyEmbargoToItemBitstreams(context, item, accessStartDate); } /** - * Write the target embargo state onto the {@code Anonymous}/{@code READ} policy of every ORIGINAL bitstream - * of the item, leaving exactly one such policy per bitstream, as a second undated one would defeat the - * embargo. Where there is no such policy none is created: the bitstream was not public, and creating one - * would widen access instead of re-dating it. + * Write the target embargo state onto the {@code Anonymous}/{@code READ} policy of every bitstream in + * {@link SafEmbargoConstants#EMBARGOED_BUNDLE_NAMES}, leaving exactly one such policy per bitstream, as a + * second undated one would defeat the embargo. Where there is no such policy none is created: the bitstream + * was not public, and creating one would widen access instead of re-dating it. The embargo belongs to the + * item, so the derived bundles get the same start date as the file they were derived from. * * @param context DSpace context - * @param item item whose ORIGINAL bitstreams are synchronised + * @param item item whose bitstreams are synchronised * @param startDate day the files become publicly readable; a day in the past takes effect immediately * @throws SQLException if a database error occurs * @throws AuthorizeException if the policy update is not permitted @@ -815,58 +821,99 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta return; } - for (Bundle bundle : item.getBundles(Constants.CONTENT_BUNDLE_NAME)) { - for (Bitstream bitstream : bundle.getBitstreams()) { - // Located by (group, action) rather than by rpName: policies written under earlier names - // have to be adopted and normalised instead of being left behind next to a new one. - List anonymousReadPolicies = new ArrayList<>(); - for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream, Constants.READ)) { - if (anonymousGroup.equals(policy.getGroup())) { - anonymousReadPolicies.add(policy); - } + for (String bundleName : SafEmbargoConstants.EMBARGOED_BUNDLE_NAMES) { + for (Bundle bundle : item.getBundles(bundleName)) { + for (Bitstream bitstream : bundle.getBitstreams()) { + applyEmbargoToBitstream(context, item, bundleName, bitstream, anonymousGroup, startDate); } + } + } + } - if (anonymousReadPolicies.isEmpty()) { - prErr("Bitstream '" + bitstream.getName() + "' (" + bitstream.getID() + ") of item " - + itemLabel(item) + " has no " + Group.ANONYMOUS + " READ policy, so there is" - + " nothing to re-date and its embargo could not be synchronised. No policy is" - + " created: that would grant access nobody ever granted. " - + BULK_ACCESS_CONTROL_HINT); - embargoSyncFailures++; - continue; - } + /** + * Synchronise the {@code Anonymous}/{@code READ} policy of a single bitstream, reporting the bundle it + * belongs to so that the operator sees which file of the item a problem is about. + * + * @param context DSpace context + * @param item item the bitstream belongs to + * @param bundleName bundle the bitstream lives in + * @param bitstream bitstream to synchronise + * @param anonymousGroup the {@code Anonymous} group + * @param startDate day the file becomes publicly readable + * @throws SQLException if a database error occurs + * @throws AuthorizeException if the policy update is not permitted + */ + protected void applyEmbargoToBitstream(Context context, Item item, String bundleName, Bitstream bitstream, + Group anonymousGroup, Date startDate) throws SQLException, AuthorizeException { + // Located by (group, action) rather than by rpName: policies written under earlier names have to be + // adopted and normalised instead of being left behind next to a new one. + List anonymousReadPolicies = new ArrayList<>(); + for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream, Constants.READ)) { + if (anonymousGroup.equals(policy.getGroup())) { + anonymousReadPolicies.add(policy); + } + } - // A policy with an end date is a lease, not an embargo: re-dating it would either close the - // file for good or drop the end date, and either way decide access for the operator. - ResourcePolicy leasePolicy = firstPolicyWithEndDate(anonymousReadPolicies); - if (leasePolicy != null) { - prErr("Bitstream '" + bitstream.getName() + "' (" + bitstream.getID() + ") of item " - + itemLabel(item) + " has an " + Group.ANONYMOUS + " READ policy with an end" - + " date (policy #" + leasePolicy.getID() + ", rpName='" - + leasePolicy.getRpName() + "'), which this tool does not manage, so its" - + " embargo could not be synchronised and the bitstream is left untouched. " - + BULK_ACCESS_CONTROL_HINT); - embargoSyncFailures++; - continue; - } + if (anonymousReadPolicies.isEmpty()) { + prErr("Bitstream " + bitstreamLabel(bundleName, bitstream) + " of item " + itemLabel(item) + + " has no " + Group.ANONYMOUS + " READ policy, so there is nothing to re-date and its" + + " embargo could not be synchronised. No policy is created: that would grant access" + + " nobody ever granted. " + accessChangeHint(bundleName)); + embargoSyncFailures++; + return; + } - ResourcePolicy survivor = selectSurvivorPolicy(anonymousReadPolicies); - survivor.setStartDate(startDate); - survivor.setRpType(ResourcePolicy.TYPE_CUSTOM); - survivor.setRpName(SafEmbargoConstants.EMBARGO_POLICY_NAME); - resourcePolicyService.update(context, survivor); - - // The duplicates go only once the survivor is stored. Reference identity rather than - // equals(): ResourcePolicy.equals compares values, which the lines above just changed. - for (ResourcePolicy policy : anonymousReadPolicies) { - if (policy != survivor) { - resourcePolicyService.delete(context, policy); - } - } + // A policy with an end date is a lease, not an embargo: re-dating it would either close the file for + // good or drop the end date, and either way decide access for the operator. + ResourcePolicy leasePolicy = firstPolicyWithEndDate(anonymousReadPolicies); + if (leasePolicy != null) { + prErr("Bitstream " + bitstreamLabel(bundleName, bitstream) + " of item " + itemLabel(item) + + " has an " + Group.ANONYMOUS + " READ policy with an end date (policy #" + + leasePolicy.getID() + ", rpName='" + leasePolicy.getRpName() + "'), which this tool" + + " does not manage, so its embargo could not be synchronised and the bitstream is left" + + " untouched. " + accessChangeHint(bundleName)); + embargoSyncFailures++; + return; + } + + ResourcePolicy survivor = selectSurvivorPolicy(anonymousReadPolicies); + survivor.setStartDate(startDate); + survivor.setRpType(ResourcePolicy.TYPE_CUSTOM); + survivor.setRpName(SafEmbargoConstants.EMBARGO_POLICY_NAME); + resourcePolicyService.update(context, survivor); + + // The duplicates go only once the survivor is stored. Reference identity rather than equals(): + // ResourcePolicy.equals compares values, which the lines above just changed. + for (ResourcePolicy policy : anonymousReadPolicies) { + if (policy != survivor) { + resourcePolicyService.delete(context, policy); } } } + /** + * Bundle, name and id of a bitstream, so that a report identifies exactly one file of the item. + * + * @param bundleName bundle the bitstream lives in + * @param bitstream bitstream to describe + * @return label of the form {@code BUNDLE/name (uuid)} + */ + protected String bitstreamLabel(String bundleName, Bitstream bitstream) { + return bundleName + "/'" + bitstream.getName() + "' (" + bitstream.getID() + ")"; + } + + /** + * How to change access to a bitstream this tool refuses to touch. {@code bulk-access-control} walks the + * ORIGINAL bundle only, so it cannot be recommended for a derived one. + * + * @param bundleName bundle the bitstream lives in + * @return the hint matching the bundle + */ + protected String accessChangeHint(String bundleName) { + return Constants.CONTENT_BUNDLE_NAME.equals(bundleName) ? BULK_ACCESS_CONTROL_HINT + : DERIVATIVE_ACCESS_HINT; + } + /** * First policy of the list that expires by itself. * diff --git a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java index 212e6777da21..bcd5940edb9c 100644 --- a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java +++ b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java @@ -7,6 +7,12 @@ */ package org.dspace.app.util; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +import org.dspace.core.Constants; + /** * Constants of the embargo resource policies written by the SAF batch tools, declared once so that * {@code dspace import} and {@code dspace itemupdate} cannot drift apart. @@ -20,6 +26,19 @@ public final class SafEmbargoConstants { */ public static final String EMBARGO_POLICY_NAME = "embargo"; + /** Bundle holding the text extracted from a file; {@link Constants} has no name for it. */ + public static final String TEXT_BUNDLE_NAME = "TEXT"; + + /** Bundle holding the thumbnail rendered from a file; {@link Constants} has no name for it. */ + public static final String THUMBNAIL_BUNDLE_NAME = "THUMBNAIL"; + + /** + * Bundles an embargo covers: the file itself and everything derived from it, because the thumbnail and the + * extracted full text disclose the embargoed file. {@code filter.*.publicPermission} is ignored on purpose. + */ + public static final List EMBARGOED_BUNDLE_NAMES = Collections.unmodifiableList(Arrays.asList( + Constants.CONTENT_BUNDLE_NAME, TEXT_BUNDLE_NAME, THUMBNAIL_BUNDLE_NAME)); + private SafEmbargoConstants() { } } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java new file mode 100644 index 000000000000..8faf4a4405e4 --- /dev/null +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java @@ -0,0 +1,707 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.app.itemupdate; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.PrintStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.SQLException; +import java.time.LocalDate; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Date; +import java.util.List; +import java.util.Set; +import java.util.TreeSet; +import java.util.stream.Collectors; + +import org.apache.commons.io.file.PathUtils; +import org.dspace.AbstractIntegrationTestWithDatabase; +import org.dspace.authorize.ResourcePolicy; +import org.dspace.authorize.factory.AuthorizeServiceFactory; +import org.dspace.authorize.service.AuthorizeService; +import org.dspace.authorize.service.ResourcePolicyService; +import org.dspace.builder.BitstreamBuilder; +import org.dspace.builder.CollectionBuilder; +import org.dspace.builder.CommunityBuilder; +import org.dspace.builder.ItemBuilder; +import org.dspace.builder.MetadataFieldBuilder; +import org.dspace.builder.ResourcePolicyBuilder; +import org.dspace.content.Bitstream; +import org.dspace.content.Bundle; +import org.dspace.content.Collection; +import org.dspace.content.Item; +import org.dspace.content.MetadataField; +import org.dspace.content.MetadataSchema; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.ItemService; +import org.dspace.content.service.MetadataFieldService; +import org.dspace.content.service.MetadataSchemaService; +import org.dspace.core.Constants; +import org.dspace.eperson.EPerson; +import org.dspace.eperson.Group; +import org.dspace.eperson.factory.EPersonServiceFactory; +import org.dspace.eperson.service.GroupService; +import org.dspace.handle.factory.HandleServiceFactory; +import org.dspace.handle.service.HandleService; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +/** + * Embargo synchronisation of the bundles derived from an embargoed file. While an item is under embargo + * neither its extracted full text (TEXT) nor its thumbnail (THUMBNAIL) may be readable, and both have to be + * re-opened when the embargo ends. Bundles that are not derived from the file, LICENSE above all, stay out + * of scope. + */ +public class EmbargoDerivativesIT extends AbstractIntegrationTestWithDatabase { + + /** Policy name the synchronisation writes. */ + private static final String EMBARGO_POLICY_NAME = "embargo"; + + private static final String OPEN_ACCESS = "openAccess"; + private static final String EMBARGOED_ACCESS = "embargoedAccess"; + private static final String RESTRICTED_ACCESS = "restrictedAccess"; + + private static final String TEXT_BUNDLE = "TEXT"; + private static final String THUMBNAIL_BUNDLE = "THUMBNAIL"; + private static final String LICENSE_BUNDLE = "LICENSE"; + private static final String CC_LICENSE_BUNDLE = "CC-LICENSE"; + + private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); + private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); + private final ResourcePolicyService resourcePolicyService = + AuthorizeServiceFactory.getInstance().getResourcePolicyService(); + private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); + private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); + private final MetadataSchemaService metadataSchemaService = + ContentServiceFactory.getInstance().getMetadataSchemaService(); + private final MetadataFieldService metadataFieldService = + ContentServiceFactory.getInstance().getMetadataFieldService(); + + private Collection collection; + private Group anonymousGroup; + private Path tempDir; + private String previousHandlePrefix; + + @Before + @Override + public void setUp() throws Exception { + super.setUp(); + context.turnOffAuthorisationSystem(); + + parentCommunity = CommunityBuilder.createCommunity(context) + .withName("Parent Community") + .build(); + collection = CollectionBuilder.createCollection(context, parentCommunity) + .withName("Collection") + .build(); + + // neither field exists in the test metadata registry, AddMetadataAction needs both + ensureMetadataFieldExists("rights", "access"); + ensureMetadataFieldExists("date", "embargoend"); + + anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); + // ItemArchive resolves items through this mutable static, it is restored in destroy() + previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; + ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); + + context.restoreAuthSystemState(); + + tempDir = Files.createTempDirectory("embargoDerivativesIT"); + } + + @After + @Override + public void destroy() throws Exception { + ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; + if (tempDir != null) { + PathUtils.deleteDirectory(tempDir); + } + super.destroy(); + } + + /** + * Verifies that a running embargo closes the thumbnail and the extracted full text as well. The TEXT + * bundle holds the whole content of the file, so leaving it public publishes the embargoed work. + */ + @Test + public void futureEmbargoAlsoClosesTextAndThumbnail() throws Exception { + LocalDate embargoEnd = LocalDate.now().plusMonths(6); + LocalDate accessStart = embargoEnd.plusDays(1); + + Item item = createItem("Embargoed Derivatives Thesis"); + List files = createOriginalWithDerivatives(item, "thesis.pdf"); + + for (Bitstream file : files) { + assertTrue("fixture precondition: [" + label(file) + "] must start publicly readable" + + describe(file), + anonymousCanRead(file)); + } + + Run run = runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, embargoEnd.toString())); + item = context.reloadEntity(item); + reloadAll(files); + + assertExitCode("future embargo on an item with derivatives", 0, run); + for (Bitstream file : files) { + ResourcePolicy policy = onlyAnonymousReadPolicy(file, "the embargo run"); + assertNotNull("[" + label(file) + "] the surviving Anonymous READ policy carries no start date, so" + + " the embargo was never written onto this bundle." + describe(file), + policy.getStartDate()); + assertEquals("[" + label(file) + "] the embargo belongs to the item, so every bundle derived from" + + " the embargoed file has to carry the same start date." + describe(file), + accessStart, toLocalDate(policy.getStartDate())); + assertFalse("[" + label(file) + "] is publicly readable while the item is under embargo until " + + embargoEnd + ". Neither the thumbnail nor the extracted full text of an" + + " embargoed file may be reachable." + describe(file), + anonymousCanRead(file)); + } + } + + /** + * Verifies that an expired embargo re-opens the derivatives too. Closing them is only half the job; a + * published thesis whose thumbnail stays hidden is just as wrong. + */ + @Test + public void expiredEmbargoReopensTextAndThumbnail() throws Exception { + LocalDate embargoEnd = LocalDate.now().minusMonths(2); + LocalDate accessStart = embargoEnd.plusDays(1); + + Item item = createItem("Expired Derivatives Thesis"); + List files = createOriginalWithDerivatives(item, "expired.pdf"); + + // the state a finished embargo run leaves behind: one dated Anonymous READ policy on every bundle + for (Bitstream file : files) { + replaceAnonymousReadPolicy(file, startOfDayUtc(LocalDate.now().plusYears(1)), EMBARGO_POLICY_NAME); + } + reloadAll(files); + for (Bitstream file : files) { + assertFalse("fixture precondition: [" + label(file) + "] must start closed" + describe(file), + anonymousCanRead(file)); + } + + Run run = runItemUpdate(item, dublinCore(item, OPEN_ACCESS, embargoEnd.toString())); + item = context.reloadEntity(item); + reloadAll(files); + + assertExitCode("expired embargo on an item with derivatives", 0, run); + for (Bitstream file : files) { + ResourcePolicy policy = onlyAnonymousReadPolicy(file, "the expired embargo run"); + assertNotNull("[" + label(file) + "] the Anonymous READ policy lost its start date instead of being" + + " re-dated to the day the embargo ended." + describe(file), + policy.getStartDate()); + assertEquals("[" + label(file) + "] wrong start date after an embargo that ended on " + embargoEnd + + describe(file), + accessStart, toLocalDate(policy.getStartDate())); + assertTrue("[" + label(file) + "] is still closed although the embargo ended on " + embargoEnd + + ". An expired embargo publishes the file together with everything derived from" + + " it." + describe(file), + anonymousCanRead(file)); + } + } + + /** + * Verifies that the derivative policy is re-dated in place, the rule the ORIGINAL bundle already follows: + * a delete plus create leaves the bitstream without any policy if the run breaks in between. + */ + @Test + public void derivativePolicyIsMutatedNotRecreated() throws Exception { + LocalDate embargoEnd = LocalDate.now().plusMonths(3); + LocalDate accessStart = embargoEnd.plusDays(1); + + Item item = createItem("Mutated Derivatives Thesis"); + List files = createOriginalWithDerivatives(item, "mutate.pdf"); + List before = snapshotAll(files); + + Run run = runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, embargoEnd.toString())); + item = context.reloadEntity(item); + reloadAll(files); + + assertExitCode("future embargo on an item with derivatives", 0, run); + for (int i = 0; i < files.size(); i++) { + Bitstream file = files.get(i); + ResourcePolicy policy = onlyAnonymousReadPolicy(file, "the embargo run"); + assertEquals("[" + label(file) + "] the set of policy ids changed, so the Anonymous READ policy was" + + " deleted and recreated instead of being re-dated." + describe(file), + before.get(i).ids, policyIds(file)); + assertNotNull("[" + label(file) + "] the policy was kept but never re-dated, so this bundle stayed" + + " outside the embargo." + describe(file), + policy.getStartDate()); + assertEquals("[" + label(file) + "] wrong embargo start date" + describe(file), + accessStart, toLocalDate(policy.getStartDate())); + assertEquals("[" + label(file) + "] the mutated policy has to be named '" + EMBARGO_POLICY_NAME + + "', otherwise the next run cannot tell it apart from a foreign one." + + describe(file), + EMBARGO_POLICY_NAME, policy.getRpName()); + } + } + + /** + * Verifies that the scope stops at the derived bundles. A licence is not derived from the embargoed file + * and has to stay readable, whatever the embargo end date says. + */ + @Test + public void licenseBundleIsNeverTouched() throws Exception { + LocalDate futureEnd = LocalDate.now().plusMonths(4); + LocalDate pastEnd = LocalDate.now().minusMonths(4); + + Item item = createItem("Licensed Thesis"); + Bitstream original = createBitstreamInBundle(item, "licensed.pdf", Constants.CONTENT_BUNDLE_NAME); + List licences = new ArrayList<>(); + licences.add(createBitstreamInBundle(item, "license.txt", LICENSE_BUNDLE)); + licences.add(createBitstreamInBundle(item, "license_rdf", CC_LICENSE_BUNDLE)); + List before = snapshotAll(licences); + + Run embargoRun = runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEnd.toString())); + item = context.reloadEntity(item); + original = context.reloadEntity(original); + reloadAll(licences); + + assertExitCode("future embargo on an item with licence bundles", 0, embargoRun); + assertFalse("sanity check: this run has to embargo the ORIGINAL bitstream, otherwise nothing happened" + + " at all and the licence bundles are untested." + describe(original), + anonymousCanRead(original)); + assertUntouched("future embargo end date", before, licences); + + Run expiredRun = runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEnd.toString())); + item = context.reloadEntity(item); + original = context.reloadEntity(original); + reloadAll(licences); + + assertExitCode("expired embargo on an item with licence bundles", 0, expiredRun); + assertTrue("sanity check: the expired embargo has to publish the ORIGINAL bitstream" + + describe(original), + anonymousCanRead(original)); + assertUntouched("expired embargo end date", before, licences); + for (Bitstream licence : licences) { + assertTrue("[" + label(licence) + "] the licence text must stay publicly readable" + + describe(licence), + anonymousCanRead(licence)); + } + } + + /** + * Verifies that a derivative without an {@code Anonymous}/{@code READ} policy does not get one. Creating + * it would grant access nobody granted, so the run reports the bitstream instead. + */ + @Test + public void derivativeWithoutAnonymousReadIsNotPublished() throws Exception { + LocalDate embargoEnd = LocalDate.now().plusMonths(5); + + Item item = createItem("Closed Text Thesis"); + Bitstream original = createBitstreamInBundle(item, "closed.pdf", Constants.CONTENT_BUNDLE_NAME); + Bitstream text = createBitstreamInBundle(item, "closed.pdf.txt", TEXT_BUNDLE); + + context.turnOffAuthorisationSystem(); + deleteReadPolicies(text); + context.restoreAuthSystemState(); + text = context.reloadEntity(text); + + assertTrue("fixture precondition: the TEXT bitstream must carry no READ policy" + describe(text), + readPolicies(text).isEmpty()); + + Run run = runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, embargoEnd.toString())); + item = context.reloadEntity(item); + original = context.reloadEntity(original); + text = context.reloadEntity(text); + + assertTrue("[TEXT] a READ policy was created for a bitstream that had none. Synchronising an embargo" + + " re-dates existing access, it never grants access nobody granted." + describe(text), + readPolicies(text).isEmpty()); + assertFalse("[TEXT] became publicly readable although it carried no policy before" + describe(text), + anonymousCanRead(text)); + assertTrue("the run has to report the TEXT bitstream " + text.getID() + " it could not synchronise," + + " otherwise the operator never learns that the extracted text was left behind." + + " Console output was:" + System.lineSeparator() + run.console, + run.console.contains(text.getID().toString())); + assertFalse("a derivative that cannot be synchronised must not stop the ORIGINAL bitstream from being" + + " embargoed" + describe(original), + anonymousCanRead(original)); + } + + /** + * Verifies that the guards which keep itemupdate away from an item cover the derived bundles too: an + * access right the tool does not manage and a withdrawn item both leave every policy where it was. + */ + @Test + public void guardsAlsoProtectDerivatives() throws Exception { + Item restricted = createItem("Restricted Derivatives Thesis"); + List restrictedFiles = createOriginalWithDerivatives(restricted, "restricted.pdf"); + List restrictedBefore = snapshotAll(restrictedFiles); + + for (String embargoEnd : Arrays.asList(LocalDate.now().plusYears(1).toString(), + LocalDate.now().minusYears(1).toString())) { + Run run = runItemUpdate(restricted, dublinCore(restricted, RESTRICTED_ACCESS, embargoEnd)); + restricted = context.reloadEntity(restricted); + reloadAll(restrictedFiles); + + assertTrue("itemupdate has to refuse the item because of dc.rights.access=" + RESTRICTED_ACCESS + + ". Nothing in the console output says so, so some other guard stopped the run." + + " Console output was:" + System.lineSeparator() + run.console, + run.console.contains(RESTRICTED_ACCESS)); + assertUntouched(RESTRICTED_ACCESS + " with embargo end " + embargoEnd, restrictedBefore, + restrictedFiles); + } + + Item withdrawn = createItem("Withdrawn Derivatives Thesis"); + List withdrawnFiles = createOriginalWithDerivatives(withdrawn, "withdrawn.pdf"); + + context.turnOffAuthorisationSystem(); + itemService.withdraw(context, withdrawn); + context.restoreAuthSystemState(); + withdrawn = context.reloadEntity(withdrawn); + reloadAll(withdrawnFiles); + + assertTrue("fixture precondition: the item must be withdrawn", withdrawn.isWithdrawn()); + List withdrawnBefore = snapshotAll(withdrawnFiles); + + Run run = runItemUpdate(withdrawn, + dublinCore(withdrawn, EMBARGOED_ACCESS, LocalDate.now().plusYears(1).toString())); + withdrawn = context.reloadEntity(withdrawn); + reloadAll(withdrawnFiles); + + assertTrue("itemupdate has to refuse the item because it is withdrawn. Console output was:" + + System.lineSeparator() + run.console, + run.console.contains("is withdrawn")); + assertUntouched("withdrawn item with a future embargo end date", withdrawnBefore, withdrawnFiles); + for (Bitstream file : withdrawnFiles) { + assertFalse("[" + label(file) + "] a withdrawn file must not be publicly readable, the takedown" + + " covers everything derived from it." + describe(file), + anonymousCanRead(file)); + } + } + + /** + * Compares policy identity (ids) and policy content (fingerprints) of every bitstream: the ids catch + * delete-and-recreate, the fingerprints catch in-place mutation. + */ + private void assertUntouched(String scenario, List before, List bitstreams) + throws SQLException { + for (int i = 0; i < bitstreams.size(); i++) { + Bitstream bitstream = bitstreams.get(i); + assertEquals("[" + scenario + "][" + label(bitstream) + "] the set of resource policy ids changed:" + + " policies were deleted and/or recreated although this bundle is out of scope." + + describe(bitstream), + before.get(i).ids, policyIds(bitstream)); + assertEquals("[" + scenario + "][" + label(bitstream) + "] a surviving resource policy was modified" + + " in place although this bundle is out of scope." + describe(bitstream), + before.get(i).fingerprints, policyFingerprints(bitstream)); + } + } + + /** + * Runs itemupdate with both embargo fields as targets, the combination that triggers embargo + * synchronisation. + * + * @return the console output of the run and the number of embargo problems it counted + */ + private Run runItemUpdate(Item item, String dublinCoreContent) throws Exception { + Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); + // without this marker processArchive writes an undo archive next to the source directory + Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); + + Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); + Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); + + ItemUpdate itemUpdate = new ItemUpdate(); + DeleteMetadataAction deleteAction = + (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); + deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + AddMetadataAction addAction = + (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); + addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + // ItemUpdate reports to the console only, so it has to be captured to assert on it + ByteArrayOutputStream consoleBuffer = new ByteArrayOutputStream(); + PrintStream originalOut = System.out; + PrintStream originalErr = System.err; + PrintStream captureStream = new PrintStream(consoleBuffer, true, StandardCharsets.UTF_8); + + context.turnOffAuthorisationSystem(); + System.setOut(captureStream); + System.setErr(captureStream); + try { + itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); + } finally { + captureStream.flush(); + System.setOut(originalOut); + System.setErr(originalErr); + context.restoreAuthSystemState(); + } + + context.uncacheEntity(item); + + String consoleOutput = consoleBuffer.toString(StandardCharsets.UTF_8); + // replay it so the failsafe -output.txt still holds the full ItemUpdate log + System.out.println(consoleOutput); + return new Run(consoleOutput, itemUpdate.embargoSyncFailures); + } + + /** + * What a finished {@code itemupdate} run is judged by: its console output and its failure count. + */ + private static final class Run { + private final String console; + private final int embargoSyncFailures; + + private Run(String console, int embargoSyncFailures) { + this.console = console; + this.embargoSyncFailures = embargoSyncFailures; + } + } + + /** + * Identity and content of every policy of one bitstream, taken before a run. + */ + private static final class Snapshot { + private final Set ids; + private final List fingerprints; + + private Snapshot(Set ids, List fingerprints) { + this.ids = ids; + this.fingerprints = fingerprints; + } + } + + private List snapshotAll(List bitstreams) throws SQLException { + List snapshots = new ArrayList<>(); + for (Bitstream bitstream : bitstreams) { + snapshots.add(new Snapshot(policyIds(bitstream), policyFingerprints(bitstream))); + } + return snapshots; + } + + /** + * Asserts the exit code the run would have produced; without it a skipped item looks synchronised. + */ + private void assertExitCode(String scenario, int expectedFailures, Run run) { + assertEquals("[" + scenario + "] wrong number of reported embargo problems, so ItemUpdate.main() would" + + " exit with " + ItemUpdate.exitStatus(0, run.embargoSyncFailures) + " instead of " + + ItemUpdate.exitStatus(0, expectedFailures) + ". Console output was:" + + System.lineSeparator() + run.console, + expectedFailures, run.embargoSyncFailures); + } + + /** + * Builds a SAF {@code dublin_core.xml} carrying one {@code dc.rights.access} and one + * {@code dc.date.embargoend} value. + */ + private String dublinCore(Item item, String accessRight, String embargoEndDate) { + return "\n" + + "\n" + + " " + + ItemUpdate.HANDLE_PREFIX + item.getHandle() + "\n" + + " " + accessRight + "\n" + + " " + embargoEndDate + "\n" + + ""; + } + + private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { + MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); + MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); + if (existingField == null) { + MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); + } + } + + private Item createItem(String title) throws Exception { + context.turnOffAuthorisationSystem(); + Item item = ItemBuilder.createItem(context, collection) + .withTitle(title) + .build(); + context.restoreAuthSystemState(); + return item; + } + + /** + * The state filter-media leaves behind on a public item: the file plus its extracted text and its + * thumbnail, each with the one undated Anonymous READ policy inherited from the collection. + */ + private List createOriginalWithDerivatives(Item item, String fileName) throws Exception { + List files = new ArrayList<>(); + files.add(createBitstreamInBundle(item, fileName, Constants.CONTENT_BUNDLE_NAME)); + files.add(createBitstreamInBundle(item, fileName + ".txt", TEXT_BUNDLE)); + files.add(createBitstreamInBundle(item, fileName + ".jpg", THUMBNAIL_BUNDLE)); + return files; + } + + private Bitstream createBitstreamInBundle(Item item, String name, String bundleName) throws Exception { + context.turnOffAuthorisationSystem(); + Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, + new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8)), + bundleName) + .withName(name) + .withMimeType("text/plain") + .build(); + context.restoreAuthSystemState(); + return bitstream; + } + + /** + * Replaces every READ policy of the bitstream with a single dated Anonymous READ policy. + */ + private ResourcePolicy replaceAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) + throws Exception { + context.turnOffAuthorisationSystem(); + deleteReadPolicies(bitstream); + ResourcePolicy policy = ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) + .withAction(Constants.READ) + .withDspaceObject(bitstream) + .withName(name) + .withPolicyType(ResourcePolicy.TYPE_CUSTOM) + .withStartDate(startDate) + .build(); + context.restoreAuthSystemState(); + return policy; + } + + /** + * Deletes READ policies one by one, because the bulk removal helpers issue an HQL delete and leave the + * in-memory collection stale. + */ + private void deleteReadPolicies(Bitstream bitstream) throws Exception { + for (ResourcePolicy policy : readPolicies(bitstream)) { + resourcePolicyService.delete(context, policy); + } + } + + private List readPolicies(Bitstream bitstream) throws SQLException { + return new ArrayList<>(resourcePolicyService.find(context, bitstream, Constants.READ)); + } + + private List anonymousReadPolicies(Bitstream bitstream) throws SQLException { + return readPolicies(bitstream).stream() + .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) + .collect(Collectors.toList()); + } + + private ResourcePolicy onlyAnonymousReadPolicy(Bitstream bitstream, String what) throws SQLException { + List policies = anonymousReadPolicies(bitstream); + assertEquals("[" + label(bitstream) + "] exactly one Anonymous READ policy must remain after " + what + + ", a second one would defeat the embargo." + describe(bitstream), + 1, policies.size()); + return policies.get(0); + } + + private Set policyIds(Bitstream bitstream) throws SQLException { + Set ids = new TreeSet<>(); + for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream)) { + ids.add(policy.getID()); + } + return ids; + } + + private List policyFingerprints(Bitstream bitstream) throws SQLException { + List fingerprints = new ArrayList<>(); + for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream)) { + fingerprints.add(fingerprint(policy)); + } + Collections.sort(fingerprints); + return fingerprints; + } + + private String fingerprint(ResourcePolicy policy) { + return String.format("id=%s action=%s group=%s eperson=%s rpType=%s rpName=%s start=%s end=%s", + policy.getID(), + Constants.actionText[policy.getAction()], + policy.getGroup() == null ? "" : policy.getGroup().getName(), + policy.getEPerson() == null ? "" : policy.getEPerson().getEmail(), + policy.getRpType(), + policy.getRpName(), + day(policy.getStartDate()), + day(policy.getEndDate())); + } + + /** + * Renders the current policies of the bitstream for failure messages, so a red build shows which policy + * moved. + */ + private String describe(Bitstream bitstream) throws SQLException { + StringBuilder sb = new StringBuilder(System.lineSeparator()) + .append(" bitstream=").append(label(bitstream)).append(" (").append(bitstream.getID()) + .append(")").append(System.lineSeparator()) + .append(" anonymousCanRead=").append(anonymousCanRead(bitstream)) + .append(System.lineSeparator()); + List fingerprints = policyFingerprints(bitstream); + if (fingerprints.isEmpty()) { + sb.append(" ").append(System.lineSeparator()); + } + for (String fingerprint : fingerprints) { + sb.append(" ").append(fingerprint).append(System.lineSeparator()); + } + return sb.toString(); + } + + /** + * Bundle and file name of a bitstream, so a failure message says which bundle is wrong. + */ + private String label(Bitstream bitstream) throws SQLException { + List bundles = bitstream.getBundles(); + return (bundles.isEmpty() ? "" : bundles.get(0).getName()) + "/" + bitstream.getName(); + } + + /** + * Tells whether a visitor who is not logged in may read the bitstream. The authorisation state is a stack + * the builders push and pop, so it is drained first - otherwise every read looks allowed. + */ + private boolean anonymousCanRead(Bitstream bitstream) throws SQLException { + EPerson savedUser = context.getCurrentUser(); + int popped = 0; + while (context.ignoreAuthorization()) { + context.restoreAuthSystemState(); + popped++; + } + context.setCurrentUser(null); + try { + return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); + } finally { + context.setCurrentUser(savedUser); + for (int i = 0; i < popped; i++) { + context.turnOffAuthorisationSystem(); + } + } + } + + private void reloadAll(List bitstreams) throws SQLException { + for (int i = 0; i < bitstreams.size(); i++) { + bitstreams.set(i, context.reloadEntity(bitstreams.get(i))); + } + } + + private Date startOfDayUtc(LocalDate day) { + return Date.from(day.atStartOfDay(ZoneOffset.UTC).toInstant()); + } + + /** + * Start dates come back from the database as {@code java.sql.Date}, so they are compared at calendar day + * granularity instead of as instants. + */ + private LocalDate toLocalDate(Date date) { + if (date instanceof java.sql.Date) { + return ((java.sql.Date) date).toLocalDate(); + } + return date.toInstant().atZone(ZoneId.systemDefault()).toLocalDate(); + } + + private String day(Date date) { + return date == null ? "" : toLocalDate(date).toString(); + } +} diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java index e78f8ae22abd..849a045b2975 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java @@ -69,7 +69,7 @@ * Life cycle of an embargo driven by {@link ItemUpdate}: setting it, re-running the same SAF archive, and * ending it with a {@code dc.date.embargoend} in the past, including on policies written by earlier versions. * Covers the invariants that one {@code Anonymous}/{@code READ} policy survives every run, that it is mutated - * rather than recreated, and that bitstreams outside the ORIGINAL bundle are left alone. + * rather than recreated, and that the bundles derived from an embargoed file follow it. */ public class EmbargoLifecycleIT extends AbstractIntegrationTestWithDatabase { @@ -525,64 +525,74 @@ public void multipleBitstreamsAllGetSameState() throws Exception { } /** - * Verifies that only ORIGINAL bitstreams are synchronised. Derivatives are produced and re-protected by - * filter-media, and the bundle objects carry policies of their own. + * Verifies that the derived bundles follow the embargo of the file they were derived from: the thumbnail + * and the extracted full text of an embargoed file must not be readable, and both re-open with it. The + * bundle objects themselves stay out of scope, only their bitstreams are synchronised. */ @Test - public void derivativeBundlesAreNotTouchedDirectly() throws Exception { - String futureEmbargoEnd = LocalDate.now().plusMonths(2).toString(); - String pastEmbargoEnd = LocalDate.now().minusMonths(2).toString(); + public void derivativeBundlesFollowTheEmbargo() throws Exception { + LocalDate futureEmbargoEnd = LocalDate.now().plusMonths(2); + LocalDate pastEmbargoEnd = LocalDate.now().minusMonths(2); Item item = createItem("Derivatives Thesis"); Bitstream original = createOriginalBitstream(item, "thesis.pdf"); Bitstream extractedText = createBitstreamInBundle(item, "thesis.pdf.txt", TEXT_BUNDLE); Bitstream thumbnail = createBitstreamInBundle(item, "thesis.pdf.jpg", THUMBNAIL_BUNDLE); + List files = new ArrayList<>(List.of(original, extractedText, thumbnail)); Bundle originalBundle = bundleOf(item, Constants.CONTENT_BUNDLE_NAME); - Set textPolicies = policySignatures(extractedText); - Set thumbnailPolicies = policySignatures(thumbnail); Set originalBundlePolicies = policySignatures(originalBundle); - assertFalse("fixture precondition: the TEXT bitstream must start with policies", textPolicies.isEmpty()); - assertFalse("fixture precondition: the THUMBNAIL bitstream must start with policies", - thumbnailPolicies.isEmpty()); assertFalse("fixture precondition: the ORIGINAL bundle must start with policies", originalBundlePolicies.isEmpty()); + List policyIdsBefore = new ArrayList<>(); + for (Bitstream file : files) { + policyIdsBefore.add(onlyAnonymousReadPolicy(file, "the fixture setup").getID()); + } // embargo run - assertRunSucceeded("embargoing the ORIGINAL bitstream", - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); + assertRunSucceeded("embargoing the item", + runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd.toString()))); item = context.reloadEntity(item); - original = context.reloadEntity(original); - extractedText = context.reloadEntity(extractedText); - thumbnail = context.reloadEntity(thumbnail); + reloadAll(files); originalBundle = context.reloadEntity(originalBundle); - ResourcePolicy embargoed = onlyAnonymousReadPolicy(original, "the embargo run"); - assertEquals("sanity check: this run must have embargoed the ORIGINAL bitstream", - EMBARGO_POLICY_NAME, embargoed.getRpName()); - assertEquals("TEXT bitstream policies belong to filter-media and must not be rewritten here", - textPolicies, policySignatures(extractedText)); - assertEquals("THUMBNAIL bitstream policies belong to filter-media and must not be rewritten here", - thumbnailPolicies, policySignatures(thumbnail)); + for (int i = 0; i < files.size(); i++) { + Bitstream file = files.get(i); + ResourcePolicy policy = onlyAnonymousReadPolicy(file, "the embargo run"); + assertEquals("the Anonymous READ policy was recreated instead of being re-dated" + + describePolicies(file), + policyIdsBefore.get(i), policy.getID()); + assertEquals("every bundle derived from the embargoed file carries the item embargo" + + describePolicies(file), + EMBARGO_POLICY_NAME, policy.getRpName()); + assertEquals("wrong embargo start date" + describePolicies(file), + futureEmbargoEnd.plusDays(1), toLocalDate(policy.getStartDate())); + assertFalse("a file of an item under embargo must not be publicly readable" + + describePolicies(file), + anonymousCanRead(file)); + } assertEquals("the ORIGINAL bundle's own policies must not be touched", originalBundlePolicies, policySignatures(originalBundle)); // expired embargo run assertRunSucceeded("letting the embargo expire", - runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd))); + runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd.toString()))); item = context.reloadEntity(item); - original = context.reloadEntity(original); - extractedText = context.reloadEntity(extractedText); - thumbnail = context.reloadEntity(thumbnail); + reloadAll(files); originalBundle = context.reloadEntity(originalBundle); - assertTrue("sanity check: the expired embargo must publish the ORIGINAL bitstream" - + describePolicies(original), - anonymousCanRead(original)); - assertEquals("TEXT bitstream policies must survive an expired embargo untouched", - textPolicies, policySignatures(extractedText)); - assertEquals("THUMBNAIL bitstream policies must survive an expired embargo untouched", - thumbnailPolicies, policySignatures(thumbnail)); + for (int i = 0; i < files.size(); i++) { + Bitstream file = files.get(i); + ResourcePolicy policy = onlyAnonymousReadPolicy(file, "the expired embargo run"); + assertEquals("the Anonymous READ policy was recreated instead of being re-dated" + + describePolicies(file), + policyIdsBefore.get(i), policy.getID()); + assertEquals("wrong start date after an expired embargo" + describePolicies(file), + pastEmbargoEnd.plusDays(1), toLocalDate(policy.getStartDate())); + assertTrue("an expired embargo publishes the file together with everything derived from it" + + describePolicies(file), + anonymousCanRead(file)); + } assertEquals("the ORIGINAL bundle's own policies must survive an expired embargo untouched", originalBundlePolicies, policySignatures(originalBundle)); } From c5fe8e43516cc19356f1a9375aae166f9edf15c2 Mon Sep 17 00:00:00 2001 From: milanmajchrak Date: Thu, 20 Aug 2026 10:48:45 +0200 Subject: [PATCH 08/10] VSB-TUO/Test: share the embargo integration test fixture The six ItemUpdate embargo test classes each carried their own copy of the same setup: the collection whose bitstreams inherit an undated Anonymous READ policy, the metadata field registration, the SAF archive writer that drives a run, and the policy inspection helpers. AbstractEmbargoIT now holds that fixture and each class keeps only the scenario it pins down. No test case is removed and no assertion changes. Three helpers had drifted apart between the copies and are unified on the version that was already documented as the correct one: - replaceAnonymousReadPolicies deletes policies one by one; the bulk removePoliciesActionFilter leaves the in-memory collection stale - dublinCore has one implementation instead of five - a run tees the console, so the output still reaches the failsafe report --- .../app/itemupdate/AbstractEmbargoIT.java | 517 ++++++++++++++++++ .../app/itemupdate/EmbargoDateBoundaryIT.java | 314 +---------- .../app/itemupdate/EmbargoDerivativesIT.java | 332 +---------- .../app/itemupdate/EmbargoLifecycleIT.java | 325 +---------- .../app/itemupdate/EmbargoPastDateIT.java | 204 +------ .../app/itemupdate/EmbargoSafetyIT.java | 376 +------------ .../dspace/app/itemupdate/ItemUpdateIT.java | 306 +---------- 7 files changed, 604 insertions(+), 1770 deletions(-) create mode 100644 dspace-api/src/test/java/org/dspace/app/itemupdate/AbstractEmbargoIT.java diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/AbstractEmbargoIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/AbstractEmbargoIT.java new file mode 100644 index 000000000000..474347afeaa9 --- /dev/null +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/AbstractEmbargoIT.java @@ -0,0 +1,517 @@ +/** + * The contents of this file are subject to the license and copyright + * detailed in the LICENSE and NOTICE files at the root of the source + * tree and available online at + * + * http://www.dspace.org/license/ + */ +package org.dspace.app.itemupdate; + +import static org.junit.Assert.assertEquals; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.PrintStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.SQLException; +import java.time.LocalDate; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Date; +import java.util.List; +import java.util.Set; +import java.util.TreeSet; +import java.util.stream.Collectors; + +import org.apache.commons.io.file.PathUtils; +import org.apache.commons.io.output.TeeOutputStream; +import org.dspace.AbstractIntegrationTestWithDatabase; +import org.dspace.authorize.ResourcePolicy; +import org.dspace.authorize.factory.AuthorizeServiceFactory; +import org.dspace.authorize.service.AuthorizeService; +import org.dspace.authorize.service.ResourcePolicyService; +import org.dspace.builder.BitstreamBuilder; +import org.dspace.builder.CollectionBuilder; +import org.dspace.builder.CommunityBuilder; +import org.dspace.builder.ItemBuilder; +import org.dspace.builder.MetadataFieldBuilder; +import org.dspace.builder.ResourcePolicyBuilder; +import org.dspace.content.Bitstream; +import org.dspace.content.Collection; +import org.dspace.content.Item; +import org.dspace.content.MetadataField; +import org.dspace.content.MetadataSchema; +import org.dspace.content.MetadataValue; +import org.dspace.content.factory.ContentServiceFactory; +import org.dspace.content.service.BundleService; +import org.dspace.content.service.ItemService; +import org.dspace.content.service.MetadataFieldService; +import org.dspace.content.service.MetadataSchemaService; +import org.dspace.core.Constants; +import org.dspace.eperson.EPerson; +import org.dspace.eperson.Group; +import org.dspace.eperson.factory.EPersonServiceFactory; +import org.dspace.eperson.service.GroupService; +import org.dspace.handle.factory.HandleServiceFactory; +import org.dspace.handle.service.HandleService; +import org.junit.After; +import org.junit.Before; + +/** + * Fixture shared by the {@link ItemUpdate} embargo integration tests: a collection whose bitstreams inherit + * an undated {@code Anonymous}/{@code READ} policy, the metadata fields the embargo actions target, and the + * plumbing needed to drive a {@code dspace itemupdate} run against a SAF archive built on the fly. + * + *

    Assertions specific to one scenario stay in the test class that makes them; only fixture building and + * policy inspection live here.

    + */ +public abstract class AbstractEmbargoIT extends AbstractIntegrationTestWithDatabase { + + /** The single normalised rpName, matching the access condition name in access-conditions.xml. */ + protected static final String EMBARGO_POLICY_NAME = "embargo"; + + /** rpName written by earlier versions; fixtures use it so that normalisation is exercised. */ + protected static final String LEGACY_EMBARGO_POLICY_NAME = "Standard Embargo"; + + /** Bundle holding the text extracted from a file. */ + protected static final String TEXT_BUNDLE = "TEXT"; + + /** Bundle holding the thumbnail rendered from a file. */ + protected static final String THUMBNAIL_BUNDLE = "THUMBNAIL"; + + /** + * Sentinel for {@link #deletePolicies(Bitstream, int)} meaning "every action", picked so it can never + * collide with a real value of {@link Constants#actionText}. + */ + protected static final int ALL_ACTIONS = -1; + + protected final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); + protected final BundleService bundleService = ContentServiceFactory.getInstance().getBundleService(); + protected final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); + protected final ResourcePolicyService resourcePolicyService = + AuthorizeServiceFactory.getInstance().getResourcePolicyService(); + protected final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); + protected final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); + protected final MetadataSchemaService metadataSchemaService = + ContentServiceFactory.getInstance().getMetadataSchemaService(); + protected final MetadataFieldService metadataFieldService = + ContentServiceFactory.getInstance().getMetadataFieldService(); + + protected Collection collection; + protected Group anonymousGroup; + protected Path tempDir; + + private String previousHandlePrefix; + + @Before + @Override + public void setUp() throws Exception { + super.setUp(); + context.turnOffAuthorisationSystem(); + + parentCommunity = CommunityBuilder.createCommunity(context) + .withName("Parent Community") + .build(); + collection = CollectionBuilder.createCollection(context, parentCommunity) + .withName("Collection") + .build(); + + // none of these exist in the test metadata registry, the update actions need them as targets + ensureMetadataFieldExists("rights", "access"); + ensureMetadataFieldExists("date", "embargoend"); + ensureMetadataFieldExists("identifier", "thesis"); + + anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); + // ItemArchive resolves items through this mutable static, it is restored in destroy() + previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; + ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); + + context.restoreAuthSystemState(); + + tempDir = Files.createTempDirectory(getClass().getSimpleName()); + } + + @After + @Override + public void destroy() throws Exception { + ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; + if (tempDir != null) { + PathUtils.deleteDirectory(tempDir); + } + super.destroy(); + } + + protected void ensureMetadataFieldExists(String element, String qualifier) throws Exception { + MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); + MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); + if (existingField == null) { + MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); + } + } + + /** + * @param metadataTriples element, qualifier and value of extra {@code dc} metadata, in groups of three + */ + protected Item createItem(String title, String... metadataTriples) throws Exception { + context.turnOffAuthorisationSystem(); + ItemBuilder builder = ItemBuilder.createItem(context, collection).withTitle(title); + for (int i = 0; i + 2 < metadataTriples.length; i += 3) { + builder.withMetadata("dc", metadataTriples[i], metadataTriples[i + 1], metadataTriples[i + 2]); + } + Item item = builder.build(); + context.restoreAuthSystemState(); + return item; + } + + /** + * A bitstream in the ORIGINAL bundle. It inherits the collection DEFAULT_BITSTREAM_READ and so carries one + * undated Anonymous READ policy, the state a freshly imported SAF item is in. + */ + protected Bitstream createOriginalBitstream(Item item, String name) throws Exception { + return createBitstreamInBundle(item, name, Constants.CONTENT_BUNDLE_NAME); + } + + protected Bitstream createBitstreamInBundle(Item item, String name, String bundleName) throws Exception { + context.turnOffAuthorisationSystem(); + Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, + new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8)), bundleName) + .withName(name) + .withMimeType("text/plain") + .build(); + context.restoreAuthSystemState(); + return bitstream; + } + + protected void reloadAll(List bitstreams) throws SQLException { + for (int i = 0; i < bitstreams.size(); i++) { + bitstreams.set(i, context.reloadEntity(bitstreams.get(i))); + } + } + + /** + * Tells whether a visitor who is not logged in may read the bitstream. The authorisation state is a stack + * the builders push and pop, so it is drained first - otherwise every read looks allowed. + */ + protected boolean anonymousCanRead(Bitstream bitstream) throws SQLException { + EPerson savedUser = context.getCurrentUser(); + int popped = 0; + while (context.ignoreAuthorization()) { + context.restoreAuthSystemState(); + popped++; + } + context.setCurrentUser(null); + try { + return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); + } finally { + context.setCurrentUser(savedUser); + for (int i = 0; i < popped; i++) { + context.turnOffAuthorisationSystem(); + } + } + } + + protected List allPolicies(Bitstream bitstream) throws SQLException { + return new ArrayList<>(resourcePolicyService.find(context, bitstream)); + } + + protected List policiesForAction(Bitstream bitstream, int actionId) throws SQLException { + return new ArrayList<>(resourcePolicyService.find(context, bitstream, actionId)); + } + + protected List readPolicies(Bitstream bitstream) throws SQLException { + return policiesForAction(bitstream, Constants.READ); + } + + protected List anonymousReadPolicies(Bitstream bitstream) throws SQLException { + return readPolicies(bitstream).stream() + .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) + .collect(Collectors.toList()); + } + + /** + * Every resource policy id of the bitstream. Ids rather than counts, so a policy that was deleted and + * re-created is visible. + */ + protected Set policyIds(Bitstream bitstream) throws SQLException { + Set ids = new TreeSet<>(); + for (ResourcePolicy policy : allPolicies(bitstream)) { + ids.add(policy.getID()); + } + return ids; + } + + /** + * Deletes policies one by one, because the bulk removal helpers issue an HQL delete and leave the + * in-memory collection stale. + * + * @param actionId action to delete, or {@link #ALL_ACTIONS} for every policy regardless of action + */ + protected void deletePolicies(Bitstream bitstream, int actionId) throws Exception { + List doomed = actionId == ALL_ACTIONS + ? allPolicies(bitstream) + : policiesForAction(bitstream, actionId); + for (ResourcePolicy policy : doomed) { + resourcePolicyService.delete(context, policy); + } + } + + protected ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) + throws Exception { + return addAnonymousReadPolicy(bitstream, startDate, name, null); + } + + protected ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name, + String policyType) throws Exception { + context.turnOffAuthorisationSystem(); + ResourcePolicyBuilder builder = ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) + .withAction(Constants.READ) + .withDspaceObject(bitstream) + .withName(name); + if (startDate != null) { + builder.withStartDate(startDate); + } + if (policyType != null) { + builder.withPolicyType(policyType); + } + ResourcePolicy policy = builder.build(); + context.restoreAuthSystemState(); + return policy; + } + + /** + * Leaves the bitstream with a single Anonymous READ policy, removing the collection's undated default + * first; without that an "embargoed" fixture is not embargoed at all. + */ + protected ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name) + throws Exception { + return replaceAnonymousReadPolicies(bitstream, startDate, name, null); + } + + protected ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name, + String policyType) throws Exception { + context.turnOffAuthorisationSystem(); + deletePolicies(bitstream, Constants.READ); + context.restoreAuthSystemState(); + return addAnonymousReadPolicy(bitstream, startDate, name, policyType); + } + + protected List metadataValues(Item item, String element, String qualifier) { + return itemService.getMetadata(item, "dc", element, qualifier, Item.ANY).stream() + .map(MetadataValue::getValue) + .collect(Collectors.toList()); + } + + protected String singleMetadataValue(Item item, String element, String qualifier) { + List values = metadataValues(item, element, qualifier); + return values.isEmpty() ? null : values.get(0); + } + + /** + * Calendar day of a date. {@code ResourcePolicy.startDate} is mapped as {@code @Temporal(DATE)}, so after + * a round trip through the database it comes back as a day-granular {@code java.sql.Date}. + */ + protected LocalDate toLocalDate(Date date) { + if (date instanceof java.sql.Date) { + return ((java.sql.Date) date).toLocalDate(); + } + return date.toInstant().atZone(ZoneId.systemDefault()).toLocalDate(); + } + + /** + * The same day rendered for a failure message, {@code null} included. + */ + protected String day(Date date) { + return date == null ? "" : toLocalDate(date).toString(); + } + + /** + * Full identity and content of one policy. Comparing these rather than counts makes both + * delete-and-recreate and in-place mutation visible. + */ + protected String fingerprint(ResourcePolicy policy) { + return String.format("id=%s action=%s group=%s eperson=%s rpType=%s rpName=%s start=%s end=%s", + policy.getID(), + Constants.actionText[policy.getAction()], + policy.getGroup() == null ? "" : policy.getGroup().getName(), + policy.getEPerson() == null ? "" : policy.getEPerson().getEmail(), + policy.getRpType(), + policy.getRpName(), + day(policy.getStartDate()), + day(policy.getEndDate())); + } + + protected List policyFingerprints(Bitstream bitstream) throws SQLException { + List fingerprints = new ArrayList<>(); + for (ResourcePolicy policy : allPolicies(bitstream)) { + fingerprints.add(fingerprint(policy)); + } + Collections.sort(fingerprints); + return fingerprints; + } + + protected Date startOfDayUtc(LocalDate day) { + return Date.from(day.atStartOfDay(ZoneOffset.UTC).toInstant()); + } + + protected String pastDate() { + return LocalDate.now().minusMonths(1).toString(); + } + + /** + * A future end date, the branch that closes the file; a past one opens it again. + */ + protected String futureDate() { + return LocalDate.now().plusYears(1).toString(); + } + + /** + * Builds a SAF {@code dublin_core.xml}. {@code ItemArchive.create} resolves the item by + * {@code dc.identifier.uri == ItemUpdate.HANDLE_PREFIX + handle}. + * + * @param rightsAccess value for {@code dc.rights.access}, or {@code null} to omit the element entirely + * @param embargoEndDate {@code null} omits {@code dc.date.embargoend} entirely, the empty string writes a + * blank value + */ + protected String dublinCore(Item item, String rightsAccess, String embargoEndDate) { + return dublinCoreWithEndDates(item, rightsAccess, embargoEndDate); + } + + /** + * The same document with more than one {@code dc.date.embargoend} value, emitted in the given order. + */ + protected String dublinCoreWithEndDates(Item item, String rightsAccess, String... embargoEndDates) { + return dublinCoreWithAccessRights(item, rightsAccess == null ? Collections.emptyList() + : Collections.singletonList(rightsAccess), embargoEndDates); + } + + /** + * The same document carrying zero or more {@code dc.rights.access} values, for the contradictory metadata + * an operator can put in a package. + */ + protected String dublinCoreWithAccessRights(Item item, List accessRights, String... embargoEndDates) { + StringBuilder sb = new StringBuilder(); + sb.append("\n") + .append("\n") + .append(" ") + .append(ItemUpdate.HANDLE_PREFIX).append(item.getHandle()) + .append("\n"); + + for (String accessRight : accessRights) { + sb.append(" ") + .append(accessRight) + .append("\n"); + } + + for (String embargoEndDate : embargoEndDates) { + if (embargoEndDate == null) { + continue; + } + sb.append(" ") + // an empty XML element is dropped by the parser, a single space survives as a blank value + .append(embargoEndDate.isEmpty() ? " " : embargoEndDate) + .append("\n"); + } + + sb.append(""); + return sb.toString(); + } + + /** + * Runs itemupdate with both embargo fields as targets, the combination that triggers embargo + * synchronisation. {@code main()} is not used because it ends in {@code System.exit}. + */ + protected Run runItemUpdate(Item item, String dublinCoreContent) throws Exception { + return runItemUpdate(new ItemUpdate(), item, dublinCoreContent); + } + + /** + * The same run driven by a caller supplied {@link ItemUpdate}, so that a test can make one step of the + * synchronisation fail where a database error would. + */ + protected Run runItemUpdate(ItemUpdate itemUpdate, Item item, String dublinCoreContent) throws Exception { + Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); + // without this marker processArchive writes an undo archive next to the source directory + Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); + + Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); + Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); + + DeleteMetadataAction deleteAction = + (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); + deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + AddMetadataAction addAction = + (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); + addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); + + // ItemUpdate reports to System.out only, so the console has to be captured to assert on it; the + // stream is teed so that the failsafe -output.txt still holds the full log + ByteArrayOutputStream captured = new ByteArrayOutputStream(); + PrintStream originalOut = System.out; + PrintStream captureStream = + new PrintStream(new TeeOutputStream(originalOut, captured), true, StandardCharsets.UTF_8); + + context.turnOffAuthorisationSystem(); + System.setOut(captureStream); + try { + itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); + } finally { + captureStream.flush(); + System.setOut(originalOut); + context.restoreAuthSystemState(); + } + + context.uncacheEntity(item); + return new Run(captured.toString(StandardCharsets.UTF_8), itemUpdate.embargoSyncFailures); + } + + /** + * A run whose failure count is asserted straight away. + * + * @param expectedFailures number of embargo problems the run has to count; anything but 0 makes + * {@code ItemUpdate.main()} exit with 1 + * @return everything the run printed + */ + protected String runItemUpdateExpecting(Item item, String dublinCoreContent, int expectedFailures) + throws Exception { + Run run = runItemUpdate(item, dublinCoreContent); + assertExitCode("itemupdate run", expectedFailures, run); + return run.console; + } + + /** + * @return the number of embargo problems the run reported, which {@link ItemUpdate#exitStatus(int, int)} + * turns into the exit code of {@code dspace itemupdate} + */ + protected int runItemUpdateFailures(Item item, String dublinCoreContent) throws Exception { + return runItemUpdate(item, dublinCoreContent).embargoSyncFailures; + } + + /** + * Asserts the exit code the run would have produced; without it a skipped item looks synchronised. + */ + protected void assertExitCode(String scenario, int expectedFailures, Run run) { + assertEquals("[" + scenario + "] wrong number of reported embargo problems, so ItemUpdate.main() would" + + " exit with " + ItemUpdate.exitStatus(0, run.embargoSyncFailures) + " instead of " + + ItemUpdate.exitStatus(0, expectedFailures) + ". Console output was:" + + System.lineSeparator() + run.console, + expectedFailures, run.embargoSyncFailures); + } + + /** + * What a finished {@code itemupdate} run is judged by: its console output and its failure count. + */ + protected static final class Run { + final String console; + final int embargoSyncFailures; + + private Run(String console, int embargoSyncFailures) { + this.console = console; + this.embargoSyncFailures = embargoSyncFailures; + } + } +} diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java index 0b0d3ed068e1..39d3a05b644e 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java @@ -15,12 +15,6 @@ import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; -import java.io.PrintStream; -import java.nio.charset.StandardCharsets; -import java.nio.file.Files; -import java.nio.file.Path; import java.time.LocalDate; import java.time.ZoneId; import java.time.ZoneOffset; @@ -28,40 +22,12 @@ import java.util.List; import java.util.Locale; import java.util.Set; -import java.util.TreeSet; -import java.util.stream.Collectors; -import org.apache.commons.io.file.PathUtils; -import org.apache.commons.io.output.TeeOutputStream; -import org.dspace.AbstractIntegrationTestWithDatabase; import org.dspace.authorize.ResourcePolicy; -import org.dspace.authorize.factory.AuthorizeServiceFactory; -import org.dspace.authorize.service.AuthorizeService; -import org.dspace.authorize.service.ResourcePolicyService; -import org.dspace.builder.BitstreamBuilder; -import org.dspace.builder.CollectionBuilder; -import org.dspace.builder.CommunityBuilder; -import org.dspace.builder.ItemBuilder; -import org.dspace.builder.MetadataFieldBuilder; import org.dspace.content.Bitstream; -import org.dspace.content.Collection; import org.dspace.content.Item; -import org.dspace.content.MetadataField; -import org.dspace.content.MetadataSchema; -import org.dspace.content.MetadataValue; -import org.dspace.content.factory.ContentServiceFactory; -import org.dspace.content.service.ItemService; -import org.dspace.content.service.MetadataFieldService; -import org.dspace.content.service.MetadataSchemaService; import org.dspace.core.Constants; -import org.dspace.eperson.EPerson; import org.dspace.eperson.Group; -import org.dspace.eperson.factory.EPersonServiceFactory; -import org.dspace.eperson.service.GroupService; -import org.dspace.handle.factory.HandleServiceFactory; -import org.dspace.handle.service.HandleService; -import org.junit.After; -import org.junit.Before; import org.junit.Test; /** @@ -70,67 +36,11 @@ * that an embargo end date already in the past opens the file. All dates are derived from * {@code LocalDate.now(ZoneOffset.UTC)} so the suite cannot expire. */ -public class EmbargoDateBoundaryIT extends AbstractIntegrationTestWithDatabase { - - /** Expected normalised policy name. Must stay within the 30 character {@code ResourcePolicy.rpname} column. */ - private static final String EMBARGO_POLICY_NAME = "embargo"; - - private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); - private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); - private final ResourcePolicyService resourcePolicyService = - AuthorizeServiceFactory.getInstance().getResourcePolicyService(); - private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); - private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); - private final MetadataSchemaService metadataSchemaService = - ContentServiceFactory.getInstance().getMetadataSchemaService(); - private final MetadataFieldService metadataFieldService = - ContentServiceFactory.getInstance().getMetadataFieldService(); +public class EmbargoDateBoundaryIT extends AbstractEmbargoIT { /** Human readable trace of every policy state observed during a test; appended to failure messages. */ private final StringBuilder diagnostics = new StringBuilder(); - private Collection collection; - private Group anonymousGroup; - private Path tempDir; - private String previousHandlePrefix; - - @Before - @Override - public void setUp() throws Exception { - super.setUp(); - context.turnOffAuthorisationSystem(); - - parentCommunity = CommunityBuilder.createCommunity(context) - .withName("Parent Community") - .build(); - collection = CollectionBuilder.createCollection(context, parentCommunity) - .withName("Collection") - .build(); - - // Neither field exists in the test metadata registry; AddMetadataAction would fail without them. - ensureMetadataFieldExists("rights", "access"); - ensureMetadataFieldExists("date", "embargoend"); - - anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); - previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; - ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); - - context.restoreAuthSystemState(); - - tempDir = Files.createTempDirectory("embargoDateBoundaryIT"); - } - - @After - @Override - public void destroy() throws Exception { - // HANDLE_PREFIX is a mutable public static; leaking it would poison other test classes. - ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; - if (tempDir != null) { - PathUtils.deleteDirectory(tempDir); - } - super.destroy(); - } - /** * Verifies that a future {@code dc.date.embargoend} closes the file and leaves one normalised * {@code Anonymous}/{@code READ} policy starting the day after the embargo end date. @@ -145,7 +55,7 @@ public void futureEmbargoEndBlocksAccess() throws Exception { dump("STEP A - fresh SAF import, before any itemupdate", bitstream); assertFreshImportBaseline(bitstream); - runItemUpdate(item, dublinCore(item, "embargoedAccess", embargoEnd.toString())); + runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", embargoEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B - after itemupdate with FUTURE dc.date.embargoend=" + embargoEnd, bitstream); @@ -177,7 +87,7 @@ public void embargoEndTodayStillBlocksToday() throws Exception { dump("STEP A - fresh SAF import, before any itemupdate", bitstream); assertFreshImportBaseline(bitstream); - runItemUpdate(item, dublinCore(item, "embargoedAccess", embargoEnd.toString())); + runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", embargoEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B - after itemupdate with dc.date.embargoend=TODAY=" + embargoEnd, bitstream); @@ -212,7 +122,7 @@ public void embargoEndYesterdayOpensAccess() throws Exception { dump("STEP A - fresh SAF import, before any itemupdate", bitstream); assertFreshImportBaseline(bitstream); - runItemUpdate(item, dublinCore(item, null, embargoEnd.toString())); + runItemUpdateExpecting(item, dublinCore(item, null, embargoEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B - after itemupdate with dc.date.embargoend=YESTERDAY=" + embargoEnd, bitstream); @@ -247,7 +157,7 @@ public void pastEmbargoEndWithOpenAccessOpensAccess() throws Exception { assertFreshImportBaseline(bitstream); // priming run: an embargo with a future end date - runItemUpdate(item, dublinCore(item, "embargoedAccess", futureEnd.toString())); + runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", futureEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B - after itemupdate with FUTURE dc.date.embargoend=" + futureEnd, bitstream); @@ -256,7 +166,7 @@ public void pastEmbargoEndWithOpenAccessOpensAccess() throws Exception { anonymousCanRead(bitstream)); // the embargo expires: past end date, item declared openAccess - runItemUpdate(item, dublinCore(item, "openAccess", pastEnd.toString())); + runItemUpdateExpecting(item, dublinCore(item, "openAccess", pastEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP C - after itemupdate with PAST dc.date.embargoend=" + pastEnd @@ -264,7 +174,7 @@ public void pastEmbargoEndWithOpenAccessOpensAccess() throws Exception { assertEmbargoEndStored(item, pastEnd.toString()); assertEquals("itemupdate did not store dc.rights.access=openAccess." + diagnostics, - "openAccess", firstMetadataValue(item, "rights", "access")); + "openAccess", singleMetadataValue(item, "rights", "access")); ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy("expired embargo " + pastEnd + " with dc.rights.access=openAccess", bitstream); @@ -295,7 +205,7 @@ public void pastEmbargoEndWithEmbargoedAccessOpensAccess() throws Exception { dump("STEP A - fresh SAF import, before any itemupdate", bitstream); assertFreshImportBaseline(bitstream); - runItemUpdate(item, dublinCore(item, "embargoedAccess", futureEnd.toString())); + runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", futureEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B - after itemupdate with FUTURE dc.date.embargoend=" + futureEnd, bitstream); @@ -303,7 +213,7 @@ public void pastEmbargoEndWithEmbargoedAccessOpensAccess() throws Exception { + " not be publicly readable." + diagnostics, anonymousCanRead(bitstream)); - runItemUpdate(item, dublinCore(item, "embargoedAccess", pastEnd.toString())); + runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", pastEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP C - after itemupdate with PAST dc.date.embargoend=" + pastEnd @@ -418,7 +328,7 @@ private void assertStoredStartDaySurvivesRoundTrip(LocalDate embargoEnd, boolean + " only part of the start date that survives - and it is the part that decides" + " access. Expected " + expectedStartDay + " (dc.date.embargoend + 1 day), stored " + stored.getStartDate() + "." + diagnostics, - expectedStartDay, toLocalDate(stored.getStartDate())); + expectedStartDay, toUtcLocalDate(stored.getStartDate())); assertNull("[" + leg + "] this tool must never write an end date: a policy that expires by itself" + " would close the file again on that day." + diagnostics, stored.getEndDate()); @@ -458,8 +368,8 @@ public void multipleEmbargoEndValuesUsesFirst() throws Exception { dump("STEP A - fresh SAF import, before any itemupdate", bitstream); assertFreshImportBaseline(bitstream); - String consoleOutput = runItemUpdate(item, - dublinCore(item, "embargoedAccess", firstEnd.toString(), secondEnd.toString())); + String consoleOutput = runItemUpdateExpecting(item, + dublinCoreWithEndDates(item, "embargoedAccess", firstEnd.toString(), secondEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B - after itemupdate with dc.date.embargoend=" + firstEnd + " AND " + secondEnd, bitstream); @@ -482,7 +392,7 @@ public void multipleEmbargoEndValuesUsesFirst() throws Exception { assertNotEquals("the SECOND dc.date.embargoend value (" + secondEnd + ") must be ignored, the embargo has" + " to follow the first one (" + firstEnd + ")." + diagnostics, - secondEnd.plusDays(1), toLocalDate(policy.getStartDate())); + secondEnd.plusDays(1), toUtcLocalDate(policy.getStartDate())); assertFalse("both dc.date.embargoend values lie in the future, so the ORIGINAL bitstream must not be" + " publicly readable." + diagnostics, anonymousCanRead(bitstream)); @@ -525,7 +435,7 @@ public void legacyPastEmbargoEndPublishesOnPurpose() throws Exception { // The priming run is what makes this a publication rather than a no-op: it leaves the single dated // policy that keeps the file closed, so opening it afterwards is a state change that can be observed. - runItemUpdate(item, dublinCore(item, "embargoedAccess", primingEnd.toString())); + runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", primingEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B - after priming itemupdate with a FUTURE dc.date.embargoend=" + primingEnd, bitstream); @@ -534,7 +444,7 @@ public void legacyPastEmbargoEndPublishesOnPurpose() throws Exception { + diagnostics, anonymousCanRead(bitstream)); - runItemUpdate(item, dublinCore(item, "openAccess", legacyValue)); + runItemUpdateExpecting(item, dublinCore(item, "openAccess", legacyValue), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP C - after itemupdate with " + scenario, bitstream); @@ -588,7 +498,7 @@ private void assertLegacyEmbargoEndClosesTheFileUntil(String legacyValue, LocalD dump("STEP A [" + legacyValue + "] - fresh SAF import, before any itemupdate", bitstream); assertFreshImportBaseline(bitstream); - runItemUpdate(item, dublinCore(item, "embargoedAccess", legacyValue)); + runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", legacyValue), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B [" + legacyValue + "] - after itemupdate", bitstream); @@ -622,16 +532,16 @@ private void assertEmbargoEndIsRefused(String rejectedValue) throws Exception { Bitstream bitstream = createOriginalBitstream(item, "unparseable.pdf"); assertFreshImportBaseline(bitstream); - runItemUpdate(item, dublinCore(item, "embargoedAccess", primingEnd.toString())); + runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", primingEnd.toString()), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP A [" + rejectedValue + "] - embargoed until " + primingEnd, bitstream); assertFalse("fixture precondition [" + scenario + "]: the file has to be closed before the unparseable" + " value is fed in." + diagnostics, anonymousCanRead(bitstream)); - Set idsBefore = allPolicyIds(bitstream); + Set idsBefore = policyIds(bitstream); - String consoleOutput = runItemUpdate(item, dublinCore(item, "embargoedAccess", rejectedValue), 1); + String consoleOutput = runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", rejectedValue), 1); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B [" + rejectedValue + "] - after itemupdate with the unparseable value", bitstream); @@ -639,12 +549,12 @@ private void assertEmbargoEndIsRefused(String rejectedValue) throws Exception { assertEquals("[" + scenario + "] the set of resource policy ids changed, so policies were deleted and/or" + " re-created although an unreadable date is no instruction at all. Console output" + " was:\n" + consoleOutput + diagnostics, - idsBefore, allPolicyIds(bitstream)); + idsBefore, policyIds(bitstream)); ResourcePolicy policy = assertExactlyOneAnonymousReadPolicy(scenario, bitstream); assertEquals("[" + scenario + "] the surviving policy was re-dated although the value could not be read." + " Whatever the tool cannot parse it must not act on." + diagnostics, - primingStartDay, toLocalDate(policy.getStartDate())); + primingStartDay, toUtcLocalDate(policy.getStartDate())); assertFalse("[" + scenario + "] the embargoed file became publicly readable after an unreadable" + " dc.date.embargoend." + diagnostics, anonymousCanRead(bitstream)); } @@ -697,7 +607,7 @@ private void assertNormalisedEmbargoPolicy(String scenario, ResourcePolicy polic assertEquals("After " + scenario + " resource policy #" + policy.getID() + " must start on the day AFTER" + " dc.date.embargoend, because the end date is the inclusive last day of the embargo." + diagnostics, - expectedStartDay, toLocalDate(policy.getStartDate())); + expectedStartDay, toUtcLocalDate(policy.getStartDate())); assertEquals("After " + scenario + " resource policy #" + policy.getID() + " must be normalised to" + " rpType=" + ResourcePolicy.TYPE_CUSTOM + "; AuthorizeServiceImpl only honours custom" + " policies on not-yet-installed items." + diagnostics, @@ -712,47 +622,7 @@ private void assertEmbargoEndStored(Item item, String expectedEmbargoEnd) { assertEquals("itemupdate did not store dc.date.embargoend on the item, so the run never really reached it" + " (ItemArchive.create may have failed to resolve it - processArchive swallows every" + " per-item exception)." + diagnostics, - expectedEmbargoEnd, firstMetadataValue(item, "date", "embargoend")); - } - - /** - * Tells whether a visitor who is not logged in may read the bitstream. The authorisation state is a stack - * the builders push and pop, so it is drained first - otherwise every read looks allowed. - */ - private boolean anonymousCanRead(Bitstream bitstream) throws Exception { - EPerson savedUser = context.getCurrentUser(); - int popped = 0; - while (context.ignoreAuthorization()) { - context.restoreAuthSystemState(); - popped++; - } - context.setCurrentUser(null); - try { - return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); - } finally { - context.setCurrentUser(savedUser); - for (int i = 0; i < popped; i++) { - context.turnOffAuthorisationSystem(); - } - } - } - - /** - * Every resource policy id of the bitstream. Ids rather than counts, so a policy that was deleted and - * re-created is visible. - */ - private Set allPolicyIds(Bitstream bitstream) throws Exception { - Set ids = new TreeSet<>(); - for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream)) { - ids.add(policy.getID()); - } - return ids; - } - - private List anonymousReadPolicies(Bitstream bitstream) throws Exception { - return resourcePolicyService.find(context, bitstream, Constants.READ).stream() - .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) - .collect(Collectors.toList()); + expectedEmbargoEnd, singleMetadataValue(item, "date", "embargoend")); } private void dump(String label, Bitstream bitstream) throws Exception { @@ -787,7 +657,7 @@ private void dump(String label, Bitstream bitstream) throws Exception { * Calendar day of a start date. {@code ResourcePolicy.startDate} is mapped as {@code @Temporal(DATE)}, so * after a round trip through the database it comes back as a day-granular {@code java.sql.Date}. */ - private LocalDate toLocalDate(Date date) { + private LocalDate toUtcLocalDate(Date date) { if (date instanceof java.sql.Date) { return ((java.sql.Date) date).toLocalDate(); } @@ -812,140 +682,4 @@ private LocalDate nextIrishSummerTimeDay() { return candidate; } - private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { - MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); - MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); - if (existingField == null) { - MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); - } - } - - private Item createItem(String title, String... metadataTriples) throws Exception { - context.turnOffAuthorisationSystem(); - ItemBuilder builder = ItemBuilder.createItem(context, collection).withTitle(title); - for (int i = 0; i + 2 < metadataTriples.length; i += 3) { - builder.withMetadata("dc", metadataTriples[i], metadataTriples[i + 1], metadataTriples[i + 2]); - } - Item item = builder.build(); - context.restoreAuthSystemState(); - return item; - } - - private Bitstream createOriginalBitstream(Item item, String name) throws Exception { - context.turnOffAuthorisationSystem(); - Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, - new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8))) - .withName(name) - .withMimeType("text/plain") - .build(); - context.restoreAuthSystemState(); - return bitstream; - } - - private List metadataValues(Item item, String element, String qualifier) { - return itemService.getMetadata(item, "dc", element, qualifier, Item.ANY).stream() - .map(MetadataValue::getValue) - .collect(Collectors.toList()); - } - - private String firstMetadataValue(Item item, String element, String qualifier) { - List values = metadataValues(item, element, qualifier); - return values.isEmpty() ? null : values.get(0); - } - - /** - * Runs itemupdate with both embargo fields as targets, the combination that triggers embargo - * synchronisation. {@code main()} is not used because it ends in {@code System.exit}. - * - * @return everything {@code ItemUpdate.pr()} printed during the run; the stream is teed, so the output still - * reaches the failsafe output file as well. - */ - private String runItemUpdate(Item item, String dublinCoreContent) throws Exception { - return runItemUpdate(item, dublinCoreContent, 0); - } - - /** - * Same run, for the scenarios {@code itemupdate} has to refuse. - * - * @param expectedEmbargoSyncFailures number of embargo problems the run has to count; anything but 0 makes - * {@code ItemUpdate.main()} exit with 1 - */ - private String runItemUpdate(Item item, String dublinCoreContent, int expectedEmbargoSyncFailures) - throws Exception { - Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); - // Without suppress_undo, processArchive writes an undo archive as a SIBLING of the source directory. - Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); - - Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); - Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); - - ItemUpdate itemUpdate = new ItemUpdate(); - DeleteMetadataAction deleteAction = - (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); - deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); - - AddMetadataAction addAction = - (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); - addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); - - ByteArrayOutputStream captured = new ByteArrayOutputStream(); - PrintStream originalOut = System.out; - System.setOut(new PrintStream(new TeeOutputStream(originalOut, captured), true, - StandardCharsets.UTF_8.name())); - context.turnOffAuthorisationSystem(); - try { - itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); - } finally { - context.restoreAuthSystemState(); - System.out.flush(); - System.setOut(originalOut); - } - - context.uncacheEntity(item); - String consoleOutput = captured.toString(StandardCharsets.UTF_8.name()); - - assertEquals("wrong number of reported embargo synchronisation problems - ItemUpdate.main() would exit" - + " with " + ItemUpdate.exitStatus(0, itemUpdate.embargoSyncFailures) + " instead of " - + ItemUpdate.exitStatus(0, expectedEmbargoSyncFailures) + ", and the exit code is the" - + " only thing an operator scripting itemupdate ever sees. Console output was:\n" - + consoleOutput, - expectedEmbargoSyncFailures, itemUpdate.embargoSyncFailures); - - return consoleOutput; - } - - /** - * Builds a SAF {@code dublin_core.xml}. {@code ItemArchive.create} resolves the item by - * {@code dc.identifier.uri == ItemUpdate.HANDLE_PREFIX + handle}. - * - * @param rightsAccess value for {@code dc.rights.access}, or {@code null} to omit the element entirely - * @param embargoEndDates zero or more {@code dc.date.embargoend} values, emitted in the given order - */ - private String dublinCore(Item item, String rightsAccess, String... embargoEndDates) { - StringBuilder sb = new StringBuilder(); - sb.append("\n") - .append("\n") - .append(" ") - .append(ItemUpdate.HANDLE_PREFIX).append(item.getHandle()) - .append("\n"); - - if (rightsAccess != null) { - sb.append(" ") - .append(rightsAccess) - .append("\n"); - } - - for (String embargoEndDate : embargoEndDates) { - if (embargoEndDate == null) { - continue; - } - sb.append(" ") - // an empty XML element is dropped by the parser, a single space survives as a blank value - .append(embargoEndDate.isEmpty() ? " " : embargoEndDate) - .append("\n"); - } - - sb.append(""); - return sb.toString(); - } } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java index 8faf4a4405e4..6dfb51d8807f 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java @@ -12,56 +12,18 @@ import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; -import java.io.PrintStream; -import java.nio.charset.StandardCharsets; -import java.nio.file.Files; -import java.nio.file.Path; import java.sql.SQLException; import java.time.LocalDate; -import java.time.ZoneId; -import java.time.ZoneOffset; import java.util.ArrayList; import java.util.Arrays; -import java.util.Collections; -import java.util.Date; import java.util.List; import java.util.Set; -import java.util.TreeSet; -import java.util.stream.Collectors; -import org.apache.commons.io.file.PathUtils; -import org.dspace.AbstractIntegrationTestWithDatabase; import org.dspace.authorize.ResourcePolicy; -import org.dspace.authorize.factory.AuthorizeServiceFactory; -import org.dspace.authorize.service.AuthorizeService; -import org.dspace.authorize.service.ResourcePolicyService; -import org.dspace.builder.BitstreamBuilder; -import org.dspace.builder.CollectionBuilder; -import org.dspace.builder.CommunityBuilder; -import org.dspace.builder.ItemBuilder; -import org.dspace.builder.MetadataFieldBuilder; -import org.dspace.builder.ResourcePolicyBuilder; import org.dspace.content.Bitstream; import org.dspace.content.Bundle; -import org.dspace.content.Collection; import org.dspace.content.Item; -import org.dspace.content.MetadataField; -import org.dspace.content.MetadataSchema; -import org.dspace.content.factory.ContentServiceFactory; -import org.dspace.content.service.ItemService; -import org.dspace.content.service.MetadataFieldService; -import org.dspace.content.service.MetadataSchemaService; import org.dspace.core.Constants; -import org.dspace.eperson.EPerson; -import org.dspace.eperson.Group; -import org.dspace.eperson.factory.EPersonServiceFactory; -import org.dspace.eperson.service.GroupService; -import org.dspace.handle.factory.HandleServiceFactory; -import org.dspace.handle.service.HandleService; -import org.junit.After; -import org.junit.Before; import org.junit.Test; /** @@ -70,73 +32,15 @@ * re-opened when the embargo ends. Bundles that are not derived from the file, LICENSE above all, stay out * of scope. */ -public class EmbargoDerivativesIT extends AbstractIntegrationTestWithDatabase { - - /** Policy name the synchronisation writes. */ - private static final String EMBARGO_POLICY_NAME = "embargo"; +public class EmbargoDerivativesIT extends AbstractEmbargoIT { private static final String OPEN_ACCESS = "openAccess"; private static final String EMBARGOED_ACCESS = "embargoedAccess"; private static final String RESTRICTED_ACCESS = "restrictedAccess"; - private static final String TEXT_BUNDLE = "TEXT"; - private static final String THUMBNAIL_BUNDLE = "THUMBNAIL"; private static final String LICENSE_BUNDLE = "LICENSE"; private static final String CC_LICENSE_BUNDLE = "CC-LICENSE"; - private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); - private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); - private final ResourcePolicyService resourcePolicyService = - AuthorizeServiceFactory.getInstance().getResourcePolicyService(); - private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); - private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); - private final MetadataSchemaService metadataSchemaService = - ContentServiceFactory.getInstance().getMetadataSchemaService(); - private final MetadataFieldService metadataFieldService = - ContentServiceFactory.getInstance().getMetadataFieldService(); - - private Collection collection; - private Group anonymousGroup; - private Path tempDir; - private String previousHandlePrefix; - - @Before - @Override - public void setUp() throws Exception { - super.setUp(); - context.turnOffAuthorisationSystem(); - - parentCommunity = CommunityBuilder.createCommunity(context) - .withName("Parent Community") - .build(); - collection = CollectionBuilder.createCollection(context, parentCommunity) - .withName("Collection") - .build(); - - // neither field exists in the test metadata registry, AddMetadataAction needs both - ensureMetadataFieldExists("rights", "access"); - ensureMetadataFieldExists("date", "embargoend"); - - anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); - // ItemArchive resolves items through this mutable static, it is restored in destroy() - previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; - ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); - - context.restoreAuthSystemState(); - - tempDir = Files.createTempDirectory("embargoDerivativesIT"); - } - - @After - @Override - public void destroy() throws Exception { - ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; - if (tempDir != null) { - PathUtils.deleteDirectory(tempDir); - } - super.destroy(); - } - /** * Verifies that a running embargo closes the thumbnail and the extracted full text as well. The TEXT * bundle holds the whole content of the file, so leaving it public publishes the embargoed work. @@ -189,7 +93,8 @@ public void expiredEmbargoReopensTextAndThumbnail() throws Exception { // the state a finished embargo run leaves behind: one dated Anonymous READ policy on every bundle for (Bitstream file : files) { - replaceAnonymousReadPolicy(file, startOfDayUtc(LocalDate.now().plusYears(1)), EMBARGO_POLICY_NAME); + replaceAnonymousReadPolicies(file, startOfDayUtc(LocalDate.now().plusYears(1)), + EMBARGO_POLICY_NAME, ResourcePolicy.TYPE_CUSTOM); } reloadAll(files); for (Bitstream file : files) { @@ -310,7 +215,7 @@ public void derivativeWithoutAnonymousReadIsNotPublished() throws Exception { Bitstream text = createBitstreamInBundle(item, "closed.pdf.txt", TEXT_BUNDLE); context.turnOffAuthorisationSystem(); - deleteReadPolicies(text); + deletePolicies(text, Constants.READ); context.restoreAuthSystemState(); text = context.reloadEntity(text); @@ -406,68 +311,6 @@ private void assertUntouched(String scenario, List before, List snapshotAll(List bitstreams) throws SQLExcepti return snapshots; } - /** - * Asserts the exit code the run would have produced; without it a skipped item looks synchronised. - */ - private void assertExitCode(String scenario, int expectedFailures, Run run) { - assertEquals("[" + scenario + "] wrong number of reported embargo problems, so ItemUpdate.main() would" - + " exit with " + ItemUpdate.exitStatus(0, run.embargoSyncFailures) + " instead of " - + ItemUpdate.exitStatus(0, expectedFailures) + ". Console output was:" - + System.lineSeparator() + run.console, - expectedFailures, run.embargoSyncFailures); - } - - /** - * Builds a SAF {@code dublin_core.xml} carrying one {@code dc.rights.access} and one - * {@code dc.date.embargoend} value. - */ - private String dublinCore(Item item, String accessRight, String embargoEndDate) { - return "\n" - + "\n" - + " " - + ItemUpdate.HANDLE_PREFIX + item.getHandle() + "\n" - + " " + accessRight + "\n" - + " " + embargoEndDate + "\n" - + ""; - } - - private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { - MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); - MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); - if (existingField == null) { - MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); - } - } - - private Item createItem(String title) throws Exception { - context.turnOffAuthorisationSystem(); - Item item = ItemBuilder.createItem(context, collection) - .withTitle(title) - .build(); - context.restoreAuthSystemState(); - return item; - } - /** * The state filter-media leaves behind on a public item: the file plus its extracted text and its * thumbnail, each with the one undated Anonymous READ policy inherited from the collection. @@ -543,56 +344,6 @@ private List createOriginalWithDerivatives(Item item, String fileName return files; } - private Bitstream createBitstreamInBundle(Item item, String name, String bundleName) throws Exception { - context.turnOffAuthorisationSystem(); - Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, - new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8)), - bundleName) - .withName(name) - .withMimeType("text/plain") - .build(); - context.restoreAuthSystemState(); - return bitstream; - } - - /** - * Replaces every READ policy of the bitstream with a single dated Anonymous READ policy. - */ - private ResourcePolicy replaceAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) - throws Exception { - context.turnOffAuthorisationSystem(); - deleteReadPolicies(bitstream); - ResourcePolicy policy = ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) - .withAction(Constants.READ) - .withDspaceObject(bitstream) - .withName(name) - .withPolicyType(ResourcePolicy.TYPE_CUSTOM) - .withStartDate(startDate) - .build(); - context.restoreAuthSystemState(); - return policy; - } - - /** - * Deletes READ policies one by one, because the bulk removal helpers issue an HQL delete and leave the - * in-memory collection stale. - */ - private void deleteReadPolicies(Bitstream bitstream) throws Exception { - for (ResourcePolicy policy : readPolicies(bitstream)) { - resourcePolicyService.delete(context, policy); - } - } - - private List readPolicies(Bitstream bitstream) throws SQLException { - return new ArrayList<>(resourcePolicyService.find(context, bitstream, Constants.READ)); - } - - private List anonymousReadPolicies(Bitstream bitstream) throws SQLException { - return readPolicies(bitstream).stream() - .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) - .collect(Collectors.toList()); - } - private ResourcePolicy onlyAnonymousReadPolicy(Bitstream bitstream, String what) throws SQLException { List policies = anonymousReadPolicies(bitstream); assertEquals("[" + label(bitstream) + "] exactly one Anonymous READ policy must remain after " + what @@ -601,35 +352,6 @@ private ResourcePolicy onlyAnonymousReadPolicy(Bitstream bitstream, String what) return policies.get(0); } - private Set policyIds(Bitstream bitstream) throws SQLException { - Set ids = new TreeSet<>(); - for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream)) { - ids.add(policy.getID()); - } - return ids; - } - - private List policyFingerprints(Bitstream bitstream) throws SQLException { - List fingerprints = new ArrayList<>(); - for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream)) { - fingerprints.add(fingerprint(policy)); - } - Collections.sort(fingerprints); - return fingerprints; - } - - private String fingerprint(ResourcePolicy policy) { - return String.format("id=%s action=%s group=%s eperson=%s rpType=%s rpName=%s start=%s end=%s", - policy.getID(), - Constants.actionText[policy.getAction()], - policy.getGroup() == null ? "" : policy.getGroup().getName(), - policy.getEPerson() == null ? "" : policy.getEPerson().getEmail(), - policy.getRpType(), - policy.getRpName(), - day(policy.getStartDate()), - day(policy.getEndDate())); - } - /** * Renders the current policies of the bitstream for failure messages, so a red build shows which policy * moved. @@ -658,50 +380,4 @@ private String label(Bitstream bitstream) throws SQLException { return (bundles.isEmpty() ? "" : bundles.get(0).getName()) + "/" + bitstream.getName(); } - /** - * Tells whether a visitor who is not logged in may read the bitstream. The authorisation state is a stack - * the builders push and pop, so it is drained first - otherwise every read looks allowed. - */ - private boolean anonymousCanRead(Bitstream bitstream) throws SQLException { - EPerson savedUser = context.getCurrentUser(); - int popped = 0; - while (context.ignoreAuthorization()) { - context.restoreAuthSystemState(); - popped++; - } - context.setCurrentUser(null); - try { - return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); - } finally { - context.setCurrentUser(savedUser); - for (int i = 0; i < popped; i++) { - context.turnOffAuthorisationSystem(); - } - } - } - - private void reloadAll(List bitstreams) throws SQLException { - for (int i = 0; i < bitstreams.size(); i++) { - bitstreams.set(i, context.reloadEntity(bitstreams.get(i))); - } - } - - private Date startOfDayUtc(LocalDate day) { - return Date.from(day.atStartOfDay(ZoneOffset.UTC).toInstant()); - } - - /** - * Start dates come back from the database as {@code java.sql.Date}, so they are compared at calendar day - * granularity instead of as instants. - */ - private LocalDate toLocalDate(Date date) { - if (date instanceof java.sql.Date) { - return ((java.sql.Date) date).toLocalDate(); - } - return date.toInstant().atZone(ZoneId.systemDefault()).toLocalDate(); - } - - private String day(Date date) { - return date == null ? "" : toLocalDate(date).toString(); - } } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java index 849a045b2975..58abbba37d19 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoLifecycleIT.java @@ -14,55 +14,19 @@ import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; -import java.io.ByteArrayInputStream; -import java.nio.charset.StandardCharsets; -import java.nio.file.Files; -import java.nio.file.Path; import java.time.LocalDate; -import java.time.ZoneId; -import java.time.ZoneOffset; import java.util.ArrayList; -import java.util.Date; import java.util.List; import java.util.Set; import java.util.TreeSet; import java.util.UUID; -import java.util.stream.Collectors; -import org.apache.commons.io.file.PathUtils; -import org.dspace.AbstractIntegrationTestWithDatabase; import org.dspace.authorize.ResourcePolicy; -import org.dspace.authorize.factory.AuthorizeServiceFactory; -import org.dspace.authorize.service.AuthorizeService; -import org.dspace.authorize.service.ResourcePolicyService; -import org.dspace.builder.BitstreamBuilder; -import org.dspace.builder.CollectionBuilder; -import org.dspace.builder.CommunityBuilder; -import org.dspace.builder.ItemBuilder; -import org.dspace.builder.MetadataFieldBuilder; -import org.dspace.builder.ResourcePolicyBuilder; import org.dspace.content.Bitstream; import org.dspace.content.Bundle; -import org.dspace.content.Collection; import org.dspace.content.DSpaceObject; import org.dspace.content.Item; -import org.dspace.content.MetadataField; -import org.dspace.content.MetadataSchema; -import org.dspace.content.MetadataValue; -import org.dspace.content.factory.ContentServiceFactory; -import org.dspace.content.service.BundleService; -import org.dspace.content.service.ItemService; -import org.dspace.content.service.MetadataFieldService; -import org.dspace.content.service.MetadataSchemaService; import org.dspace.core.Constants; -import org.dspace.eperson.EPerson; -import org.dspace.eperson.Group; -import org.dspace.eperson.factory.EPersonServiceFactory; -import org.dspace.eperson.service.GroupService; -import org.dspace.handle.factory.HandleServiceFactory; -import org.dspace.handle.service.HandleService; -import org.junit.After; -import org.junit.Before; import org.junit.Test; /** @@ -71,73 +35,13 @@ * Covers the invariants that one {@code Anonymous}/{@code READ} policy survives every run, that it is mutated * rather than recreated, and that the bundles derived from an embargoed file follow it. */ -public class EmbargoLifecycleIT extends AbstractIntegrationTestWithDatabase { +public class EmbargoLifecycleIT extends AbstractEmbargoIT { - /** Target policy name of the fix. Must stay within the 30 char {@code resourcepolicy.rpname} column. */ - private static final String EMBARGO_POLICY_NAME = "embargo"; - - /** Policy names written by earlier versions and still present in existing repositories. */ - private static final String LEGACY_STANDARD_EMBARGO = "Standard Embargo"; private static final String LEGACY_SPECIAL_CASE_EMBARGO = "Special Case Embargo"; private static final String OPEN_ACCESS = "openAccess"; private static final String EMBARGOED_ACCESS = "embargoedAccess"; - private static final String TEXT_BUNDLE = "TEXT"; - private static final String THUMBNAIL_BUNDLE = "THUMBNAIL"; - - private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); - private final BundleService bundleService = ContentServiceFactory.getInstance().getBundleService(); - private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); - private final ResourcePolicyService resourcePolicyService = - AuthorizeServiceFactory.getInstance().getResourcePolicyService(); - private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); - private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); - private final MetadataSchemaService metadataSchemaService = - ContentServiceFactory.getInstance().getMetadataSchemaService(); - private final MetadataFieldService metadataFieldService = - ContentServiceFactory.getInstance().getMetadataFieldService(); - - private Collection collection; - private Group anonymousGroup; - private Path tempDir; - private String previousHandlePrefix; - - @Before - @Override - public void setUp() throws Exception { - super.setUp(); - context.turnOffAuthorisationSystem(); - - parentCommunity = CommunityBuilder.createCommunity(context) - .withName("Parent Community") - .build(); - collection = CollectionBuilder.createCollection(context, parentCommunity) - .withName("Collection") - .build(); - - ensureMetadataFieldExists("rights", "access"); - ensureMetadataFieldExists("date", "embargoend"); - - anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); - previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; - ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); - - context.restoreAuthSystemState(); - - tempDir = Files.createTempDirectory("embargoLifecycleIT"); - } - - @After - @Override - public void destroy() throws Exception { - ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; - if (tempDir != null) { - PathUtils.deleteDirectory(tempDir); - } - super.destroy(); - } - /** * Verifies that a SAF package without {@code dc.date.embargoend} leaves every policy as it was. A missing * field says nothing about the embargo; an embargo is ended by an end date that lies in the past. @@ -151,7 +55,7 @@ public void removingEmbargoMetadataLeavesPoliciesUntouched() throws Exception { Integer importedPolicyId = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import").getID(); // the operator embargoes the item - assertRunSucceeded("setting the embargo", runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, + assertRunSucceeded("setting the embargo", runItemUpdateFailures(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -170,7 +74,7 @@ public void removingEmbargoMetadataLeavesPoliciesUntouched() throws Exception { // the next SAF package does not carry dc.date.embargoend assertRunSucceeded("running without dc.date.embargoend", - runItemUpdate(item, dublinCore(item, OPEN_ACCESS, null))); + runItemUpdateFailures(item, dublinCore(item, OPEN_ACCESS, null))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -217,7 +121,7 @@ public void foreignEmbargoIsNeverLifted() throws Exception { // a routine metadata batch: the fields are targeted, the package carries neither of them assertRunSucceeded("running a batch that does not mention the embargo", - runItemUpdate(item, dublinCore(item, null, null))); + runItemUpdateFailures(item, dublinCore(item, null, null))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -251,7 +155,7 @@ public void syncIsIdempotent() throws Exception { Integer importedPolicyId = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import").getID(); assertRunSucceeded("setting the embargo", - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); + runItemUpdateFailures(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -268,7 +172,7 @@ public void syncIsIdempotent() throws Exception { // exactly the same archive again assertRunSucceeded("re-running the identical archive", - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); + runItemUpdateFailures(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -294,7 +198,7 @@ public void syncIsIdempotent() throws Exception { */ @Test public void legacyRpNameThenFutureEmbargoIsEnforced() throws Exception { - assertLegacyPolicyIsAdoptedAndEnforced(LEGACY_STANDARD_EMBARGO, EMBARGOED_ACCESS, "legacy-standard.pdf"); + assertLegacyPolicyIsAdoptedAndEnforced(LEGACY_EMBARGO_POLICY_NAME, EMBARGOED_ACCESS, "legacy-standard.pdf"); } /** @@ -324,7 +228,7 @@ public void bornOpenItemThenFutureEmbargoIsEnforced() throws Exception { assertTrue("fixture precondition: a born open bitstream is publicly readable", anonymousCanRead(bitstream)); assertRunSucceeded("embargoing a born-open item", - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); + runItemUpdateFailures(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -361,7 +265,7 @@ public void duplicateAnonymousReadPoliciesCollapseToOne() throws Exception { Integer immediateId = onlyAnonymousReadPolicy(bitstream, "the fresh SAF import").getID(); Integer oldestDatedId = addAnonymousReadPolicy(bitstream, startOfDayUtc(LocalDate.now().minusMonths(3)), - LEGACY_STANDARD_EMBARGO).getID(); + LEGACY_EMBARGO_POLICY_NAME).getID(); Integer newestDatedId = addAnonymousReadPolicy(bitstream, startOfDayUtc(LocalDate.now().plusMonths(2)), LEGACY_SPECIAL_CASE_EMBARGO).getID(); bitstream = context.reloadEntity(bitstream); @@ -369,7 +273,7 @@ public void duplicateAnonymousReadPoliciesCollapseToOne() throws Exception { 3, anonymousReadPolicies(bitstream).size()); assertRunSucceeded("collapsing duplicate Anonymous READ policies", - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); + runItemUpdateFailures(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -424,7 +328,7 @@ public void neverZeroReadPoliciesInvariant() throws Exception { Item item = createItem("Invariant Thesis - " + label); Bitstream bitstream = createOriginalBitstream(item, "invariant.pdf"); - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, initialEmbargoEnd)); + runItemUpdateFailures(item, dublinCore(item, EMBARGOED_ACCESS, initialEmbargoEnd)); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -437,7 +341,7 @@ public void neverZeroReadPoliciesInvariant() throws Exception { continue; } - runItemUpdate(item, dublinCore(item, testCase[1], testCase[2])); + runItemUpdateFailures(item, dublinCore(item, testCase[1], testCase[2])); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -481,7 +385,7 @@ public void multipleBitstreamsAllGetSameState() throws Exception { // embargo every file of the record assertRunSucceeded("embargoing every ORIGINAL bitstream", - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); + runItemUpdateFailures(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd))); item = context.reloadEntity(item); reloadAll(bitstreams); originalBundle = context.reloadEntity(originalBundle); @@ -504,7 +408,7 @@ public void multipleBitstreamsAllGetSameState() throws Exception { // the embargo expires: the same archive is re-imported with a past date assertRunSucceeded("letting the embargo expire", - runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd))); + runItemUpdateFailures(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd))); item = context.reloadEntity(item); reloadAll(bitstreams); originalBundle = context.reloadEntity(originalBundle); @@ -551,7 +455,7 @@ public void derivativeBundlesFollowTheEmbargo() throws Exception { // embargo run assertRunSucceeded("embargoing the item", - runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd.toString()))); + runItemUpdateFailures(item, dublinCore(item, EMBARGOED_ACCESS, futureEmbargoEnd.toString()))); item = context.reloadEntity(item); reloadAll(files); originalBundle = context.reloadEntity(originalBundle); @@ -576,7 +480,7 @@ public void derivativeBundlesFollowTheEmbargo() throws Exception { // expired embargo run assertRunSucceeded("letting the embargo expire", - runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd.toString()))); + runItemUpdateFailures(item, dublinCore(item, OPEN_ACCESS, pastEmbargoEnd.toString()))); item = context.reloadEntity(item); reloadAll(files); originalBundle = context.reloadEntity(originalBundle); @@ -621,7 +525,7 @@ private void assertLegacyPolicyIsAdoptedAndEnforced(String legacyName, String ri anonymousCanRead(bitstream)); assertRunSucceeded("adopting a legacy embargo policy", - runItemUpdate(item, dublinCore(item, rightsAccess, futureEmbargoEnd))); + runItemUpdateFailures(item, dublinCore(item, rightsAccess, futureEmbargoEnd))); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -659,37 +563,6 @@ private void assertUniformState(List bitstreams, String what) throws } } - /** - * Tells whether a visitor who is not logged in may read the bitstream. - */ - private boolean anonymousCanRead(Bitstream bitstream) throws Exception { - EPerson saved = context.getCurrentUser(); - int popped = 0; - while (context.ignoreAuthorization()) { - context.restoreAuthSystemState(); - popped++; - } - context.setCurrentUser(null); - try { - return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); - } finally { - context.setCurrentUser(saved); - for (int i = 0; i < popped; i++) { - context.turnOffAuthorisationSystem(); - } - } - } - - private List readPolicies(Bitstream bitstream) throws Exception { - return resourcePolicyService.find(context, bitstream, Constants.READ); - } - - private List anonymousReadPolicies(Bitstream bitstream) throws Exception { - return readPolicies(bitstream).stream() - .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) - .collect(Collectors.toList()); - } - private ResourcePolicy onlyAnonymousReadPolicy(Bitstream bitstream, String what) throws Exception { List policies = anonymousReadPolicies(bitstream); assertEquals("exactly one Anonymous READ policy must remain after " + what + describePolicies(bitstream), @@ -762,49 +635,6 @@ private String describePolicies(Bitstream bitstream) throws Exception { return sb.toString(); } - private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { - MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); - MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); - if (existingField == null) { - MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); - } - } - - private Item createItem(String title) throws Exception { - context.turnOffAuthorisationSystem(); - Item item = ItemBuilder.createItem(context, collection) - .withTitle(title) - .build(); - context.restoreAuthSystemState(); - return item; - } - - /** - * Creates a bitstream in the ORIGINAL bundle. It inherits the collection DEFAULT_BITSTREAM_READ and so - * carries one undated Anonymous READ policy, the state a freshly imported SAF package is in. - */ - private Bitstream createOriginalBitstream(Item item, String name) throws Exception { - context.turnOffAuthorisationSystem(); - Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, - new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8))) - .withName(name) - .withMimeType("text/plain") - .build(); - context.restoreAuthSystemState(); - return bitstream; - } - - private Bitstream createBitstreamInBundle(Item item, String name, String bundleName) throws Exception { - context.turnOffAuthorisationSystem(); - Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, - new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8)), bundleName) - .withName(name) - .withMimeType("text/plain") - .build(); - context.restoreAuthSystemState(); - return bitstream; - } - private Bundle bundleOf(Item item, String bundleName) throws Exception { List bundles = itemService.getBundles(item, bundleName); assertFalse("fixture precondition: the item must have a " + bundleName + " bundle", bundles.isEmpty()); @@ -818,99 +648,6 @@ private void setPrimaryBitstream(Bundle bundle, Bitstream bitstream) throws Exce context.restoreAuthSystemState(); } - private void reloadAll(List bitstreams) throws Exception { - for (int i = 0; i < bitstreams.size(); i++) { - bitstreams.set(i, context.reloadEntity(bitstreams.get(i))); - } - } - - private ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) - throws Exception { - return addAnonymousReadPolicy(bitstream, startDate, name, null); - } - - private ResourcePolicy addAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name, - String policyType) throws Exception { - context.turnOffAuthorisationSystem(); - ResourcePolicyBuilder builder = ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) - .withAction(Constants.READ) - .withDspaceObject(bitstream) - .withName(name); - if (startDate != null) { - builder.withStartDate(startDate); - } - if (policyType != null) { - builder.withPolicyType(policyType); - } - ResourcePolicy policy = builder.build(); - context.restoreAuthSystemState(); - return policy; - } - - /** - * Replaces every READ policy of the bitstream with a single Anonymous READ policy, the state earlier - * versions left behind. - */ - private ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name) - throws Exception { - return replaceAnonymousReadPolicies(bitstream, startDate, name, null); - } - - private ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name, - String policyType) throws Exception { - context.turnOffAuthorisationSystem(); - authorizeService.removePoliciesActionFilter(context, bitstream, Constants.READ); - context.restoreAuthSystemState(); - return addAnonymousReadPolicy(bitstream, startDate, name, policyType); - } - - private String singleMetadataValue(Item item, String element, String qualifier) { - List values = itemService.getMetadata(item, "dc", element, qualifier, Item.ANY); - return values.isEmpty() ? null : values.get(0).getValue(); - } - - private Date startOfDayUtc(LocalDate day) { - return Date.from(day.atStartOfDay(ZoneOffset.UTC).toInstant()); - } - - private LocalDate toLocalDate(Date date) { - if (date instanceof java.sql.Date) { - return ((java.sql.Date) date).toLocalDate(); - } - return date.toInstant().atZone(ZoneId.systemDefault()).toLocalDate(); - } - - /** - * Runs itemupdate with both embargo fields as targets, the combination that triggers embargo - * synchronisation. - * - * @return the number of embargo problems the run reported, which is what {@code ItemUpdate.main()} turns - * into a non-zero exit code - */ - private int runItemUpdate(Item item, String dublinCoreContent) throws Exception { - Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); - Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); - - Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); - Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); - - ItemUpdate itemUpdate = new ItemUpdate(); - DeleteMetadataAction deleteAction = - (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); - deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); - - AddMetadataAction addAction = - (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); - addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); - - context.turnOffAuthorisationSystem(); - itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); - context.restoreAuthSystemState(); - - context.uncacheEntity(item); - return itemUpdate.embargoSyncFailures; - } - /** * A run the tool is supposed to carry out has to end with exit code 0. */ @@ -921,32 +658,4 @@ private void assertRunSucceeded(String what, int embargoSyncFailures) { 0, embargoSyncFailures); } - /** - * Builds a SAF {@code dublin_core.xml}. A {@code null} value omits the element entirely (that is how an - * operator removes a field), an empty string is written as a single space because an empty XML element is - * dropped by the parser. - */ - private String dublinCore(Item item, String rightsAccess, String embargoEndDate) { - StringBuilder sb = new StringBuilder(); - sb.append("\n") - .append("\n") - .append(" ") - .append(ItemUpdate.HANDLE_PREFIX).append(item.getHandle()) - .append("\n"); - - if (rightsAccess != null) { - sb.append(" ") - .append(rightsAccess) - .append("\n"); - } - - if (embargoEndDate != null) { - sb.append(" ") - .append(embargoEndDate.isEmpty() ? " " : embargoEndDate) - .append("\n"); - } - - sb.append(""); - return sb.toString(); - } } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java index a0d99faf3139..85c4251fa09a 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoPastDateIT.java @@ -12,45 +12,15 @@ import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; -import java.io.ByteArrayInputStream; -import java.nio.charset.StandardCharsets; -import java.nio.file.Files; -import java.nio.file.Path; import java.time.LocalDate; import java.util.ArrayList; import java.util.List; -import java.util.stream.Collectors; -import org.apache.commons.io.file.PathUtils; -import org.dspace.AbstractIntegrationTestWithDatabase; import org.dspace.authorize.ResourcePolicy; -import org.dspace.authorize.factory.AuthorizeServiceFactory; -import org.dspace.authorize.service.AuthorizeService; -import org.dspace.authorize.service.ResourcePolicyService; -import org.dspace.builder.BitstreamBuilder; -import org.dspace.builder.CollectionBuilder; -import org.dspace.builder.CommunityBuilder; -import org.dspace.builder.ItemBuilder; -import org.dspace.builder.MetadataFieldBuilder; import org.dspace.content.Bitstream; -import org.dspace.content.Collection; import org.dspace.content.Item; -import org.dspace.content.MetadataField; -import org.dspace.content.MetadataSchema; -import org.dspace.content.MetadataValue; -import org.dspace.content.factory.ContentServiceFactory; -import org.dspace.content.service.ItemService; -import org.dspace.content.service.MetadataFieldService; -import org.dspace.content.service.MetadataSchemaService; import org.dspace.core.Constants; -import org.dspace.eperson.EPerson; import org.dspace.eperson.Group; -import org.dspace.eperson.factory.EPersonServiceFactory; -import org.dspace.eperson.service.GroupService; -import org.dspace.handle.factory.HandleServiceFactory; -import org.dspace.handle.service.HandleService; -import org.junit.After; -import org.junit.Before; import org.junit.Test; /** @@ -58,61 +28,10 @@ * {@code dc.date.embargoend} that has already passed: the ORIGINAL bitstreams have to stay readable * for anonymous users instead of losing their last {@code READ} policy. */ -public class EmbargoPastDateIT extends AbstractIntegrationTestWithDatabase { - - private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); - private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); - private final ResourcePolicyService resourcePolicyService = - AuthorizeServiceFactory.getInstance().getResourcePolicyService(); - private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); - private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); - private final MetadataSchemaService metadataSchemaService = - ContentServiceFactory.getInstance().getMetadataSchemaService(); - private final MetadataFieldService metadataFieldService = - ContentServiceFactory.getInstance().getMetadataFieldService(); +public class EmbargoPastDateIT extends AbstractEmbargoIT { private final StringBuilder diagnostics = new StringBuilder(); - private Collection collection; - private Group anonymousGroup; - private Path tempDir; - private String previousHandlePrefix; - - @Before - @Override - public void setUp() throws Exception { - super.setUp(); - context.turnOffAuthorisationSystem(); - - parentCommunity = CommunityBuilder.createCommunity(context) - .withName("Parent Community") - .build(); - collection = CollectionBuilder.createCollection(context, parentCommunity) - .withName("Collection") - .build(); - - ensureMetadataFieldExists("rights", "access"); - ensureMetadataFieldExists("date", "embargoend"); - - anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); - previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; - ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); - - context.restoreAuthSystemState(); - - tempDir = Files.createTempDirectory("embargoPastDateIT"); - } - - @After - @Override - public void destroy() throws Exception { - ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; - if (tempDir != null) { - PathUtils.deleteDirectory(tempDir); - } - super.destroy(); - } - /** * Verifies that a future embargo followed by an expired one leaves the ORIGINAL bitstream publicly * readable. @@ -137,7 +56,7 @@ public void pastEmbargoEndMustKeepFilesPublic() throws Exception { anonymousCanRead(bitstream)); // first run: embargo end date in the future - runItemUpdate(item, dublinCore(item, "embargoedAccess", futureEmbargoEnd)); + runItemUpdateExpecting(item, dublinCore(item, "embargoedAccess", futureEmbargoEnd), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP B - after itemupdate with FUTURE dc.date.embargoend=" + futureEmbargoEnd, bitstream); @@ -150,7 +69,7 @@ public void pastEmbargoEndMustKeepFilesPublic() throws Exception { assertFalse("while embargoed the file must not be publicly readable", anonymousCanRead(bitstream)); // second run: embargo end date in the past, item declared openAccess - runItemUpdate(item, dublinCore(item, "openAccess", pastEmbargoEnd)); + runItemUpdateExpecting(item, dublinCore(item, "openAccess", pastEmbargoEnd), 0); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); dump("STEP C - after itemupdate with PAST dc.date.embargoend=" + pastEmbargoEnd @@ -166,33 +85,6 @@ public void pastEmbargoEndMustKeepFilesPublic() throws Exception { anonymousCanRead(bitstream)); } - /** - * Tells whether a visitor who is not logged in may read the bitstream. - */ - private boolean anonymousCanRead(Bitstream bs) throws Exception { - EPerson saved = context.getCurrentUser(); - int popped = 0; - while (context.ignoreAuthorization()) { - context.restoreAuthSystemState(); - popped++; - } - context.setCurrentUser(null); - try { - return authorizeService.authorizeActionBoolean(context, bs, Constants.READ); - } finally { - context.setCurrentUser(saved); - for (int i = 0; i < popped; i++) { - context.turnOffAuthorisationSystem(); - } - } - } - - private List anonymousReadPolicies(Bitstream bitstream) throws Exception { - return resourcePolicyService.find(context, bitstream, Constants.READ).stream() - .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) - .collect(Collectors.toList()); - } - private void dump(String label, Bitstream bitstream) throws Exception { List lines = new ArrayList<>(); for (ResourcePolicy policy : resourcePolicyService.find(context, bitstream, Constants.READ)) { @@ -222,94 +114,4 @@ private void dump(String label, Bitstream bitstream) throws Exception { System.out.print(sb); } - private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { - MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); - MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); - if (existingField == null) { - MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); - } - } - - private Item createItem(String title) throws Exception { - context.turnOffAuthorisationSystem(); - Item item = ItemBuilder.createItem(context, collection) - .withTitle(title) - .build(); - context.restoreAuthSystemState(); - return item; - } - - private Bitstream createOriginalBitstream(Item item, String name) throws Exception { - context.turnOffAuthorisationSystem(); - Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, - new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8))) - .withName(name) - .withMimeType("text/plain") - .build(); - context.restoreAuthSystemState(); - return bitstream; - } - - private String singleMetadataValue(Item item, String element, String qualifier) { - List values = itemService.getMetadata(item, "dc", element, qualifier, Item.ANY); - return values.isEmpty() ? null : values.get(0).getValue(); - } - - /** - * Runs itemupdate with both embargo fields as targets, the combination that triggers embargo - * synchronisation. - */ - private void runItemUpdate(Item item, String dublinCoreContent) throws Exception { - Path sourceRoot = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); - Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); - - Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); - Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); - - ItemUpdate itemUpdate = new ItemUpdate(); - DeleteMetadataAction deleteAction = - (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); - deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); - - AddMetadataAction addAction = - (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); - addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); - - context.turnOffAuthorisationSystem(); - itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); - context.restoreAuthSystemState(); - - context.uncacheEntity(item); - - // embargoSyncFailures drives the exit code of ItemUpdate.main(), so a refusal that left it at - // zero would be invisible to the calling script. - assertEquals("itemupdate reported an embargo synchronisation problem, so ItemUpdate.main() would exit" - + " with " + ItemUpdate.exitStatus(0, itemUpdate.embargoSyncFailures), - 0, itemUpdate.embargoSyncFailures); - } - - private String dublinCore(Item item, String rightsAccess, String embargoEndDate) { - String identifierUri = ItemUpdate.HANDLE_PREFIX + item.getHandle(); - StringBuilder sb = new StringBuilder(); - sb.append("\n") - .append("\n") - .append(" ") - .append(identifierUri) - .append("\n"); - - if (rightsAccess != null) { - sb.append(" ") - .append(rightsAccess) - .append("\n"); - } - - if (embargoEndDate != null) { - sb.append(" ") - .append(embargoEndDate) - .append("\n"); - } - - sb.append(""); - return sb.toString(); - } } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java index db2178557d65..67a51732946f 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoSafetyIT.java @@ -11,59 +11,24 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; -import java.io.PrintStream; -import java.nio.charset.StandardCharsets; -import java.nio.file.Files; -import java.nio.file.Path; import java.sql.SQLException; import java.time.LocalDate; -import java.time.ZoneId; -import java.time.ZoneOffset; -import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.Date; import java.util.List; import java.util.Set; -import java.util.TreeSet; -import java.util.stream.Collectors; -import org.apache.commons.io.file.PathUtils; -import org.dspace.AbstractIntegrationTestWithDatabase; import org.dspace.authorize.AuthorizeException; import org.dspace.authorize.ResourcePolicy; -import org.dspace.authorize.factory.AuthorizeServiceFactory; -import org.dspace.authorize.service.AuthorizeService; -import org.dspace.authorize.service.ResourcePolicyService; -import org.dspace.builder.BitstreamBuilder; -import org.dspace.builder.CollectionBuilder; -import org.dspace.builder.CommunityBuilder; import org.dspace.builder.GroupBuilder; -import org.dspace.builder.ItemBuilder; -import org.dspace.builder.MetadataFieldBuilder; import org.dspace.builder.ResourcePolicyBuilder; import org.dspace.content.Bitstream; -import org.dspace.content.Collection; import org.dspace.content.Item; -import org.dspace.content.MetadataField; -import org.dspace.content.MetadataSchema; import org.dspace.content.MetadataValue; -import org.dspace.content.factory.ContentServiceFactory; -import org.dspace.content.service.ItemService; -import org.dspace.content.service.MetadataFieldService; -import org.dspace.content.service.MetadataSchemaService; import org.dspace.core.Constants; import org.dspace.core.Context; -import org.dspace.eperson.EPerson; import org.dspace.eperson.Group; -import org.dspace.eperson.factory.EPersonServiceFactory; -import org.dspace.eperson.service.GroupService; -import org.dspace.handle.factory.HandleServiceFactory; -import org.dspace.handle.service.HandleService; -import org.junit.After; -import org.junit.Before; import org.junit.Test; /** @@ -71,13 +36,7 @@ * rather than publish it. Every "must not touch" assertion compares policy ids and a fingerprint of every * policy, because counts hide both delete-and-recreate and in-place mutation. */ -public class EmbargoSafetyIT extends AbstractIntegrationTestWithDatabase { - - /** - * rpName written by earlier versions; the fixtures use it so that normalisation of legacy policies is - * exercised. - */ - private static final String LEGACY_EMBARGO_POLICY_NAME = "Standard Embargo"; +public class EmbargoSafetyIT extends AbstractEmbargoIT { /** * The supported way of re-opening files whose Anonymous READ policy is already gone; ItemUpdate points @@ -91,65 +50,6 @@ public class EmbargoSafetyIT extends AbstractIntegrationTestWithDatabase { */ private static final String LEASE_POLICY_NAME = "lease"; - /** - * Sentinel for {@link #deletePolicies(Bitstream, int)} meaning "every action", picked so it can never - * collide with a real value of {@link Constants#actionText}. - */ - private static final int ALL_ACTIONS = -1; - - private final ItemService itemService = ContentServiceFactory.getInstance().getItemService(); - private final HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); - private final ResourcePolicyService resourcePolicyService = - AuthorizeServiceFactory.getInstance().getResourcePolicyService(); - private final AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); - private final GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); - private final MetadataSchemaService metadataSchemaService = - ContentServiceFactory.getInstance().getMetadataSchemaService(); - private final MetadataFieldService metadataFieldService = - ContentServiceFactory.getInstance().getMetadataFieldService(); - - private Collection collection; - private Group anonymousGroup; - private Path tempDir; - private String previousHandlePrefix; - - @Before - @Override - public void setUp() throws Exception { - super.setUp(); - context.turnOffAuthorisationSystem(); - - parentCommunity = CommunityBuilder.createCommunity(context) - .withName("Parent Community") - .build(); - collection = CollectionBuilder.createCollection(context, parentCommunity) - .withName("Collection") - .build(); - - // neither field exists in the test metadata registry, AddMetadataAction needs both - ensureMetadataFieldExists("rights", "access"); - ensureMetadataFieldExists("date", "embargoend"); - - anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); - // ItemArchive resolves items through this mutable static, it is restored in destroy() - previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; - ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); - - context.restoreAuthSystemState(); - - tempDir = Files.createTempDirectory("embargoSafetyIT"); - } - - @After - @Override - public void destroy() throws Exception { - ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; - if (tempDir != null) { - PathUtils.deleteDirectory(tempDir); - } - super.destroy(); - } - /** * Verifies that a withdrawn item is left alone: withdrawal turns every READ policy into WITHDRAWN_READ, * and synchronising an embargo must not undo a takedown. @@ -178,7 +78,7 @@ public void withdrawnItemIsNeverRepublished() throws Exception { Set idsBefore = policyIds(bitstream); List policiesBefore = policyFingerprints(bitstream); - Run pastRun = runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + Run pastRun = runItemUpdate(item, dublinCore(item, "openAccess", pastDate())); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -202,7 +102,7 @@ public void withdrawnItemIsNeverRepublished() throws Exception { // The past-date run above only reaches the early return; the future-date branch is the one that // creates policies. Run futureRun = - runItemUpdate(item, dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); + runItemUpdate(item, dublinCore(item, "embargoedAccess", futureDate())); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -289,7 +189,7 @@ public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { Set idsBefore = policyIds(bitstream); List policiesBefore = policyFingerprints(bitstream); - Run pastRun = runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + Run pastRun = runItemUpdate(item, dublinCore(item, "openAccess", pastDate())); bitstream = context.reloadEntity(bitstream); @@ -304,7 +204,7 @@ public void bitstreamWithoutAnonymousReadIsNotPublished() throws Exception { // A past date only reaches the early return; the future-date branch is where a group-restricted file // could gain an Anonymous policy. Run futureRun = runItemUpdate(item, - dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); + dublinCore(item, "embargoedAccess", futureDate())); bitstream = context.reloadEntity(bitstream); @@ -351,7 +251,7 @@ public void alreadyBrokenBitstreamWithZeroPoliciesStaysZero() throws Exception { assertFalse("fixture precondition: a bitstream without policies must not be readable" + describe(bitstream), anonymousCanRead(bitstream)); - Run run = runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + Run run = runItemUpdate(item, dublinCore(item, "openAccess", pastDate())); bitstream = context.reloadEntity(bitstream); @@ -386,7 +286,7 @@ public void leasedAnonymousReadPolicyIsUntouched() throws Exception { // The future-date branch is the one that writes policies, so it is where the end date would be lost. Run futureRun = runItemUpdate(item, - dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); + dublinCore(item, "embargoedAccess", futureDate())); bitstream = context.reloadEntity(bitstream); @@ -399,7 +299,7 @@ public void leasedAnonymousReadPolicyIsUntouched() throws Exception { // An expired end date reaches the same mutation, only with a start date that has already passed. Run pastRun = runItemUpdate(item, - dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + dublinCore(item, "openAccess", pastDate())); bitstream = context.reloadEntity(bitstream); @@ -428,7 +328,7 @@ public void leaseNextToADatedEmbargoPolicyIsNotDeleted() throws Exception { List policiesBefore = policyFingerprints(bitstream); Run run = runItemUpdate(item, - dublinCore(item, Collections.singletonList("embargoedAccess"), futureDate())); + dublinCore(item, "embargoedAccess", futureDate())); bitstream = context.reloadEntity(bitstream); @@ -503,7 +403,7 @@ public void notArchivedItemIsUntouched() throws Exception { Set idsBefore = policyIds(bitstream); List policiesBefore = policyFingerprints(bitstream); - Run run = runItemUpdate(item, dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + Run run = runItemUpdate(item, dublinCore(item, "openAccess", pastDate())); bitstream = context.reloadEntity(bitstream); @@ -535,7 +435,7 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta }; Run run = runItemUpdate(itemUpdate, item, - dublinCore(item, Collections.singletonList("openAccess"), pastDate())); + dublinCore(item, "openAccess", pastDate())); bitstream = context.reloadEntity(bitstream); @@ -563,7 +463,7 @@ private Item assertEmbargoSyncIsANoOp(String scenario, List accessRights Set idsBefore = policyIds(bitstream); List policiesBefore = policyFingerprints(bitstream); - Run run = runItemUpdate(item, dublinCore(item, accessRights, embargoEndDate)); + Run run = runItemUpdate(item, dublinCoreWithAccessRights(item, accessRights, embargoEndDate)); item = context.reloadEntity(item); bitstream = context.reloadEntity(bitstream); @@ -590,150 +490,6 @@ private void assertUntouched(String scenario, Set idsBefore, List accessRights, String embargoEndDate) { - StringBuilder sb = new StringBuilder(); - sb.append("\n") - .append("\n") - .append(" ") - .append(ItemUpdate.HANDLE_PREFIX) - .append(item.getHandle()) - .append("\n"); - - for (String accessRight : accessRights) { - sb.append(" ") - .append(accessRight) - .append("\n"); - } - - if (embargoEndDate != null) { - sb.append(" ") - .append(embargoEndDate.isEmpty() ? " " : embargoEndDate) - .append("\n"); - } - - sb.append(""); - return sb.toString(); - } - - private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { - MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); - MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); - if (existingField == null) { - MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); - } - } - - private Item createItem(String title) throws Exception { - context.turnOffAuthorisationSystem(); - Item item = ItemBuilder.createItem(context, collection) - .withTitle(title) - .build(); - context.restoreAuthSystemState(); - return item; - } - - /** - * A bitstream in the ORIGINAL bundle. It inherits the collection DEFAULT_BITSTREAM_READ and so carries one - * undated Anonymous READ policy, the state a freshly imported SAF item is in. - */ - private Bitstream createOriginalBitstream(Item item, String name) throws Exception { - context.turnOffAuthorisationSystem(); - Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, - new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8))) - .withName(name) - .withMimeType("text/plain") - .build(); - context.restoreAuthSystemState(); - return bitstream; - } - /** * The state an itemupdate run with a future end date leaves: a single dated legacy policy blocking access, * and nothing else between the public and the file. @@ -779,64 +535,6 @@ private void addLeasePolicy(Bitstream bitstream) throws Exception { .build(); } - /** - * Deletes policies one by one, because the bulk removal helpers issue an HQL delete and leave the - * in-memory collection stale. - * - * @param actionId action to delete, or {@link #ALL_ACTIONS} for every policy regardless of action - */ - private void deletePolicies(Bitstream bitstream, int actionId) throws Exception { - List doomed = actionId == ALL_ACTIONS - ? allPolicies(bitstream) - : policiesForAction(bitstream, actionId); - for (ResourcePolicy policy : doomed) { - resourcePolicyService.delete(context, policy); - } - } - - private List allPolicies(Bitstream bitstream) throws SQLException { - return new ArrayList<>(resourcePolicyService.find(context, bitstream)); - } - - private List policiesForAction(Bitstream bitstream, int actionId) throws SQLException { - return new ArrayList<>(resourcePolicyService.find(context, bitstream, actionId)); - } - - private List anonymousReadPolicies(Bitstream bitstream) throws SQLException { - return policiesForAction(bitstream, Constants.READ).stream() - .filter(policy -> policy.getGroup() != null && anonymousGroup.equals(policy.getGroup())) - .collect(Collectors.toList()); - } - - private Set policyIds(Bitstream bitstream) throws SQLException { - Set ids = new TreeSet<>(); - for (ResourcePolicy policy : allPolicies(bitstream)) { - ids.add(policy.getID()); - } - return ids; - } - - private List policyFingerprints(Bitstream bitstream) throws SQLException { - List fingerprints = new ArrayList<>(); - for (ResourcePolicy policy : allPolicies(bitstream)) { - fingerprints.add(fingerprint(policy)); - } - Collections.sort(fingerprints); - return fingerprints; - } - - private String fingerprint(ResourcePolicy policy) { - return String.format("id=%s action=%s group=%s eperson=%s rpType=%s rpName=%s start=%s end=%s", - policy.getID(), - Constants.actionText[policy.getAction()], - policy.getGroup() == null ? "" : policy.getGroup().getName(), - policy.getEPerson() == null ? "" : policy.getEPerson().getEmail(), - policy.getRpType(), - policy.getRpName(), - day(policy.getStartDate()), - day(policy.getEndDate())); - } - /** * Renders the current policies of the bitstream for failure messages, so a red build shows which policy * moved. @@ -854,54 +552,4 @@ private String describe(Bitstream bitstream) throws SQLException { return sb.toString(); } - /** - * Tells whether a visitor who is not logged in may read the bitstream. The authorisation state is a stack - * the builders push and pop, so it is drained first - otherwise every read looks allowed. - */ - private boolean anonymousCanRead(Bitstream bitstream) throws SQLException { - EPerson savedUser = context.getCurrentUser(); - int popped = 0; - while (context.ignoreAuthorization()) { - context.restoreAuthSystemState(); - popped++; - } - context.setCurrentUser(null); - try { - return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); - } finally { - context.setCurrentUser(savedUser); - for (int i = 0; i < popped; i++) { - context.turnOffAuthorisationSystem(); - } - } - } - - private String pastDate() { - return LocalDate.now().minusMonths(1).toString(); - } - - /** - * A future end date, the branch that writes resource policies; the past-date branch returns early. - */ - private String futureDate() { - return LocalDate.now().plusYears(1).toString(); - } - - private Date startOfDayUtc(LocalDate day) { - return Date.from(day.atStartOfDay(ZoneOffset.UTC).toInstant()); - } - - /** - * Renders a date at calendar day granularity: start dates come back from the database as - * {@code java.sql.Date}, so comparing instants across the harness time zone would be flaky. - */ - private String day(Date date) { - if (date == null) { - return ""; - } - if (date instanceof java.sql.Date) { - return ((java.sql.Date) date).toLocalDate().toString(); - } - return date.toInstant().atZone(ZoneId.systemDefault()).toLocalDate().toString(); - } } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java index 04b579c48218..6871b4d079eb 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/ItemUpdateIT.java @@ -13,113 +13,26 @@ import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; -import java.io.ByteArrayInputStream; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; -import java.sql.SQLException; import java.time.LocalDate; import java.time.ZoneId; import java.util.Date; import java.util.List; -import java.util.stream.Collectors; +import java.util.Set; -import org.apache.commons.io.file.PathUtils; -import org.dspace.AbstractIntegrationTestWithDatabase; import org.dspace.authorize.ResourcePolicy; -import org.dspace.authorize.factory.AuthorizeServiceFactory; -import org.dspace.authorize.service.AuthorizeService; -import org.dspace.authorize.service.ResourcePolicyService; -import org.dspace.builder.BitstreamBuilder; -import org.dspace.builder.CollectionBuilder; -import org.dspace.builder.CommunityBuilder; -import org.dspace.builder.ItemBuilder; -import org.dspace.builder.MetadataFieldBuilder; -import org.dspace.builder.ResourcePolicyBuilder; import org.dspace.content.Bitstream; -import org.dspace.content.Collection; import org.dspace.content.Item; -import org.dspace.content.MetadataField; -import org.dspace.content.MetadataSchema; import org.dspace.content.MetadataValue; -import org.dspace.content.factory.ContentServiceFactory; -import org.dspace.content.service.ItemService; -import org.dspace.content.service.MetadataFieldService; -import org.dspace.content.service.MetadataSchemaService; -import org.dspace.core.Constants; -import org.dspace.eperson.EPerson; -import org.dspace.eperson.Group; -import org.dspace.eperson.factory.EPersonServiceFactory; -import org.dspace.eperson.service.GroupService; -import org.dspace.handle.factory.HandleServiceFactory; -import org.dspace.handle.service.HandleService; -import org.junit.After; -import org.junit.Before; import org.junit.Test; /** * Integration tests for {@link ItemUpdate} and {@link ItemArchive}. */ -public class ItemUpdateIT extends AbstractIntegrationTestWithDatabase { - - /** rpName written by earlier versions; it has to be adopted and normalised. */ - private static final String STANDARD_EMBARGO = "Standard Embargo"; - - /** The single normalised rpName, matching the access condition name in access-conditions.xml. */ - private static final String EMBARGO_POLICY_NAME = "embargo"; - - private ItemService itemService = ContentServiceFactory.getInstance().getItemService(); - private HandleService handleService = HandleServiceFactory.getInstance().getHandleService(); - private ResourcePolicyService resourcePolicyService = - AuthorizeServiceFactory.getInstance().getResourcePolicyService(); - private AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); - private GroupService groupService = EPersonServiceFactory.getInstance().getGroupService(); - private MetadataSchemaService metadataSchemaService = - ContentServiceFactory.getInstance().getMetadataSchemaService(); - private MetadataFieldService metadataFieldService = - ContentServiceFactory.getInstance().getMetadataFieldService(); - - private Collection collection; - private Group anonymousGroup; - private Path tempDir; - private String previousHandlePrefix; - - @Before - @Override - public void setUp() throws Exception { - super.setUp(); - context.turnOffAuthorisationSystem(); - - parentCommunity = CommunityBuilder.createCommunity(context) - .withName("Parent Community") - .build(); - collection = CollectionBuilder.createCollection(context, parentCommunity) - .withName("Collection") - .build(); - - ensureMetadataFieldExists("identifier", "thesis"); - ensureMetadataFieldExists("rights", "access"); - ensureMetadataFieldExists("date", "embargoend"); - - anonymousGroup = groupService.findByName(context, Group.ANONYMOUS); - previousHandlePrefix = ItemUpdate.HANDLE_PREFIX; - ItemUpdate.HANDLE_PREFIX = handleService.getCanonicalPrefix(); - - context.restoreAuthSystemState(); - - tempDir = Files.createTempDirectory("itemUpdateIT"); - } - - @After - @Override - public void destroy() throws Exception { - ItemUpdate.HANDLE_PREFIX = previousHandlePrefix; - if (tempDir != null) { - PathUtils.deleteDirectory(tempDir); - } - super.destroy(); - } +public class ItemUpdateIT extends AbstractEmbargoIT { /** * Verifies the step that turns {@code embargoSyncFailures} into the process exit code, which is otherwise @@ -199,9 +112,9 @@ public void syncEmbargoPoliciesDatesTheAnonymousReadPolicyAndBlocksAccess() thro Item item = createItem("Standard Embargo Item", "rights", "access", "embargoedAccess", "date", "embargoend", futureDate); - Bitstream bitstream = createBitstream(item, "standard.txt"); + Bitstream bitstream = createOriginalBitstream(item, "standard.txt"); - createAnonymousReadPolicy(bitstream, null, "Immediate Read"); + addAnonymousReadPolicy(bitstream, null, "Immediate Read"); ItemUpdate itemUpdate = new ItemUpdate(); itemUpdate.syncEmbargoPolicies(context, item); @@ -223,7 +136,7 @@ public void syncEmbargoPoliciesDatesTheAnonymousReadPolicyAndBlocksAccess() thro public void syncEmbargoPoliciesAppliesEmbargoWithoutAccessRightMetadata() throws Exception { String futureDate = LocalDate.now().plusDays(21).toString(); Item item = createItem("Special Case Embargo Item", "date", "embargoend", futureDate); - Bitstream bitstream = createBitstream(item, "special.txt"); + Bitstream bitstream = createOriginalBitstream(item, "special.txt"); ItemUpdate itemUpdate = new ItemUpdate(); itemUpdate.syncEmbargoPolicies(context, item); @@ -249,12 +162,12 @@ public void syncEmbargoPoliciesAppliesEmbargoWithoutAccessRightMetadata() throws @Test public void syncEmbargoPoliciesLeavesPoliciesUntouchedWhenEmbargoDateInvalid() throws Exception { Item item = createItem("Invalid Date Item", "date", "embargoend", ""); - Bitstream bitstream = createBitstream(item, "invalid.txt"); + Bitstream bitstream = createOriginalBitstream(item, "invalid.txt"); ResourcePolicy legacyPolicy = replaceAnonymousReadPolicies(bitstream, - new Date(System.currentTimeMillis() + 86_400_000L), STANDARD_EMBARGO); + new Date(System.currentTimeMillis() + 86_400_000L), LEGACY_EMBARGO_POLICY_NAME); bitstream = context.reloadEntity(bitstream); - List idsBefore = policyIds(bitstream); + Set idsBefore = policyIds(bitstream); assertFalse("fixture precondition: the embargoed file must not be publicly readable, otherwise the" + " 'nothing changed' assertions below say nothing about a leak", anonymousCanRead(bitstream)); @@ -272,7 +185,7 @@ public void syncEmbargoPoliciesLeavesPoliciesUntouchedWhenEmbargoDateInvalid() t // The dated policy the run could not validate is still there, unchanged, under its legacy name. ResourcePolicy reloadedLegacy = resourcePolicyService.find(context, legacyPolicy.getID()); assertNotNull(reloadedLegacy); - assertEquals(STANDARD_EMBARGO, reloadedLegacy.getRpName()); + assertEquals(LEGACY_EMBARGO_POLICY_NAME, reloadedLegacy.getRpName()); } @Test @@ -283,15 +196,15 @@ public void processArchiveUpdatesEmbargoMetadataAndResyncsEmbargoPolicy() throws Item item = createItem("Embargo Update Item", "rights", "access", "embargoedAccess", "date", "embargoend", oldEmbargoDate); - Bitstream bitstream = createBitstream(item, "update-embargo.txt"); + Bitstream bitstream = createOriginalBitstream(item, "update-embargo.txt"); LocalDate oldPolicyDate = LocalDate.parse(oldEmbargoDate).plusDays(1); Date oldPolicyStart = Date.from(oldPolicyDate.atStartOfDay(ZoneId.systemDefault()).toInstant()); - ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = addAnonymousReadPolicy(bitstream, oldPolicyStart, LEGACY_EMBARGO_POLICY_NAME); Integer legacyPolicyId = legacyPolicy.getID(); assertEquals("re-dating an embargo is not a failure", 0, - runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, "embargoedAccess", newEmbargoDate))); + runItemUpdateFailures(item, dublinCore(item, "embargoedAccess", newEmbargoDate))); Item reloadedItem = context.reloadEntity(item); Bitstream reloadedBitstream = context.reloadEntity(bitstream); @@ -326,20 +239,21 @@ public void processArchiveUpdateWithBlankEmbargoDateLeavesPoliciesUntouched() th Item item = createItem("Blank Embargo Date Update", "rights", "access", "embargoedAccess", "date", "embargoend", oldEmbargoDate); - Bitstream bitstream = createBitstream(item, "blank-embargo-date.txt"); + Bitstream bitstream = createOriginalBitstream(item, "blank-embargo-date.txt"); Date oldPolicyStart = Date.from(LocalDate.parse(oldEmbargoDate).plusDays(1) .atStartOfDay(ZoneId.systemDefault()).toInstant()); - ResourcePolicy legacyPolicy = replaceAnonymousReadPolicies(bitstream, oldPolicyStart, STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = + replaceAnonymousReadPolicies(bitstream, oldPolicyStart, LEGACY_EMBARGO_POLICY_NAME); bitstream = context.reloadEntity(bitstream); - List idsBefore = policyIds(bitstream); + Set idsBefore = policyIds(bitstream); assertFalse("fixture precondition: the embargoed file must not be publicly readable, otherwise the" + " 'nothing changed' assertions below say nothing about a leak", anonymousCanRead(bitstream)); assertEquals("a blank dc.date.embargoend has to fail the run", 1, - runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, "embargoedAccess", ""))); + runItemUpdateFailures(item, dublinCore(item, "embargoedAccess", ""))); Bitstream reloadedBitstream = context.reloadEntity(bitstream); @@ -349,7 +263,7 @@ public void processArchiveUpdateWithBlankEmbargoDateLeavesPoliciesUntouched() th ResourcePolicy reloadedLegacy = resourcePolicyService.find(context, legacyPolicy.getID()); assertNotNull(reloadedLegacy); - assertEquals(STANDARD_EMBARGO, reloadedLegacy.getRpName()); + assertEquals(LEGACY_EMBARGO_POLICY_NAME, reloadedLegacy.getRpName()); } /** @@ -363,21 +277,22 @@ public void processArchiveUpdateRemovingEmbargoMetadataLeavesPoliciesUntouched() Item item = createItem("Remove Embargo Metadata Update", "rights", "access", "embargoedAccess", "date", "embargoend", oldEmbargoDate); - Bitstream bitstream = createBitstream(item, "remove-embargo.txt"); + Bitstream bitstream = createOriginalBitstream(item, "remove-embargo.txt"); Date oldPolicyStart = Date.from(LocalDate.parse(oldEmbargoDate).plusDays(1) .atStartOfDay(ZoneId.systemDefault()).toInstant()); // The undated Anonymous READ policy from the collection default has to go, otherwise the file is // readable throughout and the assertions below prove nothing. - ResourcePolicy legacyPolicy = replaceAnonymousReadPolicies(bitstream, oldPolicyStart, STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = + replaceAnonymousReadPolicies(bitstream, oldPolicyStart, LEGACY_EMBARGO_POLICY_NAME); Integer legacyPolicyId = legacyPolicy.getID(); bitstream = context.reloadEntity(bitstream); - List idsBefore = policyIds(bitstream); + Set idsBefore = policyIds(bitstream); assertFalse("fixture precondition: the embargoed file must not be publicly readable", anonymousCanRead(bitstream)); - int failures = runEmbargoMetadataUpdate(item, dublinCore(item)); + int failures = runItemUpdateFailures(item, dublinCore(item)); Item reloadedItem = context.reloadEntity(item); Bitstream reloadedBitstream = context.reloadEntity(bitstream); @@ -396,7 +311,7 @@ public void processArchiveUpdateRemovingEmbargoMetadataLeavesPoliciesUntouched() assertEquals(legacyPolicyId, untouchedPolicy.getID()); assertNotNull("removing dc.date.embargoend must not clear the embargo start date", untouchedPolicy.getStartDate()); - assertEquals(STANDARD_EMBARGO, untouchedPolicy.getRpName()); + assertEquals(LEGACY_EMBARGO_POLICY_NAME, untouchedPolicy.getRpName()); assertFalse("removing dc.date.embargoend published an embargoed file", anonymousCanRead(reloadedBitstream)); // "No instruction" is not a failure - the batch has to keep its exit code 0. @@ -411,15 +326,15 @@ public void processArchiveUpdateWithEmbargoDateAndNoRightsAppliesEmbargo() throw Item item = createItem("Special Case Update", "rights", "access", "embargoedAccess", "date", "embargoend", oldEmbargoDate); - Bitstream bitstream = createBitstream(item, "special-case-update.txt"); + Bitstream bitstream = createOriginalBitstream(item, "special-case-update.txt"); Date oldPolicyStart = Date.from(LocalDate.parse(oldEmbargoDate).plusDays(1) .atStartOfDay(ZoneId.systemDefault()).toInstant()); - ResourcePolicy legacyPolicy = createAnonymousReadPolicy(bitstream, oldPolicyStart, STANDARD_EMBARGO); + ResourcePolicy legacyPolicy = addAnonymousReadPolicy(bitstream, oldPolicyStart, LEGACY_EMBARGO_POLICY_NAME); Integer legacyPolicyId = legacyPolicy.getID(); assertEquals("an embargo end date without dc.rights.access is not a failure", 0, - runEmbargoMetadataUpdate(item, dublinCoreWithEmbargo(item, null, newEmbargoDate))); + runItemUpdateFailures(item, dublinCore(item, null, newEmbargoDate))); Item reloadedItem = context.reloadEntity(item); Bitstream reloadedBitstream = context.reloadEntity(bitstream); @@ -440,111 +355,6 @@ public void processArchiveUpdateWithEmbargoDateAndNoRightsAppliesEmbargo() throw assertFalse(anonymousCanRead(reloadedBitstream)); } - private void ensureMetadataFieldExists(String element, String qualifier) throws Exception { - MetadataSchema dcSchema = metadataSchemaService.find(context, "dc"); - MetadataField existingField = metadataFieldService.findByElement(context, dcSchema, element, qualifier); - if (existingField == null) { - MetadataFieldBuilder.createMetadataField(context, dcSchema, element, qualifier, null).build(); - } - } - - private Item createItem(String title, String... metadataTriples) throws Exception { - context.turnOffAuthorisationSystem(); - - ItemBuilder builder = ItemBuilder.createItem(context, collection) - .withTitle(title); - - for (int i = 0; i + 2 < metadataTriples.length; i += 3) { - builder.withMetadata("dc", metadataTriples[i], metadataTriples[i + 1], metadataTriples[i + 2]); - } - - Item item = builder.build(); - context.restoreAuthSystemState(); - return item; - } - - private Bitstream createBitstream(Item item, String name) throws Exception { - context.turnOffAuthorisationSystem(); - Bitstream bitstream = BitstreamBuilder.createBitstream(context, item, - new ByteArrayInputStream(("content-" + name).getBytes(StandardCharsets.UTF_8))) - .withName(name) - .withMimeType("text/plain") - .build(); - context.restoreAuthSystemState(); - return bitstream; - } - - /** - * Leaves the bitstream with one Anonymous READ policy, removing the collection's undated default first; - * without that an "embargoed" fixture is not embargoed at all. - */ - private ResourcePolicy replaceAnonymousReadPolicies(Bitstream bitstream, Date startDate, String name) - throws Exception { - context.turnOffAuthorisationSystem(); - authorizeService.removePoliciesActionFilter(context, bitstream, Constants.READ); - context.restoreAuthSystemState(); - return createAnonymousReadPolicy(bitstream, startDate, name); - } - - private ResourcePolicy createAnonymousReadPolicy(Bitstream bitstream, Date startDate, String name) - throws Exception { - context.turnOffAuthorisationSystem(); - ResourcePolicyBuilder builder = ResourcePolicyBuilder.createResourcePolicy(context, null, anonymousGroup) - .withAction(Constants.READ) - .withDspaceObject(bitstream) - .withName(name); - - if (startDate != null) { - builder.withStartDate(startDate); - } - ResourcePolicy policy = builder.build(); - context.restoreAuthSystemState(); - return policy; - } - - private boolean isAnonymousPolicy(ResourcePolicy policy) { - return policy.getGroup() != null && policy.getGroup().equals(anonymousGroup); - } - - private List anonymousReadPolicies(Bitstream bitstream) throws Exception { - return resourcePolicyService.find(context, bitstream, Constants.READ).stream() - .filter(this::isAnonymousPolicy) - .collect(Collectors.toList()); - } - - /** - * Identity of every resource policy on the bitstream. Ids rather than counts, so a policy that was deleted - * and re-created is visible. - */ - private List policyIds(Bitstream bitstream) throws Exception { - return authorizeService.getPolicies(context, bitstream).stream() - .map(ResourcePolicy::getID) - .sorted() - .collect(Collectors.toList()); - } - - /** - * Tells whether a visitor who is not logged in may read the bitstream, with the test's own - * turnOffAuthorisationSystem calls temporarily unwound. - */ - private boolean anonymousCanRead(Bitstream bitstream) throws SQLException { - EPerson savedUser = context.getCurrentUser(); - int popped = 0; - while (context.ignoreAuthorization()) { - context.restoreAuthSystemState(); - popped++; - } - context.setCurrentUser(null); - try { - return authorizeService.authorizeActionBoolean(context, bitstream, Constants.READ); - } finally { - context.setCurrentUser(savedUser); - for (int i = 0; i < popped; i++) { - context.turnOffAuthorisationSystem(); - } - } - } - private Path createSafItemDirectory(String dublinCoreContent) throws IOException { Path safDir = Files.createDirectory(tempDir.resolve("saf-" + System.nanoTime())); Path itemDir = Files.createDirectory(safDir.resolve("item_000")); @@ -570,37 +380,6 @@ private String dublinCore(String identifierUri, String thesisIdentifier) { return sb.toString(); } - /** - * Runs one SAF metadata update over the item. - * - * @return the number of embargo problems reported by the run, which {@link ItemUpdate#exitStatus(int, int)} - * turns into the exit code of {@code dspace itemupdate} - */ - private int runEmbargoMetadataUpdate(Item item, String dublinCoreContent) throws Exception { - Path sourceRoot = Files.createDirectory(tempDir.resolve("update-source-" + System.nanoTime())); - Files.createFile(sourceRoot.resolve(ItemUpdate.SUPPRESS_UNDO_FILENAME)); - - Path itemDir = Files.createDirectory(sourceRoot.resolve("item_000")); - Files.writeString(itemDir.resolve("dublin_core.xml"), dublinCoreContent, StandardCharsets.UTF_8); - - ItemUpdate itemUpdate = new ItemUpdate(); - DeleteMetadataAction deleteAction = - (DeleteMetadataAction) itemUpdate.actionMgr.getUpdateAction(DeleteMetadataAction.class); - deleteAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); - - AddMetadataAction addAction = - (AddMetadataAction) itemUpdate.actionMgr.getUpdateAction(AddMetadataAction.class); - addAction.addTargetFields(new String[] { "dc.rights.access", "dc.date.embargoend" }); - - context.turnOffAuthorisationSystem(); - itemUpdate.processArchive(context, sourceRoot.toString(), null, null, true, false, true); - context.restoreAuthSystemState(); - - // Force entity reload in caller assertions after update transaction. - context.uncacheEntity(item); - return itemUpdate.embargoSyncFailures; - } - private String dublinCore(Item item) { String identifierUri = ItemUpdate.HANDLE_PREFIX + item.getHandle(); return "\n" @@ -609,35 +388,4 @@ private String dublinCore(Item item) { + ""; } - private String dublinCoreWithEmbargo(Item item, String rightsAccess, String embargoEndDate) { - String identifierUri = ItemUpdate.HANDLE_PREFIX + item.getHandle(); - StringBuilder sb = new StringBuilder(); - sb.append("\n") - .append("\n") - .append(" ") - .append(identifierUri) - .append("\n"); - - if (rightsAccess != null) { - sb.append(" ") - .append(rightsAccess) - .append("\n"); - } - - if (embargoEndDate != null) { - sb.append(" ") - .append(embargoEndDate.isEmpty() ? " " : embargoEndDate) - .append("\n"); - } - - sb.append(""); - return sb.toString(); - } - - private LocalDate toLocalDate(Date date) { - if (date instanceof java.sql.Date) { - return ((java.sql.Date) date).toLocalDate(); - } - return date.toInstant().atZone(ZoneId.systemDefault()).toLocalDate(); - } } \ No newline at end of file From 2419fa0647edeb215896f4f224d52438935a996f Mon Sep 17 00:00:00 2001 From: milanmajchrak Date: Thu, 20 Aug 2026 11:15:15 +0200 Subject: [PATCH 09/10] VSB-TUO/Fix: delete the half built submission when embargo metadata is refused The batch import path completes its context in a finally block, so a workspace item left behind by a refused package was committed as an orphan submission with its bitstreams. The command line path aborts its context and was never affected. Mirrors the cleanup the install failure path in the same method already does. --- .../app/itemimport/ItemImportServiceImpl.java | 5 +++ .../app/itemimport/EmbargoImportIT.java | 36 +++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java index 943a97e6a2c0..0597b88daa83 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java @@ -819,6 +819,11 @@ protected Item addItem(Context c, List mycollections, String path, try { processEmbargoMetadata(c, myitem); } catch (EmbargoMetadataException e) { + // The half built submission goes with the failure: the batch import path completes its context + // in a finally block, so anything left behind here would be committed as an orphan. + if (wi != null) { + workspaceItemService.deleteAll(c, wi); + } // The operator needs the package directory, not the item id: the package is what they fix. throw new EmbargoMetadataException("SAF package '" + itemname + "': " + e.getMessage(), e); } diff --git a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java index b31e2f9d9395..2d9bff42b0f6 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java @@ -25,6 +25,7 @@ import java.time.LocalDate; import java.time.YearMonth; import java.time.ZoneOffset; +import java.util.Collections; import java.util.Date; import java.util.Iterator; import java.util.List; @@ -35,6 +36,7 @@ import org.apache.commons.io.file.PathUtils; import org.dspace.AbstractIntegrationTestWithDatabase; +import org.dspace.app.itemimport.factory.ItemImportServiceFactory; import org.dspace.authorize.ResourcePolicy; import org.dspace.authorize.factory.AuthorizeServiceFactory; import org.dspace.authorize.service.AuthorizeService; @@ -50,6 +52,7 @@ import org.dspace.content.factory.ContentServiceFactory; import org.dspace.content.service.ItemService; import org.dspace.content.service.MetadataSchemaService; +import org.dspace.content.service.WorkspaceItemService; import org.dspace.core.Constants; import org.dspace.eperson.EPerson; import org.dspace.eperson.Group; @@ -90,6 +93,8 @@ public class EmbargoImportIT extends AbstractIntegrationTestWithDatabase { private static final String EMBARGO_POLICY_NAME = "embargo"; private ItemService itemService = ContentServiceFactory.getInstance().getItemService(); + private WorkspaceItemService workspaceItemService = + ContentServiceFactory.getInstance().getWorkspaceItemService(); private ResourcePolicyService resourcePolicyService = AuthorizeServiceFactory.getInstance().getResourcePolicyService(); private AuthorizeService authorizeService = AuthorizeServiceFactory.getInstance().getAuthorizeService(); @@ -606,6 +611,37 @@ public void testBlankEmbargoEndIsRefused() throws Exception { assertBrokenEmbargoPackageIsRefused("", "an empty dc.date.embargoend"); } + /** + * Verifies that a refused package leaves no workspace item behind. The command line path aborts its + * context, but the batch import path completes its own in a finally block, so a half built submission + * left here would be committed as an orphan submission with its bitstreams. + * + *

    {@code addItem} is driven directly because that is the unit that has to clean up after itself; going + * through the script would only exercise the caller that aborts anyway.

    + */ + @Test + public void testRefusedPackageLeavesNoWorkspaceItem() throws Exception { + Path itemDir = safPackage("embargoedAccess", "not-a-date", "TEST CONTENT ORPHAN"); + ItemImportServiceImpl importService = + (ItemImportServiceImpl) ItemImportServiceFactory.getInstance().getItemImportService(); + importService.setTest(false); + + context.turnOffAuthorisationSystem(); + try { + importService.addItem(context, Collections.singletonList(collection), + itemDir.getParent().toString(), itemDir.getFileName().toString(), null, false); + fail("an unparseable dc.date.embargoend has to be reported instead of importing the package"); + } catch (EmbargoMetadataException expected) { + // the failure the operator is given + } finally { + context.restoreAuthSystemState(); + } + + assertTrue("the refused package must leave no workspace item behind, the batch import path would" + + " commit it as an orphan submission", + workspaceItemService.findByCollection(context, collection).isEmpty()); + } + /** * Verifies that a bare year keeps the day {@code DCDate} mapped it to, 1 January; reading it as * 31 December would extend embargoes that repositories already live with. From 9bee40e190cd722fbce360c2dfb1be559016461e Mon Sep 17 00:00:00 2001 From: milanmajchrak Date: Thu, 20 Aug 2026 11:42:45 +0200 Subject: [PATCH 10/10] VSB-TUO/Fix: embargo every bundle that holds the work, not only ORIGINAL A SAF contents file can route its payload into a bundle of its own with the "bundle:" marker. The import path only looked at ORIGINAL, so such a package was archived with public files although its own metadata declared it closed, and itemupdate left that bundle public for the whole embargo. Both tools now cover every bundle except LICENSE, CC-LICENSE and METADATA, the same three DefaultEmbargoSetter leaves world readable. Two smaller fixes on the same paths: - SafEmbargoDateParser accepts an unpadded year-month such as 2027-2, which SimpleDateFormat read and existing SAF packages therefore contain - itemupdate reports an expired embargo as published only after every bitstream is synchronised, so a bitstream it had to refuse is not announced as public --- .../app/itemimport/ItemImportServiceImpl.java | 17 ++++---- .../org/dspace/app/itemupdate/ItemUpdate.java | 22 ++++++----- .../dspace/app/util/SafEmbargoConstants.java | 22 +++++++++-- .../dspace/app/util/SafEmbargoDateParser.java | 17 +++++++- .../app/itemimport/EmbargoImportIT.java | 36 ++++++++++++++++- .../app/itemupdate/EmbargoDateBoundaryIT.java | 3 ++ .../app/itemupdate/EmbargoDerivativesIT.java | 39 +++++++++++++++++++ 7 files changed, 131 insertions(+), 25 deletions(-) diff --git a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java index 0597b88daa83..7a6b8f069ec4 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java +++ b/dspace-api/src/main/java/org/dspace/app/itemimport/ItemImportServiceImpl.java @@ -2680,17 +2680,14 @@ protected void applyEmbargoToItemBitstreams(Context c, Item item, Date accessSta + " default policies and no embargo at all."); } - // Only process ORIGINAL bundles to avoid affecting system bundles - List originalBundles = item.getBundles("ORIGINAL"); - if (originalBundles.isEmpty()) { - // Known limitation: a contents file can route its files into another bundle with the - // "bundle:" marker; only the ORIGINAL bundle is embargoed at import time. - logInfo("Embargo: No ORIGINAL bundles found, no embargo applied"); - return; - } - + // Every bundle that holds the work itself, licence and metadata bundles excepted: a contents file + // can route its files into a bundle of its own with the "bundle:" marker, and leaving those + // out would archive the package the operator declared closed with public files. int bitstreamsProcessed = 0; - for (Bundle bundle : originalBundles) { + for (Bundle bundle : item.getBundles()) { + if (!SafEmbargoConstants.isEmbargoed(bundle.getName())) { + continue; + } for (Bitstream bitstream : bundle.getBitstreams()) { // Create ResourcePolicy for READ access with start date = embargo end date + 1 day ResourcePolicy policy = resourcePolicyService.create(c, null, anonymousGroup); diff --git a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java index 19922d650c07..847b1db3ed32 100644 --- a/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java +++ b/dspace-api/src/main/java/org/dspace/app/itemupdate/ItemUpdate.java @@ -788,14 +788,16 @@ protected void syncEmbargoPolicies(Context context, Item item) throws SQLExcepti LocalDate accessStartDay = embargoEndDay.plusDays(1); Date accessStartDate = Date.from(accessStartDay.atStartOfDay(ZoneOffset.UTC).toInstant()); - if (!accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { + int failuresBefore = embargoSyncFailures; + applyEmbargoToItemBitstreams(context, item, accessStartDate); + + if (embargoSyncFailures == failuresBefore && !accessStartDay.isAfter(LocalDate.now(ZoneOffset.UTC))) { // An expired embargo is a publication, not a deletion: the start date that has already passed - // makes the policy effective immediately. + // makes the policy effective immediately. Reported once every bitstream is done, so that a + // bitstream the tool had to refuse is never announced as public. pr("Embargo of item " + itemLabel(item) + " already expired on " + embargoEndDay + ", its bitstreams are public since " + accessStartDay + "."); } - - applyEmbargoToItemBitstreams(context, item, accessStartDate); } /** @@ -821,11 +823,13 @@ protected void applyEmbargoToItemBitstreams(Context context, Item item, Date sta return; } - for (String bundleName : SafEmbargoConstants.EMBARGOED_BUNDLE_NAMES) { - for (Bundle bundle : item.getBundles(bundleName)) { - for (Bitstream bitstream : bundle.getBitstreams()) { - applyEmbargoToBitstream(context, item, bundleName, bitstream, anonymousGroup, startDate); - } + for (Bundle bundle : item.getBundles()) { + String bundleName = bundle.getName(); + if (!SafEmbargoConstants.isEmbargoed(bundleName)) { + continue; + } + for (Bitstream bitstream : bundle.getBitstreams()) { + applyEmbargoToBitstream(context, item, bundleName, bitstream, anonymousGroup, startDate); } } } diff --git a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java index bcd5940edb9c..84f18213217c 100644 --- a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java +++ b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoConstants.java @@ -12,6 +12,7 @@ import java.util.List; import org.dspace.core.Constants; +import org.dspace.license.service.CreativeCommonsService; /** * Constants of the embargo resource policies written by the SAF batch tools, declared once so that @@ -33,11 +34,24 @@ public final class SafEmbargoConstants { public static final String THUMBNAIL_BUNDLE_NAME = "THUMBNAIL"; /** - * Bundles an embargo covers: the file itself and everything derived from it, because the thumbnail and the - * extracted full text disclose the embargoed file. {@code filter.*.publicPermission} is ignored on purpose. + * Bundles an embargo never covers, the same three {@code DefaultEmbargoSetter} leaves world readable. + * Everything else is covered: the file, the thumbnail and the extracted full text disclose the embargoed + * work, and so does a file an operator routed into a bundle of their own with the SAF + * {@code bundle:} marker. {@code filter.*.publicPermission} is ignored on purpose. */ - public static final List EMBARGOED_BUNDLE_NAMES = Collections.unmodifiableList(Arrays.asList( - Constants.CONTENT_BUNDLE_NAME, TEXT_BUNDLE_NAME, THUMBNAIL_BUNDLE_NAME)); + public static final List NON_EMBARGOED_BUNDLE_NAMES = Collections.unmodifiableList(Arrays.asList( + Constants.LICENSE_BUNDLE_NAME, CreativeCommonsService.CC_BUNDLE_NAME, + Constants.METADATA_BUNDLE_NAME)); + + /** + * Whether an embargo covers the bundle. + * + * @param bundleName name of the bundle + * @return false for the licence and metadata bundles, true for everything that holds the work itself + */ + public static boolean isEmbargoed(String bundleName) { + return !NON_EMBARGOED_BUNDLE_NAMES.contains(bundleName); + } private SafEmbargoConstants() { } diff --git a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java index 4ef48bc8b576..9902f84c92bb 100644 --- a/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java +++ b/dspace-api/src/main/java/org/dspace/app/util/SafEmbargoDateParser.java @@ -56,6 +56,15 @@ public final class SafEmbargoDateParser { .appendValue(ChronoField.DAY_OF_MONTH) .toFormatter().withResolverStyle(ResolverStyle.STRICT); + /** + * {@code yyyy-M} with an unpadded month, which {@code SimpleDateFormat} accepted for the same reason. + */ + private static final DateTimeFormatter UNPADDED_YEAR_MONTH = new DateTimeFormatterBuilder() + .appendValue(ChronoField.YEAR, 4, 10, SignStyle.EXCEEDS_PAD) + .appendLiteral('-') + .appendValue(ChronoField.MONTH_OF_YEAR) + .toFormatter().withResolverStyle(ResolverStyle.STRICT); + /** Listed in operator messages, so that the two tools describe the same set of values. */ public static final String ACCEPTED_FORMATS = "yyyy-MM-dd, yyyy-MM (first of the month), yyyy (1 January) or yyyy-MM-dd'T'HH[:mm[:ss]][Z]"; @@ -104,8 +113,14 @@ public static LocalDate parseEmbargoEndDay(String value) { try { return LocalDate.parse(trimmed, UNPADDED_DATE); } catch (DateTimeParseException notAnUnpaddedDay) { + // ditto + } + + try { + return YearMonth.parse(trimmed, UNPADDED_YEAR_MONTH).atDay(1); + } catch (DateTimeParseException notAnUnpaddedYearMonth) { throw new DateTimeParseException("Unparseable embargo end date '" + value + "', expected " - + ACCEPTED_FORMATS, trimmed, notAnUnpaddedDay.getErrorIndex()); + + ACCEPTED_FORMATS, trimmed, notAnUnpaddedYearMonth.getErrorIndex()); } } } diff --git a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java index 2d9bff42b0f6..a2b32029de3f 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemimport/EmbargoImportIT.java @@ -611,6 +611,30 @@ public void testBlankEmbargoEndIsRefused() throws Exception { assertBrokenEmbargoPackageIsRefused("", "an empty dc.date.embargoend"); } + /** + * Verifies that a package whose payload sits in a bundle of its own is embargoed too. Only the licence + * and metadata bundles stay world readable, as in {@code DefaultEmbargoSetter}; anything else holds the + * work and would otherwise be published by the collection default policies. + */ + @Test + public void testCustomBundlePayloadIsEmbargoed() throws Exception { + Path itemDir = safPackage("embargoedAccess", EMBARGOEND_DATE_FUTURE, "TEST CONTENT CUSTOM BUNDLE", + "SUPPLEMENT"); + + assertNull("a package whose payload is in a custom bundle has to import", runImport(itemDir.getParent())); + + Iterator archived = itemService.findByMetadataField(context, "dc", "title", null, ITEM_TITLE); + assertTrue("the package has to be archived", archived.hasNext()); + List bundles = archived.next().getBundles("SUPPLEMENT"); + assertFalse("fixture precondition: the payload must be in the SUPPLEMENT bundle", bundles.isEmpty()); + + for (Bitstream bitstream : bundles.get(0).getBitstreams()) { + assertFalse("the package says dc.rights.access=embargoedAccess, so a file in a bundle of its own" + + " must not be readable by an anonymous visitor either: " + describe(bitstream), + anonymousCanRead(bitstream)); + } + } + /** * Verifies that a refused package leaves no workspace item behind. The command line path aborts its * context, but the batch import path completes its own in a finally block, so a half built submission @@ -724,6 +748,15 @@ public void testMissingAnonymousGroupIsNotSwallowed() throws Exception { * @return the item directory; its parent is the source directory to hand to the import */ private Path safPackage(String accessRight, String embargoEnd, String content) throws Exception { + return safPackage(accessRight, embargoEnd, content, null); + } + + /** + * The same package with its payload routed into a bundle of its own, as the SAF + * {@code bundle:} marker does. + */ + private Path safPackage(String accessRight, String embargoEnd, String content, String bundleName) + throws Exception { Path safDir = Files.createDirectory(Path.of(tempDir.toString() + "/test")); Path itemDir = Files.createDirectory(Path.of(safDir.toString() + "/item_000")); @@ -742,7 +775,8 @@ private Path safPackage(String accessRight, String embargoEnd, String content) t dublinCore.append(""); Files.writeString(Path.of(itemDir.toString() + "/dublin_core.xml"), dublinCore.toString()); - Files.writeString(Files.createFile(Path.of(itemDir.toString() + "/contents")), "test.txt"); + Files.writeString(Files.createFile(Path.of(itemDir.toString() + "/contents")), + bundleName == null ? "test.txt" : "test.txt\tbundle:" + bundleName); Files.writeString(Files.createFile(Path.of(itemDir.toString() + "/test.txt")), content); return itemDir; } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java index 39d3a05b644e..13f8139cb72c 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDateBoundaryIT.java @@ -411,6 +411,9 @@ public void legacyEmbargoEndShapesKeepTheirDcDateDay() throws Exception { assertLegacyEmbargoEndClosesTheFileUntil(String.valueOf(nextYear), LocalDate.of(nextYear, 1, 1)); // a bare month is the 1st of it, for the same reason assertLegacyEmbargoEndClosesTheFileUntil(nextYear + "-05", LocalDate.of(nextYear, 5, 1)); + // SimpleDateFormat did not require the month to be padded, so packages contain this shape too + assertLegacyEmbargoEndClosesTheFileUntil(nextYear + "-5", LocalDate.of(nextYear, 5, 1)); + assertLegacyEmbargoEndClosesTheFileUntil(nextYear + "-5-9", LocalDate.of(nextYear, 5, 9)); // the shape DSpace exports write; the time of day is dropped, the UTC day is the last closed day assertLegacyEmbargoEndClosesTheFileUntil(tomorrow + "T00:00:00Z", tomorrow); } diff --git a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java index 6dfb51d8807f..12b713be2415 100644 --- a/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java +++ b/dspace-api/src/test/java/org/dspace/app/itemupdate/EmbargoDerivativesIT.java @@ -202,6 +202,45 @@ public void licenseBundleIsNeverTouched() throws Exception { } } + /** + * Verifies that a file an operator routed into a bundle of its own follows the embargo. SAF packages can + * do that with the {@code bundle:} marker, and {@code DefaultEmbargoSetter} covers every bundle but + * the licence and metadata ones, so leaving it public would disclose the embargoed work. + */ + @Test + public void customBundleFollowsTheEmbargo() throws Exception { + LocalDate futureEnd = LocalDate.now().plusMonths(4); + LocalDate pastEnd = LocalDate.now().minusMonths(4); + + Item item = createItem("Thesis with a supplement"); + Bitstream original = createBitstreamInBundle(item, "thesis.pdf", Constants.CONTENT_BUNDLE_NAME); + Bitstream supplement = createBitstreamInBundle(item, "dataset.csv", "SUPPLEMENT"); + + Run embargoRun = runItemUpdate(item, dublinCore(item, EMBARGOED_ACCESS, futureEnd.toString())); + item = context.reloadEntity(item); + original = context.reloadEntity(original); + supplement = context.reloadEntity(supplement); + + assertExitCode("future embargo on an item with a custom bundle", 0, embargoRun); + assertFalse("sanity check: the ORIGINAL bitstream has to be closed" + describe(original), + anonymousCanRead(original)); + assertFalse("a file in a bundle of its own is part of the work and must be closed by the embargo" + + describe(supplement), + anonymousCanRead(supplement)); + + Run expiredRun = runItemUpdate(item, dublinCore(item, OPEN_ACCESS, pastEnd.toString())); + item = context.reloadEntity(item); + original = context.reloadEntity(original); + supplement = context.reloadEntity(supplement); + + assertExitCode("expired embargo on an item with a custom bundle", 0, expiredRun); + assertTrue("sanity check: the expired embargo has to publish the ORIGINAL bitstream" + + describe(original), + anonymousCanRead(original)); + assertTrue("the custom bundle has to open together with the file it belongs to" + describe(supplement), + anonymousCanRead(supplement)); + } + /** * Verifies that a derivative without an {@code Anonymous}/{@code READ} policy does not get one. Creating * it would grant access nobody granted, so the run reports the bitstream instead.