Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:
options: [dev, prod]
default: dev
schedule_enabled:
description: Enable the weekly Friday schedule
description: Enable the recurring schedule (Mon/Wed/Fri)
type: choice
options: ["DISABLED", "ENABLED"]
default: "DISABLED"
Expand Down Expand Up @@ -182,7 +182,7 @@ jobs:
echo "| Stack | \`${STACK_NAME}\` |"
echo "| Image | \`${IMAGE_URI}\` |"
echo "| Database | \`${RDSDB}\` |"
echo "| Weekly schedule | \`${SCHEDULE_ENABLED}\` |"
echo "| Recurring schedule | \`${SCHEDULE_ENABLED}\` |"
echo ""
echo "Deploying does not run the minter. Use the **Run DOI Minting**"
echo "workflow to trigger a run."
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/run-minting.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: Run DOI Minting

# The manual replacement for the Friday terminal session. The weekly run itself
# is fired by EventBridge Scheduler inside AWS; this workflow is for ad hoc runs
# The manual replacement for the Friday terminal session. The scheduled runs
# are fired by EventBridge Scheduler inside AWS; this workflow is for ad hoc runs
# and for verifying the pipeline after a deploy.
on:
workflow_dispatch:
Expand Down Expand Up @@ -110,7 +110,7 @@ jobs:
run: |
set -euo pipefail
# `wait tasks-stopped` polls for up to 10 minutes per call; a large
# weekly batch can run longer, so retry until the task really stops.
# batch can run longer, so retry until the task really stops.
for _ in $(seq 1 9); do
if aws ecs wait tasks-stopped --cluster "$CLUSTER" --tasks "$TASK_ARN" \
--region "$AWS_REGION" 2>/dev/null; then
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ sequenceDiagram
participant NeotomaDB@{ "type" : "database" }
participant DataCite@{"type":"boundary"}

Note left of Batch: Trigger to run weekly using Fargate
Note left of Batch: Trigger to run Mon/Wed/Fri using Fargate
Batch->>neotomaPyDOI: Trigger a run (cron)
neotomaPyDOI->>NeotomaDB: Check for new datasets without DOIs
NeotomaDB->>neotomaPyDOI: Return datasetids
Expand Down Expand Up @@ -85,7 +85,7 @@ uv run ndbdoi.py --tank
## Automated Minting

The manual Friday routine — a sandbox pass followed by a production mint — runs
itself on AWS. The `infrastructure/doi-minter.yaml` CloudFormation stack builds
itself on AWS, on Mondays, Wednesdays and Fridays. The `infrastructure/doi-minter.yaml` CloudFormation stack builds
out the architecture described in the sequence diagram above.

```
Expand All @@ -94,7 +94,7 @@ GitHub Actions (deploy.yml) AWS
write credentials ────────────────► Secrets Manager
deploy stack ─────────────────────► CloudFormation
├─ EventBridge Scheduler ── cron(0 14 ? * FRI *)
├─ EventBridge Scheduler ── cron(0 0 ? * MON,WED,FRI *)
│ │
│ ▼
├─ ECS Fargate task (private subnets)
Expand Down Expand Up @@ -385,10 +385,10 @@ submitted less than two days ago should stay a manual act from a workstation.

### Changing the schedule

`ScheduleExpression` defaults to `cron(0 14 ? * FRI *)` — Fridays at 14:00 UTC,
which is 09:00 US Central in summer and 08:00 in winter. EventBridge Scheduler
supports `ScheduleExpressionTimezone`, so setting it to `America/Chicago` would
hold a constant local time year-round.
`ScheduleExpression` defaults to `cron(0 0 ? * MON,WED,FRI *)`, read in the
`ScheduleTimezone` parameter which defaults to `Los Angeles`; ie midnight Pacific on Mondays, Wednesdays and Fridays.
Comment on lines +388 to +389

Running three times a week shortens how long a new dataset waits for its DOI, keeping the 48 hours window for datasets to settle.

## Neotoma DOI Metadata

Expand Down
2 changes: 1 addition & 1 deletion entrypoint.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
#
# The weekly Neotoma DOI minting run, as previously performed by hand:
# The Neotoma DOI minting run, as previously performed by hand each Friday:
#
# uv run ndbdoi.py # DataCite sandbox pass, used here as a gate
# uv run ndbdoi.py -m # the real mint against api.datacite.org
Expand Down
24 changes: 17 additions & 7 deletions infrastructure/doi-minter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ AWSTemplateFormatVersion: "2010-09-09"
Description: >
Scheduled Neotoma DOI minting. Runs neotomapydoi as an ECS Fargate task inside
the VPC (so it can reach the Neotoma RDS instance on its private subnets),
triggered weekly by EventBridge Scheduler. Logs go to CloudWatch and S3;
triggered on a schedule by EventBridge Scheduler. Logs go to CloudWatch and S3;
a non-zero task exit publishes to an SNS topic that emails a human.

Parameters:
Expand Down Expand Up @@ -47,18 +47,28 @@ Parameters:

ScheduleExpression:
Type: String
Default: cron(0 14 ? * FRI *)
Default: cron(0 0 ? * MON,WED,FRI *)
Description: >
When to run, in UTC. The default is Fridays at 14:00 UTC, which is 09:00
US Central during daylight saving and 08:00 outside it.
When to run, interpreted in ScheduleTimezone. The default is midnight on
Mondays, Wednesdays and Fridays. Note the selection query only picks up
datasets more than two days old, so running more often shortens the wait
before a new dataset is minted rather than minting anything sooner.

ScheduleTimezone:
Type: String
Default: America/Los_Angeles
Description: >
IANA timezone the schedule is read in. Using a named zone rather than UTC
keeps the local run time fixed across daylight saving changes, instead of
drifting by an hour twice a year.

ScheduleEnabled:
Type: String
Default: "DISABLED"
AllowedValues: ["ENABLED", "DISABLED"]
Description: >
Leave DISABLED until a manual run has been verified end to end, then
redeploy with ENABLED to hand the Friday job over to the scheduler.
redeploy with ENABLED to hand the recurring job over to the scheduler.

Resources:

Expand Down Expand Up @@ -245,10 +255,10 @@ Resources:
Type: AWS::Scheduler::Schedule
Properties:
Name: !Sub "neotoma-doi-minting-${Environment}"
Description: Weekly Neotoma DOI minting run.
Description: Neotoma DOI minting run (midnight, Mondays/Wednesdays/Fridays).
State: !Ref ScheduleEnabled
ScheduleExpression: !Ref ScheduleExpression
ScheduleExpressionTimezone: UTC
ScheduleExpressionTimezone: !Ref ScheduleTimezone
FlexibleTimeWindow:
Mode: "OFF"
Target:
Expand Down