Skip to content

Commit ea459df

Browse files
authored
Add scheduled tasks feature and delayed entity signals (#160)
* Add scheduled tasks feature and delayed entity signals Add a recurring schedule feature (durabletask.scheduled) for parity with durabletask-dotnet, built on durable entities and a helper orchestrator. Enable it on a worker with configure_scheduled_tasks(worker) and manage schedules from the client via ScheduledTaskClient / ScheduleClient (create, describe, list, update, pause, resume, delete). To support the schedule entity's self-rearming, add an optional signal_time parameter to entity, orchestration-context, and client signal_entity methods, and make the in-memory backend honor delayed (scheduled) entity signals. Includes unit tests, in-memory E2E tests, and live DTS E2E tests for both the schedule feature and delayed signals, plus an example and changelog entries. * Let to_json hook take precedence over dataclass asdict in serializer The JSON serializer encoded dataclasses via dataclasses.asdict before ever checking for a to_json hook, so a dataclass could not override its own serialization -- a problem for dataclasses whose fields are not JSON-native (e.g. timedelta/datetime). The read path already consults from_json before its dataclass branch; this makes the write path symmetric by checking the to_json hook first and falling back to asdict/SimpleNamespace. * Use to_json/from_json hooks for schedule options With the serializer now honoring the to_json hook for dataclasses, the schedule option types expose to_json/from_json instead of bespoke to_dict/from_dict. The schedule entity operations can again annotate their input as the option dataclass and let the worker reconstruct it via the from_json hook, removing the manual _coerce_options shim and the Any-typed parameter workaround. * Serialization improvements (WIP) * Revert JSON stuff to base * Revert serialization test * Update scheduled tasks with new serialization fixes * PR Feedback * Address PR feedback: capability flag, datetime filter fix, helper consolidation - Advertise WORKER_CAPABILITY_SCHEDULED_TASKS from configure_scheduled_tasks via a new TaskHubGrpcWorker.add_capability(), mirroring .NET's UseScheduledTasks (currently inert against the DTS backend, which only gates on HistoryStreaming). - Fix naive-vs-aware datetime TypeError in schedule list filtering: ScheduleQuery normalizes its created_from/created_to bounds to aware UTC, and the client filter defensively normalizes the stored timestamp. Bounds remain exclusive to match the .NET ScheduledTasks implementation. - Consolidate the duplicated _ensure_aware helpers into a single public helpers.ensure_aware, removing the cross-module private-usage warning. - Document that list_schedules applies status/created filters client-side, so pages may be underfilled (matches .NET). - Add unit tests for query normalization, exclusive filter bounds, and the scheduled-tasks capability advertisement.
1 parent f6bfd44 commit ea459df

32 files changed

Lines changed: 2410 additions & 26 deletions

CHANGELOG.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,18 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
99

1010
ADDED
1111

12+
- Added `durabletask.scheduled`, a recurring schedule feature built on durable
13+
entities. Use `configure_scheduled_tasks(worker)` to enable it on a worker,
14+
then manage schedules from the client via `ScheduledTaskClient` (and the
15+
per-schedule `ScheduleClient`). Supports creating, describing, listing,
16+
updating, pausing, resuming, and deleting schedules with configurable
17+
`interval`, `start_at`, `end_at`, and `start_immediately_if_late` options.
18+
- Added an optional `signal_time` parameter to `EntityContext.signal_entity`
19+
and `DurableEntity.signal_entity`, allowing an entity signal to be scheduled
20+
for future delivery.
21+
- Added an optional `signal_time` parameter to `OrchestrationContext.signal_entity`
22+
and to the client `signal_entity` methods (sync and async), allowing entity
23+
signals to be scheduled for future delivery from orchestrations and clients.
1224
- Added a pluggable `DataConverter` (`durabletask.serialization`) accepted by
1325
`TaskHubGrpcWorker`, `TaskHubGrpcClient`, and `AsyncTaskHubGrpcClient` via a
1426
`data_converter` argument. Every payload boundary (inputs, outputs, events,

durabletask/client.py

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -825,8 +825,10 @@ def purge_orchestrations_by(self,
825825
def signal_entity(self,
826826
entity_instance_id: EntityInstanceId,
827827
operation_name: str,
828-
input: Any | None = None) -> None:
829-
req = build_signal_entity_req(entity_instance_id, operation_name, input, self._data_converter)
828+
input: Any | None = None,
829+
signal_time: datetime | None = None) -> None:
830+
req = build_signal_entity_req(
831+
entity_instance_id, operation_name, input, signal_time, self._data_converter)
830832
self._logger.info(f"Signaling entity '{entity_instance_id}' operation '{operation_name}'.")
831833
if self._payload_store is not None:
832834
payload_helpers.externalize_payloads(
@@ -1308,8 +1310,10 @@ async def purge_orchestrations_by(self,
13081310
async def signal_entity(self,
13091311
entity_instance_id: EntityInstanceId,
13101312
operation_name: str,
1311-
input: Any | None = None) -> None:
1312-
req = build_signal_entity_req(entity_instance_id, operation_name, input, self._data_converter)
1313+
input: Any | None = None,
1314+
signal_time: datetime | None = None) -> None:
1315+
req = build_signal_entity_req(
1316+
entity_instance_id, operation_name, input, signal_time, self._data_converter)
13131317
self._logger.info(f"Signaling entity '{entity_instance_id}' operation '{operation_name}'.")
13141318
if self._payload_store is not None:
13151319
await payload_helpers.externalize_payloads_async(

durabletask/entities/durable_entity.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
# Licensed under the MIT License.
33

44
from typing import Any, TypeVar, overload
5+
from datetime import datetime
56

67
from durabletask.entities.entity_context import EntityContext
78
from durabletask.entities.entity_instance_id import EntityInstanceId
@@ -52,7 +53,9 @@ def set_state(self, state: Any):
5253
"""
5354
self.entity_context.set_state(state)
5455

55-
def signal_entity(self, entity_instance_id: EntityInstanceId, operation: str, input: Any | None = None) -> None:
56+
def signal_entity(self, entity_instance_id: EntityInstanceId, operation: str,
57+
input: Any | None = None,
58+
signal_time: datetime | None = None) -> None:
5659
"""Signal another entity to perform an operation.
5760
5861
Parameters
@@ -63,8 +66,12 @@ def signal_entity(self, entity_instance_id: EntityInstanceId, operation: str, in
6366
The operation to perform on the entity.
6467
input : Any, optional
6568
The input to provide to the entity for the operation.
69+
signal_time : datetime, optional
70+
The time at which the signal should be delivered. If None, the signal is
71+
delivered as soon as possible. Use this to schedule a future operation,
72+
for example to have an entity wake itself up at a later time.
6673
"""
67-
self.entity_context.signal_entity(entity_instance_id, operation, input)
74+
self.entity_context.signal_entity(entity_instance_id, operation, input, signal_time)
6875

6976
def schedule_new_orchestration(self, orchestration_name: str, input: Any | None = None, instance_id: str | None = None) -> str:
7077
"""Schedule a new orchestration instance.

durabletask/entities/entity_context.py

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
# Copyright (c) Microsoft Corporation.
22
# Licensed under the MIT License.
33

4+
from datetime import datetime
45
from typing import TYPE_CHECKING, Any, TypeVar, overload
56
import uuid
7+
from google.protobuf import timestamp_pb2
68
from durabletask.entities.entity_instance_id import EntityInstanceId
79
from durabletask.internal import helpers
810
from durabletask.internal.entity_state_shim import StateShim
@@ -88,7 +90,9 @@ def set_state(self, new_state: Any) -> None:
8890
"""
8991
self._state.set_state(new_state)
9092

91-
def signal_entity(self, entity_instance_id: EntityInstanceId, operation: str, input: Any | None = None) -> None:
93+
def signal_entity(self, entity_instance_id: EntityInstanceId, operation: str,
94+
input: Any | None = None,
95+
signal_time: datetime | None = None) -> None:
9296
"""Signal another entity to perform an operation.
9397
9498
Parameters
@@ -99,15 +103,23 @@ def signal_entity(self, entity_instance_id: EntityInstanceId, operation: str, in
99103
The operation to perform on the entity.
100104
input : Any, optional
101105
The input to provide to the entity for the operation.
106+
signal_time : datetime, optional
107+
The time at which the signal should be delivered. If None, the signal is
108+
delivered as soon as possible. Use this to schedule a future operation,
109+
for example to have an entity wake itself up at a later time.
102110
"""
103111
encoded_input: str | None = self._data_converter.serialize(input)
112+
scheduled_time: timestamp_pb2.Timestamp | None = None
113+
if signal_time is not None:
114+
scheduled_time = timestamp_pb2.Timestamp()
115+
scheduled_time.FromDatetime(signal_time)
104116
self._state.add_operation_action(
105117
pb.OperationAction(
106118
sendSignal=pb.SendSignalAction(
107119
instanceId=str(entity_instance_id),
108120
name=operation,
109121
input=helpers.get_string_value(encoded_input),
110-
scheduledTime=None,
122+
scheduledTime=scheduled_time,
111123
requestTime=None,
112124
parentTraceContext=None,
113125
)

durabletask/internal/client_helpers.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,14 +205,16 @@ def build_signal_entity_req(
205205
entity_instance_id: EntityInstanceId,
206206
operation_name: str,
207207
input: Any | None = None,
208+
signal_time: datetime | None = None,
208209
data_converter: DataConverter | None = None) -> pb.SignalEntityRequest:
209210
"""Build a SignalEntityRequest for signaling an entity."""
211+
scheduled_time = helpers.new_timestamp(signal_time) if signal_time is not None else None
210212
return pb.SignalEntityRequest(
211213
instanceId=str(entity_instance_id),
212214
name=operation_name,
213215
input=helpers.get_string_value(_serialize(input, data_converter)),
214216
requestId=str(uuid.uuid4()),
215-
scheduledTime=None,
217+
scheduledTime=scheduled_time,
216218
parentTraceContext=None,
217219
requestTime=helpers.new_timestamp(datetime.now(timezone.utc))
218220
)

durabletask/internal/helpers.py

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Licensed under the MIT License.
33

44
import traceback
5-
from datetime import datetime
5+
from datetime import datetime, timezone
66

77
from google.protobuf import timestamp_pb2, wrappers_pb2
88

@@ -255,11 +255,13 @@ def new_signal_entity_action(id: int,
255255
entity_id: EntityInstanceId,
256256
operation: str,
257257
encoded_input: str | None,
258-
request_id: str) -> pb.OrchestratorAction:
258+
request_id: str,
259+
signal_time: datetime | None = None) -> pb.OrchestratorAction:
260+
scheduled_time = new_timestamp(signal_time) if signal_time is not None else None
259261
return pb.OrchestratorAction(id=id, sendEntityMessage=pb.SendEntityMessageAction(entityOperationSignaled=pb.EntityOperationSignaledEvent(
260262
requestId=request_id,
261263
operation=operation,
262-
scheduledTime=None,
264+
scheduledTime=scheduled_time,
263265
input=get_string_value(encoded_input),
264266
targetInstanceId=get_string_value(str(entity_id)),
265267
)))
@@ -300,6 +302,21 @@ def new_timestamp(dt: datetime) -> timestamp_pb2.Timestamp:
300302
return ts
301303

302304

305+
def ensure_aware(value: datetime | None) -> datetime | None:
306+
"""Return ``value`` as a timezone-aware datetime, assuming UTC when naive.
307+
308+
A naive datetime is tagged as UTC; an already-aware datetime is returned
309+
unchanged. Useful before comparing user-supplied datetimes against the
310+
SDK's always-aware-UTC timestamps to avoid "can't compare offset-naive and
311+
offset-aware datetimes".
312+
"""
313+
if value is None:
314+
return None
315+
if value.tzinfo is None:
316+
return value.replace(tzinfo=timezone.utc)
317+
return value
318+
319+
303320
def new_create_sub_orchestration_action(
304321
id: int,
305322
name: str,

durabletask/scheduled/__init__.py

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Copyright (c) Microsoft Corporation.
2+
# Licensed under the MIT License.
3+
4+
"""Scheduled tasks support for the Durable Task SDK.
5+
6+
This package provides a recurring schedule feature built on top of durable
7+
entities and a helper orchestrator. Register the entity and orchestrator with a
8+
worker via :func:`configure_scheduled_tasks`, then manage schedules from the
9+
client via :class:`ScheduledTaskClient`.
10+
"""
11+
12+
from durabletask.scheduled.client import ScheduleClient, ScheduledTaskClient
13+
from durabletask.scheduled.exceptions import (ScheduleClientValidationError,
14+
ScheduleError,
15+
ScheduleInvalidTransitionError,
16+
ScheduleNotFoundError)
17+
from durabletask.scheduled.models import (ScheduleCreationOptions,
18+
ScheduleDescription, ScheduleQuery,
19+
ScheduleUpdateOptions)
20+
from durabletask.scheduled.registration import configure_scheduled_tasks
21+
from durabletask.scheduled.schedule_status import ScheduleStatus
22+
23+
__all__ = [
24+
"ScheduledTaskClient",
25+
"ScheduleClient",
26+
"ScheduleCreationOptions",
27+
"ScheduleUpdateOptions",
28+
"ScheduleDescription",
29+
"ScheduleQuery",
30+
"ScheduleStatus",
31+
"ScheduleError",
32+
"ScheduleNotFoundError",
33+
"ScheduleClientValidationError",
34+
"ScheduleInvalidTransitionError",
35+
"configure_scheduled_tasks",
36+
]
37+
38+
PACKAGE_NAME = "durabletask.scheduled"

durabletask/scheduled/client.py

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Copyright (c) Microsoft Corporation.
2+
# Licensed under the MIT License.
3+
4+
import logging
5+
6+
from durabletask.client import (EntityQuery, OrchestrationStatus,
7+
TaskHubGrpcClient)
8+
from durabletask.entities import EntityInstanceId
9+
from durabletask.internal.helpers import ensure_aware
10+
from durabletask.scheduled import transitions
11+
from durabletask.scheduled.exceptions import ScheduleNotFoundError
12+
from durabletask.scheduled.models import (ScheduleCreationOptions,
13+
ScheduleDescription, ScheduleQuery,
14+
ScheduleState, ScheduleUpdateOptions)
15+
from durabletask.scheduled.orchestrator import (
16+
ScheduleOperationRequest, execute_schedule_operation_orchestrator)
17+
from durabletask.scheduled.schedule_entity import (DELETE_OPERATION,
18+
ENTITY_NAME)
19+
20+
logger = logging.getLogger("durabletask.scheduled")
21+
22+
23+
class ScheduleClient:
24+
"""Client for managing a single schedule instance."""
25+
26+
def __init__(self, client: TaskHubGrpcClient, schedule_id: str,
27+
*, operation_timeout: float = 60):
28+
if not schedule_id:
29+
raise ValueError("schedule_id cannot be empty.")
30+
self._client = client
31+
self._schedule_id = schedule_id
32+
self._entity_id = EntityInstanceId(ENTITY_NAME, schedule_id)
33+
self._operation_timeout = operation_timeout
34+
35+
@property
36+
def schedule_id(self) -> str:
37+
"""Gets the ID of this schedule."""
38+
return self._schedule_id
39+
40+
def _run_operation(self, operation_name: str, input: object | None = None) -> None:
41+
request = ScheduleOperationRequest(
42+
entity_id=str(self._entity_id),
43+
operation_name=operation_name,
44+
input=input,
45+
)
46+
instance_id = self._client.schedule_new_orchestration(
47+
execute_schedule_operation_orchestrator, input=request)
48+
state = self._client.wait_for_orchestration_completion(
49+
instance_id, timeout=self._operation_timeout)
50+
if state is None or state.runtime_status != OrchestrationStatus.COMPLETED:
51+
failure = state.failure_details if state else None
52+
message = failure.message if failure else "unknown error"
53+
raise RuntimeError(
54+
f"Failed to '{operation_name}' schedule '{self._schedule_id}': {message}")
55+
56+
def create(self, options: ScheduleCreationOptions) -> None:
57+
"""Create or update this schedule with the given configuration."""
58+
self._run_operation(transitions.CREATE_SCHEDULE, options)
59+
60+
def update(self, options: ScheduleUpdateOptions) -> None:
61+
"""Update this schedule's configuration."""
62+
self._run_operation(transitions.UPDATE_SCHEDULE, options)
63+
64+
def pause(self) -> None:
65+
"""Pause this schedule."""
66+
self._run_operation(transitions.PAUSE_SCHEDULE)
67+
68+
def resume(self) -> None:
69+
"""Resume this schedule."""
70+
self._run_operation(transitions.RESUME_SCHEDULE)
71+
72+
def delete(self) -> None:
73+
"""Delete this schedule."""
74+
self._run_operation(DELETE_OPERATION)
75+
76+
def describe(self) -> ScheduleDescription:
77+
"""Retrieve the current details of this schedule."""
78+
metadata = self._client.get_entity(self._entity_id, include_state=True)
79+
if metadata is None:
80+
raise ScheduleNotFoundError(self._schedule_id)
81+
state = metadata.get_typed_state(ScheduleState)
82+
if state is None:
83+
raise ScheduleNotFoundError(self._schedule_id)
84+
return state.to_description()
85+
86+
87+
class ScheduledTaskClient:
88+
"""Client for managing scheduled tasks in a Durable Task application."""
89+
90+
def __init__(self, client: TaskHubGrpcClient, *, operation_timeout: float = 60):
91+
self._client = client
92+
self._operation_timeout = operation_timeout
93+
94+
def get_schedule_client(self, schedule_id: str) -> ScheduleClient:
95+
"""Get a handle to manage a specific schedule."""
96+
return ScheduleClient(self._client, schedule_id,
97+
operation_timeout=self._operation_timeout)
98+
99+
def create_schedule(self, options: ScheduleCreationOptions) -> ScheduleClient:
100+
"""Create a new schedule and return a client for managing it."""
101+
schedule_client = self.get_schedule_client(options.schedule_id)
102+
schedule_client.create(options)
103+
return schedule_client
104+
105+
def get_schedule(self, schedule_id: str) -> ScheduleDescription | None:
106+
"""Get a schedule description by ID, or None if it does not exist."""
107+
try:
108+
return self.get_schedule_client(schedule_id).describe()
109+
except ScheduleNotFoundError:
110+
return None
111+
112+
def list_schedules(self, schedule_query: ScheduleQuery | None = None) -> list[ScheduleDescription]:
113+
"""List schedules matching the given filter criteria.
114+
115+
> [!NOTE]
116+
> The ``status`` and ``created_from``/``created_to`` filters are applied
117+
> client-side after each page of entities is fetched, so an individual
118+
> page may contain fewer than ``page_size`` matches (or none) even when
119+
> more matching schedules exist. This mirrors the .NET implementation.
120+
"""
121+
prefix = schedule_query.schedule_id_prefix if schedule_query and schedule_query.schedule_id_prefix else ""
122+
page_size = (schedule_query.page_size if schedule_query and schedule_query.page_size
123+
else ScheduleQuery.DEFAULT_PAGE_SIZE)
124+
query = EntityQuery(
125+
instance_id_starts_with=f"@{ENTITY_NAME}@{prefix}",
126+
include_state=True,
127+
page_size=page_size,
128+
)
129+
results: list[ScheduleDescription] = []
130+
for metadata in self._client.get_all_entities(query):
131+
state = metadata.get_typed_state(ScheduleState)
132+
if state is None or state.schedule_configuration is None:
133+
continue
134+
if not self._matches_filter(state, schedule_query):
135+
continue
136+
results.append(state.to_description())
137+
return results
138+
139+
@staticmethod
140+
def _matches_filter(state: ScheduleState, schedule_query: ScheduleQuery | None) -> bool:
141+
if schedule_query is None:
142+
return True
143+
if schedule_query.status is not None and state.status != schedule_query.status:
144+
return False
145+
# ``ScheduleQuery`` normalizes its bounds to aware UTC; defensively
146+
# normalize the stored timestamp too (a payload could in principle carry
147+
# a naive value) so the comparison can never raise on naive-vs-aware.
148+
# Bounds are exclusive, matching the .NET ScheduledTasks implementation.
149+
created_at = ensure_aware(state.schedule_created_at)
150+
if schedule_query.created_from is not None and not (created_at and created_at > schedule_query.created_from):
151+
return False
152+
if schedule_query.created_to is not None and not (created_at and created_at < schedule_query.created_to):
153+
return False
154+
return True

0 commit comments

Comments
 (0)