Time selection for web downloads by klpoland · Pull Request #263 · spectrumx/sds-code

klpoland · 2026-03-01T21:59:01Z

Select times for web download:

Add time selection helper functions
Integrate slider JS package for range slider UI
Dynamic labels to indicate number of files and time start and end

semanticdiff-com · 2026-03-01T21:59:04Z

Review changes with

Changed Files

File	Status
gateway/sds_gateway/users/tests/test_drf_views.py	38% smaller
gateway/sds_gateway/api_methods/tasks.py	27% smaller
gateway/sds_gateway/api_methods/serializers/capture_serializers.py	7% smaller
gateway/sds_gateway/static/js/file-list.js	6% smaller
gateway/sds_gateway/templates/users/file_list.html	2% smaller
gateway/sds_gateway/static/js/actions/DownloadActionManager.js	1% smaller
gateway/sds_gateway/templates/users/partials/web_download_modal.html	1% smaller
gateway/sds_gateway/api_methods/helpers/temporal_filtering.py	0% smaller
gateway/sds_gateway/api_methods/tests/factories.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_celery_tasks.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_temporal_filtering.py	0% smaller
gateway/sds_gateway/static/js/actions/__tests__/DownloadActionManager.test.js	0% smaller
gateway/sds_gateway/templates/base.html	0% smaller
gateway/sds_gateway/templates/users/dataset_list.html	0% smaller
gateway/sds_gateway/templates/users/partials/captures_page_table.html	Unsupported file format
gateway/sds_gateway/templates/users/published_datasets_list.html	0% smaller
gateway/sds_gateway/users/views.py	0% smaller

gateway/sds_gateway/api_methods/helpers/temporal_filtering.py

Copilot

Pull request overview

Adds temporal (time-range) selection to web downloads—primarily for DigitalRF capture downloads—by introducing temporal filtering helpers, exposing timing metadata on capture API responses, and wiring a noUiSlider-based UI into the existing web download modal flow.

Changes:

Add temporal filtering backend plumbing (helper + task/view parameters) to select a subset of capture RF data files by time bounds.
Extend capture serializers to provide duration/cadence/bounds metadata needed to drive the slider UI.
Integrate noUiSlider and update templates/JS to show a time-range slider and dynamic file/size labels for capture downloads.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
gateway/sds_gateway/users/views.py	Accepts `start_time`/`end_time` POST params and forwards them to the Celery download task.
gateway/sds_gateway/api_methods/tasks.py	Threads `start_time`/`end_time` into file collection and adds DigitalRF-only temporal filtering for capture downloads.
gateway/sds_gateway/api_methods/helpers/temporal_filtering.py	New helper utilities for DigitalRF filename parsing, OpenSearch bounds lookup, cadence estimation, and time-bounded file selection.
gateway/sds_gateway/api_methods/serializers/capture_serializers.py	Adds new timing/size/count fields used by the capture download slider UI.
gateway/sds_gateway/templates/base.html	Loads noUiSlider CSS/JS from CDN.
gateway/sds_gateway/templates/users/partials/web_download_modal.html	Adds capture-only slider UI + client-side formatting and passes time bounds in the download request.
gateway/sds_gateway/static/js/actions/DownloadActionManager.js	Initializes the slider from capture button data attributes and opens the web download modal for captures.
gateway/sds_gateway/static/js/file-list.js	Adds capture timing/size/count metadata into capture-row download button attributes (dynamic table).
gateway/sds_gateway/templates/users/partials/captures_page_table.html	Adds capture timing/size/count metadata into capture-row download button attributes (server-rendered table).
gateway/sds_gateway/templates/users/file_list.html	Includes the web download modal partial for capture downloads.
gateway/sds_gateway/templates/users/dataset_list.html	Includes the web download modal partial with `item_type="dataset"`.
gateway/sds_gateway/templates/users/published_datasets_list.html	Includes the web download modal partial with `item_type="dataset"`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

gateway/sds_gateway/api_methods/serializers/capture_serializers.py

gateway/sds_gateway/api_methods/helpers/temporal_filtering.py

gateway/sds_gateway/users/views.py

gateway/sds_gateway/static/js/actions/DownloadActionManager.js

gateway/sds_gateway/api_methods/tasks.py

gateway/sds_gateway/templates/users/partials/web_download_modal.html

Copilot · 2026-03-02T16:44:20Z

gateway/sds_gateway/static/js/actions/DownloadActionManager.js

+		// Initialize temporal slider from button data attributes (clears or builds slider)
+		const durationMs = parseInt(button.getAttribute("data-length-of-capture-ms"), 10);
+		const fileCadenceMs = parseInt(button.getAttribute("data-file-cadence-ms"), 10);
+		const perDataFileSize = parseFloat(button.getAttribute("data-per-data-file-size"), 10);


parseFloat does not take a radix argument; the second parameter is ignored. This is misleading and suggests base conversion that won't happen. Use parseFloat(value) (or Number(...)) consistently here.

Suggested change

const perDataFileSize = parseFloat(button.getAttribute("data-per-data-file-size"), 10);

const perDataFileSize = parseFloat(button.getAttribute("data-per-data-file-size"));

gateway/sds_gateway/static/js/file-list.js

…r file, catch potential errors

lucaspar · 2026-03-06T21:43:37Z

gateway/sds_gateway/api_methods/tests/factories.py

+    """
+
+    uuid = Faker("uuid4")
+    directory = f"/files/{self.owner.email}/{self.capture.top_level_dir}/"


self is not defined here, this needs to be in a method, or wrapped in a LazyAttribute

lucaspar · 2026-03-06T22:08:05Z

gateway/sds_gateway/api_methods/tests/factories.py

+    channel = Faker("word")
+    capture_type = "drf"
+    top_level_dir = Faker("file_path", depth=2).replace("/", "_")
+    owner = Faker("subfactory", factory=UserFactory)
+    name = Faker("slug")


check if these attributes work

$ uv run python -c 'from factory import Faker; print(Faker("file_path", depth=2).replace("/", "_"))' Traceback (most recent call last): File "<string>", line 1, in <module> from factory import Faker; print(Faker("file_path", depth=2).replace("/", "_")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'Faker' object has no attribute 'replace'

lucaspar · 2026-03-06T22:18:43Z

gateway/sds_gateway/api_methods/serializers/capture_serializers.py

    @extend_schema_field(serializers.IntegerField)
-    def get_files_count(self, obj: dict[str, Any]) -> int:
+    def get_data_files_count(self, obj: dict[str, Any]) -> int | None:
        """Get the total count of files across all channels."""
+        if obj["capture_type"] != CaptureType.DigitalRF:
+            return None
+
        total_count = 0
        for channel_data in obj["channels"]:
            capture_uuid = channel_data["uuid"]
            capture = Capture.objects.get(uuid=capture_uuid)
-            total_count += get_capture_files(capture, include_deleted=False).count()
+            total_count += get_data_files(capture.capture_type, capture).count()
        return total_count

+    @extend_schema_field(serializers.IntegerField)
+    def get_data_files_total_size(self, obj: dict[str, Any]) -> int | None:
+        """Exact sum of data file sizes across all channels."""
+        if obj["capture_type"] != CaptureType.DigitalRF:
+            return None
+        total = 0
+        for channel_data in obj["channels"]:
+            capture_uuid = channel_data["uuid"]
+            capture = Capture.objects.get(uuid=capture_uuid)
+            data_files = get_data_files(capture.capture_type, capture)
+            result = data_files.aggregate(total_size=Sum("size"))
+            total += result.get("total_size") or 0
+        return total
+


we're doing some rework here, maybe there's a way to return both count and size in one pass for both serialized fields

lucaspar · 2026-03-06T22:54:06Z

gateway/sds_gateway/api_methods/helpers/temporal_filtering.py

+
+def get_data_files(capture_type: CaptureType, capture: Capture) -> QuerySet[File]:
+    """Get the data files in the capture."""
+    _catch_capture_type_error(capture_type)
+
+    return get_capture_files(capture).filter(name__regex=DRF_RF_FILENAME_REGEX_STR)


This is a hot path involving a DB call: it's called once per channel, then for each capture being serialized, and from time range computations, file cadence, and capture bounds.

See if we can refactor this call stack first, then cache results (lru_cache if a function, or django.utils.functional.cached_property if you make it a method of Capture).

The refactoring suggestion is because this and other functions in this file might make more sense as properties or methods of a Capture instance.

lucaspar · 2026-03-06T23:14:13Z

gateway/sds_gateway/api_methods/serializers/capture_serializers.py

+        data_files = get_data_files(capture.capture_type, capture)
+
+        if data_files.count() == 0:  
+            return None  
+
+        data_file_sizes = data_files.aggregate(total_size=Sum("size"))
+        total_size = data_file_sizes.get("total_size")  
+
+        if not total_size:  
+            return None  

+        return float(total_size) / data_files.count()
+


calling 3 queries (2x count + 1x agg); see if a single query works here

stats = data_files.aggregate( total_size=Sum("size"), count=Count("id") ) if stats["count"] == 0: return None return float(stats["total_size"]) / stats["count"]

lucaspar · 2026-03-06T23:18:58Z

gateway/sds_gateway/api_methods/serializers/capture_serializers.py

+    length_of_capture_ms = serializers.SerializerMethodField()
+    file_cadence_ms = serializers.SerializerMethodField()
+    capture_start_epoch_sec = serializers.SerializerMethodField()
+    data_files_count = serializers.SerializerMethodField()
+    data_files_total_size = serializers.SerializerMethodField()
+    per_data_file_size = serializers.SerializerMethodField()


change gateway/sds_gateway/static/js/components.js after this files_count removal

lucaspar · 2026-03-06T23:22:53Z

gateway/sds_gateway/api_methods/helpers/temporal_filtering.py

+    try:
+        start_time, end_time = get_capture_bounds(capture_type, capture_uuid)
+    except ValueError as e:
+        log.error(e)
+        raise e


try-except not needed, just let it raise, or handle it

lucaspar · 2026-03-06T23:25:57Z

gateway/sds_gateway/static/js/actions/DownloadActionManager.js

Ask the agent to replace all vars with consts (preferred, easier to reason about) and lets when they need to be mutable.

Also, see if it can break down these long functions here and add tests.

lucaspar · 2026-03-06T23:29:41Z

gateway/sds_gateway/static/js/actions/DownloadActionManager.js

+function formatBytes(bytes) {
+	const n = Number(bytes);
+	if (!Number.isFinite(n) || n < 0) return "0 bytes";
+	if (n === 0) return "0 bytes";
+	const units = ["bytes", "KB", "MB", "GB"];
+	let i = 0;
+	let v = n;
+	while (v >= 1024 && i < units.length - 1) {
+		v /= 1024;
+		i++;
+	}
+	return (i === 0 ? v : v.toFixed(2)) + " " + units[i];
+}
+


we have 4 definitions of this in multiple js files, let's start reusing it

lucaspar · 2026-03-06T23:35:18Z

gateway/sds_gateway/static/js/file-list.js

+	 * Use web download modal (with temporal slider) when DownloadActionManager is available.
+	 */
+	handleDownloadCapture(button) {
+		if (window.currentDownloadManager && document.getElementById("webDownloadModal")) {


This silently fails if the element is not there.

We can reverse logic, raise a pretty "danger" alert to user (or at least a console.err) if element is not present and return.

klpoland added 3 commits February 20, 2026 15:53

adding time based filtering and slider ui

bce8979

fix label updates

4a29f2f

fix backend issues, enhance filter labels

92d2e6b

klpoland requested a review from lucaspar March 1, 2026 21:59

klpoland self-assigned this Mar 1, 2026

klpoland added feature New feature or request gateway Gateway component javascript Pull requests that update non-trivial javascript code labels Mar 1, 2026

add start and end time inputs

6d6f479

lucaspar requested changes Mar 2, 2026

View reviewed changes

lucaspar requested a review from Copilot March 2, 2026 16:38

Copilot started reviewing on behalf of lucaspar March 2, 2026 16:38 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

klpoland added 2 commits March 6, 2026 08:16

add tests, generalize download modal, move js from template to manage…

01182c1

…r file, catch potential errors

add flatpickr to handle datetime selection for better control

7a9a87b

lucaspar requested changes Mar 6, 2026

View reviewed changes

	const perDataFileSize = parseFloat(button.getAttribute("data-per-data-file-size"), 10);
	const perDataFileSize = parseFloat(button.getAttribute("data-per-data-file-size"));

Conversation

klpoland commented Mar 1, 2026

Uh oh!

semanticdiff-com bot commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucaspar Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

semanticdiff-com bot commented Mar 1, 2026 •

edited

Loading

lucaspar Mar 6, 2026 •

edited

Loading