feat(backend): parse and fetch single-commit metadata (#2090)

.env.backend.example

@@ -25,6 +25,8 @@ PROMETHEUS_MULTIPROC_DIR=/tmp/metrics
 INGESTER_METRICS_PORT=8002
 
 BACKEND_VOLUME_DIR=/volume_data
+# Ephemeral git clones for SHA-only commit fetch (#2090). Prefer tmpfs.
+GIT_SCRATCH_DIR=/dev/shm/kernelci-git-scratch
 
 ## Variables used for the notifications command. Check docs/notifications.md
 # EMAIL_HOST_USER="youruser@host" # (optional)

.env.example

@@ -106,6 +106,8 @@ HEALTHCHECK_ID_NOTIFICATIONS_SUMMARY_MAESTRO=
 # Backend Volume
 # -----------------------------------------------------------------------------
 BACKEND_VOLUME_DIR=/volume_data
+# Ephemeral git clones for SHA-only commit fetch (#2090). Prefer tmpfs.
+GIT_SCRATCH_DIR=/dev/shm/kernelci-git-scratch
 
 # -----------------------------------------------------------------------------
 # Ingester (only needed with --profile=with_commands)

backend/Dockerfile

@@ -19,6 +19,7 @@ RUN apk update \
         libpq-dev \
         postgresql \
         curl \
+        git \
     && python3 -m venv $POETRY_HOME \
     && $POETRY_HOME/bin/pip install poetry~=2.4.0 \
     && ln -s $POETRY_HOME/bin/poetry /bin/poetry

backend/docs/update_db command.md

@@ -1,20 +1,20 @@
 # update_db Command Documentation
 
-The `update_db` command migrates data from the default database (kcidb) to the dashboard_db database within a specified time interval. All tables are updated by default, but you can select a specific one as well.
+The `update_db` command snapshots dashboard tables to a `.tar.gz` and restores them. Most tables are limited to `--start-interval` / `--end-interval`. `commits` and `commit_parents` are copied in full for now (no time filter), so git ancestry is not cut when a parent has no checkout in the window. A later change may slice those tables.
 
 The migration preserves foreign key constraints. For example, if a test A references a build B in kcidb, but the build B doesn't exist in dashboard_db, then the test A will not be inserted in dashboard_db.
 
 ## Parameters
 
 ### Required Parameters
 
-- `--start-interval`: Start interval for filtering data (format: 'x days' or 'x hours'). The format follows the SQL filtering format.
-- `--end-interval`: End interval for filtering data (format: 'x days' or 'x hours'). The format follows the SQL filtering format.
+- `--start-interval`: Start interval for filtering data (format: 'x days' or 'x hours'). The format follows the SQL filtering format. Does not apply to `commits` or `commit_parents`.
+- `--end-interval`: End interval for filtering data (format: 'x days' or 'x hours'). The format follows the SQL filtering format. Does not apply to `commits` or `commit_parents`.
 
 ### Optional Parameters
 
 - `--table`: Limit data copy to a specific table
-  - Valid options: `issues`, `checkouts`, `builds`, `tests`, `incidents`
+  - Valid options: `issues`, `checkouts`, `commits`, `commit_parents`, `builds`, `tests`, `incidents`, `latest_checkout`, `hardware_status`, `tree_listing`, `tree_tests_rollup`
   - If not provided, data from all tables will be copied
 - `--related-data-only`: Limits the selected data to data where the foreign key constraint is not broken.
   - Default: False.
@@ -35,7 +35,7 @@ python manage.py update_db --start-interval "1 days" --end-interval "0 days" --t
 
 ## Migration Process
 
-1. **Data Selection**: Selects records from the default database within the specified time range
+1. **Data Selection**: Selects records from the default database within the specified time range, except `commits` and `commit_parents` (full table)
 2. **Relationship Validation**: Ensures foreign key constraints are maintained
 3. **Data Insertion**: Inserts valid data into the dashboard db
 4. **Conflict Resolution**: Uses `ignore_conflicts=True` to handle duplicate records
@@ -44,6 +44,8 @@ python manage.py update_db --start-interval "1 days" --end-interval "0 days" --t
 
 - Migration preserves JSON fields by parsing them appropriately
 - The skipped rows count are related to rows which didn't have relationships in dashboard_db. The processed rows count are related to the remaining rows that were selected but not skipped (even if they were inserted or had a conflict, which is how django returns the `bulk_create` result)
+- `commits` and `commit_parents` are not filtered by time or origin. The full git graph is dumped so ancestors without a checkout in the window are kept. Time slicing may be added later.
+- Surrogate ids are kept as in the source. Restore runs `setval` to `MAX(id)` so the next insert does not collide.
 
 ## Performance Considerations
 

backend/kernelCI/settings.py

@@ -268,6 +268,10 @@ def get_json_env_var(name, default):
 # https://docs.djangoproject.com/en/5.0/ref/settings/#databases
 
 BACKEND_VOLUME_DIR = os.environ.get("BACKEND_VOLUME_DIR", "/volume_data")
+# Throwaway git dirs for one-shot SHA fetches (#2090). Prefer tmpfs (e.g. /dev/shm).
+GIT_SCRATCH_DIR = os.environ.get(
+    "GIT_SCRATCH_DIR", "/dev/shm/kernelci-git-scratch"
+)
 
 DATABASE_ROUTERS = ["kernelCI_app.routers.databaseRouter.DatabaseRouter"]