Added runbooks for RCA and replaced GCS URLs with the repsective paths

tools/import_validation/troubleshooting_guides/bugs_summary/470415967.md

@@ -0,0 +1,7 @@
+# Bug 470415967 Summary
+
+## Problem Description
+Discrepancies in GDP data were reported between Data Commons and the World Bank website, specifically for Nigeria, Kenya, and Hungary. Investigations revealed that the source data at the World Bank had been updated recently, causing a mismatch with the existing data in Data Commons. A refresh run failed to download the updated data, and subsequent runs encountered validation failures, including issues with deletion checks and missing reference errors.
+
+## Resolution
+The issue was resolved by correcting DCID mappings for the Channel Islands and Kosovo (changing XKX to XKS), which resolved validation and reference errors in the import pipeline. Following these corrections, the data import pipelines were successfully re-executed, which synchronized the data with the updated World Bank records.

tools/import_validation/troubleshooting_guides/bugs_summary/472258775.md

@@ -0,0 +1,7 @@
+# Bug 472258775 Summary
+
+## Problem Description
+An automated refresh of World Bank datasets led to validation failures due to missing observations for multiple countries. Root cause analysis revealed issues related to country DCID naming and name discrepancies at the source.
+
+## Resolution
+The team implemented a workaround to retain historical data for indicators that were deleted from the source run. A PR was submitted to apply the changes, resolving the validation blocks.

tools/import_validation/troubleshooting_guides/bugs_summary/472605922.md

@@ -0,0 +1,7 @@
+# Bug 472605922 Summary
+
+## Problem Description
+The auto-refresh for the `USCensusPEP_Sex` import failed. The root cause was identified as intermittent unavailability of data at the source for programmatic download.
+
+## Resolution
+The issue was resolved by manually triggering the pipeline execution once the source data became available. The manual trigger was successful and the import completed without further issues. The bug was marked as fixed after verification.

tools/import_validation/troubleshooting_guides/bugs_summary/472606851.md

@@ -0,0 +1,7 @@
+# Bug 472606851 Summary
+
+## Problem Description
+The automated refresh for the `EurostatData_Education_Attainment` import failed, leading to the deletion of several data points.
+
+## Resolution
+A Root Cause Analysis (RCA) was conducted to investigate the deletions. Following the review of the RCA and implementation of the recommended changes, the job was re-run and confirmed to be working correctly with a satisfactory validation report.

tools/import_validation/troubleshooting_guides/bugs_summary/474326901.md

@@ -0,0 +1,7 @@
+# Bug 474326901 Summary
+
+## Problem Description
+The auto-refresh job for Eurostat Education Enrollment failed. Analysis in the differ tool was performed to investigate the discrepancies and data series deletions.
+
+## Resolution
+It was determined that no schema or code modifications were required. The issue was resolved by updating the `latest_version.txt` file (following approval) and successfully re-running the job, yielding a clean differ report.

tools/import_validation/troubleshooting_guides/bugs_summary/478186511.md

@@ -0,0 +1,7 @@
+# Bug 478186511 Summary
+
+## Problem Description
+A failed data import was reported for `USCensusPEP_Sex`. Investigation revealed that while local runs were clean, automated job runs showed inconsistent results, including deletions of data series that were not reproducible locally. This anomaly suggested a discrepancy between the local and automated environments or intermittent source data issues.
+
+## Resolution
+The discrepancies were documented in an anomaly tracking spreadsheet for further analysis. After reviewing the diff outputs and confirming they met acceptable criteria despite the inconsistencies, it was decided to proceed with the current versions. The issue was marked as fixed after verification.

tools/import_validation/troubleshooting_guides/bugs_summary/479399481.md

@@ -0,0 +1,7 @@
+# Bug 479399481 Summary
+
+## Problem Description
+The automated import for `USCensusPEP_Annual_Population` failed during download, which cascade-failed the subsequent data processing stages with deletion errors.
+
+## Resolution
+The import manifest was modified to isolate the download and processing steps, and the code was updated to explicitly fail and log errors if a download fails rather than continuing with incomplete data.

tools/import_validation/troubleshooting_guides/bugs_summary/481243356.md

@@ -0,0 +1,7 @@
+# Bug 481243356 Summary
+
+## Problem Description
+The import failed due to Eurostat's transition from country/region grouping classification EA20 to EA21. The new `nuts/EA21` ID was missing from the Knowledge Graph (KG), causing linting errors and threshold violations for deleted records (since EA20 was removed).
+
+## Resolution
+The new place ID `nuts/EA21` was added to `ProvisionalNodePlaces.mcf` to address the lint errors. Additionally, `nuts/EA21` was temporarily added to `skip_places.csv`, and a historical data file was created for the deleted `nuts/EA20` records. A CL with these changes was approved and merged.

tools/import_validation/troubleshooting_guides/bugs_summary/481245546.md

@@ -0,0 +1,7 @@
+# Bug 481245546 Summary
+
+## Problem Description
+A recurring failure of the `EurostatData_Education_Attainment` import occurred, linked to a regression from a previous fix (Bug 481243356).
+
+## Resolution
+A new fix was applied, which included the creation of a historical data file and a corresponding code change. The approach was reviewed and approved, and the issue was resolved with clean differ and validation reports provided.

tools/import_validation/troubleshooting_guides/bugs_summary/482902862.md

@@ -0,0 +1,7 @@
+# Bug 482902862 Summary
+
+## Problem Description
+The auto-refresh job for the World Development Indicators import failed due to the deletion of 4 Statistical Variable (SV) observations. Investigation of the source data confirmed that these observations had been deleted from the origin by the World Bank.
+
+## Resolution
+After reviewing the source-side data deletions, it was approved to accept the change and update to the latest version of the data. The import pipeline was updated to reference the latest version, and the job was completed successfully.

tools/import_validation/troubleshooting_guides/bugs_summary/482946661.md

@@ -0,0 +1,7 @@
+# Bug 482946661 Summary
+
+## Problem Description
+There was a failure in the auto-refresh process for imports related to state-level employment data. The issue was specifically caused by a missing reference or incorrect mapping for a statistical variable representing person counts in the transportation sector.
+
+## Resolution
+The problem was resolved by updating the mapping for the statistical variable to a more specific industry classification (NAICS-based). This involved modifying the configuration that maps source data fields to the internal statistical variables. The fix was verified through local testing and the changes were integrated into the main data pipeline.

tools/import_validation/troubleshooting_guides/bugs_summary/483219293.md

@@ -0,0 +1,7 @@
+# Bug 483219293 Summary
+
+## Problem Description
+The import job for `USCensusPEP_Sex` encountered a read timeout when attempting to access a source URL. This was a recurring issue caused by the source server taking longer to respond than the default timeout settings in the import script.
+
+## Resolution
+The root cause analysis recommended increasing the timeout duration within the script code to handle slower responses from the source. The issue was resolved by implementing this timeout adjustment, and the bug was marked as fixed.

tools/import_validation/troubleshooting_guides/bugs_summary/485260648.md

@@ -0,0 +1,7 @@
+# Bug 485260648 Summary
+
+## Problem Description
+The auto-refresh process for the `USCensusPEP_By_Sex_Race` import failed because the source data URL was intermittently unavailable, causing the import to error out during the scheduled run.
+
+## Resolution
+The issue resolved itself once the source URL became accessible again. Subsequent runs of the data import were successful, and the bug was marked as fixed.

tools/import_validation/troubleshooting_guides/bugs_summary/486801970.md

@@ -0,0 +1,7 @@
+# Bug 486801970 Summary
+
+## Problem Description
+The automated refresh for the `USCensusPEP_PopulationEstimatebyRace` import failed during its scheduled execution.
+
+## Resolution
+The import job was manually re-run and completed successfully. Validation outputs, including a differ log and verification reports, were checked to confirm data integrity. The bug is verified and fixed.

tools/import_validation/troubleshooting_guides/bugs_summary/489948342.md

@@ -0,0 +1,7 @@
+# Bug 489948342 Summary
+
+## Problem Description
+The import for World Development Indicators failed because a required file (`API_IND_DS28_en_csv_v2_10671`) was missing from the source World Bank data.
+
+## Resolution
+The issue was resolved by updating the `latest_version.txt` file and addressing the deleted flags for India (IND) in the differ report. The pipeline was successfully re-run, and a validation report was generated.

tools/import_validation/troubleshooting_guides/bugs_summary/493190090.md

@@ -0,0 +1,7 @@
+# Bug 493190090 Summary
+
+## Problem Description
+The auto-refresh for the `USCensusPEP_PopulationEstimatebyRace` data import failed during its scheduled run.
+
+## Resolution
+The import issue was addressed and fixed. Successful data import was confirmed via validation reports.

tools/import_validation/troubleshooting_guides/bugs_summary/4935600377.md

@@ -0,0 +1,7 @@
+# Bug 4935600377 / 493560037 Summary
+
+## Problem Description
+The auto-refresh for the `USCensusPEP_Annual_Population` data import failed due to an error in the execution pipeline. (Note: The bug ID `4935600377` in documentation corresponds to Bug `493560037` in Buganizer).
+
+## Resolution
+This issue was identified as a duplicate of a canonical, systemic failure within the core import pipeline rather than a dataset-specific error. The pipeline issue is currently assigned and being tracked centrally.

tools/import_validation/troubleshooting_guides/bugs_summary/496059688.md

@@ -0,0 +1,7 @@
+# Bug 496059688 Summary
+
+## Problem Description
+A validation failure was encountered during the Eurostat Education Enrollment import process, blocking the pipeline completion.
+
+## Resolution
+The issue was resolved by generating a new validation report and verifying the data consistency. The fix was verified, and the validation checks passed.

tools/import_validation/troubleshooting_guides/bugs_summary/497802532.md

@@ -0,0 +1,7 @@
+# Bug 497802532 Summary
+
+## Problem Description
+Intermittent "403 Forbidden" errors occurred when attempting to download data files from census.gov. Specifically, CSV URLs were inaccessible while XLS URLs remained working, causing batch job failures and unintended data deletions.
+
+## Resolution
+The team addressed the intermittent URL accessibility issues. The fix was verified through successful re-runs of the national, state, and county-level population estimate imports.

tools/import_validation/troubleshooting_guides/bugs_summary/498154643.md

@@ -0,0 +1,7 @@
+# Bug 498154643 Summary
+
+## Problem Description
+A validation failure was encountered during the processing of Eurostat regional statistics (`EurostatData_Fertility`) due to additional data deletions in the source data.
+
+## Resolution
+The deletions were reviewed and approved by the team. The data import pipelines were updated and successfully re-executed, and the validation checks passed.

tools/import_validation/troubleshooting_guides/bugs_summary/500622108.md

@@ -0,0 +1,7 @@
+# Bug 500622108 Summary
+
+## Problem Description
+The automated refresh for the `USCensusPEP_AgeSexRace` data import was failing, and critical execution logs were missing from the logging outputs, making troubleshooting difficult.
+
+## Resolution
+A code change (PR 1950) was merged to restore logging outputs for the dataset import process. The fix was verified, and the validation report completed successfully.

tools/import_validation/troubleshooting_guides/bugs_summary/500945912.md

@@ -0,0 +1,7 @@
+# Bug 500945912 Summary
+
+## Problem Description
+A failure was reported in the download script for an import. Additionally, there were unexpected deletions of data series for certain Statistical Variables. It was identified that these deletions occurred because a specific StatVar wasn't properly recognized or was missing from the internal schema, leading the system to treat existing data as obsolete.
+
+## Resolution
+The issue was resolved by adding the `-existing_statvar_mcf` parameter to the `manifest.json` file. This ensures that the system correctly identifies and preserves the existing Statistical Variables during the import process. The fix was verified as successful in subsequent runs.

tools/import_validation/troubleshooting_guides/bugs_summary/502079403.md

@@ -0,0 +1,7 @@
+# Bug 502079403 Summary
+
+## Problem Description
+The auto-refresh cron job for the school district statistics import was missing from the scheduler, preventing automated updates.
+
+## Resolution
+The refresh configuration was analyzed. Since newer data for 2024-25 was available at the source, steps were initiated to manually download the files and run a semi-automated refresh. The issue is assigned and is being resolved.

tools/import_validation/troubleshooting_guides/bugs_summary/502090898.md

@@ -0,0 +1,7 @@
+# Bug 502090898 Summary
+
+## Problem Description
+A validation failure occurred due to API issues causing unintended deletions during an import. This was related to how historical data was being handled and flagged for deletion when not found in the current source run.
+
+## Resolution
+The fix involved a multi-step process: creating a plan for historical data retention, updating the import configuration to store historical data properly, and finally updating the latest version configurations to remove the deletion flags. This ensured that valid historical data was preserved even if not present in the most recent source run.

tools/import_validation/troubleshooting_guides/bugs_summary/504879314.md

@@ -0,0 +1,7 @@
+# Bug 504879314 Summary
+
+## Problem Description
+The automated data refresh for the `EurostatData_Education_Attainment` dataset failed due to a validation error, which blocked the data import pipeline.
+
+## Resolution
+A validation fix was implemented to resolve the failure. Following verification, the validation report confirmed that the imported data successfully passes all validation checks.

tools/import_validation/troubleshooting_guides/bugs_summary/506961224.md

@@ -0,0 +1,7 @@
+# Bug 506961224 Summary
+
+## Problem Description
+The World Bank dataset refresh failed with a "length mismatch" error in `world_bank_stat_vars.mcf` caused by new indicators and significant row deletions. The task was blocked because the executor run lacked the necessary write permissions to update production files.
+
+## Resolution
+A new deduplication logic was merged (PR 2077) to handle historical data. A workaround was identified to manually update the MCF file and copy historical data files to the production environment by a user with higher permissions.

tools/import_validation/troubleshooting_guides/bugs_summary/507394518.md

@@ -0,0 +1,7 @@
+# Bug 507394518 Summary
+
+## Problem Description
+The data import for `US_SAT_ACT_Participation` failed during automated refresh validation.
+
+## Resolution
+A code fix was merged and deployed. Validation checks successfully ran and passed, verifying that the SAT/ACT data import is stable.

tools/import_validation/troubleshooting_guides/docs_summary/bls_ces_state_deletion_resolution.md

@@ -0,0 +1,9 @@
+# Google Doc: BLS_CES_State Deletion Resolution Summary
+
+*   **Original URL:** https://docs.google.com/document/d/1QK9pMoSFR7STb78aFgN_NOupcJzU_IdWp4tyIvzrwis/edit
+*   **Purpose/Overview:** This document outlines the resolution plan for a validation failure in the `BLS_CES_State` data import on April 20, 2026.
+*   **Problem Description:** The import process failed because 4.57% (33,120 records) of the data points were deleted, which exceeded the default 0% threshold.
+*   **Root Causes:** The Bureau of Labor Statistics (BLS) announced official reductions and eliminations of several series data points starting with the release of January 2026 data as part of their annual benchmarking and sample review.
+*   **Resolutions/Findings:** 
+    *   Since these deletions are official source-side changes, the deleted records cannot be recovered.
+    *   To resolve the import failure, the deleted data was preserved by extracting and storing it in a historical file, uploading it to Content Native Storage, and adjusting the validation settings to allow the pipeline to complete.

tools/import_validation/troubleshooting_guides/docs_summary/bls_imports_issues_summary.md

@@ -0,0 +1,9 @@
+# Google Doc: BLS Imports Issues Summary
+
+*   **Original URL:** https://docs.google.com/document/d/1bQbwKMVPtcehUZ3zI3r2kGlidMiE6UjEN9faKlJgtUs/edit
+*   **Purpose/Overview:** This document provides a general overview of multiple common issues and fixes across various BLS (Bureau of Labor Statistics) data imports (CES, CPI_Category, CES_State, etc.).
+*   **Problem Description:** Multiple BLS pipelines failed due to outdated files, missing references, and blocked automated downloads.
+*   **Root Causes & Fixes:** 
+    *   **Outdated latest_version.txt:** The Cloud Batch run date was newer than the version indicated in the `latest_version.txt` file, causing false deletion detections. Fixed by manually updating the `latest_version.txt` configuration.
+    *   **Dependency on Production Schema Releases:** New Statistical Variables (SVs) added to the schema caused missing reference errors until the production release was finalized. Resolved by updating the pipeline to query the AutoPush DC schema instance.
+    *   **Source Blocked Automated Downloads:** BLS upgraded their security posture using TLS Fingerprinting, which blocked automated python scripts with a `403 Forbidden` error. Resolved by using the `curl_cffi` library to mimic a standard browser's TLS handshake signature.

tools/import_validation/troubleshooting_guides/docs_summary/dc_imports_execution_learnings.md

@@ -0,0 +1,9 @@
+# Google Sheet: DC - Imports Execution Learnings Summary
+
+*   **Original URL:** https://docs.google.com/spreadsheets/d/1lAm-EPB6o9U9btQHbdRruGhx_DFerlhQF43Ba6skI4Q/edit
+*   **Purpose/Overview:** This spreadsheet tracks ongoing execution learnings, issues, and recommendations across multiple Data Commons import pipelines.
+*   **Key Entries & Actions:**
+    *   **Unexpected Characters:** Source data issues resolved by implementing PV Mapping and processor code changes.
+    *   **Missing Places/DCIDs:** Validation failures resolved by manually adding missing places to production (e.g. `b/472258775`).
+    *   **Pipeline Automation:** Recommendation to convert Semi-Auto Refresh setups into Full Auto Refresh pipelines to minimize manual restarts (e.g. `b/474348464`).
+    *   **Hangs and Performance:** Documented cases of copy service hangs requiring manual restarts, and low parallel performance in the KG Import UI.

tools/import_validation/troubleshooting_guides/docs_summary/eia_nuclear_outages_summary.md

@@ -0,0 +1,9 @@
+# Google Doc: EIA_NuclearOutages Summary
+
+*   **Original URL:** https://docs.google.com/document/d/1GfF_sdUCg4d3fmkJoRCarQpFoGtLIPfezoKwoEXXL84/edit
+*   **Purpose/Overview:** This document details the analysis of an import failure in the `EIA_NuclearOutages` dataset on March 3, 2026.
+*   **Problem Description:** The import failed validation because it found 273 (0.01%) deleted records, which violated the 0% deletion threshold.
+*   **Root Causes:** The deletions represented pre-commercial testing phase records for Vogtle Unit 4 (from January to March 2024). Once the unit entered official commercial operation, the EIA cleaned up the dataset, removing these pre-commercial records from their historical data.
+*   **Resolutions/Findings:** 
+    *   It was decided not to preserve the deleted data as historical, since the deleted points only represented testing-phase records and the volume was negligible.
+    *   The issue was resolved by updating the `latest_version.txt` file to acknowledge the deletion as a deliberate source update, which allowed the pipeline execution to proceed.

tools/import_validation/troubleshooting_guides/docs_summary/eia_seds_summary.md

@@ -0,0 +1,11 @@
+# Google Doc: EIA_SEDS Summary
+
+*   **Original URL:** https://docs.google.com/document/d/1Kvhz-7AaWpaxSvSN1wHDP75-3CVXTfZT7dffBgZNTaY/edit
+*   **Purpose/Overview:** This document describes the validation failure of the `EIA_SEDS` data import on April 20, 2026.
+*   **Problem Description:** The import failed due to 1,248 (0.05%) deleted records and 56 missing reference warnings.
+*   **Root Causes:** 
+    *   **Deletions:** The source data removed zero-value placeholders that were not part of the NL Statistical Variables.
+    *   **Missing References:** The source introduced new Statistical Variables that were not defined in the Data Commons schema.
+*   **Resolutions/Findings:** 
+    *   Acknowledged the deletions by updating `latest_version.txt`.
+    *   Created and merged a Change List (CL) to add the definitions for the 56 new Statistical Variables in Data Commons, then ran a forced update.

tools/import_validation/troubleshooting_guides/docs_summary/eurostat_employment_per_sector_summary.md

@@ -0,0 +1,7 @@
+# Google Doc: EurostatData_Employment_Per_Sector Summary
+
+*   **Original URL:** https://docs.google.com/document/d/1CPyqi9DBjv0t4eWanNYMRfJTIGjhubN6Op2Sao5S-9Y/edit
+*   **Purpose/Overview:** This document analyzes an import failure for the `EurostatData_Employment_Per_Sector` dataset on February 16, 2026.
+*   **Problem Description:** The process failed because it found 6,645 deleted records (1.39% of the total), which exceeded the allowed 0% threshold for deletions.
+*   **Root Causes:** The failure was caused by missing source data for the years 1995–1999 across 92 NUTS/IDs. The data was deleted from the official Eurostat source and is not expected to be restored.
+*   **Resolutions/Findings:** To resolve the validation error, the document recommends preserving the missing data by identifying it through a comparison with the previous production version, storing it in a historical file, and uploading it to Content Native Storage.