Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions repo-troubleshooting/list-repos/CSV_SCHEMA.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,16 @@ sort
| `repos-with-skipped-files.csv` | `url` |
| `skipped-files-reason-details.csv` | `repository.name`, `rev`, `reason`, `file.extension`, `file.path` |

The optional `--count-commits` and `--run-search` flags append extra
columns to the repo-listing CSVs above, excluding the `--stats`
files and the skipped-file reason detail CSV, in this order: main
columns → per-CSV extras → commit-count columns → run-search columns
The optional `--count-commits`, `--run-search`, and repair flags
(`--fetch`, `--reclone`, `--reindex`) append extra columns to the
repo-listing CSVs above, excluding the `--stats` files and the
skipped-file reason detail CSV, in this order: main columns → per-CSV
extras → commit-count columns → run-search columns → action columns

`--failed` narrows every repo-listing CSV to repos with a cloning error,
using Sourcegraph's server-side `failedFetch`, `corrupted`, and
`cloneStatus: NOT_CLONED` filters, so `repos.csv` and
`repos-with-cloning-errors.csv` then list the same repos

## Main columns

Expand All @@ -65,6 +71,8 @@ These are written to every repo-listing CSV file
| `mirrorInfo.nextSyncAt` | timestamp | | Timestamp the repo is next scheduled to be synced from upstream |
| `mirrorInfo.secondsUntilNextSyncAt` | integer | | Integer seconds remaining until `mirrorInfo.nextSyncAt` |
| `mirrorInfo.updateSchedule.intervalSeconds` | integer | | Interval, in seconds, between scheduled mirror updates. Default max is 28800 seconds (8 hours), but is shortened for busy / popular repos |
| `mirrorInfo.updateQueue.index` | integer | | Position of the repo in repo-updater's update queue. Repos being updated are moved to the end of the queue, so ignore this when `mirrorInfo.updateQueue.updating` is `True` |
| `mirrorInfo.updateQueue.updating` | boolean | | `True` while repo-updater has a fetch or clone of this repo in progress |
| `mirrorInfo.shard` | string | true | Pod name of the gitserver shard which holds this repo's clone |
| `textSearchIndex.status` | enum (indexed, not_indexed) | | Search-index state, derived locally: `indexed` if Zoekt has built an index for this repo, `not_indexed` otherwise |
| `textSearchIndex.lastIndexStatus` | enum (SUCCESS, FAILURE) | | Most recent persisted text search indexing attempt result. Blank when no attempt was reported |
Expand Down Expand Up @@ -144,6 +152,17 @@ Appended to CSV files when `--run-search PATTERN` is used
| `runSearch.limitHit` | boolean | | `True` when the search hit a limit, so the results are incomplete |
| `runSearch.alertTitle` | string | | Title of the search-API alert when the server's `timeout:` budget was exceeded or the query was malformed |

## Action columns

Appended to CSV files when `--fetch`, `--reclone`, or `--reindex` is used.
Mutations are sent in aliased batches of 10 per GraphQL
request, `--concurrency` requests at a time

| Column | Type | Requires admin | Description |
| --- | --- | --- | --- |
| `action` | string | true | What the script did to this repo: `listed` when no mutation applied, otherwise `<fetch\|reclone\|reindex> <triggered\|skipped\|failed>`; semicolon-joined when several mutations applied |
| `result` | string | true | GraphQL error or skip message for a `skipped` or `failed` action; blank when triggered |

## `--stats` files

- Written when `--stats` is used
Expand Down
71 changes: 49 additions & 22 deletions repo-troubleshooting/list-repos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
Export repository health and size metadata from a Sourcegraph instance to CSV

The script is meant for support and troubleshooting work: it streams the repo
list through Sourcegraph's GraphQL API, writes endpoint-prefixed CSV files, and
keeps memory use flat on large instances
list through Sourcegraph's GraphQL API, writes CSV files per run, and keeps
memory use flat on large instances

## Requirements

Expand Down Expand Up @@ -71,13 +71,33 @@ python3 list-repos.py --run-search 'TODO patternType:literal'
python3 list-repos.py --statistics
```

Site admins can also trigger repair mutations:
### Failed repos

`--failed` narrows the run to repos with cloning errors, using server-side
filters (`failedFetch`, `corrupted`, `cloneStatus: NOT_CLONED`) instead of
scanning every repo. The same client-side error detection is then applied, so
the result matches `repos-with-cloning-errors.csv` from a full run, in a
fraction of the time on large instances

```sh
# Reclone every repo currently in a cloning-error state
python3 list-repos.py --reclone
# List only repos with cloning errors
python3 list-repos.py --failed

# Trigger a fetch (updateMirrorRepository) on every failed repo
python3 list-repos.py --failed --fetch

# Reclone (recloneRepository) every failed repo
python3 list-repos.py --failed --reclone
```

### Repair mutations

# Reclone one repo, whether in an error state or not
Site admins can trigger repair mutations. `--fetch` and `--reclone` are
mutually exclusive, and without a `REPO` they require `--failed`:

```sh
# Fetch or reclone one repo, whether in an error state or not
python3 list-repos.py --fetch github.com/org/repo
python3 list-repos.py --reclone github.com/org/repo

# Reindex every cloned repo missing a search index
Expand All @@ -87,25 +107,32 @@ python3 list-repos.py --reindex
python3 list-repos.py --reindex github.com/org/repo
```

## Output files
Mutations are sent in aliased batches of 10 per GraphQL request, with up to
`--concurrency` requests in flight. Recloning is expensive on gitserver, so
lower `--concurrency` when recloning thousands of repos

- Output files are written in the current directory
- Filenames are prefixed with the hostname from `SRC_ENDPOINT`
Each repo's outcome lands in the `action` and `result` CSV columns, for example
`reclone triggered`, `reclone skipped` (another reclone is already in
progress), or `reclone failed` with the server's error message

Possible output files:

| File | When written |
| ----------------------------------------- | --------------------------------------------------------- |
| `<prefix>-repos.csv` | Every normal listing run |
| `<prefix>-repos-with-cloning-errors.csv` | When one or more repos have a cloning or corruption error |
| `<prefix>-repos-with-indexing-errors.csv` | When one or more cloned repos are missing a search index |
| `<prefix>-repos-with-skipped-files.csv` | With `--skipped-files` and one or more skipped-file repos |
| `<prefix>-stats-*.csv` | With `--statistics` |
| `<prefix>-<repo>-<rev>-skipped-files.csv` | With `--skipped-files-reason REPO[@REV]` |
| `<prefix>-<repo>-<rev>-skipped-stats.csv` | With `--skipped-files-reason REPO[@REV]` |
## Output files

- Optional columns from `--count-commits` and `--run-search` are appended to the
per-repo CSVs
Each run writes to `list-repos-runs/<endpoint>/<timestamp>/`, so runs never
overwrite each other

| File | When written |
| ----------------------------------- | --------------------------------------------------------- |
| `list-repos.log` | Every run |
| `repos.csv` | Every listing run |
| `repos-with-cloning-errors.csv` | When one or more repos have a cloning or corruption error |
| `repos-with-indexing-errors.csv` | When one or more cloned repos are missing a search index |
| `repos-with-skipped-files.csv` | With `--skipped-files` and one or more skipped-file repos |
| `stats-*.csv` | With `--statistics` |
| `skipped-files-reason-details.csv` | With `--skipped-files-reason REPO[@REV]` |
| `skipped-files-reason-stats.csv` | With `--skipped-files-reason REPO[@REV]` |

- Optional columns from `--count-commits`, `--run-search`, and the repair
mutations are appended to the per-repo CSVs
- See [`CSV_SCHEMA.md`](CSV_SCHEMA.md) for the exact columns, types, and
admin-only fields

Expand Down
8 changes: 5 additions & 3 deletions repo-troubleshooting/list-repos/dev/TODO.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# TODO

- Verify whether `recloneRepository` and `reindexRepository` are idempotent. Until
then, avoid retrying a mutation after an ambiguous network failure, or verify
its resulting state before retrying.
- Mutation batches (`--fetch`, `--reclone`, `--reindex`) go through the same
HTTP retry logic as queries. A retried reclone is reported as `skipped`
("another reclone is in progress"); fetch and reindex only re-enqueue. Confirm
this against a real instance under a forced retry (e.g. a 503) before
relying on it.
- Stream aggregate skipped-file results as each indexed ref completes to reduce
peak client memory. First confirm that lowering retained results provides a
meaningful improvement over the current bounded queue.
Expand Down
Loading
Loading