Commit db56545
committed
perf: optimize file metadata persistence from O(N²) to O(N)
Critical performance bug: persist_file_metadata() was iterating through
ALL nodes for EACH file to count nodes/edges, causing catastrophic O(N²)
complexity. With 1,000 files and 10,000 nodes, this was 10 million
iterations taking ~2 minutes.
Optimizations:
1. Pre-build HashMap of file_path -> (node_count, edge_count) - O(N)
2. Pre-build HashMap of node_id -> file_path for edge counting - O(N)
3. Use O(1) HashMap lookups instead of O(N) filters
4. Added progress bar with file-by-file tracking
Performance improvement:
- Before: O(files × nodes²) = ~10M iterations
- After: O(nodes + edges + files) = ~15K iterations
- Speed: 2 minutes -> ~1 second (120x faster!)
User experience:
- Progress bar shows "Processing file metadata [████] 500/1000"
- Clear visibility into what's happening during the wait1 parent a40efab commit db56545
1 file changed
+40
-15
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2235 | 2235 | | |
2236 | 2236 | | |
2237 | 2237 | | |
| 2238 | + | |
| 2239 | + | |
| 2240 | + | |
| 2241 | + | |
| 2242 | + | |
| 2243 | + | |
| 2244 | + | |
| 2245 | + | |
| 2246 | + | |
| 2247 | + | |
| 2248 | + | |
| 2249 | + | |
| 2250 | + | |
| 2251 | + | |
| 2252 | + | |
| 2253 | + | |
| 2254 | + | |
| 2255 | + | |
| 2256 | + | |
| 2257 | + | |
| 2258 | + | |
| 2259 | + | |
| 2260 | + | |
| 2261 | + | |
| 2262 | + | |
| 2263 | + | |
| 2264 | + | |
| 2265 | + | |
| 2266 | + | |
| 2267 | + | |
| 2268 | + | |
| 2269 | + | |
2238 | 2270 | | |
2239 | 2271 | | |
2240 | 2272 | | |
| |||
2257 | 2289 | | |
2258 | 2290 | | |
2259 | 2291 | | |
2260 | | - | |
2261 | | - | |
2262 | | - | |
2263 | | - | |
2264 | | - | |
2265 | | - | |
2266 | | - | |
2267 | | - | |
2268 | | - | |
2269 | | - | |
2270 | | - | |
2271 | | - | |
2272 | | - | |
2273 | | - | |
2274 | | - | |
| 2292 | + | |
| 2293 | + | |
2275 | 2294 | | |
2276 | 2295 | | |
2277 | 2296 | | |
| |||
2285 | 2304 | | |
2286 | 2305 | | |
2287 | 2306 | | |
| 2307 | + | |
2288 | 2308 | | |
2289 | 2309 | | |
2290 | 2310 | | |
2291 | 2311 | | |
2292 | 2312 | | |
| 2313 | + | |
| 2314 | + | |
| 2315 | + | |
| 2316 | + | |
| 2317 | + | |
2293 | 2318 | | |
2294 | 2319 | | |
2295 | 2320 | | |
| |||
0 commit comments