I've updated to v1.19.5 and have run the indexing over my repository. I've found that as the index gets larger and larger, the store phase starts taking a huge amount of time. At the beginning it was taking a couple of seconds for 50 chunks, and now it's taking 10+ minutes for 50 chunks. Is there any way to accelerate this?
For instance: Store phase: 13 files stored in 1391.8s (running total 17090 chunks)
My data folder is 6+ GB at the moment:
[5.7G] data
├── [5.6G] chunks.lance
│ ├── [219M] data
│ ├── [174K] _deletions
│ ├── [ 12M] _transactions
│ └── [5.4G] _versions
├── [ 38] index.lock
├── [ 46M] keyword-index.json
├── [ 13K] __manifest
│ ├── [4.4K] _transactions
│ └── [4.8K] _versions
├── [1.0M] manifest.json
└── [ 28M] opencode-rag.log
Also, with the new change to interleave the stages, it would be nice if some of them could be run in parallel. For instance, the storing stage doesn't use the GPU, so it could theoretically compute the next stage that uses the GPU at the same time.
I've updated to v1.19.5 and have run the indexing over my repository. I've found that as the index gets larger and larger, the store phase starts taking a huge amount of time. At the beginning it was taking a couple of seconds for 50 chunks, and now it's taking 10+ minutes for 50 chunks. Is there any way to accelerate this?
For instance:
Store phase: 13 files stored in 1391.8s (running total 17090 chunks)My data folder is 6+ GB at the moment:
Also, with the new change to interleave the stages, it would be nice if some of them could be run in parallel. For instance, the storing stage doesn't use the GPU, so it could theoretically compute the next stage that uses the GPU at the same time.