Commit df90924
committed
Add chunked batch processing and stability improvements to index rebuild command
Implemented efficient chunked processing in encryption:index-rebuild to handle large datasets safely without excessive memory usage.
• Each batch of model records (default: 100) is processed sequentially, ensuring constant memory consumption and reliable performance for millions of records.
• Added progress feedback (Processing in chunks of 100...) for better CLI visibility.
• Improved command resilience by validating model existence and method availability before rebuilding indexes.
• This change enhances scalability and production readiness of the encrypted search index rebuild process.1 parent f7c195a commit df90924
File tree
3 files changed
+62
-79
lines changed- src
- Console
- Observers
- Traits
3 files changed
+62
-79
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
14 | | - | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
15 | 15 | | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
| 16 | + | |
| 17 | + | |
21 | 18 | | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | 19 | | |
35 | 20 | | |
36 | 21 | | |
37 | 22 | | |
38 | 23 | | |
39 | 24 | | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | 25 | | |
44 | 26 | | |
45 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
46 | 30 | | |
47 | 31 | | |
48 | 32 | | |
| |||
54 | 38 | | |
55 | 39 | | |
56 | 40 | | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
| 41 | + | |
65 | 42 | | |
66 | 43 | | |
67 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
68 | 55 | | |
69 | | - | |
| 56 | + | |
70 | 57 | | |
71 | 58 | | |
72 | | - | |
| 59 | + | |
73 | 60 | | |
74 | 61 | | |
75 | 62 | | |
76 | 63 | | |
| 64 | + | |
| 65 | + | |
77 | 66 | | |
78 | 67 | | |
79 | 68 | | |
80 | 69 | | |
81 | | - | |
82 | | - | |
| 70 | + | |
| 71 | + | |
83 | 72 | | |
84 | 73 | | |
85 | 74 | | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
91 | 79 | | |
92 | 80 | | |
93 | 81 | | |
| 82 | + | |
94 | 83 | | |
95 | 84 | | |
96 | 85 | | |
97 | 86 | | |
98 | 87 | | |
99 | | - | |
| 88 | + | |
100 | 89 | | |
101 | 90 | | |
102 | 91 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
98 | | - | |
| 98 | + | |
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
| |||
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
116 | | - | |
| 116 | + | |
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | 50 | | |
68 | 51 | | |
69 | 52 | | |
| |||
98 | 81 | | |
99 | 82 | | |
100 | 83 | | |
101 | | - | |
| 84 | + | |
102 | 85 | | |
103 | | - | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
104 | 89 | | |
105 | 90 | | |
106 | 91 | | |
107 | 92 | | |
108 | 93 | | |
109 | 94 | | |
110 | | - | |
111 | | - | |
112 | | - | |
| 95 | + | |
| 96 | + | |
113 | 97 | | |
114 | 98 | | |
115 | | - | |
116 | 99 | | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
122 | 104 | | |
123 | 105 | | |
124 | | - | |
125 | | - | |
126 | | - | |
| 106 | + | |
127 | 107 | | |
128 | 108 | | |
129 | 109 | | |
130 | | - | |
131 | | - | |
| 110 | + | |
| 111 | + | |
132 | 112 | | |
133 | 113 | | |
134 | 114 | | |
| |||
140 | 120 | | |
141 | 121 | | |
142 | 122 | | |
143 | | - | |
144 | | - | |
| 123 | + | |
| 124 | + | |
145 | 125 | | |
146 | 126 | | |
147 | 127 | | |
| |||
236 | 216 | | |
237 | 217 | | |
238 | 218 | | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
239 | 233 | | |
0 commit comments