Skip to content

Use less memory in processing JSON #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mason-sharp merged 1 commit into from
Feb 23, 2026
Merged

Use less memory in processing JSON #82

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions internal/consistency/diff/table_diff.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -873,7 +873,6 @@ func (t *TableDiffTask) RunChecks(skipValidation bool) (err error) {
if err != nil {
return fmt.Errorf("failed to connect to node %s: %w", hostname, err)
}
defer conn.Close()

currCols, err := queries.GetColumns(t.Ctx, conn, schema, table)
if err != nil {
Expand Down Expand Up @@ -930,6 +929,8 @@ func (t *TableDiffTask) RunChecks(skipValidation bool) (err error) {
if t.TableFilter != "" {
logger.Info("Applying table filter for diff: %s", t.TableFilter)
}

conn.Close()
}

logger.Info("Connections successful to nodes in cluster")
Expand Down Expand Up @@ -1001,13 +1002,14 @@ func (t *TableDiffTask) cleanupFilteredView() {
logger.Warn("table-diff: failed to get connection for filtered view cleanup on node %s: %v", name, err)
continue
}
defer pool.Close()

if _, err := pool.Exec(t.Ctx, dropSQL); err != nil {
logger.Warn("table-diff: failed to drop filtered view %s.%s on node %s: %v", t.Schema, t.FilteredViewName, name, err)
} else {
logger.Info("table-diff: dropped filtered view %s.%s on node %s", t.Schema, t.FilteredViewName, name)
}

pool.Close()
}

t.FilteredViewCreated = false
Expand Down
23 changes: 15 additions & 8 deletions pkg/common/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1216,15 +1216,22 @@ func WriteDiffReport(diffResult types.DiffOutput, schema, table, format string)
)
jsonFileName := outputPrefix + ".json"

jsonData, err := json.MarshalIndent(diffResult, "", " ")
// Stream JSON directly to file — avoids holding a second full copy in memory
f, err := os.Create(jsonFileName)
if err != nil {
logger.Error("ERROR marshalling diff output to JSON: %v", err)
return "", "", fmt.Errorf("failed to marshal diffs: %w", err)
}

if err := os.WriteFile(jsonFileName, jsonData, 0644); err != nil {
logger.Error("ERROR writing diff output to file %s: %v", jsonFileName, err)
return "", "", fmt.Errorf("failed to write diffs file: %w", err)
logger.Error("ERROR creating diff output file %s: %v", jsonFileName, err)
return "", "", fmt.Errorf("failed to create diffs file: %w", err)
}
enc := json.NewEncoder(f)
enc.SetIndent("", " ")
if err := enc.Encode(diffResult); err != nil {
Copy link

@coderabbitai coderabbitai bot Feb 23, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

json.Encoder.Encode appends a trailing newline — behavioral change

enc.Encode(v) always writes a \n after the JSON payload; the previous json.MarshalIndent + os.WriteFile did not. Any downstream consumer or test that reads the file and compares its exact byte content will be affected. This is likely harmless for JSON parsers, but worth confirming no tooling does a byte-exact check on the diff output.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@pkg/common/utils.go` at line 1229, The current use of enc.Encode(diffResult)
writes a trailing newline which changes prior behavior (json.MarshalIndent +
os.WriteFile) and may break byte-exact comparisons; update the write so the file
contains the exact JSON bytes without the extra '\n' — either replace
enc.Encode(diffResult) with json.MarshalIndent(diffResult, "", "  ") and write
the result using os.WriteFile (preserving previous indentation and no trailing
newline), or encode into a buffer then trim a single trailing newline before
writing; refer to enc.Encode(diffResult), json.MarshalIndent, os.WriteFile and
the diffResult variable to locate and modify the code.

Copy link
Member Author

@mason-sharp mason-sharp Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is ok. It is just at the end of the file, everything else remains the same.

Copy link

@coderabbitai coderabbitai bot Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mason-sharp, understood! Thanks for confirming that the trailing newline is acceptable for the JSON diff output files. The streaming approach still achieves the memory optimization goal while keeping the output functionally equivalent for consumers.

logger.Error("ERROR writing diff output to JSON: %v", err)
f.Close()
return "", "", fmt.Errorf("failed to write diffs: %w", err)
}
if err := f.Close(); err != nil {
logger.Error("ERROR closing diff output file %s: %v", jsonFileName, err)
return "", "", fmt.Errorf("failed to close diffs file: %w", err)
}

logger.Warn("%s TABLES DO NOT MATCH", CrossMark)
Expand Down
Loading