Updating gogen from v1.19.7 to v1.20.2 causes test failures in TestFromTestdata

[fennoai]

Problem

Updating github.com/goplus/gogen from v1.19.7 to v1.20.2 causes multiple TestFromTestdata tests to fail with compilation errors related to block comment formatting.

How to Reproduce

1. In go.mod, change gogen dependency:

- github.com/goplus/gogen v1.19.7
+ github.com/goplus/gogen v1.20.2

2. Run go mod tidy

3. Run tests:

go test ./cl/internal/convert/ -run TestFromTestdata -v

Error Messages

TestFromTestdata/gettext

expected declaration, found The

TestFromTestdata/gpgerror

expected declaration, found '*'

TestFromTestdata/issue507

Multiple block comment syntax errors

TestFromTestdata/keepcomment

comment not terminated

Root Cause

These errors occur because multi-line C block comments like:

/* Create an iterator for traversing a domain
   The domain NULL denotes the default domain */

Are split into multiple ast.Comment nodes (one per line) instead of a single node. With gogen v1.20.2, the Go printer now strictly follows the ast.Comment specification and produces invalid Go code from these incorrectly split comments.

Example of invalid generated output with gogen v1.20.2:

/* Create an iterator for traversing a domain
*/   The domain NULL denotes the default domain */
*/func ExampleFunction() {
}

Related

• Bug: gogen v1.20.2 update exposes comment parsing issue causing test failures #618 - Original issue tracking this bug

[luoliwoshang]

like cjson, with the v1.20.2 will generate like

/* The cJSON structure: */
*/type JSON struct {
	Next        *JSON
	Prev        *JSON
	Child       *JSON
	Type        c.Int
	Valuestring *c.Char
	Valueint    c.Int
	Valuedouble c.Double
	String      *c.Char
}

type Hooks struct {
	MallocFn c.Pointer
	FreeFn   c.Pointer
}
type Bool c.Int
/* returns the version of cJSON as a string */
*///go:linkname Version C.cJSON_Version
func Version() *c.Char
/* Supply malloc, realloc and free functions to cJSON */
*/// llgo:link (*Hooks).InitHooks C.cJSON_InitHooks
func (recv_ *Hooks) InitHooks() {
}
/* Memory Management: the caller is always responsible to free the results from all variants of cJSON_Parse (with
*/ * cJSON_Delete) and cJSON_Print (with stdlib free, cJSON_Hooks.free_fn, or cJSON_free as appropriate). The exception is
*/ * cJSON_PrintPreallocated, where the caller has full responsibility of the buffer. */
*//* Supply a block of JSON, and this returns a cJSON object you can interrogate. */
*///go:linkname Parse C.cJSON_Parse
func Parse(value *c.Char) *JSON
//go:linkname ParseWithLength C.cJSON_ParseWithLength
func ParseWithLength(value *c.Char, buffer_length c.SizeT) *JSON
/* ParseWithOpts allows you to require (and check) that the JSON is null terminated, and to retrieve the pointer to the
*/ * final byte parsed. */
*//* If you supply a ptr in return_parse_end and parsing fails, then return_parse_end will contain a pointer to the error
*/ * so will match cJSON_GetErrorPtr(). */
*///go:linkname ParseWithOpts C.cJSON_ParseWithOpts
func ParseWithOpts(value *c.Char, return_parse_end **c.Char, require_null_terminated Bool) *JSON

but we expect

/* The cJSON structure: */

type JSON struct {
	Next        *JSON
	Prev        *JSON
	Child       *JSON
	Type        c.Int
	Valuestring *c.Char
	Valueint    c.Int
	Valuedouble c.Double
	String      *c.Char
}

type Hooks struct {
	MallocFn c.Pointer
	FreeFn   c.Pointer
}
type Bool c.Int

/* returns the version of cJSON as a string */
//go:linkname Version C.cJSON_Version
func Version() *c.Char

/* Supply malloc, realloc and free functions to cJSON */
// llgo:link (*Hooks).InitHooks C.cJSON_InitHooks
func (recv_ *Hooks) InitHooks() {
}

/* Memory Management: the caller is always responsible to free the results from all variants of cJSON_Parse (with cJSON_Delete) and cJSON_Print (with stdlib free, cJSON_Hooks.free_fn, or cJSON_free as appropriate). The exception is cJSON_PrintPreallocated, where the caller has full responsibility of the buffer. */
/* Supply a block of JSON, and this returns a cJSON object you can interrogate. */
//go:linkname Parse C.cJSON_Parse
func Parse(value *c.Char) *JSON

//go:linkname ParseWithLength C.cJSON_ParseWithLength
func ParseWithLength(value *c.Char, buffer_length c.SizeT) *JSON

/* ParseWithOpts allows you to require (and check) that the JSON is null terminated, and to retrieve the pointer to the final byte parsed. */
/* If you supply a ptr in return_parse_end and parsing fails, then return_parse_end will contain a pointer to the error so will match cJSON_GetErrorPtr(). */
//go:linkname ParseWithOpts C.cJSON_ParseWithOpts
func ParseWithOpts(value *c.Char, return_parse_end **c.Char, require_null_terminated Bool) *JSON

[luoliwoshang]

related issue goplus/gogen#589

[luoliwoshang]

goplus/gogen#589 (comment)

this behavior is the go/printer 's change lead

package main

import (
	"bytes"
	"fmt"
	"go/ast"
	"go/printer"
	"go/token"
)

func main() {
	fset := token.NewFileSet()
	f := &ast.File{
		Name: ast.NewIdent("main"),
		Decls: []ast.Decl{
			&ast.FuncDecl{
				Doc: &ast.CommentGroup{List: []*ast.Comment{{
					Slash: token.NoPos,
					Text:  "/* returns the version of cJSON as a string */\n",
				}}},
				Name: ast.NewIdent("main"),
				Type: &ast.FuncType{Params: &ast.FieldList{}},
				Body: &ast.BlockStmt{},
			},
		},
	}
	var b bytes.Buffer
	cfg := printer.Config{Mode: printer.UseSpaces | printer.TabIndent, Tabwidth: 8}
	if err := cfg.Fprint(&b, fset, f); err != nil {
		panic(err)
	}
	fmt.Print(b.String())
}

llcppg/_demo/printf on  main [$!?⇕] via 🐹 v1.24.10 
❯ go run main.go
package main
/* returns the version of cJSON as a string */
*/func main() {
}

so our generate ast.Comment actually every list item is a single line comment, ast.Comment generate from parser need not a aditional \n

{
      "Doc": {
        "List": [
          {
            "Text": "// not read doc 1\n",
            "_Type": "Comment"
          }
        ],
        "_Type": "CommentGroup"
      },
}

@xgopilot create a issue to report we need remove the ast.Comment 's additional \n,because a comment node already known as one line!

[fennoai]

Fixed: Remove trailing \n from ast.Comment.Text

PR created: #637

Changes

_xtool/internal/parser/parser.go — ParseComment function:

• Removed + "\n" when creating ast.Comment nodes
• Added empty line skip to avoid creating empty comment nodes from trailing newlines in strings.Split output

cl/internal/convert/comments.go — NewCommentGroupFromC function:

• Added defensive strings.TrimRight(comment.Text, "\n") when converting C AST comments to Go AST comments, ensuring any remaining trailing newlines are stripped at the boundary

_xtool/internal/parser/testdata/comment/expect.json:

• Updated all "Text" fields to remove trailing \n to match the new behavior

Root Cause

Each ast.Comment represents a single line. ParseComment was splitting by \n then re-adding \n to each line. With go/printer in gogen v1.20.2, this produces invalid Go code like */func main() {.

Verification

• go build ./... — passes
• go fmt ./... — no changes
• go vet ./... — passes
• Unit tests — all pass
• TestFromTestdata requires llcppsigfetch (CI will validate)