Skip to content

[chore]: Replaced Hand-written DTOs with Protobuff - #241

Merged
yash-khaitan merged 22 commits into
mainfrom
Introducing-protobuff-for-autoGeneration-of-DTO
Nov 15, 2026
Merged

yash-khaitan merged 22 commits into
mainfrom
Introducing-protobuff-for-autoGeneration-of-DTO

Conversation

@yash-khaitan

Copy link
Copy Markdown
Contributor

SUMMARY

What this pr is for?

  • This PR replaces four hand-written Java DTO classes (PipelineDTO, StageDTO, PipelineBuildDTO, StageBuildDTO) with proto-generated classes from .proto schema files.
  • All mappers, services, controllers, and tests are updated to use the new immutable proto builder pattern.

Issue Link

Link the Issue for which this PR was raised
#217 (comment)

Changes Summary

What changes has been done to solve the issue

  • Replaced all hand-written DTOs with proto-generated classes.

Testing Done

How has the solution been tested?

  • All 138 unit tests pass (mvn test) covering mappers, services, controllers end-to-end
  • Verified docker compose up --build compiles proto files and starts the application without errors
  • Manually tested all pipeline features via UI: create pipeline, view pipeline list, run a build, view build status

Pull Request Checklist

Tasks Yes/No
Issue Linked?
Do these changes affect local development in anyway? (if yes mention effects in additional comments)

Additional Comments

Anything else regarding this PR

@SuvabrataChowdhury SuvabrataChowdhury left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of unnecessary formating changes has been sent. please check that. Also refer to how the openapi doc is used. that is also using similer concept of autogenerated thing. we can keep the .proto files under /apis/protobuff so that every service has a unified location.

Comment thread pipeline/pom.xml Resolved

@yash-khaitan yash-khaitan left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document

@SuvabrataChowdhury SuvabrataChowdhury left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good. one ask is just keep the .proto files itself under root of autobuild just like /apis/openapi-spec/ keeps openapi specifications under unified path. This helps in keeping all such definition easily sharable amongst multiple services.

Comment thread pipeline/pom.xml Resolved
Comment thread pipeline/pom.xml Resolved

@SuvabrataChowdhury SuvabrataChowdhury left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good to me. One ask is to include a little bit of documentation around it. Like if you used any extensions for syntax highlighting, docs you referred etc, so that anyone who needs to develop later, they can read and have a better understanding.

Also one more thing is that which maven build stage does the generation of these files come? is it compilation?

One more thing, please add the gen package in .gitignore to prevent manual edits of the generated files.

Comment thread pipeline/pom.xml Resolved
@yash-khaitan

Copy link
Copy Markdown
Contributor Author

Mostly looks good to me. One ask is to include a little bit of documentation around it. Like if you used any extensions for syntax highlighting, docs you referred etc, so that anyone who needs to develop later, they can read and have a better understanding.

Also one more thing is that which maven build stage does the generation of these files come? is it compilation?

One more thing, please add the gen package in .gitignore to prevent manual edits of the generated files.

Documents added.

The files are generated in the generate-sources phase before compile phase... This info has also been added in the documentation.

Updated .gitignore

@SuvabrataChowdhury SuvabrataChowdhury left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Little modifications on the wordings.

Also please take the latest pull and resolve the merge conflicts.

@yash-khaitan
yash-khaitan force-pushed the Introducing-protobuff-for-autoGeneration-of-DTO branch from 74c8195 to 9411321 Compare November 10, 2026 07:02

@SuvabrataChowdhury SuvabrataChowdhury left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Please create a new issue so that we can use the conventional openapi specs. This will help in onboarding any new service as those won't have to implement additional unnecessary transformation logic.

@Sam007Yeah Sam007Yeah left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good changes wise.
Major issue is the communication with the client. After testing there seems to be some problems.

  1. I think you need to test it once from your side and observe what's happening. THe generated openAPI spec based on the generated DTO has some problems. the UI doesn't get the requested data through the API's and results in 500 errors throughout.
{
    "type": "about:blank",
    "title": "Internal Server Error",
    "status": 500,
    "detail": "Direct self-reference leading to cycle (through reference chain: java.util.ImmutableCollections$ListN[0]->com.autobuild.pipeline.definiton.gen.dto.Pipeline[\"unknownFields\"]->com.google.protobuf.UnknownFieldSet[\"defaultInstanceForType\"])",
    "instance": "/api/v1/pipeline"
}
  1. Even though the database saves these changes we can't see them on the UI becuase of the same issue.

Let's have a round of testing with the client once, labelling this as DO_NOT_MERGE for now till we find a solution

@Sam007Yeah Sam007Yeah added the do not merge This is an experimental PR that tests something. Do not merge the PR if it has this label. label Nov 12, 2026
@yash-khaitan

Copy link
Copy Markdown
Contributor Author

Everything looks good changes wise. Major issue is the communication with the client. After testing there seems to be some problems.

  1. I think you need to test it once from your side and observe what's happening. THe generated openAPI spec based on the generated DTO has some problems. the UI doesn't get the requested data through the API's and results in 500 errors throughout.
{
    "type": "about:blank",
    "title": "Internal Server Error",
    "status": 500,
    "detail": "Direct self-reference leading to cycle (through reference chain: java.util.ImmutableCollections$ListN[0]->com.autobuild.pipeline.definiton.gen.dto.Pipeline[\"unknownFields\"]->com.google.protobuf.UnknownFieldSet[\"defaultInstanceForType\"])",
    "instance": "/api/v1/pipeline"
}
  1. Even though the database saves these changes we can't see them on the UI becuase of the same issue.

Let's have a round of testing with the client once, labelling this as DO_NOT_MERGE for now till we find a solution

yess.. looking into it

@Sam007Yeah Sam007Yeah removed the do not merge This is an experimental PR that tests something. Do not merge the PR if it has this label. label Nov 12, 2026
@Sam007Yeah
Sam007Yeah self-requested a review November 12, 2026 08:13
Sam007Yeah
Sam007Yeah previously approved these changes Nov 12, 2026

@Sam007Yeah Sam007Yeah left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, @SuvabrataChowdhury will also check and see, we're almost done I guess

@SuvabrataChowdhury SuvabrataChowdhury left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested it out. It's working fine. Some suggestions.

  1. Can we keep the existing DTOs and Mappers? We need to unify the service structure and as this service exposes itself to UI through REST, we definitely at the edge of this service where the existing DTOs and Mappers will be required.
  2. Please Raise an Issue for future reference regarding clean contracts.

@yash-khaitan
yash-khaitan force-pushed the Introducing-protobuff-for-autoGeneration-of-DTO branch from 3c02cb2 to 4454403 Compare November 15, 2026 06:55

@SuvabrataChowdhury SuvabrataChowdhury left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM..

@yash-khaitan

Copy link
Copy Markdown
Contributor Author

I tested it out. It's working fine. Some suggestions.

  1. Can we keep the existing DTOs and Mappers? We need to unify the service structure and as this service exposes itself to UI through REST, we definitely at the edge of this service where the existing DTOs and Mappers will be required.
  2. Please Raise an Issue for future reference regarding clean contracts.

Done. #268

@yash-khaitan
yash-khaitan merged commit 15de62c into main Nov 15, 2026
2 checks passed
@yash-khaitan
yash-khaitan deleted the Introducing-protobuff-for-autoGeneration-of-DTO branch November 15, 2026 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants