Repository navigation
Init message queue integration for async pipeline execution - #264
WusatSami03 wants to merge 3 commits into
Conversation
Sam007Yeah
left a comment
There was a problem hiding this comment.
Some minor changes, mainly I'm a little concerned about the architecture a little and the flow of the entire thing, would mainly be able to test once we see the consumer side outside of this service.
| /** | ||
| * Consumes pipeline build messages from the queue and triggers execution. | ||
| * | ||
| * @author Suvabrata Chowdhury |
| /** | ||
| * Publishes pipeline build IDs to the message queue for async execution. | ||
| * | ||
| * @author Suvabrata Chowdhury |
There was a problem hiding this comment.
according to the architecture, are we consuming the messages here ? Please check once
| * @param id the pipeline build UUID | ||
| * @return an Optional containing the build with stage builds loaded | ||
| */ | ||
| @Query("SELECT pb FROM PipelineBuild pb JOIN FETCH pb.stageBuilds WHERE pb.id = :id") |
There was a problem hiding this comment.
can you test this once?, I have doubts...
SuvabrataChowdhury
left a comment
There was a problem hiding this comment.
I believe the architecture was not clear. The idea is to decouple the PipelineBuild's management from this java service. The java service owns only the definition of a pipeline i.e., Pipeline entity.
On execute request of Pipeline, the java service should put the pipeline's definition in the BUILD_REQUEST_QUEUE and return a generated build id. It will be the subscriber's responsibility to pick it up and start the build.
| }, | ||
| "defaultVariant": "off" | ||
| }, | ||
| "ENABLE_MESSAGE_QUEUE": { |
There was a problem hiding this comment.
Let's not have it as message queue specific. It's for onboarding the new microservice and the whole flow to the go service should be controlled by this flag. Maybe we can call it ENABLE_NEW_ARCHITECTURE or something like that.
| public enum FeatureFlag { | ||
| ENABLE_EDIT_PIPELINE("ENABLE_EDIT_PIPELINE"); | ||
| ENABLE_EDIT_PIPELINE("ENABLE_EDIT_PIPELINE"), | ||
| ENABLE_MESSAGE_QUEUE("ENABLE_MESSAGE_QUEUE"); |
There was a problem hiding this comment.
same here.. please change the flag's name.
| /** | ||
| * RabbitMQ messaging configuration. | ||
| * | ||
| * @author Suvabrata Chowdhury |
There was a problem hiding this comment.
wait me 😅? put your name as author or omit it.
| import org.springframework.context.annotation.Configuration; | ||
|
|
||
| /** | ||
| * RabbitMQ messaging configuration. |
There was a problem hiding this comment.
Let's not write specific config here.. RabbitMQ is the implementation details. I see this class is a simple spring config so lets not mention RabbitMQ here.
| * RabbitMQ messaging configuration. | |
| Message queue configuration |
| public class MessagingConfig { | ||
|
|
||
| /** Name of the queue used to dispatch pipeline build execution requests. */ | ||
| public static final String PIPELINE_BUILD_QUEUE = "pipeline-build-queue"; |
There was a problem hiding this comment.
It's more like a build request than the build itself.
| public static final String PIPELINE_BUILD_QUEUE = "pipeline-build-queue"; | |
| public static final String PIPELINE_BUILD_REQUEST_QUEUE = "pipeline-build-request-queue"; |
| * | ||
| * @param buildId the ID of the saved {@code PipelineBuild} to execute | ||
| */ | ||
| public void publish(UUID buildId) { |
There was a problem hiding this comment.
We are not passing the build ID btw. This service owns the pipeline entity and operations related to the pipeline entity. On the execution operation on a pipeline, this service should put the pipeline's definition on the queue and forgets about it.
So the go service is the ultimate owner of the pipelineBuild entity which acts as the consumer in this scenario.
| @Override | ||
| public void afterCommit() { | ||
| pipelineExecutor.executePipeline(savedBuild); | ||
| if (featureFlagService.getBooleanValue(FeatureFlag.ENABLE_MESSAGE_QUEUE.getKey())) { |
There was a problem hiding this comment.
Not here. the flag check should do the following,
- If it's enabled then,
- Just put the Pipeline definition and build id on the queue and return with build id. Ideally the subscriber will pickup this request and start the build.
- else
- Do the previous flow i.e., let the java service handle the build.
Idea is to decouple the whole build cycle from this service.
| exclude: | ||
| - org.springframework.boot.autoconfigure.h2.H2ConsoleAutoConfiguration | ||
|
|
||
| rabbitmq: |
There was a problem hiding this comment.
This is specific to rabbitmq. We can't easily switch to new queues if configurations are mentioned like this. Check How the db configurations are defines.. those take a simple url. Can we have something like that?
| AUTH_PORT: 8080 | ||
| OTEL_SERVICE_NAME: pipeline-app | ||
| FEATURE_PROVIDER_URI: http://flagd:8016 | ||
| RABBITMQ_HOST: rabbitmq |
There was a problem hiding this comment.
This should not be the env var. Check if it's possible to just pass the url. Make configurations generic.
SuvabrataChowdhury
left a comment
There was a problem hiding this comment.
Changes are better now. Please keep configuration names generic so that we can swap vendors in future. Also even if we are moving to this new queue based implementation we still need to support existing os native script functionality till the new service is fully introduced; therefore the entirety of existing logic should be behind the feature flag so that we do not hamper anyone's development.
| application/json: | ||
| schema: | ||
| $ref: "#/components/schemas/PipelineBuild" | ||
| type: object |
There was a problem hiding this comment.
shouldn't it point to PipelineBuild?
| required: true | ||
| responses: | ||
| "203": | ||
| description: Non-Authoritative Information |
There was a problem hiding this comment.
interesting.. didn't know it was set to 203 in the openapi spec 😅
| } | ||
|
|
||
| @Bean | ||
| public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) { |
There was a problem hiding this comment.
can we have it as a conditional bean? like if rabbitmq is being used then only create this bean. Otherwise an unnecessary bean will remain in spring pool.
| PipelineBuildDTO build = service.executePipeline(request); | ||
|
|
||
| return ResponseEntity.accepted().body(build); | ||
| public ResponseEntity<Map<String, UUID>> executePipeline(@RequestBody @Valid PipelineExecuteRequest request) { |
There was a problem hiding this comment.
I get the Map<String,UUID> response.. Please have it as some DTO so that both OpenAPI spec and UI objects have a predictable schema.
Please test with UI as well.. I feel this might break the execute flow as we were previously returning the full build object but now we are returning just the UUID.
|
|
||
| @Override | ||
| public void publish(PipelineBuildRequestMessage message) { | ||
| rabbitTemplate.convertAndSend(MessagingConfig.PIPELINE_BUILD_REQUEST_QUEUE, message); |
There was a problem hiding this comment.
does it throw some exception? please check if any runtime exception it throws that needs to be handled properly.
There was a problem hiding this comment.
It might throw AmqpException so adding a check will help
| @Getter | ||
| @Setter | ||
| public class StageDefinition { | ||
|
|
There was a problem hiding this comment.
Let's add stageId also here for reference purpose.
| PipelineBuild pipelineBuild = createBuild(pipeline); | ||
|
|
||
| createEmptyLogFiles(pipelineBuild); | ||
|
|
||
| PipelineBuild savedBuild = buildRepository.save(pipelineBuild); | ||
| UUID buildId = UUID.randomUUID(); | ||
|
|
||
| // Strictly run executing pipeline after build is commited | ||
| TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() { | ||
| @Override | ||
| public void afterCommit() { | ||
| pipelineExecutor.executePipeline(savedBuild); | ||
| } | ||
| }); | ||
|
|
||
| return mapper.entityToDto(savedBuild); | ||
| } | ||
|
|
||
| private void createEmptyLogFiles(PipelineBuild pipelineBuild) throws IOException { | ||
| for (StageBuild stageBuild : pipelineBuild.getStageBuilds()) { | ||
| try { | ||
| stageBuild.setLogPath(pipelineFileService.createLogFile(pipelineBuild.getId(), stageBuild.getId())); | ||
| } catch (IOException e) { | ||
| pipelineFileService.removeLogFiles(pipelineBuild); | ||
| } | ||
| if (featureFlagService.getBooleanValue(FeatureFlag.ENABLE_NEW_ARCHITECTURE.getKey())) { | ||
| PipelineBuildRequestMessage message = buildMessage(buildId, pipeline); | ||
| messagePublisher.publish(message); | ||
| } else { | ||
| pipelineExecutor.executePipeline(null); | ||
| } | ||
| } | ||
|
|
||
| // TODO: abstract this logic | ||
| private PipelineBuild createBuild(final Pipeline pipeline) { | ||
| List<StageBuild> stageBuilds = pipeline.getStages() | ||
| .stream() | ||
| .map(stage -> { | ||
| StageBuild stageBuild = new StageBuild(stage); | ||
| stageBuild.setId(UUID.randomUUID()); | ||
|
|
||
| return stageBuild; | ||
| }).toList(); | ||
|
|
||
| PipelineBuild pipelineBuild = new PipelineBuild(pipeline, stageBuilds); | ||
| pipelineBuild.setId(UUID.randomUUID()); | ||
| return buildId; | ||
| } | ||
|
|
||
| return pipelineBuild; |
There was a problem hiding this comment.
Shouldn't this logic be behind the feature flag? so if the feature flag of it won't be able to do it's existing operation right?
| exclude: | ||
| - org.springframework.boot.autoconfigure.h2.H2ConsoleAutoConfiguration | ||
|
|
||
| rabbitmq: |
There was a problem hiding this comment.
Does the property name has to be rabbitmq? can we call it something generic and inject it as value? It's good that url is a generic one but the property name also should be generic.
| AUTH_PORT: 8080 | ||
| OTEL_SERVICE_NAME: pipeline-app | ||
| FEATURE_PROVIDER_URI: http://flagd:8016 | ||
| RABBITMQ_URL: amqp://guest:guest@rabbitmq:5672 |
There was a problem hiding this comment.
Let's call the env a generic name.. something like MQ_URL. This will help us swap the queue vendor in future if required.
| private PipelineFileService pipelineFileService; | ||
| private PipelineExecutor pipelineExecutor; | ||
|
|
||
| @Transactional |
There was a problem hiding this comment.
Do not remove Transactional as it's still required for current implementation.
| return mapper.entityToDto(savedBuild); | ||
| } | ||
|
|
||
| private void createEmptyLogFiles(PipelineBuild pipelineBuild) throws IOException { |
There was a problem hiding this comment.
Why have we removed these methods? Kindly check the changes
Baibhab23
left a comment
There was a problem hiding this comment.
Once fixed the comments kindly resolve them, i see a lot of outdated comments
SUMMARY
Introduces async pipeline execution via RabbitMQ message queue. The pipeline service now publishes the build ID to a queue instead of directly calling the executor, controlled by a feature flag.
Issue Link
> Link the Issue for which this PR was raised
Changes Summary
Testing Done
Pull Request Checklist
Additional Comments