Skip to content

Init message queue integration for async pipeline execution - #264

Open
WusatSami03 wants to merge 3 commits into
mainfrom
Init-Message-Queue
Open

WusatSami03 wants to merge 3 commits into
mainfrom
Init-Message-Queue

Conversation

@WusatSami03

Copy link
Copy Markdown
Contributor

SUMMARY

Introduces async pipeline execution via RabbitMQ message queue. The pipeline service now publishes the build ID to a queue instead of directly calling the executor, controlled by a feature flag.

Issue Link

> Link the Issue for which this PR was raised

Changes Summary

  1. Added ENABLE_MESSAGE_QUEUE feature flag (default: off) via OpenFeature/flagd
  2. Added MessagingConfig — declares the pipeline-build-queue RabbitMQ queue
  3. Added PipelineBuildMessagePublisher — publishes build ID to the queue after transaction commit
  4. Added PipelineBuildMessageConsumer — consumes build ID from the queue and triggers execution
  5. Modified PipelineExecutorService — when flag is ON, publishes to queue; when OFF, uses existing direct execution path
  6. Added findByIdWithStageBuilds to PipelineBuildRepository to eagerly fetch stage builds for the consumer
  7. Added RabbitMQ service to docker-compose.yml
  8. Added spring.rabbitmq.* config to application-demo.yaml
  9. Excluded RabbitAutoConfiguration in application-test.yaml so unit tests don't require a broker

Testing Done

  1. Tested end-to-end locally with Docker Compose: created a pipeline via UI, triggered a build via Bruno, verified message published and consumed in RabbitMQ management UI, build reached SUCCESS state
  2. Verified flag OFF path still works (existing direct execution unchanged)

Pull Request Checklist

Tasks Yes/No
Issue Linked? Yes
Timeline of the Issue is updated?
Do these changes affect local development in anyway? (if yes mention effects in additional comments)

Additional Comments

Anything else regarding this PR

@Sam007Yeah Sam007Yeah left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor changes, mainly I'm a little concerned about the architecture a little and the flow of the entire thing, would mainly be able to test once we see the consumer side outside of this service.

/**
* Consumes pipeline build messages from the queue and triggers execution.
*
* @author Suvabrata Chowdhury

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same thing here...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done...🙂

/**
* Publishes pipeline build IDs to the message queue for async execution.
*
* @author Suvabrata Chowdhury

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done...🙂

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to the architecture, are we consuming the messages here ? Please check once

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed it.

* @param id the pipeline build UUID
* @return an Optional containing the build with stage builds loaded
*/
@Query("SELECT pb FROM PipelineBuild pb JOIN FETCH pb.stageBuilds WHERE pb.id = :id")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you test this once?, I have doubts...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted it.

@SuvabrataChowdhury SuvabrataChowdhury left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the architecture was not clear. The idea is to decouple the PipelineBuild's management from this java service. The java service owns only the definition of a pipeline i.e., Pipeline entity.

On execute request of Pipeline, the java service should put the pipeline's definition in the BUILD_REQUEST_QUEUE and return a generated build id. It will be the subscriber's responsibility to pick it up and start the build.

Comment thread flags/feature-flags.json Outdated
},
"defaultVariant": "off"
},
"ENABLE_MESSAGE_QUEUE": {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not have it as message queue specific. It's for onboarding the new microservice and the whole flow to the go service should be controlled by this flag. Maybe we can call it ENABLE_NEW_ARCHITECTURE or something like that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

public enum FeatureFlag {
ENABLE_EDIT_PIPELINE("ENABLE_EDIT_PIPELINE");
ENABLE_EDIT_PIPELINE("ENABLE_EDIT_PIPELINE"),
ENABLE_MESSAGE_QUEUE("ENABLE_MESSAGE_QUEUE");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here.. please change the flag's name.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

/**
* RabbitMQ messaging configuration.
*
* @author Suvabrata Chowdhury

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait me 😅? put your name as author or omit it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done...🙂

import org.springframework.context.annotation.Configuration;

/**
* RabbitMQ messaging configuration.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not write specific config here.. RabbitMQ is the implementation details. I see this class is a simple spring config so lets not mention RabbitMQ here.

Suggested change
* RabbitMQ messaging configuration.
Message queue configuration

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

public class MessagingConfig {

/** Name of the queue used to dispatch pipeline build execution requests. */
public static final String PIPELINE_BUILD_QUEUE = "pipeline-build-queue";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more like a build request than the build itself.

Suggested change
public static final String PIPELINE_BUILD_QUEUE = "pipeline-build-queue";
public static final String PIPELINE_BUILD_REQUEST_QUEUE = "pipeline-build-request-queue";

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

*
* @param buildId the ID of the saved {@code PipelineBuild} to execute
*/
public void publish(UUID buildId) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not passing the build ID btw. This service owns the pipeline entity and operations related to the pipeline entity. On the execution operation on a pipeline, this service should put the pipeline's definition on the queue and forgets about it.

So the go service is the ultimate owner of the pipelineBuild entity which acts as the consumer in this scenario.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@Override
public void afterCommit() {
pipelineExecutor.executePipeline(savedBuild);
if (featureFlagService.getBooleanValue(FeatureFlag.ENABLE_MESSAGE_QUEUE.getKey())) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not here. the flag check should do the following,

  • If it's enabled then,
    • Just put the Pipeline definition and build id on the queue and return with build id. Ideally the subscriber will pickup this request and start the build.
  • else
    • Do the previous flow i.e., let the java service handle the build.

Idea is to decouple the whole build cycle from this service.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

exclude:
- org.springframework.boot.autoconfigure.h2.H2ConsoleAutoConfiguration

rabbitmq:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is specific to rabbitmq. We can't easily switch to new queues if configurations are mentioned like this. Check How the db configurations are defines.. those take a simple url. Can we have something like that?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment thread docker-compose.yml Outdated
AUTH_PORT: 8080
OTEL_SERVICE_NAME: pipeline-app
FEATURE_PROVIDER_URI: http://flagd:8016
RABBITMQ_HOST: rabbitmq

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be the env var. Check if it's possible to just pass the url. Make configurations generic.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@SuvabrataChowdhury SuvabrataChowdhury left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes are better now. Please keep configuration names generic so that we can swap vendors in future. Also even if we are moving to this new queue based implementation we still need to support existing os native script functionality till the new service is fully introduced; therefore the entirety of existing logic should be behind the feature flag so that we do not hamper anyone's development.

application/json:
schema:
$ref: "#/components/schemas/PipelineBuild"
type: object

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't it point to PipelineBuild?

required: true
responses:
"203":
description: Non-Authoritative Information

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting.. didn't know it was set to 203 in the openapi spec 😅

}

@Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have it as a conditional bean? like if rabbitmq is being used then only create this bean. Otherwise an unnecessary bean will remain in spring pool.

PipelineBuildDTO build = service.executePipeline(request);

return ResponseEntity.accepted().body(build);
public ResponseEntity<Map<String, UUID>> executePipeline(@RequestBody @Valid PipelineExecuteRequest request) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get the Map<String,UUID> response.. Please have it as some DTO so that both OpenAPI spec and UI objects have a predictable schema.

Please test with UI as well.. I feel this might break the execute flow as we were previously returning the full build object but now we are returning just the UUID.


@Override
public void publish(PipelineBuildRequestMessage message) {
rabbitTemplate.convertAndSend(MessagingConfig.PIPELINE_BUILD_REQUEST_QUEUE, message);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it throw some exception? please check if any runtime exception it throws that needs to be handled properly.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might throw AmqpException so adding a check will help

@Getter
@Setter
public class StageDefinition {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add stageId also here for reference purpose.

Comment on lines -66 to -107
PipelineBuild pipelineBuild = createBuild(pipeline);

createEmptyLogFiles(pipelineBuild);

PipelineBuild savedBuild = buildRepository.save(pipelineBuild);
UUID buildId = UUID.randomUUID();

// Strictly run executing pipeline after build is commited
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
@Override
public void afterCommit() {
pipelineExecutor.executePipeline(savedBuild);
}
});

return mapper.entityToDto(savedBuild);
}

private void createEmptyLogFiles(PipelineBuild pipelineBuild) throws IOException {
for (StageBuild stageBuild : pipelineBuild.getStageBuilds()) {
try {
stageBuild.setLogPath(pipelineFileService.createLogFile(pipelineBuild.getId(), stageBuild.getId()));
} catch (IOException e) {
pipelineFileService.removeLogFiles(pipelineBuild);
}
if (featureFlagService.getBooleanValue(FeatureFlag.ENABLE_NEW_ARCHITECTURE.getKey())) {
PipelineBuildRequestMessage message = buildMessage(buildId, pipeline);
messagePublisher.publish(message);
} else {
pipelineExecutor.executePipeline(null);
}
}

// TODO: abstract this logic
private PipelineBuild createBuild(final Pipeline pipeline) {
List<StageBuild> stageBuilds = pipeline.getStages()
.stream()
.map(stage -> {
StageBuild stageBuild = new StageBuild(stage);
stageBuild.setId(UUID.randomUUID());

return stageBuild;
}).toList();

PipelineBuild pipelineBuild = new PipelineBuild(pipeline, stageBuilds);
pipelineBuild.setId(UUID.randomUUID());
return buildId;
}

return pipelineBuild;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this logic be behind the feature flag? so if the feature flag of it won't be able to do it's existing operation right?

exclude:
- org.springframework.boot.autoconfigure.h2.H2ConsoleAutoConfiguration

rabbitmq:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the property name has to be rabbitmq? can we call it something generic and inject it as value? It's good that url is a generic one but the property name also should be generic.

Comment thread docker-compose.yml
AUTH_PORT: 8080
OTEL_SERVICE_NAME: pipeline-app
FEATURE_PROVIDER_URI: http://flagd:8016
RABBITMQ_URL: amqp://guest:guest@rabbitmq:5672

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's call the env a generic name.. something like MQ_URL. This will help us swap the queue vendor in future if required.

private PipelineFileService pipelineFileService;
private PipelineExecutor pipelineExecutor;

@Transactional

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not remove Transactional as it's still required for current implementation.

return mapper.entityToDto(savedBuild);
}

private void createEmptyLogFiles(PipelineBuild pipelineBuild) throws IOException {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have we removed these methods? Kindly check the changes

@Baibhab23 Baibhab23 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once fixed the comments kindly resolve them, i see a lot of outdated comments

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants