Skip to content

Druv lodu #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NakulChauhan2001 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Druv lodu #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ce86fd3
fixed rakshit shit code--dhrv mkc
NakulChauhan2001 Mar 5, 2025
27e1597
fixed rakshit code 3.0 rakshit sucks
NakulChauhan2001 Mar 7, 2025
a94c13a
fixed minor mistakes
NakulChauhan2001 Mar 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,19 @@ docker compose -f docker-compose.dev.yml up --build
# Run database migrations
docker exec -i postgres-db psql -U admin -d taskmaster < db/schema.sql


#connect to postgres cli
docker exec -it postgres-db psql -U admin -d taskmaster


# Run tests
go test ./... -v
```

## 📋 Implementation Progress

### 1. Core Infrastructure ✅

- [x] Project structure and module setup
- [x] PostgreSQL database integration
- [x] Redis for state management
Expand All @@ -30,6 +36,7 @@ go test ./... -v
### 2. Job Processing System 🚧

#### Basic Features ✅

- [x] REST API endpoints (Fiber)
- [x] Job creation and storage
- [x] Kafka producer implementation
Expand All @@ -38,6 +45,7 @@ go test ./... -v
- [x] Simple retry mechanism

#### Core Processing Features 🚧

- [ ] Job type registry system
- [ ] Payload validation
- [ ] Configurable retry policies
Expand All @@ -50,6 +58,7 @@ go test ./... -v
- [ ] Result storage

#### Advanced Processing Features 🚧

- [ ] Distributed locking
- [ ] Job batching
- [ ] Workflow engine
Expand All @@ -62,6 +71,7 @@ go test ./... -v
### 3. Developer Experience 🚧

#### Documentation & Tools

- [ ] Job type documentation
- [ ] Debugging tools
- [ ] Testing/simulation tools
Expand All @@ -70,6 +80,7 @@ go test ./... -v
- [ ] Hooks/middleware system

#### Web Dashboard

- [ ] React + Tailwind UI
- [ ] Real-time updates (WebSocket)
- [ ] Job filtering and search
Expand All @@ -81,6 +92,7 @@ go test ./... -v
### 4. Operations & Monitoring 🚧

#### Observability

- [x] Structured logging (slog)
- [ ] Prometheus metrics
- [ ] Grafana dashboards
Expand All @@ -89,6 +101,7 @@ go test ./... -v
- [ ] Resource monitoring

#### Operational Tools

- [ ] Job archival
- [ ] Cleanup policies
- [ ] Audit logging
Expand All @@ -100,6 +113,7 @@ go test ./... -v
### 5. Production Deployment 🚧

#### Infrastructure

- [ ] Kubernetes setup
- [ ] Helm charts
- [ ] Worker auto-scaling
Expand All @@ -108,6 +122,7 @@ go test ./... -v
- [ ] Blue-green deployments

#### Cloud Integration

- [ ] Terraform configurations
- [ ] AWS/GCP deployment
- [ ] Cost optimization
Expand All @@ -116,17 +131,21 @@ go test ./... -v
## 🔄 System Architecture

### Components

1. **API Service**

- REST/gRPC endpoints
- Request validation
- Job creation & queuing

2. **Message Queue**

- Kafka-based processing
- Job distribution
- Order guarantee

3. **Worker Service**

- Job execution
- Status management
- Error handling
Expand All @@ -137,6 +156,7 @@ go test ./... -v
- Kafka: Message queue

### Basic Job Flow

1. Submit job via API
2. Store in PostgreSQL
3. Queue in Kafka
Expand All @@ -147,11 +167,13 @@ go test ./... -v
## 🛠 Development

### Prerequisites

- Go 1.23+
- Docker & Docker Compose
- Make (optional)

### Environment Setup

```bash
DATABASE_URL=postgres://admin:admin@postgres-db:5432/taskmaster?sslmode=disable
KAFKA_BROKER=kafka:9092
Expand All @@ -160,6 +182,7 @@ JWT_SECRET=supersecretkey
```

### API Examples

```bash
# Authentication
POST /api/login
Expand All @@ -184,11 +207,13 @@ GET /api/jobs
```

## 📝 Contributing

1. Fork repository
2. Create feature branch
3. Commit changes
4. Push to branch
5. Open Pull Request

## 📄 License
MIT License

MIT License
3 changes: 2 additions & 1 deletion db/schema.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@ CREATE TABLE IF NOT EXISTS jobs (
status TEXT CHECK (status IN ('pending', 'processing', 'completed', 'failed')) DEFAULT 'pending',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
type TEXT NOT NULL,
payload JSON
payload JSON,
response JSON
);
8 changes: 6 additions & 2 deletions docker-compose.dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
version: '3.9'
version: "3.9"

services:
postgres:
Expand Down Expand Up @@ -73,6 +73,8 @@ services:
REDIS_ADDR: ${REDIS_ADDR}
GO_ENV: ${GO_ENV}
JWT_SECRET: ${JWT_SECRET}
GEMINI_API_KEY: ${GEMINI_API_KEY}
LLAMA_API_KEY: ${LLAMA_API_KEY}
Comment on lines +76 to +77
Copy link

@gemini-code-assist gemini-code-assist bot Mar 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Storing API keys directly in docker-compose.dev.yml is insecure. Consider using a more secure method, such as a secrets management solution or environment variables that are not checked into version control.

# Consider using a .env file or a secrets management solution
# GEMINI_API_KEY: ${GEMINI_API_KEY}
# LLAMA_API_KEY: ${LLAMA_API_KEY}

networks:
- app-network-dev
depends_on:
Expand All @@ -91,6 +93,8 @@ services:
KAFKA_BROKER: ${KAFKA_BROKER}
REDIS_ADDR: ${REDIS_ADDR}
GO_ENV: ${GO_ENV}
GEMINI_API_KEY: ${GEMINI_API_KEY}
LLAMA_API_KEY: ${LLAMA_API_KEY}
restart: on-failure
networks:
- app-network-dev
Expand All @@ -103,4 +107,4 @@ networks:

volumes:
pgdata-dev:
go-modules:
go-modules:
30 changes: 30 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ require (
)

require (
cloud.google.com/go v0.115.0 // indirect
cloud.google.com/go/ai v0.8.0 // indirect
cloud.google.com/go/auth v0.6.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.2 // indirect
cloud.google.com/go/compute/metadata v0.3.0 // indirect
cloud.google.com/go/longrunning v0.5.7 // indirect
github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302 // indirect
github.com/andybalholm/brotli v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
Expand All @@ -26,9 +32,18 @@ require (
github.com/eapache/go-resiliency v1.7.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/generative-ai-go v0.19.0 // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.5 // indirect
github.com/hashicorp/errwrap v1.0.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
Expand Down Expand Up @@ -58,9 +73,24 @@ require (
github.com/valyala/fasthttp v1.51.0 // indirect
github.com/valyala/tcplisten v1.0.0 // indirect
github.com/yuin/gopher-lua v1.1.1 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.51.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.51.0 // indirect
go.opentelemetry.io/otel v1.26.0 // indirect
go.opentelemetry.io/otel/metric v1.26.0 // indirect
go.opentelemetry.io/otel/trace v1.26.0 // indirect
golang.org/x/crypto v0.32.0 // indirect
golang.org/x/net v0.34.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/sync v0.10.0 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/text v0.21.0 // indirect
golang.org/x/time v0.5.0 // indirect
google.golang.org/api v0.186.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240617180043-68d350f18fd4 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240617180043-68d350f18fd4 // indirect
google.golang.org/grpc v1.64.1 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect

)
Loading