184 lines
6 KiB
Markdown
184 lines
6 KiB
Markdown
# NATS to RabbitMQ Migration Summary
|
|
|
|
## Migration Overview
|
|
|
|
Successfully migrated the jobs-scraper project from NATS JetStream to RabbitMQ message queue system.
|
|
|
|
## Changes Made
|
|
|
|
### 1. Dependencies Updated
|
|
- **Removed**: `github.com/nats-io/nats.go v1.46.1`
|
|
- **Added**: `github.com/rabbitmq/amqp091-go v1.10.0`
|
|
- **Cleaned up**: Removed NATS-related indirect dependencies (`nats-io/nkeys`, `nats-io/nuid`)
|
|
|
|
### 2. Infrastructure Changes
|
|
- **Directory**: Renamed `infrastructure/nats/` → `infrastructure/rabbitmq/`
|
|
- **File**: Renamed `nats.go` → `rabbitmq.go`
|
|
- **Client**: `NatsClient` → `RabbitMQClient`
|
|
|
|
### 3. RabbitMQ Implementation Features
|
|
|
|
#### Message Queue Architecture
|
|
- **Main Exchange**: `scraper_exchange` (topic exchange)
|
|
- **Dead Letter Exchange**: `scraper_dlx` (direct exchange)
|
|
- **Queues**:
|
|
- `scraper.linkedin`
|
|
- `scraper.indeed`
|
|
- `scraper.bayt`
|
|
- `scraper.tokyodev`
|
|
- `scraper.japandev`
|
|
- **Dead Letter Queues**: Each queue has corresponding `_dlq` for failed messages
|
|
|
|
#### Production-Ready Features
|
|
- **Durable Queues**: Messages persist across server restarts
|
|
- **Message Persistence**: `DeliveryMode: amqp.Persistent`
|
|
- **Manual Acknowledgment**: Messages only removed after successful processing
|
|
- **Dead Letter Exchange**: Failed messages routed to DLX after max retries
|
|
- **Message TTL**: 24-hour expiration to prevent queue buildup
|
|
- **QoS Control**: Prefetch count of 1 for controlled message delivery
|
|
- **Retry Logic**: Up to 3 delivery attempts before DLX routing
|
|
|
|
### 4. Code Changes
|
|
|
|
#### Files Modified
|
|
1. **`api/main.go`**
|
|
- Import: `infrastructure/nats` → `infrastructure/rabbitmq`
|
|
- Client: `nats.NewNatsClient()` → `rabbitmq.NewRabbitMQClient()`
|
|
- Variable: `nc` → `rmq`
|
|
|
|
2. **`api/app/app.go`**
|
|
- Import: `infrastructure/nats` → `infrastructure/rabbitmq`
|
|
- Parameter: `*nats.NatsClient` → `*rabbitmq.RabbitMQClient`
|
|
- Variable: `nc` → `rmq`
|
|
|
|
3. **`api/commands/job/create-job.go`**
|
|
- Import: `infrastructure/nats` → `infrastructure/rabbitmq`
|
|
- Struct field: `nc *nats.NatsClient` → `rmq *rabbitmq.RabbitMQClient`
|
|
- Method call: `nats.LinkedInSubTopic` → `rabbitmq.LinkedInQueue`
|
|
|
|
4. **`scraper/main.go`**
|
|
- Import: `infrastructure/nats` → `infrastructure/rabbitmq`
|
|
- Removed: `github.com/nats-io/nats.go` import
|
|
- Client: `NewNatsClient()` → `NewRabbitMQClient()`
|
|
- Subscribe method: Changed from NATS message handler to RabbitMQ message handler
|
|
- Handler signature: `func(msg *nats.Msg)` → `func(data []byte) error`
|
|
|
|
5. **`inspect_nats.go` → `inspect_rabbitmq.go`**
|
|
- Complete rewrite for RabbitMQ queue inspection
|
|
- Features: Queue status, message counts, dead letter queue monitoring
|
|
|
|
6. **`Makefile`**
|
|
- Target: `nats-server` → `rabbitmq-server`
|
|
- Command: NATS Docker command → RabbitMQ Docker command
|
|
|
|
### 5. Handler Interface Changes
|
|
|
|
#### NATS Handler (Old)
|
|
```go
|
|
func(msg *nats.Msg) {
|
|
// Process msg.Data
|
|
msg.Ack() // or msg.Nak()
|
|
}
|
|
```
|
|
|
|
#### RabbitMQ Handler (New)
|
|
```go
|
|
func(data []byte) error {
|
|
// Process data
|
|
return nil // or return error for retry
|
|
}
|
|
```
|
|
|
|
### 6. Configuration Changes
|
|
|
|
#### Environment Variables
|
|
- **Added**: `RABBITMQ_URL=amqp://guest:guest@localhost:5672/`
|
|
- **Default**: Falls back to localhost if not set
|
|
|
|
#### Connection Settings
|
|
- **Retry Logic**: 5 connection attempts with 5-second delays
|
|
- **Auto-reconnect**: Built into RabbitMQ client
|
|
- **Channel Management**: Single channel per client instance
|
|
|
|
## Migration Benefits
|
|
|
|
### 1. Enhanced Reliability
|
|
- **Message Persistence**: Messages survive server restarts
|
|
- **Dead Letter Exchange**: Failed messages captured for analysis
|
|
- **Durable Queues**: Queue definitions persist across restarts
|
|
- **Manual Acknowledgment**: Prevents message loss
|
|
|
|
### 2. Better Monitoring
|
|
- **Management UI**: Web interface at http://localhost:15672
|
|
- **Queue Metrics**: Message counts, consumer counts, processing rates
|
|
- **Dead Letter Monitoring**: Track failed message patterns
|
|
- **Custom Inspection Tool**: `inspect_rabbitmq.go` for queue status
|
|
|
|
### 3. Production Readiness
|
|
- **Horizontal Scaling**: Multiple consumers per queue
|
|
- **Load Balancing**: Round-robin message distribution
|
|
- **Backpressure Control**: QoS prefetch limits
|
|
- **Message TTL**: Prevents infinite queue growth
|
|
|
|
### 4. Operational Improvements
|
|
- **Industry Standard**: RabbitMQ is widely adopted
|
|
- **Rich Ecosystem**: Extensive tooling and monitoring
|
|
- **Documentation**: Comprehensive official documentation
|
|
- **Community Support**: Large community and resources
|
|
|
|
## Testing Verification
|
|
|
|
✅ **Build Tests**: All components compile successfully
|
|
- `go build ./api/...` - ✅ Success
|
|
- `go build ./scraper` - ✅ Success
|
|
- `go build ./inspect_rabbitmq.go` - ✅ Success
|
|
|
|
✅ **Dependency Management**: `go mod tidy` completed without errors
|
|
|
|
✅ **Import Resolution**: All RabbitMQ imports resolve correctly
|
|
|
|
## Usage Instructions
|
|
|
|
### 1. Start RabbitMQ
|
|
```bash
|
|
make rabbitmq-server
|
|
# OR
|
|
docker run --rm -p 5672:5672 -p 15672:15672 rabbitmq:3-management
|
|
```
|
|
|
|
### 2. Start API Server
|
|
```bash
|
|
cd api && go run main.go
|
|
```
|
|
|
|
### 3. Start Scraper Worker
|
|
```bash
|
|
cd scraper && go run main.go
|
|
```
|
|
|
|
### 4. Monitor Queues
|
|
```bash
|
|
go run inspect_rabbitmq.go
|
|
# OR visit http://localhost:15672 (guest/guest)
|
|
```
|
|
|
|
## Rollback Plan
|
|
|
|
If rollback is needed:
|
|
1. Revert `go.mod` changes
|
|
2. Restore `infrastructure/nats/` directory
|
|
3. Revert all import statements
|
|
4. Restore NATS-specific handler signatures
|
|
5. Run `go mod tidy`
|
|
|
|
## Next Steps
|
|
|
|
1. **Test with RabbitMQ**: Start RabbitMQ server and test message flow
|
|
2. **Monitor Performance**: Compare performance with previous NATS implementation
|
|
3. **Configure Production**: Set up RabbitMQ cluster for production deployment
|
|
4. **Update Documentation**: Ensure all documentation reflects RabbitMQ usage
|
|
5. **Team Training**: Brief team on RabbitMQ management and monitoring
|
|
|
|
## Migration Status: ✅ COMPLETE
|
|
|
|
The migration from NATS to RabbitMQ has been successfully completed. All code compiles, dependencies are resolved, and the system is ready for testing with RabbitMQ.
|