Jobs Scraper Monorepo

A modern monorepo architecture for job scraping and analysis services.

🏗️ Architecture

jobs-scraper/
├── go.mod                    # Root go.mod with module name
├── Makefile                  # Common build/test tasks
├── .github/workflows/        # CI/CD pipelines
├── tools/                    # Build tools and utilities
├── internal/                 # Private shared packages
│   ├── pkg/
│   │   ├── domain/          # Core domain models
│   │   ├── infrastructure/  # Database, RabbitMQ, HTTP
│   │   └── utils/           # Common utilities
│   └── migrations/          # Database migrations
├── libs/                     # Shared libraries
│   ├── ports/               # Command interfaces
│   ├── repo/                # Data repositories
│   └── server/              # Server utilities
├── services/                 # Deployable services
│   ├── api/                 # HTTP API service
│   ├── scraper-linkedin/    # LinkedIn scraper service
│   └── scraper-glassdoor/   # Glassdoor scraper service
├── apps/                     # Client applications
│   ├── cron-analyzer/       # Job analysis cron
│   └── cv-analyzer/         # CV analysis utility
└── scripts/                  # Deployment and utility scripts

🚀 Services

API Service (services/api/)

• Purpose: HTTP API server with CQRS pattern
• Tech Stack: Go, Gorilla Mux, PostgreSQL, RabbitMQ
• Features: Job management, Swagger documentation, CORS support

LinkedIn Scraper (services/scraper-linkedin/)

• Purpose: Scrapes job postings from LinkedIn
• Tech Stack: Go, Playwright
• Features: Automated job scraping, RabbitMQ integration

Glassdoor Scraper (services/scraper-glassdoor/)

• Purpose: Scrapes job postings from Glassdoor
• Tech Stack: TypeScript, Node.js
• Features: Web scraping, data processing

📱 Applications

Cron Analyzer (apps/cron-analyzer/)

• Purpose: Analyzes jobs with AI on a schedule
• Tech Stack: Go, OpenRouter API
• Features: Job analysis, status updates

CV Analyzer (apps/cv-analyzer/)

• Purpose: Analyzes CVs against job requirements
• Tech Stack: Go, OpenRouter API
• Features: CV matching, skill analysis

🛠️ Development

Prerequisites

• Go 1.24+
• Node.js 18+
• Docker & Docker Compose
• PostgreSQL 15+
• RabbitMQ 3+

Quick Start

1. Clone the repository

   git clone <repository-url>
   cd jobs-scraper
   

2. Start infrastructure

   make rabbitmq-server
   # In another terminal:
   docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:15
   

3. Run services

   # API Service
   make run-api
   
   # LinkedIn Scraper
   make run-scraper-linkedin
   
   # Glassdoor Scraper
   make run-scraper-glassdoor
   
   # Cron Analyzer
   make run-cron-analyzer
   

Available Commands

Services

• make run-api - Run API service
• make build-api - Build API service
• make run-scraper-linkedin - Run LinkedIn scraper
• make build-scraper-linkedin - Build LinkedIn scraper
• make run-scraper-glassdoor - Run Glassdoor scraper
• make build-scraper-glassdoor - Build Glassdoor scraper

Applications

• make run-cron-analyzer - Run cron analyzer
• make build-cron-analyzer - Build cron analyzer
• make run-cv-analyzer - Run CV analyzer
• make build-cv-analyzer - Build CV analyzer

Infrastructure

• make rabbitmq-server - Start RabbitMQ server
• make swagger - Generate API documentation
• make clean - Clean build artifacts

Legacy Aliases (Backward Compatibility)

• make run - Alias for run-api
• make scraper - Alias for run-scraper-linkedin
• make cron - Alias for run-cron-analyzer

🐳 Docker

Each service has its own Dockerfile and can be built independently:

# Build API service
cd services/api
docker build -t jobs-scraper-api .

# Build LinkedIn scraper
cd services/scraper-linkedin
docker build -t jobs-scraper-linkedin .

🔧 Configuration

Environment variables are managed through .env files:

• .env - Production configuration
• .local.env - Local development configuration

🧪 Testing

Run tests for individual services:

# API Service
cd services/api && go test ./...

# LinkedIn Scraper
cd services/scraper-linkedin && go test ./...

# Cron Analyzer
cd apps/cron-analyzer && go test ./...

📊 Monitoring

• API Documentation: http://localhost:8080/swagger/
• RabbitMQ Management: http://localhost:15672 (guest/guest)
• Health Checks: Built into each service

🚀 Deployment

The monorepo includes GitHub Actions workflows for:

• Automated testing
• Docker image building
• Multi-service deployment

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

[Add your license information here]

🆘 Support

For support and questions, please create an issue or contact the development team.