8ac872bdc6
- Reorganized project structure into services/ and apps/ directories for better separation of concerns - Added comprehensive CI/CD pipeline with GitHub Actions for testing and Docker builds - Created .dockerignore file to optimize container builds - Updated Makefile with new targets for each service and application - Added detailed README with architecture overview, setup instructions and development guidelines - Moved cron-analyzer to dedicate
190 lines
5.1 KiB
Markdown
190 lines
5.1 KiB
Markdown
# Jobs Scraper Monorepo
|
|
|
|
A modern monorepo architecture for job scraping and analysis services.
|
|
|
|
## 🏗️ Architecture
|
|
|
|
```
|
|
jobs-scraper/
|
|
├── go.mod # Root go.mod with module name
|
|
├── Makefile # Common build/test tasks
|
|
├── .github/workflows/ # CI/CD pipelines
|
|
├── tools/ # Build tools and utilities
|
|
├── internal/ # Private shared packages
|
|
│ ├── pkg/
|
|
│ │ ├── domain/ # Core domain models
|
|
│ │ ├── infrastructure/ # Database, RabbitMQ, HTTP
|
|
│ │ └── utils/ # Common utilities
|
|
│ └── migrations/ # Database migrations
|
|
├── libs/ # Shared libraries
|
|
│ ├── ports/ # Command interfaces
|
|
│ ├── repo/ # Data repositories
|
|
│ └── server/ # Server utilities
|
|
├── services/ # Deployable services
|
|
│ ├── api/ # HTTP API service
|
|
│ ├── scraper-linkedin/ # LinkedIn scraper service
|
|
│ └── scraper-glassdoor/ # Glassdoor scraper service
|
|
├── apps/ # Client applications
|
|
│ ├── cron-analyzer/ # Job analysis cron
|
|
│ └── cv-analyzer/ # CV analysis utility
|
|
└── scripts/ # Deployment and utility scripts
|
|
```
|
|
|
|
## 🚀 Services
|
|
|
|
### API Service (`services/api/`)
|
|
- **Purpose**: HTTP API server with CQRS pattern
|
|
- **Tech Stack**: Go, Gorilla Mux, PostgreSQL, RabbitMQ
|
|
- **Features**: Job management, Swagger documentation, CORS support
|
|
|
|
### LinkedIn Scraper (`services/scraper-linkedin/`)
|
|
- **Purpose**: Scrapes job postings from LinkedIn
|
|
- **Tech Stack**: Go, Playwright
|
|
- **Features**: Automated job scraping, RabbitMQ integration
|
|
|
|
### Glassdoor Scraper (`services/scraper-glassdoor/`)
|
|
- **Purpose**: Scrapes job postings from Glassdoor
|
|
- **Tech Stack**: TypeScript, Node.js
|
|
- **Features**: Web scraping, data processing
|
|
|
|
## 📱 Applications
|
|
|
|
### Cron Analyzer (`apps/cron-analyzer/`)
|
|
- **Purpose**: Analyzes jobs with AI on a schedule
|
|
- **Tech Stack**: Go, OpenRouter API
|
|
- **Features**: Job analysis, status updates
|
|
|
|
### CV Analyzer (`apps/cv-analyzer/`)
|
|
- **Purpose**: Analyzes CVs against job requirements
|
|
- **Tech Stack**: Go, OpenRouter API
|
|
- **Features**: CV matching, skill analysis
|
|
|
|
## 🛠️ Development
|
|
|
|
### Prerequisites
|
|
- Go 1.24+
|
|
- Node.js 18+
|
|
- Docker & Docker Compose
|
|
- PostgreSQL 15+
|
|
- RabbitMQ 3+
|
|
|
|
### Quick Start
|
|
|
|
1. **Clone the repository**
|
|
```bash
|
|
git clone <repository-url>
|
|
cd jobs-scraper
|
|
```
|
|
|
|
2. **Start infrastructure**
|
|
```bash
|
|
make rabbitmq-server
|
|
# In another terminal:
|
|
docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:15
|
|
```
|
|
|
|
3. **Run services**
|
|
```bash
|
|
# API Service
|
|
make run-api
|
|
|
|
# LinkedIn Scraper
|
|
make run-scraper-linkedin
|
|
|
|
# Glassdoor Scraper
|
|
make run-scraper-glassdoor
|
|
|
|
# Cron Analyzer
|
|
make run-cron-analyzer
|
|
```
|
|
|
|
### Available Commands
|
|
|
|
#### Services
|
|
- `make run-api` - Run API service
|
|
- `make build-api` - Build API service
|
|
- `make run-scraper-linkedin` - Run LinkedIn scraper
|
|
- `make build-scraper-linkedin` - Build LinkedIn scraper
|
|
- `make run-scraper-glassdoor` - Run Glassdoor scraper
|
|
- `make build-scraper-glassdoor` - Build Glassdoor scraper
|
|
|
|
#### Applications
|
|
- `make run-cron-analyzer` - Run cron analyzer
|
|
- `make build-cron-analyzer` - Build cron analyzer
|
|
- `make run-cv-analyzer` - Run CV analyzer
|
|
- `make build-cv-analyzer` - Build CV analyzer
|
|
|
|
#### Infrastructure
|
|
- `make rabbitmq-server` - Start RabbitMQ server
|
|
- `make swagger` - Generate API documentation
|
|
- `make clean` - Clean build artifacts
|
|
|
|
#### Legacy Aliases (Backward Compatibility)
|
|
- `make run` - Alias for `run-api`
|
|
- `make scraper` - Alias for `run-scraper-linkedin`
|
|
- `make cron` - Alias for `run-cron-analyzer`
|
|
|
|
## 🐳 Docker
|
|
|
|
Each service has its own Dockerfile and can be built independently:
|
|
|
|
```bash
|
|
# Build API service
|
|
cd services/api
|
|
docker build -t jobs-scraper-api .
|
|
|
|
# Build LinkedIn scraper
|
|
cd services/scraper-linkedin
|
|
docker build -t jobs-scraper-linkedin .
|
|
```
|
|
|
|
## 🔧 Configuration
|
|
|
|
Environment variables are managed through `.env` files:
|
|
|
|
- `.env` - Production configuration
|
|
- `.local.env` - Local development configuration
|
|
|
|
## 🧪 Testing
|
|
|
|
Run tests for individual services:
|
|
|
|
```bash
|
|
# API Service
|
|
cd services/api && go test ./...
|
|
|
|
# LinkedIn Scraper
|
|
cd services/scraper-linkedin && go test ./...
|
|
|
|
# Cron Analyzer
|
|
cd apps/cron-analyzer && go test ./...
|
|
```
|
|
|
|
## 📊 Monitoring
|
|
|
|
- **API Documentation**: http://localhost:8080/swagger/
|
|
- **RabbitMQ Management**: http://localhost:15672 (guest/guest)
|
|
- **Health Checks**: Built into each service
|
|
|
|
## 🚀 Deployment
|
|
|
|
The monorepo includes GitHub Actions workflows for:
|
|
- Automated testing
|
|
- Docker image building
|
|
- Multi-service deployment
|
|
|
|
## 🤝 Contributing
|
|
|
|
1. Fork the repository
|
|
2. Create a feature branch
|
|
3. Make your changes
|
|
4. Add tests
|
|
5. Submit a pull request
|
|
|
|
## 📄 License
|
|
|
|
[Add your license information here]
|
|
|
|
## 🆘 Support
|
|
|
|
For support and questions, please [create an issue](issues) or contact the development team.
|