bugblitz77/jobs-monorepo
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

remove readme

Browse source
This commit is contained in:
Elshimy Ziad Magdy Taha 2025-09-28 21:05:46 +05:00
parent d637f5f112
commit a0e1728276
1 changed files with 6 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
README.md
View file

@ -1,4 +1,4 @@
# LinkedIn Jobs Scraper <!-- # LinkedIn Jobs Scraper
A robust Go application that scrapes LinkedIn job postings with **retry logic**, **exponential backoff**, and **concurrent processing** using a streaming pipeline architecture. A robust Go application that scrapes LinkedIn job postings with **retry logic**, **exponential backoff**, and **concurrent processing** using a streaming pipeline architecture.
@ -218,7 +218,7 @@ err = jobPipeline.ProcessJobsStreaming(ctx, 10, jobRepo, jobDescRepo, searchPara
### Before Optimization (Sequential + No Retries) ### Before Optimization (Sequential + No Retries)
- Scrape 100 jobs: ~2 minutes - Scrape 100 jobs: ~2 minutes
- Process descriptions: ~5 minutes - Process descriptions: ~5 minutes
- **Failures**: High failure rate due to network issues - **Failures**: High failure rate due to network issues
- **Total: ~7+ minutes** (with manual retries) - **Total: ~7+ minutes** (with manual retries)
@ -230,7 +230,7 @@ err = jobPipeline.ProcessJobsStreaming(ctx, 10, jobRepo, jobDescRepo, searchPara
### Key Improvements ### Key Improvements
1. **Streaming Architecture**: Jobs processed immediately as scraped 1. **Streaming Architecture**: Jobs processed immediately as scraped
2. **Concurrent Workers**: 5 workers processing descriptions simultaneously 2. **Concurrent Workers**: 5 workers processing descriptions simultaneously
3. **Intelligent Retries**: Automatic retry with exponential backoff 3. **Intelligent Retries**: Automatic retry with exponential backoff
4. **Resilient Network Handling**: Graceful handling of temporary failures 4. **Resilient Network Handling**: Graceful handling of temporary failures
5. **Smart Error Classification**: Skip permanent errors, retry temporary ones 5. **Smart Error Classification**: Skip permanent errors, retry temporary ones
@ -252,7 +252,7 @@ Saved 250 job descriptions to database
``` ```
Request attempt 1 failed with status 503 Request attempt 1 failed with status 503
Retrying in 1s... (attempt 1/3) Retrying in 1s... (attempt 1/3)
Request attempt 2 failed with status 502 Request attempt 2 failed with status 502
Retrying in 2s... (attempt 2/3) Retrying in 2s... (attempt 2/3)
Request attempt 3 succeeded Request attempt 3 succeeded
``` ```
@ -273,7 +273,7 @@ Request attempt 3 succeeded
- Check if IP is temporarily blocked - Check if IP is temporarily blocked
- Verify User-Agent header is set correctly - Verify User-Agent header is set correctly
**Database Connection Issues** **Database Connection Issues**
- Verify PostgreSQL is running: `pg_ctl status` - Verify PostgreSQL is running: `pg_ctl status`
- Check connection string in `.env` file - Check connection string in `.env` file
- Ensure database exists and migrations ran - Ensure database exists and migrations ran
@ -330,4 +330,4 @@ This project is licensed under the MIT License - see the LICENSE file for detail
## ⚠️ Disclaimer ## ⚠️ Disclaimer
This tool is for educational purposes. Please respect LinkedIn's Terms of Service and robots.txt. Use responsibly and consider rate limiting to avoid being blocked. This tool is for educational purposes. Please respect LinkedIn's Terms of Service and robots.txt. Use responsibly and consider rate limiting to avoid being blocked. -->