divar-airflow-pipeline

This project is a simple Airflow DAG that fetches the list of cities from Divar API and saves them as a CSV file.
🌐 API Source: Divar API

---

🚀 Features

• PythonOperator is used to call the API and process the data.
• DAG has been tested on a Linux environment and confirmed to work correctly.
• CSV output is stored in data/divar_cities.csv.
• The data/ folder includes .gitkeep to maintain the folder structure in GitHub.

Note: This project is mainly educational and serves as a sample Airflow workflow, ready for further development or database integration.

---

📁 Project Structure

divar-airflow-pipeline/
│
├── dags/
│   └── divar_city_list_pipeline.py
│
├── data/
│   └── .gitkeep
│
├── requirements.txt
└── README.md

---

📄 Sample CSV Output

ID	Name	Slug	Radius	Latitude	Longitude
1	Tehran	tehran	50000	35.6892	51.3890
2	Mashhad	mashhad	30000	36.2605	59.6168
3	Isfahan	isfahan	40000	32.6546	51.6680

---

🛠 Tech Stack

• Python
• Apache Airflow
• Pandas
• Requests

---

📦 Requirements

apache-airflow==2.8.1
pandas==2.1.0
requests==2.31.0

---

💻 Install dependencies

pip install -r requirements.txt

---

▶️ How to run

1. Install Airflow and activate your Python virtual environment.
2. Place the DAG file in the dags/ folder.
3. Initialize the database and start scheduler & webserver:

airflow db init
airflow scheduler
airflow webserver

4. Enable the DAG Get_City_List from the Airflow UI to fetch city data and save it to data/divar_cities.csv.

---

📄 License

MIT License (optional)

---

✨ Author

Samira Siavash

🔗 GitHub: https://github.com/SamiraSiavash

🔗 LinkedIn: https://linkedin.com/in/samira-siavash