Skip to content

README.md

Latest commit

 

History

History
118 lines (89 loc) · 3.04 KB

README.md

File metadata and controls

118 lines (89 loc) · 3.04 KB

Instagram Video Downloader Lambda

A serverless AWS Lambda function that downloads Instagram videos and stores them in S3. The project uses Supabase for metadata management and AWS SQS for message queuing.

Features

  • Downloads Instagram videos at specified resolutions
  • Stores videos in AWS S3
  • Uses Supabase for metadata management
  • Implements AWS SQS for message queuing
  • Includes CloudWatch metrics for monitoring
  • Handles retries and error cases
  • Supports batch processing of videos

Prerequisites

  • AWS Account with appropriate permissions
  • Supabase account and project
  • Python 3.10 or higher
  • AWS CLI configured with appropriate credentials

Environment Variables

Create a .env file with the following variables:

SUPABASE_URL=your_supabase_url
SUPABASE_KEY=your_supabase_key
S3_BUCKET=your_s3_bucket_name
MAX_RETRIES=3
DOWNLOAD_TIMEOUT=300
CHUNK_SIZE=1048576

Installation

  1. Clone the repository:
git clone <repository-url>
cd InstagramVideoDownloader-lambda
  1. Create and activate a virtual environment:
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
  1. Install dependencies:
pip install -r requirements.txt

Project Structure

  • lambda_function.py: Main Lambda function code
  • trigger_sqs_batches.py: Script to trigger SQS message batches
  • scripts/pack_lambda.sh: Script to package Lambda function
  • requirements.txt: Python dependencies
  • .env: Environment variables (not committed to git)

Usage

Deploying the Lambda Function

  1. Package the Lambda function:
./scripts/pack_lambda.sh
  1. Upload the lambda_function.zip to AWS Lambda

Triggering Video Downloads

Run the batch processor to send videos to the queue:

python trigger_sqs_batches.py

CloudWatch Metrics

The following metrics are available in CloudWatch under the namespace 'InstagramCollector':

  • ProcessingStarted: Number of videos processing started
  • ProcessingSuccess: Number of successfully processed videos
  • ProcessingFailed: Number of failed video processing attempts
  • DownloadTime: Time taken to download videos
  • UploadTime: Time taken to upload videos to S3
  • TotalProcessingTime: Total processing time per video
  • SupabaseVideoFound: Number of videos found in Supabase
  • SupabaseVideoNotFound: Number of videos not found in Supabase
  • SupabaseError: Number of Supabase errors
  • DownloadTimeout: Number of download timeouts
  • DownloadError: Number of download errors
  • S3UploadError: Number of S3 upload errors
  • InvalidInput: Number of invalid input messages

Error Handling

The system implements comprehensive error handling:

  • Retries for failed downloads
  • Error logging to CloudWatch
  • Metrics tracking for various failure scenarios
  • Graceful cleanup of temporary files

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Commit your changes
  4. Push to the branch
  5. Create a Pull Request

Support

For issues and feature requests, please create an issue in the repository.

Maintained by Oriane XYZ