Instagram System Design

Ever wondered what happens behind the scenes when you hit “send” on a messaging app or upload a photo on Instagram? Billions of users interact with these platforms every day, yet few understand the complex, highly scalable architecture that powers these seemingly simple actions. In this article, we’ll peel back the layers of modern system design — from real-time image uploads and follower notifications to comment queues and feed generation.

Learn how Instagram works under the hood.

---

System Design Refresher :

1. 10 System design trade-offs

2. Rate Limiting Algorithms Explained with Code

3. System Design of Reddit/Quora

4. Stateful vs Stateless Architecture

5. Best Practices for Developing Microservices

6. 10 Problems of Distributed Systems

7. 20 System Design Concepts Every Developer Should Know - Part - I

8. How Shopify handles 16,000 Request per second

9. Software Architecture Pattern - Layered Architecture

10. How Enterprise Applications Exchange Data Using Messaging

11. Microservices Design Pattern - Event Sourcing Pattern

12. Improve API Performance 🚀

13. Distributed System Learning Roadmap

14. 15 Scaling Techniques in Distributed Systems

---

💡 Functional and Non-Functional Requirements

Functional:

1. Image Upload: Users should be able to upload images from their mobile devices.
   The system should support different image formats and sizes.

2. Like and Comment: Users should be able to like and comment on images.

   View the likes and comments on images. Receive notifications when their images are liked or commented.

3. Follow: Users should be able to follow other users.

Users should receive notifications when they are followed by other users.

Technical Requirement

1. The system should support image formats such as JPEG, PNG, and GIF.

2. The maximum allowed image size should be 10MB.

3. The system should be able to handle at least 100,000 concurrent uploads.

4. Upload 10 images/day, with a maximum of 500 images/month.

5. Handle at least 1 million concurrent likes and comments.

6. Handle at least 1 billion registered users.

7. Users should be able to follow at least 1,000 other users.

Non-Functional:

• Low latency

• High availability

• Scalability (millions to billions of users)

• Message durability

Capacity Estimation

1. Storage Capacity:

Average image size = 5MB

Each user uploads = 10 images/day

Maximum storage capacity = 10,000,000 GB (10 petabytes)

1,000,000 GB (1 exabyte) per month

2. Network bandwidth:

Average image upload time = 5 seconds

Concurrent uploads = 100,000

Min network bandwidth = 50 Gbps.

Database Design

Queries

1. Number of likes for a post:

SELECT COUNT(*) FROM Like WHERE post_id = [post_id];

2. Get all the posts for a specific user:

SELECT * FROM Post WHERE user_id = [user_id];

3. Who follows user ByteMonk:

SELECT * FROM Follow WHERE following_user_id = [user_ByteMonk_id];

---

High-Level Design

to upload images, retrieve feeds, and display content to the end users.

Load Balancer

To ensure high availability and distribute incoming traffic evenly, we use load balancers. They automatically route requests to healthy backend servers, ensuring seamless user experience even if one or more servers go down.

Image Service

This backend service handles all image-related operations. It provides APIs to:

• Upload user images

• Retrieve metadata for uploaded images

The metadata includes the S3 path of the image, which clients use to render the image within the application.

Amazon S3

Uploaded images are stored in Amazon S3, a highly scalable and cost-effective object storage service. It's a reliable choice for storing large volumes of image data. By integrating S3 with CloudFront, we can speed up image delivery to end users.

CloudFront (CDN)

Amazon CloudFront is a content delivery network used to distribute images globally with low latency. It caches images close to users, ensuring faster loading times in the application.

Image Metadata Storage (Image DDB)

We use AWS DynamoDB to store metadata about uploaded images. This includes information like image path, upload time, user ID, etc. It's a fast and scalable NoSQL database ideal for handling metadata at scale.

SNS (Simple Notification Service)

Each time an image is uploaded, an event is published via AWS SNS. This allows other systems (like feed generation or analytics) to be notified in real-time and act upon the upload event.

SQS (Simple Queue Service)

SQS is used to queue upload events from SNS. The Feed Generation Service subscribes to this queue and consumes these events to generate user feeds. This decouples services and ensures reliable, asynchronous processing.

Feed Generation Service

This service listens to the SQS queue for new image uploads. Based on these events, it generates or updates user feeds. It's designed to handle millions of events efficiently and can be horizontally scaled. (Detailed LLD of this service will be discussed in a future article.)

Feed Data Storage (Feed DDB)

User feed data is stored in another DynamoDB table. The Feed Generation Service writes data to this table, which is later used to serve user feeds.

Redis Cache

To reduce read latency and improve performance, we introduce a caching layer using Redis. When a client requests a user’s feed:

• It first checks Redis for cached data.

• If not found, it fetches the feed from DynamoDB and stores it in Redis for future access.

Backend Services

1. Image Service Functionality: The image service in Instagram manages image processing and storage, facilitating actions like uploads, retrievals, and deletions. It relies on The Blob store for handling image files and ensures timely updates to image metadata in the database.

2. Metadata Management: The system employs a metadata service that maintains crucial information like captions, locations, and tags for images, ensuring that users have access to relevant details whenever needed.

3. Fan Out Service: The fan out service plays a crucial role by consuming events from the queue and disseminating them to various downstream services that track user interactions, thereby updating the relevant feed services accordingly.

4. User Feed Generation: The field service creates personalised news feeds by analysing user interactions such as follows and likes. It retrieves user data from the metadata database to curate content that aligns with user preferences.

5. Asynchronous Comment Processing: Instagram’s comment system utilises a distributed queue to handle comment requests, allowing for asynchronous processing. This ensures that the platform can manage a high volume of comments without affecting the overall response time.

6. Metadata Database Interaction: Each comment operation, such as adding or removing comments, involves real-time updates to the metadata database to keep the comments count accurate for each image.

7. Scalability and Performance: The design prioritises scalability, enabling the system to efficiently accommodate numerous concurrent comment requests while maintaining optimal performance for other user interactions.

How Image Upload Works