Notification system architecture for real-time and async delivery

# Notification System Architecture ## System Overview This design handles real-time, email, and push notifications for a platform serving 500K users with an expected peak of 10K notifications/minute. --- ## Architecture Diagram ``` ┌──────────────┐ │ Event Source │ (API actions, cron jobs, webhooks) └──────┬───────┘ │ publish ▼ ┌────────────────┐ │ Message Queue │ (Redis Streams / BullMQ) │ notification.* │ └────────┬───────┘ │ consume ▼ ┌────────────────────────┐ │ Notification Worker │ (Node.js, horizontally scalable) │ │ │ 1. Load user prefs │ │ 2. Filter by channel │ │ 3. Render templates │ │ 4. Route to channels │ └──┬──────┬──────┬──────┘ │ │ │ ┌───────┘ │ └───────┐ ▼ ▼ ▼ ┌─────────────┐ ┌──────────┐ ┌──────────────┐ │ WebSocket │ │ Email │ │ Push (FCM/ │ │ Gateway │ │ Queue │ │ APNs) │ │ (Socket.io) │ │ (SES) │ │ │ └──────┬──────┘ └─────┬────┘ └──────┬───────┘ │ │ │ ▼ ▼ ▼ In-app live Email inbox Mobile device ``` --- ## Database Schema ```sql -- Core notification record CREATE TABLE notifications ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), recipient_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE, type VARCHAR(50) NOT NULL, -- 'comment', 'like', 'follow', 'mention', etc. title VARCHAR(200) NOT NULL, body TEXT NOT NULL, data JSONB DEFAULT '{}', -- payload (post_id, comment_id, etc.) is_read BOOLEAN DEFAULT FALSE, read_at TIMESTAMPTZ, channels_sent VARCHAR(20)[] DEFAULT '{}', -- ['in_app', 'email', 'push'] created_at TIMESTAMPTZ DEFAULT NOW(), -- Indexes for fast queries CONSTRAINT idx_notif_recipient_created UNIQUE (recipient_id, created_at DESC, id) ); CREATE INDEX idx_notif_unread ON notifications (recipient_id, is_read) WHERE is_read = FALSE; -- User notification preferences CREATE TABLE notification_preferences ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE, event_type VARCHAR(50) NOT NULL, -- 'comment', 'like', 'follow', etc. in_app BOOLEAN DEFAULT TRUE, email BOOLEAN DEFAULT TRUE, push BOOLEAN DEFAULT TRUE, email_digest VARCHAR(10) DEFAULT 'instant', -- 'instant', 'daily', 'weekly', 'off' UNIQUE (user_id, event_type) ); -- Email digest queue (batched sends) CREATE TABLE email_digest_queue ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE, notification_id UUID NOT NULL REFERENCES notifications(id) ON DELETE CASCADE, digest_type VARCHAR(10) NOT NULL, -- 'daily' or 'weekly' scheduled_for TIMESTAMPTZ NOT NULL, sent_at TIMESTAMPTZ, INDEX idx_digest_pending (scheduled_for, sent_at) WHERE sent_at IS NULL ); -- Push device tokens CREATE TABLE push_tokens ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE, token TEXT NOT NULL UNIQUE, platform VARCHAR(10) NOT NULL, -- 'ios', 'android', 'web' is_active BOOLEAN DEFAULT TRUE, created_at TIMESTAMPTZ DEFAULT NOW() ); ``` --- ## Event Flow ### 1. Event Published When a user action triggers a notification (e.g., someone comments on a post): ```typescript // In the comments service, after creating the comment: await this.notificationQueue.add('notification.comment', { type: 'comment', recipientId: post.authorId, actorId: currentUser.id, data: { postId: post.id, commentId: comment.id }, title: `${currentUser.displayName} commented on your post`, body: comment.body.substring(0, 200), }); ``` ### 2. Worker Processes Event ```typescript async processNotification(job: Job<NotificationEvent>) { const { type, recipientId, actorId, data, title, body } = job.data; // Skip self-notifications if (recipientId === actorId) return; // Load user preferences const prefs = await this.getPreferences(recipientId, type); // Create the notification record const notification = await this.prisma.notification.create({ data: { recipientId, type, title, body, data, channelsSent: [] }, }); const channels: string[] = []; // Route to enabled channels if (prefs.inApp) { await this.sendInApp(recipientId, notification); channels.push('in_app'); } if (prefs.email) { if (prefs.emailDigest === 'instant') { await this.sendEmail(recipientId, notification); channels.push('email'); } else { await this.queueDigest(recipientId, notification.id, prefs.emailDigest); } } if (prefs.push) { await this.sendPush(recipientId, notification); channels.push('push'); } // Update channels sent await this.prisma.notification.update({ where: { id: notification.id }, data: { channelsSent: channels }, }); } ``` ### 3. Real-Time Delivery (WebSocket) ```typescript async sendInApp(userId: string, notification: Notification) { // Publish to Redis pub/sub for the user's channel await this.redis.publish( `user:${userId}:notifications`, JSON.stringify(notification), ); // The WebSocket gateway subscribes to this channel // and pushes to the connected client } ``` --- ## API Endpoints ``` GET /api/notifications # List (paginated, newest first) ?page=1&limit=20&unreadOnly=true GET /api/notifications/unread-count # Badge count PATCH /api/notifications/:id/read # Mark one as read POST /api/notifications/mark-all-read # Batch mark all as read DELETE /api/notifications/:id # Dismiss one GET /api/notification-preferences # Get user preferences PUT /api/notification-preferences # Update preferences Body: { comment: { inApp: true, email: false, push: true, emailDigest: 'daily' } } ``` --- ## Technology Choices | Component | Choice | Why | |-----------|--------|-----| | Queue | BullMQ (Redis) | Already using Redis. BullMQ gives retries, concurrency control, and delayed jobs for digests. | | WebSocket | Socket.io with Redis adapter | Handles reconnection, rooms (per-user), and scales horizontally via Redis pub/sub. | | Email | Amazon SES + MJML templates | SES is cheap at scale ($0.10/1K emails). MJML compiles to responsive email HTML. | | Push | Firebase Cloud Messaging | Handles both iOS (via APNs bridge) and Android. Free tier covers 500K users. | | Database | PostgreSQL | Transactional consistency for notification records. JSONB for flexible data payloads. | | Caching | Redis | Unread count cached per user. Invalidated on new notification or mark-as-read. TTL: 60s. | --- ## Scaling Considerations - **Workers**: Horizontally scale notification workers. BullMQ distributes jobs automatically. - **WebSocket**: Use Redis adapter so Socket.io works across multiple server instances. - **Database**: The composite index on (recipient_id, created_at DESC) ensures fast cursor-based pagination. Partition by month if notifications table exceeds 100M rows. - **Rate limiting**: Batch notifications of the same type within 5 minutes (e.g., "3 people liked your post" instead of 3 separate notifications). - **Cleanup**: Cron job deletes read notifications older than 90 days.