Get A Team Get A Team
← Back to Blog Email Gateway Architecture: From SMTP to AI Understanding

Email Gateway Architecture: From SMTP to AI Understanding

By Joseph Benguira • 2025-11-05 • 17 min read

TL;DR: Building an email gateway for AI agents isn't just about parsing SMTP. We went from receiving raw RFC822 messages to delivering structured, context-aware prompts to Claude sessions. This article breaks down the real technical challenges: MIME multipart hell, attachment handling, threading context, spam filtering, and maintaining conversational state across dozens of concurrent AI employees.

Why This Matters Now

GetATeam launched its production email gateway 10 days ago. Our AI employees now have real email addresses like `sydney.sweeney@getateam.org` and `joseph.benguira@getateam.org`. In the first week, we received 847 emails, ranging from simple text replies to complex multipart messages with PDF attachments and inline images.

The challenge wasn't just "receive email and forward it to AI." It was:

This is what we learned.

Architecture Overview

High-Level Flow

┌─────────────────────────────────────────────────────────────┐
│                     External Email (SMTP)                    │
│                  someone@company.com                         │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
          ┌──────────────────────────────┐
          │  Postfix SMTP Server         │
          │  (Port 25, TLS)              │
          │  - SPF/DKIM validation       │
          │  - Basic spam filtering      │
          └──────────────┬───────────────┘
                         │
                         ▼
          ┌──────────────────────────────┐
          │  Email Processor Service     │
          │  (Node.js + mailparser)      │
          │  - Parse MIME structure      │
          │  - Extract attachments       │
          │  - Thread detection          │
          │  - Save to queue as .eml     │
          └──────────────┬───────────────┘
                         │
                         ▼
          ┌──────────────────────────────┐
          │  Agent Router                │
          │  - Match recipient           │
          │  - Find agent session        │
          │  - Format prompt             │
          └──────────────┬───────────────┘
                         │
                         ▼
          ┌──────────────────────────────┐
          │  VibeCoder Admin WebSocket   │
          │  wss://172.16.0.3:2108/admin │
          │  - Send prompt to session    │
          │  - Validate with \r          │
          └──────────────┬───────────────┘
                         │
                         ▼
          ┌──────────────────────────────┐
          │  AI Agent Session (Claude)   │
          │  - Reads email prompt        │
          │  - Generates response        │
          │  - Uses email-sender skill   │
          └──────────────────────────────┘

Key Components

1. Postfix SMTP Server

2. Email Processor

3. Agent Router

4. AI Agent Response

Deep Dive: MIME Parsing Hell

The Problem

Email isn't just text. A typical modern email looks like this:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="----=_Part_123"

------=_Part_123
Content-Type: multipart/alternative; boundary="----=_Part_456"

------=_Part_456
Content-Type: text/plain; charset="UTF-8"

Plain text version here

------=_Part_456
Content-Type: text/html; charset="UTF-8"

<html><body>HTML version here</body></html>

------=_Part_456--

------=_Part_123
Content-Type: application/pdf; name="document.pdf"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="document.pdf"

JVBERi0xLjcKCjEgMCBvYmoKPDwvVHlwZS9DYXRhbG9nL1BhZ2VzIDIgMCBSPj4K...

------=_Part_123--

Challenges:

  1. Nested multipart structures - Alternative inside mixed, related inside alternative
  2. Boundary parsing - Can't just split on `--boundary`, need proper MIME parser
  3. Encoding hell - base64, quoted-printable, 7bit, 8bit, binary
  4. Character sets - UTF-8, ISO-8859-1, Windows-1252, etc.
  5. Malformed headers - Real-world emails violate RFC specs constantly

Our Solution

We use `mailparser` (npm package) which handles the complexity:

const { simpleParser } = require('mailparser');
const fs = require('fs').promises;

async function parseEmail(emlPath) {
  const emlContent = await fs.readFile(emlPath, 'utf8');
  
  const parsed = await simpleParser(emlContent);
  
  return {
    from: parsed.from.text,
    to: parsed.to.text,
    subject: parsed.subject,
    messageId: parsed.messageId,
    inReplyTo: parsed.inReplyTo,
    references: parsed.references,
    date: parsed.date,
    
    // Text content (prioritize)
    text: parsed.text,
    html: parsed.html,
    
    // Attachments with decoded content
    attachments: parsed.attachments.map(att => ({
      filename: att.filename,
      contentType: att.contentType,
      size: att.size,
      content: att.content // Buffer (already decoded)
    }))
  };
}

Why mailparser:

Metrics:

Attachment Handling

The Challenge

Attachments can be:

Questions:

  1. Should AI read the attachment content?
  2. How to present binary data (PDFs, images) to a text-based AI?
  3. Storage: Save permanently or temporary?
  4. Security: Malware scanning?

Our Approach

Decision Matrix:

Attachment Type Action Why
Images (< 5MB) Send to Claude (base64) Claude can analyze images
PDFs (< 10MB) Extract text + send pages Claude can read PDFs
Text files Send content Direct processing
Office docs Extract text AI needs content
Archives List contents only Security + complexity
Executables Block + warn Security risk

Implementation:

async function processAttachments(attachments) {
  const processed = [];
  
  for (const att of attachments) {
    // Security check
    if (isExecutable(att.filename)) {
      processed.push({
        filename: att.filename,
        status: 'blocked',
        reason: 'Executable files not allowed'
      });
      continue;
    }
    
    // Size check
    if (att.size > 10 * 1024 * 1024) { // 10MB
      processed.push({
        filename: att.filename,
        status: 'too_large',
        size: att.size
      });
      continue;
    }
    
    // Process by type
    if (att.contentType.startsWith('image/')) {
      // Claude can view images
      const base64 = att.content.toString('base64');
      processed.push({
        filename: att.filename,
        type: 'image',
        contentType: att.contentType,
        base64: base64
      });
    } else if (att.contentType === 'application/pdf') {
      // Extract text from PDF
      const text = await extractPdfText(att.content);
      processed.push({
        filename: att.filename,
        type: 'pdf',
        text: text
      });
    } else if (att.contentType.startsWith('text/')) {
      // Plain text
      processed.push({
        filename: att.filename,
        type: 'text',
        content: att.content.toString('utf8')
      });
    } else {
      // Unknown type - save metadata only
      processed.push({
        filename: att.filename,
        type: 'other',
        contentType: att.contentType,
        size: att.size
      });
    }
  }
  
  return processed;
}

function isExecutable(filename) {
  const dangerous = ['.exe', '.bat', '.cmd', '.sh', '.app', '.dmg', 
                    '.dll', '.so', '.dylib', '.scr', '.vbs', '.js', 
                    '.jar', '.apk'];
  return dangerous.some(ext => filename.toLowerCase().endsWith(ext));
}

Storage Strategy:

// Save attachments to agent-specific directory
const attachmentPath = \`/opt/app/virtualemployees/agents/\${agentFolder}/mails/attachments/\${messageId}/\`;
await fs.mkdir(attachmentPath, { recursive: true });

for (const att of attachments) {
  const filePath = path.join(attachmentPath, att.filename);
  await fs.writeFile(filePath, att.content);
}

Metrics (first 10 days):

Thread Context Detection

The Problem

Emails are conversational. When someone replies to a previous message, the AI needs context:

Without context, the AI might:

Email Threading Headers

RFC822 defines headers for threading:

Message-ID: <abc123@example.com>
In-Reply-To: <xyz789@example.com>
References: <first@example.com> <second@example.com> <xyz789@example.com>

How it works:

  1. Original email: Only has `Message-ID`
  2. First reply: Has `In-Reply-To` (points to original) and `References` (list of all previous)
  3. Subsequent replies: Update `References` with full chain

Our Implementation

async function detectThread(parsed) {
  const thread = {
    isReply: false,
    threadId: null,
    previousMessages: []
  };
  
  // Check if this is a reply
  if (parsed.inReplyTo || (parsed.references && parsed.references.length > 0)) {
    thread.isReply = true;
    
    // Use first message in References as thread ID
    thread.threadId = parsed.references 
      ? parsed.references[0] 
      : parsed.inReplyTo;
    
    // Find previous messages in this thread
    thread.previousMessages = await findThreadMessages(thread.threadId);
  } else {
    // New thread - use this message's ID
    thread.threadId = parsed.messageId;
  }
  
  return thread;
}

async function findThreadMessages(threadId) {
  const inboxPath = \`/opt/app/virtualemployees/agents/\${agentFolder}/mails/inbox/\`;
  const files = await fs.readdir(inboxPath);
  
  const threadMessages = [];
  
  for (const file of files) {
    if (!file.endsWith('.eml')) continue;
    
    const emlPath = path.join(inboxPath, file);
    const parsed = await parseEmail(emlPath);
    
    // Check if this message belongs to the thread
    if (parsed.messageId === threadId || 
        (parsed.references && parsed.references.includes(threadId))) {
      threadMessages.push({
        messageId: parsed.messageId,
        from: parsed.from.text,
        subject: parsed.subject,
        date: parsed.date,
        text: parsed.text.substring(0, 500) // Preview only
      });
    }
  }
  
  // Sort by date
  return threadMessages.sort((a, b) => a.date - b.date);
}

Constructing Context-Aware Prompts

When sending the prompt to the AI agent:

function buildEmailPrompt(parsed, thread, attachments) {
  let prompt = '📧 NEW EMAIL TASK\n\n';
  
  // Basic info
  prompt += \`From: \${parsed.from.text}\n\`;
  prompt += \`To: \${parsed.to.text}\n\`;
  prompt += \`Subject: \${parsed.subject}\n\`;
  prompt += \`Date: \${parsed.date.toISOString()}\n\n\`;
  
  // Thread context if applicable
  if (thread.isReply && thread.previousMessages.length > 0) {
    prompt += '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n';
    prompt += '📜 CONVERSATION HISTORY\n';
    prompt += '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n';
    
    thread.previousMessages.forEach((msg, idx) => {
      prompt += \`Message \${idx + 1} (\${msg.date.toDateString()}):\n\`;
      prompt += \`From: \${msg.from}\n\`;
      prompt += \`Preview: \${msg.text}\n\n\`;
    });
    
    prompt += '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n';
  }
  
  // Attachments summary
  if (attachments.length > 0) {
    prompt += '📎 ATTACHMENTS:\n';
    attachments.forEach(att => {
      prompt += \`- \${att.filename} (\${att.type}, \${formatBytes(att.size)})\n\`;
    });
    prompt += '\n';
  }
  
  // Email body
  prompt += '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n';
  prompt += '📄 EMAIL CONTENT\n';
  prompt += '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n';
  prompt += parsed.text;
  prompt += '\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n';
  
  // Instructions
  prompt += 'REQUIRED ACTIONS:\n';
  prompt += '1. Read and understand the email\n';
  prompt += '2. Check conversation history if this is a reply\n';
  prompt += '3. Review any attachments\n';
  prompt += '4. Respond appropriately using email-sender.js skill\n';
  prompt += '5. Save response to mails/sent/ directory\n';
  
  return prompt;
}

Example output:

📧 NEW EMAIL TASK

From: Alice Smith <alice@company.com>
To: Joseph Benguira <joseph.benguira@getateam.org>
Subject: Re: GetATeam Integration Question
Date: 2025-11-05T09:15:00.000Z

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📜 CONVERSATION HISTORY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Message 1 (Nov 4, 2025):
From: Alice Smith <alice@company.com>
Preview: Hi Joseph, I saw your blog post about multi-agent coordination...

Message 2 (Nov 4, 2025):
From: Joseph Benguira <joseph.benguira@getateam.org>
Preview: Hey Alice, thanks for reaching out! The coordination layer...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📎 ATTACHMENTS:
- architecture-diagram.png (image, 245 KB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📄 EMAIL CONTENT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Perfect! I've attached our current architecture. Where do you think
the email gateway should fit?

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Metrics:

Spam Filtering Without Breaking AI Communication

The Challenge

AI employees need to receive legitimate emails, but spam will flood them. Traditional spam filters (SpamAssassin, etc.) are trained for humans, not AI agents.

Problems with standard filters:

Our Multi-Layer Approach

Layer 1: SMTP Level (Postfix)

# /etc/postfix/main.cf
smtpd_recipient_restrictions =
    permit_mynetworks,
    reject_non_fqdn_recipient,
    reject_unknown_recipient_domain,
    reject_unauth_destination,
    reject_rbl_client zen.spamhaus.org,
    reject_rbl_client bl.spamcop.net,
    permit

smtpd_sender_restrictions =
    reject_non_fqdn_sender,
    reject_unknown_sender_domain,
    permit

# SPF checking
smtpd_recipient_restrictions = 
    ...
    check_policy_service unix:private/policyd-spf

# DKIM verification
milter_default_action = accept
milter_protocol = 6
smtpd_milters = inet:localhost:8891
non_smtpd_milters = inet:localhost:8891

Rejects:

Layer 2: Content Analysis

async function analyzeEmailContent(parsed) {
  const signals = {
    spam_score: 0,
    reasons: []
  };
  
  // Check subject line
  if (hasSpamKeywords(parsed.subject)) {
    signals.spam_score += 3;
    signals.reasons.push('Spam keywords in subject');
  }
  
  // Check for excessive links
  const linkCount = (parsed.html || '').match(/https?:\/\//g)?.length || 0;
  if (linkCount > 10) {
    signals.spam_score += 2;
    signals.reasons.push(\`Too many links (\${linkCount})\`);
  }
  
  // Check for URL shorteners (common in spam)
  if (hasUrlShorteners(parsed.text)) {
    signals.spam_score += 2;
    signals.reasons.push('URL shorteners detected');
  }
  
  // Check From domain vs Return-Path domain
  const fromDomain = parsed.from.value[0].address.split('@')[1];
  const returnPath = parsed.headers.get('return-path');
  if (returnPath && !returnPath.includes(fromDomain)) {
    signals.spam_score += 1;
    signals.reasons.push('From/Return-Path mismatch');
  }
  
  // Check for unusual character encoding tricks
  if (hasEncodingTricks(parsed.text)) {
    signals.spam_score += 2;
    signals.reasons.push('Suspicious character encoding');
  }
  
  // Threshold
  signals.isSpam = signals.spam_score >= 5;
  
  return signals;
}

function hasSpamKeywords(text) {
  const keywords = [
    /viagra/i, /cialis/i, /casino/i, /lottery/i,
    /enlarge/i, /click here now/i, /act now/i,
    /limited time/i, /free money/i, /nigerian prince/i,
    /crypto.*investment/i, /double.*bitcoin/i
  ];
  return keywords.some(regex => regex.test(text));
}

Layer 3: Behavioral Analysis

// Track sender history
const senderHistory = new Map();

function checkSenderReputation(fromAddress) {
  if (!senderHistory.has(fromAddress)) {
    senderHistory.set(fromAddress, {
      emailCount: 0,
      spamCount: 0,
      lastSeen: null
    });
  }
  
  const history = senderHistory.get(fromAddress);
  history.emailCount++;
  history.lastSeen = new Date();
  
  // New sender with suspicious pattern
  if (history.emailCount === 1) {
    return { trustLevel: 'unknown', reason: 'First email' };
  }
  
  // Frequent sender with no spam
  if (history.emailCount > 5 && history.spamCount === 0) {
    return { trustLevel: 'trusted', reason: 'Established sender' };
  }
  
  // High spam ratio
  if (history.spamCount / history.emailCount > 0.3) {
    return { trustLevel: 'suspicious', reason: 'High spam ratio' };
  }
  
  return { trustLevel: 'neutral' };
}

Metrics (10 days):

Performance & Scaling

Current Setup

# docker-compose.yml
services:
  email-gateway:
    image: node:22-alpine
    volumes:
      - ./email-processor:/app
      - ./queue:/queue
      - ./agents:/agents
    environment:
      - SMTP_HOST=172.17.0.1
      - SMTP_PORT=25
      - VIBECODER_HOST=172.16.0.3
      - VIBECODER_PORT=2108
    restart: always

  postfix:
    image: boky/postfix
    ports:
      - "25:25"
    volumes:
      - ./postfix-config:/etc/postfix/custom.d
    environment:
      - ALLOWED_SENDER_DOMAINS=getateam.org
    restart: always

Performance Benchmarks

Email Processing Pipeline:

Stage Time (avg) Time (p95) Notes
SMTP receive 120ms 250ms Postfix + SPF/DKIM
MIME parsing 35ms 180ms 180ms = with attachments
Spam analysis 8ms 15ms Content + behavior checks
Thread detection 22ms 65ms Searching inbox history
Queue save 12ms 25ms Write .eml to disk
Total (no AI) 197ms 535ms Just email processing
Agent routing 45ms 120ms WebSocket connection
Claude response 8.5s 25s AI generation time
Response email 150ms 320ms Create .eml + SMTP send
Total (with AI) 8.9s 26s End-to-end

Resource Usage (per email):

Concurrent Processing:

Currently handling 24 concurrent AI agents. Each agent can receive emails simultaneously.

// Queue processor with concurrency control
const CONCURRENCY = 24;
const queue = new PQueue({ concurrency: CONCURRENCY });

async function processQueue() {
  const files = await fs.readdir('/queue');
  const emlFiles = files.filter(f => f.endsWith('.eml')).sort();
  
  for (const file of emlFiles) {
    queue.add(() => processEmail(path.join('/queue', file)));
  }
  
  await queue.onIdle();
}

Scaling Projections:

Based on current performance:

Lessons Learned

1. MIME Parsing is Harder Than You Think

Mistake: Initially tried to parse MIME manually with regex.

Reality: MIME has too many edge cases. Nested multipart, malformed boundaries, mixed encodings. After 2 days of debugging, switched to `mailparser` library. Problem solved in 30 minutes.

Lesson: Use battle-tested libraries for complex specs.

2. Thread Context is Critical

Mistake: Initially sent emails to AI without context. AI would ask questions already answered in previous messages.

Fix: Parse References/In-Reply-To headers, fetch previous messages, include in prompt.

Impact: User satisfaction went from "AI seems dumb" to "Wow, it remembers our conversation!"

3. Attachments Need Type-Specific Handling

Mistake: Sent all attachments as base64 to Claude.

Problem: Large PDFs exceeded context limits, binary files were useless.

Fix: Implement smart attachment handling:

Result: Claude can actually use attachment content meaningfully.

4. Spam Filtering Must Be Aggressive But Smart

Mistake: Started with minimal spam filtering ("AI can handle it").

Reality: AI agents spent time analyzing spam, wasting tokens and time.

Fix: Multi-layer filtering (SMTP + content + behavior). Block 99.7% of spam before reaching AI.

Savings: $127/month in Claude API costs (estimated).

5. Email is Asynchronous, AI is Synchronous

Challenge: Email arrives at any time, but Claude sessions need to be active to receive prompts.

Solution:

This decouples email reception from AI processing.

Production Metrics (10 Days)

Email Volume:

Response Times:

Attachments:

Thread Detection:

Error Rate:

Cost Analysis:

What's Next

Improvements In Progress

1. Better Attachment Intelligence

2. Multi-Language Support

3. Priority Routing

4. Analytics Dashboard

Conclusion

Building an email gateway for AI agents is much more than "receive email, forward to AI." It's about:

✅ Robust MIME parsing that handles real-world chaos ✅ Smart attachment handling based on content type ✅ Thread context detection for conversational continuity ✅ Aggressive spam filtering without false positives ✅ Async processing that scales to dozens of concurrent agents ✅ Detailed logging for debugging and optimization

We went from 0 to 847 emails processed in 10 days, with a 99.4% success rate. Our AI employees now have real email addresses and handle customer inquiries, internal communications, and collaboration requests.

The architecture is solid, scalable, and production-ready.

Want to build your own? Start here:

  1. Use `mailparser` for MIME parsing
  2. Implement multi-layer spam filtering
  3. Build thread context detection
  4. Queue emails to disk for async processing
  5. Use WebSocket to deliver prompts to AI sessions
  6. Measure, optimize, iterate

Next article: How we built conversational state management across 24 concurrent AI agents (hint: it's not just Redis).

About GetATeam: We're building the platform for AI employees. Our agents have email addresses, handle conversations, and collaborate like real team members. Follow our build-in-public journey: github.com/getateam

Written by: Joseph Benguira (Founder & CTO)
Date: November 5, 2025
Category: Engineering

Enjoyed this article?

Try GetATeam Free