TrailSync - Salesforce CRM Middleware Service

A production-ready Node.js middleware service that securely synchronizes CRM data between MongoDB and Salesforce using OAuth 2.0 authentication and the Salesforce Bulk API.

🚀 Overview

TrailSync is designed as a robust middleware layer that bridges MongoDB and Salesforce, providing:

Secure OAuth 2.0 Authentication: Enterprise-grade authentication following Salesforce Trust principles
Bulk Data Synchronization: Efficient bidirectional sync using Salesforce Bulk API 2.0
Governor Limit Awareness: Intelligent handling of Salesforce API governor limits
MongoDB Persistence: Reliable data storage and audit trails
RESTful API: Complete CRUD operations for CRM objects
Comprehensive Logging: Full audit trails for compliance and debugging

📋 Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Client Applications                       │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│           TrailSync Express REST API Layer                   │
│  ┌─────────────┬──────────────┬──────────────────────────┐  │
│  │ Auth Routes │ Account APIs │ Sync Management Routes   │  │
│  └─────────────┴──────────────┴──────────────────────────┘  │
└────────────────────────┬────────────────────────────────────┘
                         │
        ┌────────────────┼────────────────┐
        │                │                │
        ▼                ▼                ▼
┌─────────────┐  ┌──────────────┐  ┌──────────────────┐
│  MongoDB    │  │ OAuth 2.0    │  │ Bulk API Client  │
│  Database   │  │ Auth Service │  │ & Sync Engine    │
└─────────────┘  └──────────────┘  └──────────────────┘
                         │
        ┌────────────────┴────────────────┐
        └────────────────┬────────────────┘
                         ▼
┌──────────────────────────────────────────────────────────────┐
│              Salesforce Organization                         │
│  ┌────────────┐    ┌──────────┐    ┌─────────────────────┐ │
│  │ OAuth 2.0  │    │ Bulk API │    │ Account/Contact     │ │
│  │ Endpoints  │    │ v2.0     │    │ Opportunity Objects │ │
│  └────────────┘    └──────────┘    └─────────────────────┘ │
└──────────────────────────────────────────────────────────────┘

🔐 Security Features

Salesforce Trust & Security

TrailSync implements comprehensive security measures aligned with Salesforce Trust principles:

OAuth 2.0 Authentication

Secure Token Exchange: Uses OAuth 2.0 authorization code flow
Token Refresh: Automatic token refresh mechanism prevents session timeouts
Credential Protection: All credentials stored in environment variables, never in code
Redirect URI Validation: Strict validation of OAuth redirect URIs
Client Secret Protection: Server-side token handling, never exposed to clients

// OAuth 2.0 Flow
1. User clicks "Login with Salesforce"
2. Redirected to Salesforce login page
3. User authorizes TrailSync application
4. Salesforce redirects to callback with authorization code
5. TrailSync exchanges code for access token (secure server-to-server)
6. Access token stored securely, refresh token for long-running sync

API Security

Rate Limiting: Prevents API abuse and excessive Salesforce API calls
CORS Protection: Configured cross-origin resource sharing
Helmet.js: Sets secure HTTP headers
Input Validation: Joi schema validation for all requests
Error Handling: Secure error messages that don't leak sensitive info

Governor Limit Management

Salesforce enforces governor limits to maintain platform stability. TrailSync actively manages these:

Key Governor Limits Monitored:

API Calls: 15,000 calls per 15-minute period (Developer Edition)
Batch Requests: 100 concurrent batch requests
Query Rows: 50,000 SOQL query result rows per 15 minutes
Short-Term Requests: 10,000 requests per second (brief overages allowed)
Streaming API Clients: Maximum concurrent connections
Batch Size: 10,000 records per batch in Bulk API

TrailSync's Governor Limit Strategy:

Pre-Sync Checks

// Check available API calls before sync
const limits = await checkGovernorLimits(connection);
if (limits.apiCalls.used / limits.apiCalls.total > 0.9) {
  // Throttle sync operations
  logger.warning('Governor limits approaching, reducing batch size');
}

Batch Processing
- Uses Bulk API 2.0 instead of REST API for bulk operations
- Each Bulk API job counts as 1 API call (vs. 1 per record with REST)
- Configurable batch sizes to stay within limits
- Default 10,000 records per batch
Intelligent Throttling
- 5-second poll intervals during job processing
- Exponential backoff on rate limit errors
- Queue management for pending operations
- Priority-based sync scheduling

Monitoring & Alerts

// Every sync operation logs governor limit usage
{
  "apiCalls": { "total": 15000, "used": 12450, "remaining": 2550 },
  "batchRequests": { "total": 100, "used": 45 },
  "queryRows": { "total": 50000, "used": 38234 }
}

🏗️ Project Structure

TrailSync/
├── src/
│   ├── config/
│   │   ├── salesforce.js          # OAuth 2.0 & Bulk API config
│   │   └── database.js            # MongoDB connection
│   ├── models/
│   │   ├── Account.js             # Account schema
│   │   └── SyncLog.js             # Sync audit trail
│   ├── controllers/
│   │   ├── authController.js      # OAuth endpoints
│   │   ├── accountController.js   # Account CRUD
│   │   └── syncController.js      # Sync operations
│   ├── services/
│   │   ├── syncService.js         # Bulk sync logic
│   │   └── salesforceService.js   # Salesforce queries
│   ├── routes/
│   │   ├── authRoutes.js          # Auth endpoints
│   │   ├── accountRoutes.js       # Account endpoints
│   │   └── syncRoutes.js          # Sync endpoints
│   ├── utils/
│   │   └── logger.js              # Winston logger
│   └── server.js                  # Express app
├── .env.example                   # Environment template
├── package.json                   # Dependencies
└── README.md                      # This file

📦 Installation

Prerequisites

Node.js 16.0.0+
npm 8.0.0+
MongoDB 4.4+
Salesforce account with API access

Setup

Clone and navigate to project
```
cd TrailSync
```
Install dependencies
```
npm install
```
Create .env file from template
```
cp .env.example .env
```
Configure Salesforce Connected App
- Log in to Salesforce as admin
- Setup → Apps → App Manager → New Connected App
- Enable OAuth 2.0 settings:
  - Callback URL: http://localhost:3000/auth/salesforce/callback
  - Selected OAuth Scopes:
    - api (Access and manage your data)
    - refresh_token (Obtain refresh token)
    - offline_access (Perform requests on your behalf)
- Create OAuth2 credentials (client ID and secret)

Update .env with Salesforce credentials

SALESFORCE_CLIENT_ID=your_client_id
SALESFORCE_CLIENT_SECRET=your_client_secret
SALESFORCE_LOGIN_URL=https://login.salesforce.com
SALESFORCE_REDIRECT_URI=http://localhost:3000/auth/salesforce/callback
MONGODB_URI=mongodb://localhost:27017/trailsync

Start MongoDB

# macOS with Homebrew
brew services start mongodb-community

# Linux
sudo systemctl start mongodb

# Docker
docker run -d -p 27017:27017 --name mongodb mongo:latest

Start TrailSync service
```
npm start
```
Or for development with auto-reload:
```
npm run dev
```

🔑 API Endpoints

Authentication Endpoints

Initiate OAuth 2.0 Flow

GET /auth/salesforce

Redirects user to Salesforce login page.

OAuth Callback Handler

GET /auth/salesforce/callback?code=<auth_code>

Handles OAuth callback from Salesforce and exchanges authorization code for access token.

Check Authentication Status

GET /auth/status

Response:

{
  "success": true,
  "authenticated": true,
  "user": {
    "instanceUrl": "https://instance.salesforce.com"
  }
}

Account Endpoints

Get All Accounts

GET /api/v1/accounts?syncStatus=SYNCED&limit=20&page=1

Create Account

POST /api/v1/accounts
Content-Type: application/json

{
  "name": "Acme Corporation",
  "type": "Customer",
  "industry": "Technology",
  "website": "www.acme.com",
  "phone": "+1-555-1234",
  "employees": 500,
  "revenue": 10000000
}

Get Account by ID

GET /api/v1/accounts/:id

Update Account

PUT /api/v1/accounts/:id
Content-Type: application/json

{
  "phone": "+1-555-5678",
  "revenue": 12000000
}

Delete Account

DELETE /api/v1/accounts/:id

Synchronization Endpoints

Trigger Bulk Synchronization

POST /api/v1/sync/bulk
Content-Type: application/json

{
  "objectType": "Account",
  "batchSize": 10000,
  "operation": "upsert"
}

Response:

{
  "success": true,
  "data": {
    "syncId": "sync-Account-1702100000000",
    "jobId": "7501Y000000bNXQQAM-1702100000000",
    "syncStatus": "IN_PROGRESS",
    "updatedRecords": 950,
    "failedRecords": 23,
    "durationMs": 45000
  }
}

Get Synchronization Status

GET /api/v1/sync/status?syncId=<sync-id>

Get Sync Logs

GET /api/v1/sync/logs?objectType=Account&status=SUCCESS&limit=50&skip=0

Get Specific Sync Details

GET /api/v1/sync/:syncId

Retry Failed Sync

POST /api/v1/sync/:syncId/retry

🔄 Data Synchronization Flow

Upload (MongoDB → Salesforce)

1. Query pending records from MongoDB
   └─ SELECT * FROM Account WHERE syncStatus = 'PENDING_SYNC'

2. Format records for Salesforce Bulk API
   └─ Convert MongoDB fields to Salesforce API names
   └─ Generate CSV for bulk upload

3. Create Bulk API 2.0 job
   └─ Call POST /services/async/bulk/2.0/jobs/Account

4. Upload CSV data
   └─ Call PUT /services/async/bulk/2.0/jobs/{jobId}

5. Monitor job completion
   └─ Poll job status every 5 seconds
   └─ Respect governor limits

6. Process results
   └─ Update MongoDB records with Salesforce IDs
   └─ Log sync results and any errors
   └─ Mark records as SYNCED or ERROR

Download (Salesforce → MongoDB)

1. Query Salesforce using SOQL
   └─ SELECT Id, Name, ... FROM Account 
   └─ WHERE LastModifiedDate >= :lastSync

2. Fetch records from Salesforce
   └─ Respect API governor limits
   └─ Use pagination for large result sets

3. Format and validate records
   └─ Ensure data integrity
   └─ Handle null/missing values

4. Upsert to MongoDB
   └─ Create new or update existing records
   └─ Mark as SYNCED

5. Track sync history
   └─ Log operation in SyncLog
   └─ Record timestamps and statistics

⚙️ Configuration

Environment Variables

# API Server
NODE_ENV=development
PORT=3000
API_VERSION=v1

# MongoDB
MONGODB_URI=mongodb://localhost:27017/trailsync
MONGODB_USER=
MONGODB_PASSWORD=

# Salesforce OAuth 2.0
SALESFORCE_CLIENT_ID=your_client_id
SALESFORCE_CLIENT_SECRET=your_client_secret
SALESFORCE_LOGIN_URL=https://login.salesforce.com
SALESFORCE_REDIRECT_URI=http://localhost:3000/auth/salesforce/callback

# Salesforce Instance
SALESFORCE_INSTANCE_URL=https://your-instance.salesforce.com
SALESFORCE_USERNAME=user@example.com
SALESFORCE_PASSWORD=password123

# Bulk API
SALESFORCE_BATCH_SIZE=10000
SALESFORCE_POLL_INTERVAL=5000
SALESFORCE_MAX_RETRIES=3

# Security
JWT_SECRET=your_jwt_secret
CORS_ORIGIN=http://localhost:3000

# Logging
LOG_LEVEL=info

# Rate Limiting
RATE_LIMIT_WINDOW_MS=900000
RATE_LIMIT_MAX_REQUESTS=100

🛡️ Best Practices

1. Secure Credential Management

// ✅ DO: Use environment variables
const clientId = process.env.SALESFORCE_CLIENT_ID;

// ❌ DON'T: Hard-code credentials
const clientId = "3MVG9QDx78ZZZ.z_QcW_bGSNqLiVvhN1VlKM3SjLa";

2. Token Handling

// ✅ DO: Refresh tokens automatically
const newToken = await refreshAccessToken(oldToken);

// ✅ DO: Store tokens securely server-side
session.accessToken = token;

// ❌ DON'T: Expose tokens to clients
res.json({ accessToken: token });

3. Governor Limit Management

// ✅ DO: Check limits before API calls
const limits = await checkGovernorLimits(connection);
if (limits.apiCalls.remaining < 100) {
  return res.status(429).json({ error: 'Rate limit exceeded' });
}

// ✅ DO: Use Bulk API for large operations
const result = await executeBulkSync('Account', { batchSize: 10000 });

// ❌ DON'T: Make unlimited sequential REST API calls
for (let i = 0; i < 100000; i++) {
  await create_record(); // This would exceed governor limits!
}

4. Error Handling & Logging

// ✅ DO: Log all sync operations
logger.info('Sync completed', {
  objectType: 'Account',
  recordsProcessed: 1000,
  recordsFailed: 5,
  durationMs: 45000
});

// ✅ DO: Retry with exponential backoff
const maxRetries = 3;
let retryCount = 0;
while (retryCount < maxRetries) {
  try {
    return await syncOperation();
  } catch (error) {
    retryCount++;
    const delay = Math.pow(2, retryCount) * 1000; // 2s, 4s, 8s
    await sleep(delay);
  }
}

// ❌ DON'T: Expose internal errors to clients
res.json({ error: err.query.column }); // Leaks information!

5. Rate Limiting in Production

// Apply stricter limits for sync operations
const syncLimiter = rateLimit({
  windowMs: 1 * 60 * 1000, // 1 minute
  max: 10, // 10 sync requests per minute
  message: 'Too many sync requests, please try again later'
});

app.post('/api/v1/sync/bulk', syncLimiter, triggerBulkSync);

📊 Monitoring & Observability

Logging

TrailSync uses Winston for structured logging:

// Logs are written to:
// - Console (development)
// - logs/error.log (errors only)
// - logs/combined.log (all levels)

logger.info('Account sync completed', {
  accountId: '001xx000003DHP',
  duration: 120,
  status: 'SUCCESS'
});

Sync History Tracking

All synchronization operations are logged in MongoDB SyncLog collection:

{
  syncId: "sync-Account-1702100000000",
  syncType: "UPLOAD",
  objectType: "Account",
  status: "SUCCESS",
  recordsProcessed: 1050,
  recordsSucceeded: 1025,
  recordsFailed: 25,
  batchJobId: "7501Y000000bNXQQAM",
  durationMs: 45000,
  createdAt: "2024-01-10T10:30:00Z"
}

Health Monitoring

# Check service health
curl http://localhost:3000/health

# Response
{
  "status": "healthy",
  "service": "TrailSync",
  "timestamp": "2024-01-10T10:30:00Z",
  "environment": "production"
}

🚨 Common Issues & Solutions

Issue: "Governor limit exceeded"

Cause: Too many API calls to Salesforce in short period.

Solution:

# 1. Reduce batch size in .env
SALESFORCE_BATCH_SIZE=5000

# 2. Increase poll interval
SALESFORCE_POLL_INTERVAL=10000

# 3. Enable rate limiting
RATE_LIMIT_MAX_REQUESTS=50

Issue: "OAuth token expired"

Cause: Access token expired, refresh token needed.

Solution: TrailSync automatically refreshes tokens. Ensure offline_access scope is enabled in Connected App.

Issue: "MongoDB connection failed"

Cause: MongoDB service not running or connection string incorrect.

Solution:

# Verify MongoDB is running
mongo --version

# Check connection string
echo $MONGODB_URI

# Test connection
mongosh "mongodb://localhost:27017/trailsync"

📈 Performance Optimization

Sync Performance

Bulk API 2.0: Handles 1000+ records per job
Batch Size: Default 10,000 records
Average Processing: ~45 seconds for 1000 records
API Efficiency: 1 API call per batch vs. 1 per record

Database Optimization

// Indexes created for fast queries
db.accounts.createIndex({ syncStatus: 1 });
db.accounts.createIndex({ sfId: 1 }, { unique: true });
db.synclogs.createIndex({ objectType: 1, status: 1 });

🔄 Deployment

Docker Deployment

FROM node:18-alpine

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

COPY src ./src
ENV NODE_ENV=production

EXPOSE 3000
CMD ["npm", "start"]

# Build
docker build -t trailsync:1.0.0 .

# Run
docker run -p 3000:3000 \
  -e SALESFORCE_CLIENT_ID=xxx \
  -e SALESFORCE_CLIENT_SECRET=xxx \
  -e MONGODB_URI=mongodb://mongo:27017/trailsync \
  trailsync:1.0.0

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trailsync
spec:
  replicas: 3
  selector:
    matchLabels:
      app: trailsync
  template:
    metadata:
      labels:
        app: trailsync
    spec:
      containers:
      - name: trailsync
        image: trailsync:1.0.0
        ports:
        - containerPort: 3000
        env:
        - name: SALESFORCE_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: salesforce-creds
              key: client-id
        - name: SALESFORCE_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: salesforce-creds
              key: client-secret
        - name: MONGODB_URI
          value: mongodb://mongodb:27017/trailsync
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 5

📚 Resources

📝 License

MIT License - See LICENSE file for details

🤝 Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch
Make your changes
Submit a pull request

📞 Support

For issues and questions:

Check the Configuration section above
Review Salesforce documentation
Open an issue on GitHub

TrailSync - Secure CRM data synchronization with Salesforce and MongoDB ✨

Built with security, scalability, and Salesforce Trust principles in mind.

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
src		src
Dockerfile		Dockerfile
PROJECT_SUMMARY.md		PROJECT_SUMMARY.md
QUICKSTART.md		QUICKSTART.md
README.md		README.md
docker-compose.yml		docker-compose.yml
package.json		package.json

Folders and files

Latest commit

History

Repository files navigation