Data Sources Integration Guide

Connect your external data sources to enrich your AI agent's knowledge. Integrate product catalogs, knowledge bases, and other data to provide more relevant and helpful responses to your customers.

What are Data Sources?

Data Sources allow you to connect external APIs containing your business data (products, courses, articles, services, etc.) to your Chatplugify AI agent. This enables your AI to provide specific, accurate information about your offerings during customer conversations.

External API Integration

Connect any API that follows our format

Automatic Sync

Hourly synchronization keeps data current

AI Enhancement

AI agents can reference your real data

How It Works

1

Configure Your API Endpoint

Set up your external API endpoint that returns your data in the required format. Configure authentication headers if needed.

2

Test Connection

Test the connection to ensure your API is responding correctly and data format is valid.

3

Automatic Synchronization

Our system automatically syncs your data every hour, calling your API with pagination parameters.

4

AI Enhancement

Your AI agent can now reference your real data during conversations, providing accurate product information, pricing, and recommendations.

API Requirements

Required Response Format

Your API endpoint must return data in the following JSON format:

API Response Example
{
  "items": [
    {
      "id": "prod_001",
      "title": "Premium Wireless Headphones",
      "description": "High-quality wireless headphones with noise cancellation",
      "category": "Electronics",
      "type": "product",
      "price": "299.99",
      "url": "https://example.com/headphones",
      "image": "https://example.com/headphones.jpg",
      "metadata": {
        "brand": "AudioTech",
        "color": "Black",
        "weight": "250g",
        "rating": 4.5,
        "inStock": true
      }
    },
    {
      "id": "course_001",
      "title": "Complete JavaScript Course",
      "description": "Learn JavaScript from basics to advanced concepts",
      "category": "Programming",
      "type": "course",
      "price": "99.99",
      "url": "https://example.com/js-course",
      "image": "https://example.com/js-course.jpg",
      "metadata": {
        "duration": "40 hours",
        "difficulty": "Beginner",
        "instructor": "John Developer",
        "certification": true
      }
    }
  ],
  "hasMore": true,
  "nextCursor": "course_001"
}

Required Fields

idUnique identifier for the item
titleDisplay name of the item
descriptionDetailed description
categoryItem category
typeItem type (product, course, article, etc.)
priceItem price (can be "0" or "Free")
urlDirect link to the item
imageURL to item image

Optional Metadata

The metadata object is optional but highly recommended for better AI responses:

Physical Products:

  • • brand, color, size, weight
  • • rating, reviews, inStock
  • • material, dimensions

Courses/Services:

  • • duration, difficulty, instructor
  • • certification, language
  • • location, availability

Pagination Support

Our system calls your API with pagination parameters to handle large datasets efficiently:

How We Call Your API

POST https://your-api.com/items
Content-Type: application/json
Your-API-Key: your-secret-key

{
  "after": null,        // First request
  "limit": 100         // Items per page
}

// Subsequent requests:
{
  "after": "item_100_id",  // Last item ID from previous response
  "limit": 100
}

Your Response Format

  • items: Array of your items in the required format
  • hasMore: Boolean indicating if there are more items
  • nextCursor: ID of the last item in current batch (for pagination)

Authentication

Configure custom authentication headers to secure your API endpoint:

Common Header Patterns

x-api-key: abc123def456
Authorization: Bearer token123
api-token: my-secret-token

Security Notes

  • • API keys are encrypted at rest
  • • Only HTTPS endpoints are supported
  • • Workspace isolation ensures data security

Implementation Examples

Node.js / Express Implementation

server.js
const express = require('express');
const app = express();

app.use(express.json());

// Your API endpoint that Chatplugify will call
app.post('/api/items', async (req, res) => {
  try {
    const { after, limit } = req.body;
    
    // Get items from your database
    const items = await getItemsFromDatabase(after, limit);
    
    // Check if there are more items
    const hasMore = await checkIfMoreItems(items[items.length - 1]?.id);
    
    // Format response according to Chatplugify requirements
    const response = {
      items: items.map(item => ({
        id: item.id,
        title: item.title,
        description: item.description,
        category: item.category,
        type: item.type,
        price: item.price.toString(),
        url: item.url,
        image: item.image,
        metadata: {
          brand: item.brand,
          rating: item.rating,
          inStock: item.stock > 0,
          // Add any other relevant metadata
        }
      })),
      hasMore: hasMore,
      nextCursor: items.length > 0 ? items[items.length - 1].id : null
    };
    
    res.json(response);
  } catch (error) {
    res.status(500).json({ error: 'Internal server error' });
  }
});

async function getItemsFromDatabase(after, limit) {
  // Your database query logic here
  // Example with SQL:
  let query = 'SELECT * FROM products';
  let params = [];
  
  if (after) {
    query += ' WHERE id > ?';
    params.push(after);
  }
  
  query += ' ORDER BY id LIMIT ?';
  params.push(limit);
  
  return await db.query(query, params);
}

app.listen(3000, () => {
  console.log('API server running on port 3000');
});

Python / Flask Implementation

app.py
from flask import Flask, request, jsonify
import sqlite3

app = Flask(__name__)

@app.route('/api/items', methods=['POST'])
def get_items():
    try:
        data = request.get_json()
        after = data.get('after')
        limit = data.get('limit', 100)
        
        # Get items from your database
        items = get_items_from_database(after, limit)
        
        # Check if there are more items
        has_more = check_if_more_items(items[-1]['id'] if items else None)
        
        # Format response according to Chatplugify requirements
        response = {
            "items": [
                {
                    "id": item['id'],
                    "title": item['title'],
                    "description": item['description'],
                    "category": item['category'],
                    "type": item['type'],
                    "price": str(item['price']),
                    "url": item['url'],
                    "image": item['image'],
                    "metadata": {
                        "brand": item.get('brand'),
                        "rating": item.get('rating'),
                        "inStock": item.get('stock', 0) > 0,
                        # Add any other relevant metadata
                    }
                }
                for item in items
            ],
            "hasMore": has_more,
            "nextCursor": items[-1]['id'] if items else None
        }
        
        return jsonify(response)
    
    except Exception as e:
        return jsonify({"error": "Internal server error"}), 500

def get_items_from_database(after, limit):
    conn = sqlite3.connect('products.db')
    cursor = conn.cursor()
    
    if after:
        cursor.execute("""
            SELECT * FROM products 
            WHERE id > ? 
            ORDER BY id 
            LIMIT ?
        """, (after, limit))
    else:
        cursor.execute("""
            SELECT * FROM products 
            ORDER BY id 
            LIMIT ?
        """, (limit,))
    
    items = cursor.fetchall()
    conn.close()
    
    return [dict(zip([col[0] for col in cursor.description], item)) for item in items]

if __name__ == '__main__':
    app.run(debug=True, port=3000)

Best Practices

Data Quality

Ensure your data is well-structured with clear titles and descriptions. The AI performs better with high-quality, detailed information.

Consistent Formatting

Use consistent formatting for prices, categories, and metadata across all items for better AI understanding.

Performance Optimization

Implement efficient pagination and caching to handle large datasets. Response times should be under 5 seconds.

Error Handling

Implement proper error handling and return appropriate HTTP status codes. Failed syncs will be retried automatically.

Troubleshooting

Common Issues

Connection test fails

  • • Verify your API endpoint URL is correct and accessible
  • • Check authentication headers are properly configured
  • • Ensure your API responds to POST requests
  • • Verify the response format matches our requirements

Data not syncing

  • • Check that required fields (id, title, description) are present
  • • Verify pagination parameters are handled correctly
  • • Ensure hasMore and nextCursor are returned properly
  • • Check server logs for any API errors

Performance issues

  • • Optimize database queries for pagination
  • • Implement caching for frequently accessed data
  • • Consider using database indexes on relevant fields
  • • Monitor API response times (should be under 5 seconds)

Ready to Connect Your Data?

Head to your dashboard to set up your first data source and enhance your AI agent's capabilities.