Historical Data Backfill

Your data follows you to Pigeon Perch. Import email engagement history, web tracking events, and location signals from your previous platform so you can pick up right where you left off.

Overview

Switching email platforms usually means starting from scratch. You lose years of open and click history, your engagement scores reset to zero, and your carefully built segments go blank. Backfill solves that problem.

With Pigeon Perch's backfill endpoints, you can import historical events with their original timestamps. Once the data is in, your analytics dashboards, engagement scores, decay calculations, and dynamic segments all work as if those events happened natively in Pigeon Perch. There is no difference between a backfilled event and a live one.

Whether you are migrating from Mailchimp, Klaviyo, SendGrid, or a custom system, backfill gives you a clean migration path that preserves your behavioral history.

How It Works

Every event table in Pigeon Perch has an occurredAt field. This timestamp represents when the event actually happened in the real world, not when it was recorded in the database. When you send a batch of historical events through the backfill API, you provide the original timestamp for each event.

Once imported, those timestamps flow through the entire system:

• Engagement scoring uses occurredAt to calculate time-weighted decay. A click from last week counts more than a click from six months ago, regardless of when it was imported.
• Analytics charts plot events on their original dates, so your historical performance graphs look accurate from day one.
• Dynamic segments evaluate rules like "opened an email in the last 30 days" against occurredAt, meaning contacts qualify for segments based on their real activity timeline.
• Location intelligence uses occurredAt to weight recent signals more heavily in interest scoring.

Backfill Endpoints

Pigeon Perch provides four batch endpoints for importing historical data. Each accepts an array of events in a single request.

All endpoints require API key authentication via the X-API-Key header. Events are validated, inserted, and the response tells you exactly how many succeeded, how many were skipped, and what went wrong with any failures.

Email Event Backfill

Use POST /api/v1/events/email/batch to import historical email engagement data. Each event needs a contact identifier, an event type, and the original timestamp.

Valid Event Types

sent, delivered, opened, clicked, bounced, unsubscribed, complained

Example Request

POST /api/v1/events/email/batch
X-API-Key: your-api-key

{
  "events": [
    {
      "contactEmail": "sarah@example.com",
      "type": "opened",
      "occurredAt": "2025-11-15T10:30:00Z",
      "metadata": {
        "externalCampaignId": "mc_campaign_4829",
        "subject": "Holiday Gift Guide"
      }
    },
    {
      "contactEmail": "sarah@example.com",
      "type": "clicked",
      "occurredAt": "2025-11-15T10:31:22Z",
      "metadata": {
        "externalCampaignId": "mc_campaign_4829",
        "url": "https://shop.example.com/gifts"
      }
    },
    {
      "contactEmail": "james@example.com",
      "type": "bounced",
      "occurredAt": "2025-11-14T08:00:00Z",
      "metadata": {
        "bounceType": "hard",
        "externalCampaignId": "mc_campaign_4828"
      }
    }
  ]
}

The metadata field is optional and accepts any key-value pairs. It is a good place to store your old platform's campaign IDs, subject lines, or link URLs so you can cross-reference later. Metadata is stored as-is and appears in contact activity timelines.

Web Event Backfill

Use POST /api/v1/events/web/batch to import historical website activity. This is especially useful if you tracked page views, conversions, or form submissions on your previous platform and want that history reflected in engagement scores and segments.

Valid Event Types

page_view, conversion, click, form_submit, custom

Example Request

POST /api/v1/events/web/batch
X-API-Key: your-api-key

{
  "events": [
    {
      "contactEmail": "sarah@example.com",
      "type": "page_view",
      "occurredAt": "2025-12-01T14:22:00Z",
      "metadata": {
        "url": "https://shop.example.com/pricing",
        "referrer": "https://google.com"
      }
    },
    {
      "contactEmail": "sarah@example.com",
      "type": "conversion",
      "occurredAt": "2025-12-01T14:25:00Z",
      "metadata": {
        "url": "https://shop.example.com/checkout/complete",
        "value": 129.99,
        "orderId": "ORD-9182"
      }
    },
    {
      "contactEmail": "james@example.com",
      "type": "form_submit",
      "occurredAt": "2025-11-28T09:15:00Z",
      "metadata": {
        "formName": "Contact Us",
        "url": "https://shop.example.com/contact"
      }
    }
  ]
}

Location Signal Backfill

If you have historical data about where your contacts have shown interest (search queries for cities, listing views in specific areas, form submissions mentioning locations), you can import those signals using the location interest batch endpoint.

Location signals use occurredAt to calculate decay-weighted interest scores. A contact who searched for "Austin homes" last week will have a higher Austin interest score than someone who searched six months ago. By importing signals with accurate timestamps, you preserve this recency weighting from day one.

Example: Location Interests

POST /api/v1/contacts/location-interests/batch
X-API-Key: your-api-key

{
  "signals": [
    {
      "contactEmail": "sarah@example.com",
      "location": "Austin, TX",
      "signalType": "search",
      "strength": 0.8,
      "occurredAt": "2025-12-10T16:00:00Z"
    },
    {
      "contactEmail": "sarah@example.com",
      "location": "Denver, CO",
      "signalType": "listing_view",
      "strength": 0.5,
      "occurredAt": "2025-11-20T12:30:00Z"
    }
  ]
}

Example: Location History

POST /api/v1/contacts/location-history/batch
X-API-Key: your-api-key

{
  "observations": [
    {
      "contactEmail": "sarah@example.com",
      "location": "Austin, TX",
      "source": "ip_geolocation",
      "occurredAt": "2025-12-05T09:00:00Z"
    },
    {
      "contactEmail": "james@example.com",
      "location": "Chicago, IL",
      "source": "form_submission",
      "occurredAt": "2025-11-30T11:00:00Z"
    }
  ]
}

Triggering Score Refresh

After importing historical events, you will want to recalculate engagement scores and location interest scores for the affected contacts. Pigeon Perch does not automatically recompute scores on every batch import (doing so mid-migration would waste processing time). Instead, call the refresh endpoint once you have finished importing all your data.

POST /api/v1/contacts/backfill/refresh
X-API-Key: your-api-key

{
  "contactIds": [
    "c_abc123",
    "c_def456",
    "c_ghi789"
  ]
}

You can pass up to 1,000 contact IDs per request. The endpoint kicks off an asynchronous recomputation that updates engagement scores, location interest scores, and segment memberships for each contact. Processing typically completes within a few seconds for small batches and a few minutes for larger ones.

If you imported data for thousands of contacts, call this endpoint in batches of 1,000 until you have covered everyone. You do not need to wait for one batch to finish before sending the next.

Validation Rules

Every event in a batch is validated individually. Events that fail validation are skipped (not imported), and the response tells you exactly which ones failed and why. Valid events in the same batch are still imported successfully.

Response Format

All batch endpoints return the same response structure, giving you a clear breakdown of what happened with each event in the batch.

{
  "imported": 847,
  "skipped": 3,
  "errors": [
    {
      "index": 12,
      "contactEmail": "unknown@example.com",
      "reason": "Contact not found"
    },
    {
      "index": 45,
      "contactEmail": "sarah@example.com",
      "reason": "occurredAt is in the future"
    },
    {
      "index": 199,
      "contactEmail": "james@example.com",
      "reason": "Invalid event type: 'forward'"
    }
  ]
}

• imported is the number of events that were successfully stored.
• skipped is the number of events that failed validation and were not imported.
• errors is an array with details about each skipped event, including its position in the original array (index), the contact it referenced, and a human-readable reason for the failure.

Migration Guide

Follow these steps to migrate your full history from another platform into Pigeon Perch. The order matters because events reference contacts, and location signals reference contacts too.

1. Import your contacts first. Use the POST /api/v1/contacts/batch endpoint or CSV upload in the dashboard to create all your contacts. Every contact needs to exist before you can attach events to them.
2. Import email events. Export your send, open, click, bounce, and unsubscribe history from your old platform. Map each event to the Pigeon Perch format and send them in batches of up to 10,000 using /api/v1/events/email/batch.
3. Import web events. If you have page view, conversion, or form submission data, import it using /api/v1/events/web/batch.
4. Import location signals. If you tracked location interest data (searches, listing views, geographic preferences), import those using the location interest and location history batch endpoints.
5. Trigger a score refresh. Once all your data is imported, call /api/v1/contacts/backfill/refresh with the IDs of all affected contacts. This recalculates engagement scores, location scores, and segment memberships.
6. Verify in the dashboard. Open a few contact profiles and check that their activity timelines, engagement scores, and segment memberships look correct. Spot-check your analytics charts to confirm historical data is plotting on the right dates.

Chunking Large Datasets

If you have hundreds of thousands or millions of events to import, break them into chunks that fit within the batch limits. A simple approach:

• Split your export file into chunks of 10,000 events (or 5,000 for location data).
• Send each chunk as a separate API request. You can send requests in parallel to speed things up, but keep it to 5-10 concurrent requests to avoid hitting rate limits.
• Log the response for each chunk. If any events were skipped, review the errors and fix the source data before retrying those specific events.
• After all chunks are processed, trigger the score refresh for all affected contacts.

For very large migrations (over 1 million events), consider running the import overnight or during off-peak hours. The backfill endpoints are designed to handle high throughput, but spreading the load gives you time to monitor for any issues.

Quick Reference