Webhooks Guide

This document explains how DeDuplica securely delivers webhooks and how customers can verify that incoming webhook requests are authentic, untampered, and recent.

The approach described here is industry-standard and aligns with patterns used by platforms such as Stripe, GitHub, and Slack.

1. How DeDuplica Secures Webhooks

Every webhook sent by DeDuplica is protected using HMAC-SHA256 signatures.

This ensures that:

The webhook was sent by DeDuplica
The payload has not been modified in transit
Old or replayed requests can be safely rejected

Each webhook subscription is associated with a unique secret, shared only between DeDuplica and your system.

2. High-Level Flow

DeDuplica generates a webhook payload
The payload is signed using your webhook secret
DeDuplica sends the webhook to your endpoint
Your endpoint recomputes the signature and validates it
If valid, your system processes the webhook

3. Webhook Payload Structure

DeDuplica sends the business payload directly as the HTTP request body. There is no wrapper or envelope object.

The JSON body follows this schema:

public class WebhookPayload
{
    public string DuplicateId { get; set; }
    public DateTime DateIssued { get; set; }
    public DateTime DateFound { get; set; }
    public string SubscriptionExternalId { get; set; }
    public string JobExternalId { get; set; }
    public string JobExecutionExternalId { get; set; }
    public string TableName { get; set; }
    public double Probability { get; set; }
    public string MatchRecordJson1 { get; set; }
    public string MatchRecordJson2 { get; set; }
    public string MatchRecordId1 { get; set; }
    public string MatchRecordId2 { get; set; }
    public string MergeOutputJson { get; set; }
}

Field Summary

Field	Description
DuplicateId	Unique identifier of the detected duplicate
DateIssued	When the webhook was issued by DeDuplica (UTC)
DateFound	When the duplicate was detected (UTC)
SubscriptionExternalId	Customer-defined subscription identifier
JobExternalId	Customer-defined job identifier
JobExecutionExternalId	Customer-defined job execution identifier
TableName	Source table where the duplicate was detected
Probability	Confidence score (0.0 – 1.0)
MatchRecordJson1	First matched record (JSON string)
MatchRecordJson2	Second matched record (JSON string)
MatchRecordId1	Identifier of first matched record
MatchRecordId2	Identifier of second matched record
MergeOutputJson	Optional merge result produced by DeDuplica

Important Notes

All timestamps are sent in UTC (ISO‑8601) format
MatchRecordJson* fields may contain large JSON documents
The exact raw HTTP request body is used for signature verification

4. Signature Generation (Conceptual)

DeDuplica computes the signature using the following formula:

message = "{timestamp}.{payload}"
signature = HMAC_SHA256(secret, message)

The signature is sent as a hex-encoded string, optionally prefixed with:

sha256=<signature>

Your system must independently compute the same signature and compare it.

5. Webhook Secret

Each subscription has its own unique secret
Secrets are Base64-encoded for safe storage
Secrets must be kept confidential

When validating a webhook, always Base64-decode the secret before using it.

6. Validation Rules (Required)

Your webhook endpoint should accept a webhook only if all checks pass:

The computed signature matches the provided signature
The timestamp is within an acceptable time window (recommended: ±5 minutes)
The comparison is done using a constant-time method

If any check fails, respond with HTTP 401 Unauthorized.

7. Minimal Azure Function Examples

The following examples show minimal, production-safe webhook handlers for common Azure Function runtimes.

All examples assume:

Your webhook secret is stored in an environment variable: WEBHOOK_SECRET
A successful validation returns HTTP 200

7.1 Azure Function – Node.js

const crypto = require("crypto");

module.exports = async function (context, req) {
    const secret = Buffer.from(process.env.WEBHOOK_SECRET, "base64");

    const signature = req.headers["x-webhook-signature"] || "";
    const timestamp = req.headers["x-webhook-timestamp"];

    // Use the raw body exactly as received for verification
    const rawBody = JSON.stringify(req.body);
    const message = `${timestamp}.${rawBody}`;

    const expected = crypto
        .createHmac("sha256", secret)
        .update(message, "utf8")
        .digest("hex");

    const received = signature.replace("sha256=", "");

    const isValid = crypto.timingSafeEqual(
        Buffer.from(expected),
        Buffer.from(received)
    );

    if (!isValid) {
        context.res = { status: 401 };
        return;
    }

    // ✅ Safe to parse and use the payload after verification
    const webhook = req.body;

    context.log(`Duplicate detected: ${webhook.DuplicateId}`);
    context.log(`Probability: ${webhook.Probability}`);

    context.res = { status: 200 };
};

7.2 Azure Function – .NET 8 (Isolated)

using System.Security.Cryptography;
using System.Text;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;

public class DeduplicaWebhook
{
    [Function("DeduplicaWebhook")]
    public static async Task<HttpResponseData> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req)
    {
        var rawBody = await new StreamReader(req.Body).ReadToEndAsync();

        var timestamp = req.Headers.GetValues("X-Webhook-Timestamp").First();
        var signature = req.Headers.GetValues("X-Webhook-Signature").First();

        var secret = Convert.FromBase64String(
            Environment.GetEnvironmentVariable("WEBHOOK_SECRET")!
        );

        var message = $"{timestamp}.{rawBody}";

        using var hmac = new HMACSHA256(secret);
        var expected = Convert.ToHexString(
            hmac.ComputeHash(Encoding.UTF8.GetBytes(message))
        ).ToLowerInvariant();

        var received = signature.Replace("sha256=", "");

        var isValid = CryptographicOperations.FixedTimeEquals(
            Encoding.UTF8.GetBytes(expected),
            Encoding.UTF8.GetBytes(received)
        );

        if (!isValid)
        {
            return req.CreateResponse(401);
        }

        // ✅ Safe to deserialize after verification
        var webhook = System.Text.Json.JsonSerializer.Deserialize<WebhookPayload>(rawBody)!;

        Console.WriteLine($"Duplicate detected: {webhook.DuplicateId}");
        Console.WriteLine($"Probability: {webhook.Probability}");

        return req.CreateResponse(200);
    }
}

7.3 Azure Function – Python

import os
import hmac
import hashlib
import base64
import json
import azure.functions as func


def main(req: func.HttpRequest) -> func.HttpResponse:
    signature = req.headers.get("x-webhook-signature", "").replace("sha256=", "")
    timestamp = req.headers.get("x-webhook-timestamp")

    raw_body = req.get_body().decode("utf-8")

    secret = base64.b64decode(os.environ.get("WEBHOOK_SECRET"))
    message = f"{timestamp}.{raw_body}".encode("utf-8")

    expected = hmac.new(secret, message, hashlib.sha256).hexdigest()

    if not hmac.compare_digest(expected, signature):
        return func.HttpResponse(status_code=401)

    # ✅ Safe to parse after verification
    webhook = json.loads(raw_body)

    duplicate_id = webhook.get("DuplicateId")
    probability = webhook.get("Probability")

    print(f"Duplicate detected: {duplicate_id}")
    print(f"Probability: {probability}")

    return func.HttpResponse(status_code=200)

Webhook Delivery Timing and Retries

Timeout: Each webhook call has a maximum timeout of 60 seconds. If your server does not respond within this time, the attempt is considered failed. You should accept webhook call as soon as possible and then handle it on the consumer side with error handling
Success Response: To confirm successful processing, your endpoint should return any HTTP status code in the 2xx range (such as 200 OK or 204 No Content).
Retries: If a webhook delivery fails (timeout or non-2xx response), DeDuplica will automatically retry up to 5 times.
Exponential Backoff: Retries use increasing wait times between attempts, starting at 10 seconds and growing up to 30 minutes (e.g., 10s, 20s, 1m, 5m, 30m).

This approach helps ensure your system receives important webhook notifications, even if your server is temporarily unavailable. If all retries fail, the webhook will be marked as undelivered in your logs.

System settings Duplicates