AWS CloudFront Integration

Ansehn's automated CloudFront integration uses Amazon Data Firehose to stream your CloudFront access logs directly to Ansehn in real time — no manual uploads, no scripts, no maintenance. Once connected, Ansehn will automatically detect and surface AI crawler activity (GPTBot, ClaudeBot, PerplexityBot, Gemini, and more) across your distribution.

Prerequisites

Before you begin, confirm you have the following:

• An active AWS account with permissions to create Amazon Data Firehose delivery streams

• A CloudFront distribution with Standard Logging v2 available (available on all distributions — no additional CloudFront configuration required beforehand)

• An Ansehn account with an active project for the domain associated with your distribution

How it works

CloudFront Distribution
        │
        │  Real-time log events (JSON)
        ▼
Amazon Data Firehose
        │
        │  HTTPS · GZIP · API Key auth
        ▼
Ansehn Ingest API  →  Your Crawler Analytics Dashboard

CloudFront streams log events to a Firehose delivery stream you control. Firehose batches and forwards those events securely to Ansehn using an API key you generate in your Ansehn dashboard. Zero latency is added to your end-user requests — all log forwarding happens asynchronously off the critical path.

Step-by-step setup

Step 1 — Generate an API key in Ansehn

note

API key generation is available in your dashboard under Project Settings → API Keys.

In your Ansehn dashboard:

1. Navigate to Projects → [Your Project] → Settings → API Keys

2. Click + New API Key

3. Give the key a descriptive name (e.g., Production CloudFront)

4. Under Scopes, select logs:ingest

5. Under Project, select the project for the domain associated with your CloudFront distribution

6. Click Generate and copy the key immediately — it is only shown once

Your key will look like: ank_a1b2c3d4e5f6...

Step 2 — Create a Firehose delivery stream

1. Sign in to the AWS Console and navigate to Amazon Data Firehose

2. Click Create Firehose stream

3. Configure the stream:

   • Source: Direct PUT

   • Destination: HTTP endpoint

   • Stream name: Something descriptive, e.g., ansehn-cloudfront-logs

Step 3 — Configure the HTTP endpoint

Within the Destination settings section:

Storing your API key securely (recommended)

We strongly recommend storing your API key in AWS Secrets Manager rather than entering it as plain text in the Firehose configuration. Firehose natively supports Secrets Manager — this keeps your key encrypted at rest, auditable via CloudTrail, and rotatable without recreating the stream.

1. Open AWS Secrets Manager and create a new secret

2. Select Other type of secret and enter the following JSON:

   { "api_key": "ank_your_key_here" }

3. Name the secret something descriptive (e.g., ansehn/cloudfront-logs/api-key)

4. Back in the Firehose configuration, select Use AWS Secrets Manager and choose the secret you just created

If you prefer to enter the key directly (e.g., for a quick test), you can paste it into the Access key field — but we recommend migrating to Secrets Manager before going to production.

Under Backup settings, create or select an S3 bucket to store any failed delivery records. This is important — the S3 backup bucket is your safety net. If any log records cannot be delivered (e.g., during a brief service interruption), they are written to this bucket. Ansehn support can help you replay these records if needed. We recommend setting a 14-day retention policy on this bucket.

Step 4 — Attach the stream to your CloudFront distribution

1. Open the CloudFront Console and select your distribution

2. Click the Logging tab

3. Click Add and select Amazon Kinesis Data Firehose (Kinesis Data Firehose is the legacy name for Amazon Data Firehose — they are the same service)

4. Select the delivery stream you created in Step 2

Step 5 — Configure log fields

In the Additional settings section, select exactly the following 12 fields. Ansehn requires all of them to parse your logs correctly — omitting any field will cause those log records to be skipped.

Set the Output format to JSON.

Pre-submit checklist

Before clicking Submit, verify the following. Misconfiguration here is the #1 cause of logs not appearing in Ansehn:

• All 12 fields above are selected (no more, no fewer)

• Output format is set to JSON (not CSV or plain text)

• The correct Firehose delivery stream is selected (the one from Step 2)

• The stream is attached to the correct CloudFront distribution

Click Submit to save.

Verification

Once the integration is live and your Firehose stream is active, you can verify everything is working by:

1. Visiting a page on your site from a browser

2. Waiting approximately 60–90 seconds (Firehose default buffer interval)

3. Checking your Ansehn Server Logs dashboard — you should see activity appear under Daily Visits

Your dashboard will also display a connection status indicator next to the tab bar showing whether logs are flowing and when the last ingest occurred. Hover over it for detailed diagnostics including total batches processed and AI crawler detection status.

What you will see in Ansehn

Once connected, your Server Logs dashboard will automatically show:

• Daily AI crawler visits — broken down by GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Gemini, Google Extended, and more

• Top crawled pages — which URLs AI search engines are indexing most frequently

• Crawler breakdown per page — understand which AI assistants are most interested in which content

• Historical trends — track changes over time as you publish new content

All data is scoped to your project domain and isolated from other Ansehn customers.

Frequently asked questions

Troubleshooting

If logs are not appearing in your Ansehn dashboard after 5 minutes, use the following diagnostics:

1. Check Firehose delivery status

1. Open the Amazon Data Firehose console

2. Select your delivery stream (e.g., ansehn-cloudfront-logs)

3. Click the Monitoring tab

4. Look at these CloudWatch metrics:

2. Check the S3 backup bucket and dashboard for error details

note

Ansehn returns HTTP 200 for permanent errors (like invalid API keys) to prevent Firehose from endlessly retrying unrecoverable issues. This means a "successful" delivery status in Firehose does not guarantee data reached your dashboard. Always verify by checking the connection status indicator in your Server Logs dashboard.

3. Verify CloudFront is generating logs

1. Open your CloudFront distribution → Logging tab

2. Confirm a logging destination is listed and its status is Enabled

3. Visit a page on your site and check the Firehose IncomingRecords metric after 60 seconds

4. Common mistakes

• Wrong field selection: Adding or removing fields from the required 12 will cause all records to be rejected. Double-check against the table in Step 5.

• Output format not JSON: If set to CSV or plain text, Ansehn cannot parse the logs. Must be JSON.

• Stream attached to wrong distribution: Verify the delivery stream is linked to the correct CloudFront distribution in the Logging tab.

• API key from wrong project: The API key must belong to the project that matches the domain on the CloudFront distribution. A key from a different project will be rejected.

Need help?

If you run into any issues during setup or have questions about the integration, reach out to us at [email protected] and mention "CloudFront Integration" in the subject line. Our team is happy to walk through the AWS configuration with you directly.