Skip to main content

Server Logs Analytics

Server Logs Analytics reveals how AI companies access your website by analyzing your raw web server access logs. This is the ground truth of AI visibility—direct evidence of which AI crawlers are visiting your pages and when. Use it to verify access, find issues with robots.txt, prioritize content, and track trends over time.

Why this matters and where to find your logs

For background on why this matters and where to find your logs, see these guides:

Ansehn - Server Logs Analytics Dashboard for AI Search Monitoring and Optimization

What you can do with Server Logs Analytics​

  • Verify AI access: Confirm which AI companies are crawling your site (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Gemini, Google-Extended, and more).
  • Identify blocked crawlers: Spot missing bots and diagnose robots.txt issues.
  • Track trends: See daily activity by crawler to monitor increases, drops, and spikes.
  • Prioritize content: Focus on pages most visited by AI crawlers.
  • Troubleshoot: Detect access errors or anomalies early.

💡 Pro tip: Compare crawler activity here with your insights in Citation Analytics to understand how crawling correlates with actual citations in AI answers.

How it works​

Upload one or more server access log files and the dashboard will parse, filter, and aggregate crawler activity. We support standard web server access logs in Apache Common Log Format (also compatible with Nginx access logs).

Where to find your Server Logs​

Log locations vary by hosting setup. Managed WordPress platforms (e.g., WP Engine, Kinsta, SiteGround) expose access/error logs in their dashboards. Shared hosting (cPanel) provides Raw Access Logs and Error Logs. On VPS/bare metal, check the filesystem (Apache under /var/log/apache2/ or /var/log/httpd/; Nginx under /var/log/nginx/). In cloud environments, logs may flow to services like AWS S3/CloudWatch (ELB/CloudFront), Google Cloud Logging, or Azure Monitor. For containers use docker logs or kubectl logs. Serverless and edge platforms (e.g., Vercel, Cloudflare) surface request/bot activity through observability dashboards.

For step-by-step instructions by platform, see the full guide: How to Find Server Logs for Any Website: A Complete Guide for All Hosting Setups.

Accepted log format​

Each line must follow this structure:

IP - - [timestamp] "method path protocol" status size "referrer" "user-agent"

Format components:

  1. IP Address — client IP (e.g., 135.234.67.1). Anonymized IPs are supported (e.g., 135.234.67.xx, xxx.xxx.xxx.xxx, or *.*.*.*).
  2. Identity — usually -
  3. User — usually -
  4. Timestamp — [DD/MMM/YYYY:HH:MM:SS +0000] (e.g., [12/Jun/2025:06:46:46 +0000])
  5. Request — "METHOD /path HTTP/version" (e.g., "GET /page.html HTTP/1.1")
  6. Status Code — e.g., 200, 404
  7. Size — response bytes (e.g., 17744)
  8. Referrer — often "-" for bot traffic
  9. User-Agent — bot identification string

Example entry:

135.234.67.1 - - [12/Jun/2025:06:46:46 +0000] "GET /kuehlen-gefrieren/kuehlschraenke/siemens-kg39n2xaf-kuehl-gefrier-kombination.html HTTP/1.1" 200 17744 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot"

Example with anonymized IP:

172.204.16.xx - - [13/Oct/2025:22:16:46 +0000] "GET /marathon-running-running-shoes-37v7jz4h6ddzy7ok HTTP/1.1" 200 38433 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"

Supported AI crawlers​

The dashboard automatically detects these crawlers (case-insensitive):

  • OpenAI: GPTBot/1.2, OAI-SearchBot/1.0, ChatGPT-User/1.0
  • Anthropic: ClaudeBot, Claude-User, Claude-SearchBot
  • Perplexity: PerplexityBot/1.0, Perplexity-User
  • Google: Gemini, Google-Extended
  • Meta: meta-externalagent (if present in your logs)
  • ... and more

Multiple file upload​

You can upload multiple log files simultaneously. We will:

  1. Combine all log entries
  2. Parse them together
  3. Display aggregated statistics across the full timespan

This is ideal for rotated logs, multi-server setups, or comparing different time windows.

Where to find your logs​

See our comprehensive guide for step-by-step instructions by platform: How to Find Server Logs for Any Website.

Common locations:

  • Apache: /var/log/apache2/access.log, /var/log/httpd/access_log
  • Nginx: /var/log/nginx/access.log
  • Managed hosting: Access via your provider dashboard (e.g., WP Engine, Kinsta, SiteGround)
  • Cloud & edge: CloudWatch, Cloud Logging, Azure Monitor, Vercel/Cloudflare dashboards

File requirements​

  • Format: .log or .txt
  • Encoding: UTF-8
  • Size: Maximum 4.5 MB per upload (due to serverless function limits). For larger files, filter to AI crawler traffic first (see CLI examples below).
  • Compression: Not supported — extract archives before upload

Parsing & filtering rules​

The parser automatically:

  • Filters to include only AI crawler traffic
  • Excludes non-200 responses from page statistics
  • Removes /robots.txt requests from page rankings
  • Groups data by date and crawler type

Dashboard overview​

Daily Visits from AI Crawlers​

Shows the daily volume of requests from AI crawlers and bot types. Each color represents a different crawler, stacked to show total daily activity.

What to look for:

  • Increasing activity: More frequent crawling due to fresh content or rising visibility
  • Sudden spikes: New content discovery, viral mentions, or model retraining cycles
  • Declines: Potential blocking via robots.txt, stale content, or technical issues
  • Missing crawlers: Verify robots.txt and allow rules for key bots

Take action:

  • Publish regularly so crawlers return more often
  • Investigate drops quickly to prevent visibility loss
  • Prioritize high-traffic days/pages for optimization
  • Update sitemaps to accelerate discovery

💡 Pro tip: Cross-reference with Citation Analytics to see if crawl volume correlates with citations in answers.

Top Pages (by AI crawler visits)​

Highlights which pages AI crawlers visit most, with a breakdown by bot type. Pages with high visits are likely candidates for training data or citations.

Use this to:

  • Prioritize updates to pages that attract AI crawlers
  • Identify underperforming pages and content gaps
  • Replicate successful structures/topics
  • Detect if critical pages are unintentionally blocked

The dashboard displays up to 100 top pages, paginated 5 at a time.

Top Crawlers​

Visualizes relative activity levels across AI crawlers. A balanced shape suggests broad AI visibility; dominance indicates one platform finds your content especially relevant.

Signals to watch:

  • Balanced distribution: Healthy visibility across platforms
  • Missing segments: Possible blocking or limited relevance
  • High “User” activity (ChatGPT-User, Perplexity-User): Real users retrieving your content via AI search

Quick start​

  1. Export or download your web server access logs.
  2. If needed, filter down to AI bot traffic (see examples below).
  3. Upload one or more .log/.txt files to Server Logs Analytics.
  4. Review Daily Visits, Top Pages, and Top Crawlers to identify trends and opportunities.
  5. Adjust robots.txt/sitemaps and content strategy, then compare before/after.

CLI examples (optional)​

Filter for AI crawlers only (Linux/Mac):

# OpenAI bots
grep -i "gptbot\|chatgpt-user\|oai-searchbot" access.log > ai-crawler.log

# All AI bots
grep -iE "(gptbot|chatgpt-user|oai-searchbot|claudebot|perplexitybot|gemini|google-extended|meta-externalagent)" access.log > ai-crawler.log

Last 7 days only:

grep "$(date -d '7 days ago' '+%d/%b/%Y')" access.log > recent.log

Troubleshooting​

"No data available"

  • Ensure your log file contains AI crawler entries
  • Check the format matches Apache Common Log Format
  • Verify the User-Agent contains a recognized crawler pattern

"Failed to upload log file"

  • Very large files may timeout — split or reduce first
  • Ensure the file is plain text, not binary
  • Verify UTF-8 encoding

"Crawler not detected"

  • Confirm the User-Agent matches supported patterns (case-insensitive)
  • If you find a new AI crawler, please let us know so we can add support

"File size exceeds the 4.5MB upload limit"

  • Filter your logs to include only AI crawler traffic using the grep command below
  • Alternatively, split large files into smaller chunks
  • See CLI examples section for commands

Learn more​