Skip to main content

Manual Upload

Upload one or more server access log files and the dashboard will parse, filter, and aggregate crawler activity. We support standard web server access logs in Apache Common Log Format (also compatible with Nginx access logs).

Quick start​

  1. Export or download your web server access logs.
  2. If needed, filter down to AI bot traffic (see CLI examples below).
  3. Upload one or more .log/.txt files to Server Logs Analytics.
  4. Review Daily Visits, Top Pages, and Top Crawlers to identify trends and opportunities.
  5. Adjust robots.txt/sitemaps and content strategy, then compare before/after.

Where to find your server logs​

Log locations vary by hosting setup. Managed WordPress platforms (e.g., WP Engine, Kinsta, SiteGround) expose access/error logs in their dashboards. Shared hosting (cPanel) provides Raw Access Logs and Error Logs. On VPS/bare metal, check the filesystem (Apache under /var/log/apache2/ or /var/log/httpd/; Nginx under /var/log/nginx/). In cloud environments, logs may flow to services like AWS S3/CloudWatch (ELB/CloudFront), Google Cloud Logging, or Azure Monitor. For containers use docker logs or kubectl logs. Serverless and edge platforms (e.g., Vercel, Cloudflare) surface request/bot activity through observability dashboards.

For step-by-step instructions by platform, see the full guide: How to Find Server Logs for Any Website: A Complete Guide for All Hosting Setups.

Common locations:

  • Apache: /var/log/apache2/access.log, /var/log/httpd/access_log
  • Nginx: /var/log/nginx/access.log
  • Managed hosting: Access via your provider dashboard (e.g., WP Engine, Kinsta, SiteGround)
  • Cloud & edge: CloudWatch, Cloud Logging, Azure Monitor, Vercel/Cloudflare dashboards

Accepted log format​

Each line must follow this structure:

IP - - [timestamp] "method path protocol" status size "referrer" "user-agent"

Format components:

  1. IP Address — client IP (e.g., 135.234.67.1). Anonymized IPs are supported (e.g., 135.234.67.xx, xxx.xxx.xxx.xxx, or *.*.*.*).
  2. Identity — usually -
  3. User — usually -
  4. Timestamp — [DD/MMM/YYYY:HH:MM:SS +0000] (e.g., [12/Jun/2025:06:46:46 +0000])
  5. Request — "METHOD /path HTTP/version" (e.g., "GET /page.html HTTP/1.1")
  6. Status Code — e.g., 200, 404
  7. Size — response bytes (e.g., 17744)
  8. Referrer — often "-" for bot traffic
  9. User-Agent — bot identification string

Example entry:

135.234.67.1 - - [12/Jun/2025:06:46:46 +0000] "GET /kuehlen-gefrieren/kuehlschraenke/siemens-kg39n2xaf-kuehl-gefrier-kombination.html HTTP/1.1" 200 17744 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot"

Example with anonymized IP:

172.204.16.xx - - [13/Oct/2025:22:16:46 +0000] "GET /marathon-running-running-shoes-37v7jz4h6ddzy7ok HTTP/1.1" 200 38433 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"

File requirements​

  • Format: .log or .txt
  • Encoding: UTF-8
  • Size: Maximum 4.5 MB per upload (due to serverless function limits). For larger files, filter to AI crawler traffic first (see CLI examples below).
  • Compression: Not supported — extract archives before upload

Multiple file upload​

You can upload multiple log files simultaneously. We will:

  1. Combine all log entries
  2. Parse them together
  3. Display aggregated statistics across the full timespan

This is ideal for rotated logs, multi-server setups, or comparing different time windows.

CLI examples (optional)​

Filter for AI crawlers only (Linux/Mac):

# OpenAI bots
grep -i "gptbot\|chatgpt-user\|oai-searchbot" access.log > ai-crawler.log

# All AI bots
grep -iE "(gptbot|chatgpt-user|oai-searchbot|claudebot|perplexitybot|gemini|google-extended|meta-externalagent)" access.log > ai-crawler.log

Last 7 days only:

grep "$(date -d '7 days ago' '+%d/%b/%Y')" access.log > recent.log

Troubleshooting​

"No data available"

  • Ensure your log file contains AI crawler entries
  • Check the format matches Apache Common Log Format
  • Verify the User-Agent contains a recognized crawler pattern

"Failed to upload log file"

  • Very large files may timeout — split or reduce first
  • Ensure the file is plain text, not binary
  • Verify UTF-8 encoding

"Crawler not detected"

  • Confirm the User-Agent matches supported patterns (case-insensitive)
  • If you find a new AI crawler, please let us know so we can add support

"File size exceeds the 4.5MB upload limit"

  • Filter your logs to include only AI crawler traffic using the grep commands above
  • Alternatively, split large files into smaller chunks
  • See CLI examples section for commands