Log File Analysis: What Server Logs Tell You

Every disagreement I have had about a website's architecture has ended the same way. We opened the server log, and within ten minutes one of us was proved wrong.

Logs are the only record of what actually happened rather than what a tool predicted should happen. In 13 years I have not found a cheaper way to settle whether a crawl problem is real before somebody pays to fix it.

This is written for the owner or marketer who has been told their site has a crawl issue and wants evidence before spending. It covers what a log records, how to tell a real bot from a fake one, how search and AI crawlers show up differently, what wasted crawl looks like in the data, and how to turn all of that into a short list of fixes.

What a server log actually records

The four fields worth caring about

A log line is a time stamp, an IP address, a request line and a user agent, usually with a status code and a response size attached. That is enough to answer most crawl questions.

Time gives you frequency and timing. IP plus user agent gives you who is asking, subject to verification. The request line gives you what was asked for, including every parameter. The status code tells you what happened when it was served.

Four fields, four questions. How often, who, what, and did it work. Almost everything useful in a log review comes from answering those four properly rather than from a dashboard.

Log formats you are likely to have

Most hosting gives you one of two shapes. The Apache combined format, which is compact and readable, and the extended W3C format, which is more verbose and easier to parse when there is a lot of it.

Ask for raw access logs rather than a summary report. If the site sits behind a content delivery network, ask for those logs too, because the request is recorded where it first lands and that is often the more complete picture.

Why crawl logs beat a simulated crawl

What a crawler tool cannot see

A crawler tool fetches your site the way a robot would, from one place, at one moment, with one user agent. It cannot tell you how often you were crawled last month, which URLs were crawled and never indexed, or whether a mobile crawler behaved differently to a desktop one.

It also cannot show you the traffic that is not a crawler at all. Most of the requests in a normal log come from somewhere other than a search engine, and no crawl tool will ever report that.

Where the two disagree

When a crawl tool and a log disagree, the log wins, because it records what happened rather than what should have happened. A tool may report a URL as reachable while the log shows thousands of requests to it and a steady trickle of server errors the tool never triggered.

I still run both. The crawler finds structural faults quickly, and the log tells me which of those faults the crawlers actually care about. That is the difference between an SEO audit that looks at server logs and an audit that lists every imperfection it can find and leaves the priorities to you.

Verifying which bots are real

Reverse DNS and published IP lists

Any request can claim to be Googlebot. Per Google's documentation on verifying crawler requests, "You can verify if a request to your server really is from Google." The published method is straightforward: "Run a reverse DNS lookup on the accessing IP address from your logs, using the host command."

The lookup returns a hostname in Google's own domain, and you then confirm that hostname resolves back to the same address. Both directions have to agree. A one way match proves nothing, because the forward record on any server can be set to whatever the owner of that server wants.

Spoofed bots and what they cost you

Fake bots matter for two reasons. They distort every conclusion you draw from the log, and they consume the same server resources as real traffic. If you count them as search crawlers, you will conclude that Google is spending far more time on your site than it is.

The scale of non human traffic is worth stating. Per Imperva's bad bot research, "Imperva Threat Research team estimates that 27.7% of online traffic is bad bots." That is the noise floor in your data, and it is the reason verification comes before analysis rather than after it.

Search bots versus AI bots in your logs

What the newer AI user agents look like

AI systems identify themselves the same way search crawlers do, with their own user agent strings. Per OpenAI's crawler documentation, "OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI."

Treat each one as a separate visitor with a separate purpose. A search crawler and a training crawler from the same company can carry different names and follow different rules, and blocking one is not the same decision as blocking the other.

What their behaviour tells you

Crawl behaviour shows what a system finds worth fetching. If an AI crawler requests your content pages but never your tag archives, that is a strong hint about which parts of the site earn attention and which do not. If it keeps requesting pages that return errors, that is a pointer at your own site rather than at the crawler.

I do not draw conclusions about how much any AI assistant sends you from log activity alone. Fetching a page is not the same as being read, quoted or recommended, and the temptation to overread this data is strong. What the log does tell you is whether a system can reach the content at all, which is the only part of that question you can control.

Finding wasted crawl

Parameters, filtered URLs and session paths

Wasted crawl is crawl effort going somewhere that cannot help you. The usual culprits are URLs carrying parameters: tracking codes, filter combinations, sort orders, session identifiers and internal search queries.

Per Google's crawl budget documentation, "The web is a nearly infinite space, exceeding Google's ability to explore every publicly accessible URL." That is why waste matters at all. Crawler attention is finite, and every pointless request is taken from something that would have earned you a visitor.

The test I apply is simple. Would I want this URL in a search result? If the answer is no, it should not be earning requests either.

Pages that keep getting requested for no reason

A page you have deliberately excluded should not be attracting requests. The same Google documentation makes the point sharply: "Don't use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time."

So I look for URLs that are requested heavily and can never appear in results. Filtered category pages, internal search results, paginated archives nobody reads, and print or email versions of pages that already exist.

The consequence of leaving them in place is set out just as plainly by the same documentation: "If Google spends too much time crawling URLs that it shouldn't", Google warns that its crawlers may not reach the rest of your site, and may not increase the crawl budget you are given. Pages you do care about are competing for attention with the ones you do not.

Reading crawl frequency like a timeline

Spikes and what triggers them

A crawl spike is a clue rather than a problem. It usually follows a change: a large content publication, a new section, a change to internal linking, a sitemap rebuilt, or a new link from an external site pointing at a page that did not exist before.

Match every spike against your own change log. Most unexplained activity turns out to be explained by something your team shipped the day before, and the ones that are not are usually the interesting ones.

Slow and erroring URLs in the data

Sort the log by response time and by status code. Slow responses under heavy crawl load can throttle how much of the site gets visited, and repeated server errors teach a crawler that a section is not worth returning to.

Browser based reporting can miss this entirely. Per Google's guide to fixing search related problems with script driven pages, "Client-side analytics may not provide a full or accurate representation of Googlebot and WRS activity on your site." The server log is the record that does not depend on anything running correctly in a browser, which is exactly why it is the one to trust when the two disagree.

Acting on what the logs show

The fixes that change crawl behaviour

The fixes that show up in a log are usually small. Remove or block parameter driven URLs you never wanted indexed. Stop generating internal links to filter combinations. Fix the endpoints that keep returning errors. Make duplicate paths redirect rather than both serving a response.

Keep the list short and behave like a scientist. One group of changes, then re-measure. If you change six things at once, the next month's log tells you nothing about which one worked.

What to recheck after the change

Re-run the same report you ran before the change, over the same length of period. Look for three signals. Wasted URLs dropping out of the top requests. Real content pages being requested more often. Error responses falling.

If the crawler comes back to your content pages but those pages still do not appear in results, the next question is about the pages rather than the crawl, and that is a different piece of work. If hosting or server configuration is where the problem actually sits, get help with hosting and server configuration rather than treating it as a marketing fix.

Building a log review routine that lasts

Frequency, retention and file sizes

Once a month is enough for most sites, and one full month of data is the minimum useful window because it contains a normal traffic cycle rather than a quiet week.

The practical constraint is size. Logs grow quickly, so keep at least 90 days of raw files somewhere you can retrieve and compress the rest. Many hosts rotate and delete without asking, and the month you most want to look at is often the one that has just been removed.

Ask your host two questions before you need the data. How long are raw logs kept, and how do I request them? The answer is usually shorter than you expect.

What to track month to month

Keep a small table with the same five numbers each month. Total crawl requests, unique URLs crawled, the share of requests going to excluded URLs, the count of error responses, and requests by crawler type.

That is enough to see a trend without becoming a reporting project. Where crawl behaviour needs connecting to what people do afterwards, connect crawl data to what visitors do using your analytics, because a log tells you what was fetched and nothing about what anybody did with it.

My advice on the order of work is the same as it is for every technical question: run the basics first, then pay for help on what the basics cannot reach. The technical SEO checklist is the first step, and it is something you or your developer can start on this week.

Frequently asked questions

Where do I get my server logs?

From whoever hosts the site. Most hosts and panels expose raw access logs, and you usually need to ask for the Apache or W3C extended format rather than a summary report. If the hosting sits behind a content delivery network, the network's logs are often more useful because that is where the request first lands.

How much log data do I need before it tells me anything?

A full month is a sensible minimum, and it needs to include a normal traffic period rather than a quiet week. One day of data will show you activity but not habits, and habits are what you are looking for.

Can I trust the user agent string in my logs?

No. User agent strings are easy to fake, which is why Google publishes a verification method based on reverse DNS lookups and documented IP ranges. The user agent tells you what a request claims to be. The verification tells you what it is.

Do AI crawlers show up in logs differently from Googlebot?

They appear as their own user agents, and providers increasingly publish both robot rules and IP ranges. Treat each bot as a separate visitor with its own purpose, because blocking one is not the same decision as blocking another.

What does wasted crawl actually look like in the data?

Repeated requests for URLs with parameters or filters that you never wanted indexed, requests for pages that respond with a noindex instruction, and heavy activity on URLs that return errors. It is visible as requests going somewhere that cannot help you.

We fixed the problem. How do I know it worked?

Compare the same report after the next crawl cycle. Look for the wasted URLs dropping out, real pages being requested more often, and error responses falling. Change the question from what did we publish to what did the crawler do differently.