The Hidden Web Analytics Data You Should Look At

by DigitalAlex on February 16, 2009

Photo by photomequickbooth - http://www.flickr.com/photos/22897538@N04/

Photo by photomequickbooth - http://www.flickr.com/photos/22897538@N04/

You could be losing visitors and business due to site issues that never register in your web analytics tool.

If you’re anything like me, you tend to look at your site’s performance mostly through web analytics data (maybe with some voice of the customer/survey data included). The problem is that web analytics tools don’t have all of the information you need.

Error Logs: Hidden Data You Need

This hidden data can be found in your website’s error log (aka apache error log). An error log is a simple file that lists all the errors your visitors encountered when trying to access your site. Errors are logged by IP address, date and time.

You can get them from your webmaster. If that’s you, the location varies but log into your host and look for the “logs” section.  Mine were at /logs/error_log.

After I got a “status 500 internal server error” screen when I tried to get to my own blog, I knew it was time to dig a little.

I found three common errors in my log:

  1. Directory index forbidden by rule
  2. WordPress database error Lost connection to MySQL server during query
  3. Premature end of script headers: php4

One of my blog’s plugins was the source of many of the issues.  That was an easy enough solution — I just deactivated and uninstalled it.  The MySQL connection was lost.  It’s a relatively simple fix for WordPress users, detailed right here.

If you’ve never seen a server log (I hadn’t, is that terrible?) , I pasted a portion of my own error log after the jump.

You can see how the error messages I listed show up repeatedly.  You may know that site speed is an issue, but you could  still be surprised how many people are encountering errors when they try to get to your site..  I was also shocked to see how much one WordPress plugin was an issue (if you’re using WordPress, I’d steer clear of Redirection)

I’m going to report back on how much (or how little) my site traffic increased/errors decreased.  Take a look at yours and comment so we can all see if you found anything interesting.

Portions of My Apache Error Logs

[Thu Jan 08 15:37:42 2009] [error] [client 67.195.37.167] Premature end of script headers: php4
[Thu Jan 08 15:38:04 2009] [error] [client 87.118.124.175] Premature end of script headers: php4
eferer: http://www.google.com/search?hl=en&q=most+popular+blog+&btnG=Search
rection` (
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `id` int(11) unsigned NOT NULL auto_increment,
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `url` mediumtext NOT NULL,
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `type` enum('301','302','307','404','410','pass') NOT NULL default '301',
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `regex` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `position` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `redirector` text NOT NULL,
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `last_count` int(10) unsigned NOT NULL default '0',
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `last_access` datetime,
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  PRIMARY KEY  (`id`)
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t) made by require, require_once, require_once, require_once, do_action, call_user_func_array, redirection->plugins_loaded, redirection->update, re_database->upgrade, re_database->install
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] WordPress database error Lost connection to MySQL server during query for query CREATE TABLE IF NOT EXISTS `wp_redirection` (
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `id` int(11) unsigned NOT NULL auto_increment,
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `url` mediumtext NOT NULL,
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `type` enum('301','302','307','404','410','pass') NOT NULL default '301',
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `regex` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `position` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `redirector` text NOT NULL,
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `last_count` int(10) unsigned NOT NULL default '0',
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  `last_access` datetime,
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t  PRIMARY KEY  (`id`)
[Thu Jan 08 15:38:27 2009] [error] [client 67.195.37.167] \t\t) made by require, require_once, require_once, require_once, do_action, call_user_func_array, redirection->plugins_loaded, redirection->update, re_database->upgrade, re_database->install
[Thu Jan 08 15:39:09 2009] [error] [client 64.157.224.138] Premature end of script headers: php4, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] WordPress database error Lost connection to MySQL server during query for query CREATE TABLE IF NOT EXISTS `wp_redirection` (, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t  `id` int(11) unsigned NOT NULL auto_increment,, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t  `url` mediumtext NOT NULL,, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t  `type` enum('301','302','307','404','410','pass') NOT NULL default '301',, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t  `regex` int(11) unsigned NOT NULL default '0',, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t  `position` int(11) unsigned NOT NULL default '0',, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t  `redirector` text NOT NULL,, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t  `last_count` int(10) unsigned NOT NULL default '0',, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t  `last_access` datetime,, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t  PRIMARY KEY  (`id`), referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:38 2009] [error] [client 64.157.224.138] \t\t) made by require, require_once, require_once, require_once, do_action, call_user_func_array, redirection->plugins_loaded, redirection->update, re_database->upgrade, re_database->install, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] WordPress database error Lost connection to MySQL server during query for query CREATE TABLE IF NOT EXISTS `wp_redirection` (
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `id` int(11) unsigned NOT NULL auto_increment,
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `url` mediumtext NOT NULL,
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `type` enum('301','302','307','404','410','pass') NOT NULL default '301',
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] WordPress database error Lost connection to MySQL server during query for query CREATE TABLE IF NOT EXISTS `wp_redirection` (
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `regex` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `position` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `redirector` text NOT NULL,
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `last_count` int(10) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `last_access` datetime,
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  PRIMARY KEY  (`id`)
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t) made by require, require_once, require_once, require_once, do_action, call_user_func_array, redirection->plugins_loaded, redirection->update, re_database->upgrade, re_database->install
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `id` int(11) unsigned NOT NULL auto_increment,
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `url` mediumtext NOT NULL,
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `type` enum('301','302','307','404','410','pass') NOT NULL default '301',
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `regex` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `position` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `redirector` text NOT NULL,
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `last_count` int(10) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  `last_access` datetime,
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t  PRIMARY KEY  (`id`)
[Thu Jan 08 15:39:29 2009] [error] [client 67.195.37.167] \t\t) made by require, require_once, require_once, require_once, do_action, call_user_func_array, redirection->plugins_loaded, redirection->update, re_database->upgrade, re_database->install
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] WordPress database error Lost connection to MySQL server during query for query CREATE TABLE IF NOT EXISTS `wp_redirection` (
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t  `id` int(11) unsigned NOT NULL auto_increment,
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t  `url` mediumtext NOT NULL,
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t  `type` enum('301','302','307','404','410','pass') NOT NULL default '301',
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t  `regex` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t  `position` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t  `redirector` text NOT NULL,
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t  `last_count` int(10) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t  `last_access` datetime,
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t  PRIMARY KEY  (`id`)
[Thu Jan 08 15:39:40 2009] [error] [client 193.65.75.85] \t\t) made by require, require_once, require_once, require_once, do_action, call_user_func_array, redirection->plugins_loaded, redirection->update, re_database->upgrade, re_database->install
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] WordPress database error Lost connection to MySQL server during query for query CREATE TABLE IF NOT EXISTS `wp_redirection` (
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t  `id` int(11) unsigned NOT NULL auto_increment,
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t  `url` mediumtext NOT NULL,
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t  `type` enum('301','302','307','404','410','pass') NOT NULL default '301',
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t  `regex` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t  `position` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t  `redirector` text NOT NULL,
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t  `last_count` int(10) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t  `last_access` datetime,
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t  PRIMARY KEY  (`id`)
[Thu Jan 08 15:39:45 2009] [error] [client 87.118.124.175] \t\t) made by require, require_once, require_once, require_once, do_action, call_user_func_array, redirection->plugins_loaded, redirection->update, re_database->upgrade, re_database->install
hu Jan 08 15:39:40 2009] [error] [client 64.157.224.133] \t\t  PRIMARY KEY  (`id`), referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:40 2009] [error] [client 64.157.224.133] \t\t) made by require, require_once, require_once, require_once, do_action, call_user_func_array, redirection->plugins_loaded, redirection->update, re_database->upgrade, re_database->install, referer: http://blogs.omniture.com/
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] WordPress database error Lost connection to MySQL server during query for query CREATE TABLE IF NOT EXISTS `wp_redirection` (
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t  `id` int(11) unsigned NOT NULL auto_increment,
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t  `url` mediumtext NOT NULL,
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t  `type` enum('301','302','307','404','410','pass') NOT NULL default '301',
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t  `regex` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t  `position` int(11) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t  `redirector` text NOT NULL,
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t  `last_count` int(10) unsigned NOT NULL default '0',
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t  `last_access` datetime,
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t  PRIMARY KEY  (`id`)
[Thu Jan 08 15:39:56 2009] [error] [client 75.101.242.96] \t\t) made by require, require_once, require_once, require_once, do_action, call_user_func_array, redirection->plugins_loaded, redirection->update, re_database->upgrade, re_database->install

Tagged as: 500 internal server error, apache error logs, server log

{ 3 comments… read them below or add one }

1 Chris G February 16, 2009 at 8:49 am

Yes! Finally somebody points out this truly overlooked source of data! Thank you Alex!

The information in these logs (which I believe can be set to be combined with access logs) can even help programmers and systems people, not just for identifying issues but for pinning them down.

2 Rich Page February 16, 2009 at 2:52 pm

This is a great post Alex. How come you don’t offer the ability to ‘share’ this post by using a social media tool like Share It. I wanted to send this to some people at work :)

3 Alex February 16, 2009 at 5:44 pm

@Chris You’re welcome! I’m pretty new to them, so if you have a top tip or 2, feel free to comment away.

@Rich – So glad you enjoyed it. There is a share link on the top of the post. Is it to inconspicuous? Did you expect to see it at the bottom? I need VOC feedback :-)

Leave a Comment

Previous post:

Next post: