Traffic Discrepancies between IHR Click-thru Reports and your Hosted Web Statistics

Sometimes a client finds that their hosted web site server statistics show less traffic from IHR.com than the IHR.com Click-thru Statistics report. IHR continually tests the accuracy of our click-thru traffic report. We find that if, for example, we click on a test link 10 times, it does in fact get recorded as exactly 10 click-thru's, not more and not less.

Why Do Discrepancies Occur?

IHR has researched these types of discrepancies and found that they can be due to the following factors:

Here's why:

• The IHR stats will count this as two click-thrus because it is two click-thrus. This is how all Pay Per Click search engines work. For example, see the the Google page - "Why do AdWords and Analytics show different figures in my reports?"
  
  NOTE:  Search marketing firms 360i and SearchIgnite reviewed more than 3.9 million users and 5.1 million clicks during the first quarter of 2006. They found that "Searchers who ultimately made a purchase clicked an average of 15% more ads than those who didn't complete a transaction. Really determined searchers - those that clicked a search marketer's ads ten times - were three times as likely to convert as those who clicked an ad only once."  Conclusion: A user that clicks on a promotional listing more than once is more likely to purchase a service or product.
   
• In contrast, the target Web server typically records this as one page view, because typically, the user's Web browsers cached (i.e. stored) the web page on the computer after the first click-thru. Then when the user views the page again, their browser displays the page from their computer. The browser doesn't go back to the server to get the page, so the server doesn't know the user has viewed the page two times, so the server doesn't record the second page view. Also, the user's ISP may cache the page to reduce their Internet traffic. By doing this, they do not have send another request to the target server. AOL uses caching extensively.

2. The users web browser did not report the "Referrer" (i.e. an IHR.com page) to the target server:

Sometimes Web browsers do not report the "Referrer" (i.e. an IHR.com page) to the target server (i.e. the IHR clients website server). The server's log file will then record this as one page view, however it will not record the Referrer (that is where the click came from). If this occurs during a click-thru from IHR.com, the client's web statistics will not see this as a click-thru from IHR.com.

Some products are sold which will remove the Referrer field in order to protect users' security:

• See for example the Afterzed.com Lessons page.
• Some firewalls strip out the Referrer information, as described in the The SEM Content Conundrum. The author says "the referrer is not reported by all browsers, and it's stripped out by some firewall software"
• Some "pre-fetch" products do not supply full information (e.g. Referrer) to the server. See Web Traffic Logs: An Imperfect Resource for Evaluation.
• Some human-used browsers, in some cases, do not supply Referrer information. For example, Jakarta Commons-HttpClient/3.0-rc3

3. A proxy or cache server asked for the web page - Sometimes a proxy or cache server will send request the web page for a user. In this case, the Referring Page will not get recorded as from IHR.com, but rather it will be recorded as from the proxy server or cache.

Special Considerations for Websites Using Google Analytics

If you are using Google Analytics (GA) for your web hosting stats, there are additional reasons for why you may see a discrepancy between your hosting stats and the IHR reported click-thru stats.

1. As discussed above, if a user click-thrus two times from IHR.com, IHR stats count that as two click-thrus, however, GA counts that as one page view.
    
2. If any of the target pages, where IHR sends traffic, are missing GA coding, then GA won't count those visits.
    
3. GA uses JavaScript, and 1-3% of users have disabled JavaScript, resulting in GA not counting those visits.
    
4. Corporate and personal firewalls can prevent Google page tagging from sending data to its servers, resulting in uncounted visits.
    
5. If a visitor quickly navigates away from the target page, before the GA page tag collects its data, this visit won't be counted.

The following Google page explains some of these issues - "Why do AdWords and Analytics show different figures in my reports?"

Overall, due to Google's "page tagging" technology, GA tends to undercount page views. IHR wishes there were a simpler way to compare apples to apples here. On the IHR side, though, we do work hard to ensure that we've removed robot click-thrus, as well as outlier human click-thrus.

Additional Information About These Discrepancies

• Problems with website statistics - the author points out that: a) "referring web sites" are often not recorded correctly in the hosted web stats, and b) caching can create discrepancies.
   
• The Web UI Master Web Usage Reports - contains lots of information explaining discrepancies and what users can and cannot know from their web statistics. In particular, review the cache section
   
• A Comprehensive Strategy for Using Web Site Statistics - page 3 - Explains the difficulties in getting accurate web hosting statistics. In particular, see the section on Caching.
   
• A Comprehensive Strategy for Using Web Site Statistics (by designandcreate.com) - See the section called "URLs and Referrer Fields" for explanation of why the Referrer field is sometimes not populated in web hosting statistics
   
• Improving The Referrer-Based Web Log Session Reconstruction - very technical article about improving Referrer data

Which of the Above Reasons Create the Most Discrepancies

IHR has developed research methods to detect when, and how often, the first two factors above occur:

1. The target web page was cached in the user's browser or the user's ISP, so the hit never got recorded on the IHR client's server: IHR reviews the occurrences of this because our server can see that the same IP address e.g. clicked two times on a link. An ISP assigns an IP address to a user. IHR can review our records to see if or when the same IP address clicks two or more times
    
2. The Referrer field is empty, so the the IHR client's server never knows that the hit came from IHR.com: Although the target server doesn't know that the hit came from IHR.com, IHR is able to see the click-thrus in our detailed click-thru reports. So we can see how often this happens.

IHR recently did some extensive research to assess the most common reason for the discrepancy. We found that the missing Referrer field was the most common, as shown in the following table.

* NOTE - In many cases, when there are duplicate entries for an IP address it is very hard to know if it represents the same user with the same IP address or different users with the same IP address. The latter situation is caused by some ISPs who dynamically assign IP addresses, so that e.g. 10 people might use one IP address over the course of a month.

If you notice any discrepancies that you wish to discuss with IHR, feel free to contact IHR.