This is a pretty cursory look into some other possiblities of refererless traffic. Given an ordinary browsing session or interaction from an end user, what else could be leading to HTTP requests without the referer header?

1) proxies

2) clever antivirus/firewall software

3) htaccess tricks will often drop headers

4) javascript

5) things like amazon silk

6) people spoofing things to make their browser work

7) anonymizer services

8) proxies

9) links followed from https -> http

10) bookmarks

The most peculiar thing is that links off of https://mail.google.com/ essentially mimics the copy & pasted links (i.e. null header.) However, https://plus.google.com, https://twitter.com, and https://facebook.com all present a referer header to the server.

There's probably room here to do some investigation as to what Google is doing to make outbound links from mail.google.com completely drop the referer. (Also, what about other web based mail clients? Yahoo? MSN? Aol? Corporate Outlook?)

The article in question claims that links followed from https facebook do have a referrer header:

>In testing links from Facebook and Twitter over HTTPS the referer is present in most cases.

That's definitely not to spec if the browser is doing it. http://tools.ietf.org/html/rfc2616#section-15.1.3

"Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol."

However, it's possible that Facebook is passing users through an HTTP gateway.

It looks like the intermediate 301's referer is being passed thru (For example, all links on Twitter get wrapped as a t.co link, and that's what shows up on the server). I'd imagine that analytics being mined are intelligent enough to collapse twitter.com and t.co as the same social origin.

Give it a try yourself: https://twitter.com/vikrum5000/status/256898972478763008 Note the protocol.

Do you imagine 1, 6, and 7 being something the mass market would be using? Or, do you suppose that it is all of these little cuts that becomes that nearly 70% of traffic being without a header?

11) There used to be an issue in select browsers (not sure how prevalent it is anymore) that when a user opened a link in a new tab/window, that the document.referrer would not be set correctly.

Off topic a bit, but I just want to say I like the layout and the general approach of your blog. If it's a template I haven't seen it before.

The "pondering" section is intriguing. What if you could accept input as to what topics readers might prefer?

Thanks! It's not from a template; it's super basic HTML I wrote up by hand with simple CSS behind it. Take a look at the source for http://5f5.org/ruminations/ — I copy and paste that into a new file and use that as a template for new posts.

The pondering section is where I keep notes to my self for future topics. There's a file in that directory that has a bunch of sentence fragments and ideas for other, not yet formed ideas.

I like the idea of suggesting new topics :)

Regarding mis-measuring "dark social": For ecommerce, Pinterest traffic is estimated to be 64% larger b/c of the popularity of the iPad app. This is significant given that Pinterest is now the 2nd largest referral source (depending on which ecommerce sites you are measuring).

http://llsocial.com/2012/07/pinterest-traffic-being-signific...

With the Facebook and Twitter apps (and iOS apps, in general), they have complete control over how the network call is being made — and the referer makes it to the server (for the post part.) One would imagine Pinterest would want to maximize this number.

I'll try posting the referer detector to some more native apps and see what happens.