The article basically uses crime as a proxy for "social population density." I'm pretty sure you could also use "restaurant reservations" as a similar proxy, but then I guess you wouldn't get to use words like 'prostitution' in the title of your blog post.
Then, Uber cab riders go to these areas that are densely social. I'm guessing they probably take other modes of transportation too.
Then, something about certain crimes being more prevalent on certain days of the week, with some pretty huge leaps of faiths made in the reasoning and no actual testable data to back it up.
I don't like to post too often on HN if I'm just going to stand there and drink some haterade, but this just seemed like such a sad attempt to put together something for pageviews that I couldn't help myself.
I think the post was a decent attempt to do something similar to OKcupid's insightful blogs.
The difference is that okcupid had data in their database that is deep, and the team has a bunch of MIT math guys, if I recall correctly.
I'm not sure about Uber, but I'm guessing to write this post they had to go out looking for this information, rather than infer it from their primary data. Still a decent post, but realize the objective here was to market Uber, and the rest makes sense.
This finding is a perfect example of the fascinating insights you can get when you combine big, seemingly disparate datasets.
I'm not sure where you would go to get an equivalent dataset for "restaurant reservations".
In the sense that it uses readily-available crime data to make inferences about possibly correlated phenomena, this analysis is another iteration on Adrian Holovaty's original mashup:
The whole article comes across very "nerdy and scientific" but indeed isn't at all. Just the difference between correlation and dependence are already something they should look at: http://en.wikipedia.org/wiki/Correlation_and_dependence
To be fair, they were pretty clear that their data was a correlation, not a causation. But the stuff about prostitution on second Wednesdays is pretty crazily unsubstantiated, and controversial.
That's because one of the side-effects of big data is the new definition "data scientist" = Ph.D. <graduate|dropout> in any field, no matter how unrelated to statistics or data mining.
I think his goal was to see if one could use an activity proxy to predict where most rides would occur. Crime data is a fairly natural proxy because it's readily available in an easy-to-use format.
It doesn't seem all that useful — it's fluffy — but it's amusing nonetheless and I don't regret reading it.
I'm curious what leaps of faith do you see in his reasoning? Crimes "caught" versus actual criminal activity seems like the obvious one, but I don't think there's really anything in the piece that makes that distinction necessary. (Police activity might actually be a better proxy for activity than criminal activity.)
Reported crimes probably correlate more strongly to police presence, so the root question may be, Are police officers more heavily patrolling areas where people also need rides, and how do they know?
That's exactly what I was thinking about. Here in San Francisco, the police were recently cracking down in North Beach (mostly around the strip clubs on Broadway). It seemed to be a pretty clear causal relationship in that case — more people meant more police meant more tickets and arrests.
Fun data, but one question - there is an implicit assumption that prostitution arrests are proportional to the committing of the crimes. I wonder if weekdays show more prostitution because the cops are busy with other crimes on weekend nights?
A fun anecdote from the Oakland Crimespotting data was that every couple weeks the cops would just do a "sweep" of San Pablo or other major streets for prostitutes, just start at one end then go to the other. I think Wednesdays are more a mark of police patterns than prostitute / john patterns.
That was the same question I had. It seems that CrimeSpotting measures police reports, which are skewed toward whatever crime the police are cracking down on.
That question and its history would need many pages to be properly explained. I'll just say that a country that looks very similar to yours can do some things in a surprisingly different manner.
These guys have looked very hard at the okcupid postings. Interesting stuff, but it doesn't really show how prostitution and alcohol ended up making uber better in some tangible way. Only that there were some interesting correlations.
Uber is able to decrease the average wait time by strategically placing cars wherever alcohol and prostitution are prevalent, thus making the service better.
They imply that looking at crime data reduces average wait time, but they don't actually present any data to show that this is the case, or to quantify how much better their demand prediction model can do when given access to the crime data.
That would require a 'before' and 'after' with some numbers that prove that they've managed to decrease the wait time in a statistically significant manner.
Here's a theory: They're only tracking the prostitution that is caught. (That's how crime data works, right?) Wednesday is close enough to the weekend for prostitutes to work, but there's not too many people around. On thursday/friday/saturday there are hoards of partiers & other night life so they can blend in and find johns easily. But on Wednesday they're more likely to be caught.
It's an interesting theory, but it doesn't explain why the second Wednesday has more prostitution. And following your theory, I would think that those Wednesdays have more people around because they have more money to spend.
This is such a great look into data-- I've long felt that consumer companies like Uber (and Google and Apple and so on...) have the strongest ability to explore the fabric of today's culture through their data, and it's sweet to see that Uber has opened up some of that understanding to the public.
I was turned off by the style of the author's writing, and couldn't get past the "shut up" line. This post didn't match my feelings of the product at all. All in all, very strange.
interesting, but wasn't there an article posted here in the last 24 hours on being too familiar? maybe they should read that. it was a pretty tiring read.
The article basically uses crime as a proxy for "social population density." I'm pretty sure you could also use "restaurant reservations" as a similar proxy, but then I guess you wouldn't get to use words like 'prostitution' in the title of your blog post.
Then, Uber cab riders go to these areas that are densely social. I'm guessing they probably take other modes of transportation too.
Then, something about certain crimes being more prevalent on certain days of the week, with some pretty huge leaps of faiths made in the reasoning and no actual testable data to back it up.
I don't like to post too often on HN if I'm just going to stand there and drink some haterade, but this just seemed like such a sad attempt to put together something for pageviews that I couldn't help myself.