"I don’t think they are filtering it. They are just storing it"
How much would it actually cost to store all the emails that American citizens send and receive? I find it difficult to visualise a system of checks and balances approving the massive budget required to house all that data. And the technical challenge of sifting through those emails would be seriously hard. I understand that this might not be just about national security, but it could also be a power game on the part of the FBI. One has to consider how much [national security | power] this really affords [US citizens | the FBI] when measured against the gargantuan expenditure required to actually pull it off. This makes me totally skeptical. Additionally, by saying 'basically the e-mails of virtually everybody in the country' Binney demonstrates his lack of conviction and uncertainty of his own claims.
So, if his words aren't the giveaway, then two minutes of critical thinking will make the interview seem alarmist and inaccurate.
Interestingly, the article explicitly mentions the FBI, not the NSA. But, if we were to unify them in our respective imaginations (for the purposes of discussion), they still need to have the budget for their operations supplied from somewhere. http://www.fbi.gov/news/testimony/fbi-budget-request-for-fis... - it appears as if the FBI's budget is just as rigidly controlled as any other governmental organisation. How they internally allocate the funding they receive is a different matter. To me it seems that if there is any form of oversight within the FBI, then the funding of a project of this cost/return ratio would be infeasible and therefore the project could possibly be shut down. There is the possibility that this is an exercise in propaganda and scare tactics: "We can read your emails, so don't try anything. Check out our power". However, because the internal mechanics of the FBI are necessarily not public knowledge this is all speculation and unsubstantiated opinion. But, the argument can be made that this very fact enables the FBI to get away with scare tactics. I still don't see an organisation with the real-world constraints of money being able to store the sheer volume of data that they claim to. For arguments sake, let's assume that the year-on-year rate of email generation remains uniform for the next decade. In one year, they need to have the ability to store X amount of emails. In a decade, they'll need 10X amount of storage. How are they going to store all that data?
> they still need to have the budget for their operations supplied from somewhere. http://www.fbi.gov/news/testimony/fbi-budget-request-for-fis.... - it appears as if the FBI's budget is just as rigidly controlled as any other governmental organisation
Also, I've commented on the NSA probably intercepting on the people in command immediately after the Petraus affair hit the media (http://news.ycombinator.com/item?id=4767644), but I had hopes that they would stop at those people. Someone in the comments reprimanded me for viewing things in a CSI-like manner (CSI the TV show), but it was not that, just the natural reflexes of a former kid who has grown in Eastern Europe with Securitate ruling and surveying everything in sight. (http://en.wikipedia.org/wiki/Securitate)
The return is potentially infinite, since they can say "we have every single criminal conversation ever in digital space recorded".
edit: conversations primarily by American Citizens.
Slightly OT:
"The Rods... so sensitive that they can be triggered by single photon"
This sort of thing is why I love HN. Irrespective of the topic, there is always some little gem I find somewhere. I was very sceptical of this claim, so I looked it up. Turns out it is possible:
So which framework comes out on top with the least developer friction metric? Not just convenient features like an ORM but also, your ability to do stuff without the framework getting in the way.
I find the best solution is as minimal a framework as possible with full featured libraries. Avoid the temptation to let a big framework choose your orm, forms library, routing, etc. just because decisions are stressful or complicated.
You try to unfairly increase the upvotes on your submission, by getting people who don't read HN to register and upvote your submission. Or, you create a bunch of HN accounts, and log into each one and upvote your submission.
Getting on the front page of HN can drive a lot of traffic to a site, which can turn into money, so there's a financial incentive to game the system.
No one can guarantee anything for you. Achieving 3 of the above criteria will not hurt you and are quite easy to achieve. They will also help you in other ways.
Worry about being great -- (Steve Martin quote is relevant here: "be so good they can't ignore you")
Cron's probably the wrong term to use there - the failures I usually see are on the code or environment side, and usually it's a PITA to try and refire missed cron events once you fix the bugs.
BTW, I've been meaning to ask for a while though: what sort of algorithm is running that detects edges and orientation so when the mouse is moved, the small white circle or quadrilateral seems to touch the objects in the image?
Are you sure? How do you know it is not an image processing/edge detection algorithm running? If they were indeed constructing a 3d-model, I'd be interested in seeing it, even if it is a low-poly approximation...
Initially this might seem different from the topic at-hand, but bear with me please:
Several years ago (probably first or second year of high school) I wrote and distributed amongst my friends a small desktop game. The program e-mailed me their new high-scores and internal game diagnostics daily. I found that this was too often so I rewrote the game to send these data daily. This way of retrieving data sucked. Often, I would get empty or unimportant messages.
Eventually I had the realisation that time-oriented polling was the wrong philosophy. Instead, there should be a "this event has happened now" (e.g. 500kb of diag. generated) algorithm that calls the polling routine/invokes the 'stuff to be done traverser'. If I understand the problem correctly, the mass e-mail example, you could make a simple counter as part of the HTTP request. When this counter hits a certain you-determined threshold value (e.g. there are 500 jobs to do), then call the traverse/processing code:
if jobs_to_do > thresh then invoke some async processing
The takeaway from this post is that, in my opinion, time-based (cron job-bish) polling algorithms are inefficient but can be replaced with a rudimentary event-driven ones.
And just BTW, 'countervailing' - I'm deifying this word. Holy s*. I'm going to use it all the time.
A comparison between this and statsd would be interesting. Any one have experience with both?
And as for the similarity between the Heroku documentation page - their github documentation page reads:
"This program is forked from heroku/heroku-docs, and originally written by @rtomayko and @adamwiggins. Later, modified by @kzk and @doryokujin."
How much would it actually cost to store all the emails that American citizens send and receive? I find it difficult to visualise a system of checks and balances approving the massive budget required to house all that data. And the technical challenge of sifting through those emails would be seriously hard. I understand that this might not be just about national security, but it could also be a power game on the part of the FBI. One has to consider how much [national security | power] this really affords [US citizens | the FBI] when measured against the gargantuan expenditure required to actually pull it off. This makes me totally skeptical. Additionally, by saying 'basically the e-mails of virtually everybody in the country' Binney demonstrates his lack of conviction and uncertainty of his own claims.
So, if his words aren't the giveaway, then two minutes of critical thinking will make the interview seem alarmist and inaccurate.