I really like the work of folks at 18F, USDS, CFPB, et al.
Bearing in mind that I'm ecstatic to see this collection of projects and someone clearly put a lot of time and effort into this already, I do have couple points of hopefully constructive feedback here:
- Please list all of your repositories and feature ones that you think deserve special recognition. Seeing all your repos is a 4-step process otherwise: click on the project, click on the link, click on the acknowledgement that I clicked a link, click on the organization name on GitHub. [1]
- Some information on contribution policies, pulling in your README.md, etc would be helpful on this site. [2]
- When I click on the link to go to your repo, I know I'm clicking a link that leaves your site. You even say so. "But you probably knew that already." Don't hijack it, please. [3]
- The activity feed doesn't provide much value in its current form. Most of these GitHub events are completely without context. [4]
"I Fought the Law" is a song written by Sonny Curtis of the Crickets and popularized by a remake by the Bobby Fuller Four, which went on to become a top-ten hit for the band in 1966 and was also recorded by the Clash in 1979.
Yes indeed you're correct. Another nice theory of mine ruined by actual evidence. I'm still hoping that the guy making the comment got it from The Clash though, just cause I'm biased towards them.
I really hope this sort of promotion of innovating in the tech sector remains growing.. with the current political climate, I doubt many sectors (besides military/finance) will see governmental spending growth in coming years.
If you're a US citizen, vote for someone that will keep it going. Some have promised to undo a lot that the current administration has done, and this might be part of that.
I'm not aware of any free code from Trump so far, btw.
Also, when his campaign has tried to code, they fail:
"A programmer named Shu Uesugi, an engineer at a California company called EdSurge, discovered a major flaw with the way Trump’s website was using jQuery. Instead of downloading the open-source code from GitHub and running it off a server they controlled, the developers who built Trump’s website simply ran the code off GitHub directly, Uesugi found.
While the code’s location might seem like a minor detail, running it off GitHub meant that the person who controlled the code on GitHub could change the code at his whim, and those changes would take hold on Trump’s website. Since GitHub is for open-source projects, it also meant that any user could submit a request to modify the code and impact Trump’s website, if the change was approved by the plug-in’s author, a developer in Lisbon named Igor Escobar."
> Since GitHub is for open-source projects, it also meant that any user could submit a request to modify the code and impact Trump’s website, if the change was approved by the plug-in’s author, a developer in Lisbon named Igor Escobar."
That's how it works for any open source project you use, regardless of where it's been hosted. Unless you review the entire codebase (as well as all changes made in new releases), you're trusting the maintainers' judgement.
That's not exactly the same as choosing to use something that is hosted on someone else's server, which they could then subsequently modify now that you are using it in a very high-profile project.
Of course, judging candidates' by the code quality of their campaign websites is a rather obscure and somewhat useless pastime.
I agree that it's still bad practice. My main objection was to the statement "Since GitHub is for open-source projects, it also meant that any user could submit a request to modify the code and impact Trump’s website". If don't trust the maintainers' judgement in merging PRs, hosting it yourself isn't a solution (short of reviewing the entire project yourself).
I think the concern here is that the maintainer could subsequently merge a malicious PR knowing who was using the library from GitHub. That wouldn't be an issue if that group was hosting a version themselves (before the maintainer might find out who was using it).
This still wasn't a good idea, but for a different reason - it's relying on the demo at https://igorescobar.github.io/jQuery-Mask-Plugin/ continuing to exist, and continuing to host the plugin at that same path.
Full disclosure, this mistake was not made by the campaign directly, but rather by Revv.co, who is the payments software provider (not processor, DJT is using Stripe).
I admire the work that 18F is doing, and live in the DC area, even contracted for the government early in my career -- but I can never go back. I just fundamentally disagree with the security clearance apparatus is in place and never want to subject myself to that again. Especially when purely private companies pay more and don't care what you do in your personal time.
Working for the government felt like being a child again.
Thank you! I didn't have the words to express it but that is it precisely. It is infantalizing in a way that makes ball pits and arcades in startup culture seem tame
They want to know deeply personal details about your life, show up at your neighbour's and friends doors to question them, have explicit control over where you can travel on your vacation, deeply care about what you do in your free time, and mishandled all of it -- all in the vague guise of preventing blackmail and sabotage.
The blackmail and sabotage argument, as near as I can tell, is a farce because our enemies and "friends" are engaged in massive corporate espionage. If it were a real concern to them than they would use thier powers to require the same level of scrutiny on the private sector.
In addition to that they say be truthful so you don't have anything to hide to prevent possible blackmail by foreign powers, but in reality it's the truthful ones who don't make it through the process.
I've come to the conclusion that it's really just about having an excuse deny employment to "people not like us."
Its pretty backwards if you think about it, if the places you go or people you know or mistakes you make, can easily make you lose your security clearance and therefore your career could be ruined, it makes you much easier to blackmail than under a system that was more laissez-faire
I think the idea is more like if you are able to pass those clearances , you are less likely to find yourself in a future position that would compromise them. Similar to the belief that a high credit score shows that you're more likely to pay your bills and not abuse your credit - because you haven't so far.
> show up at your neighbour's and friends doors to question them
I think there's a lot of people that would like the associated intrigue that would come with this, it may be a cheaper way of hiring good developers that have a bit of a vanity streak.
Ironically, the Federal Government still ends up paying the price despite all of their efforts. I know around 10 top tier programmers who use drugs of some sort, have left of mainstream politics or just don't fit the "government worker" bill, but are no less patriotic and competent people. The loss of these people from the upper echelons of the state tech system is almost palpable. The CIA director was hacked by a bunch of teenagers from Eastern Europe, for Christ's sake!
And then on top of that, even with all the screening they do, they still stop the Snowdens from infiltrating, still can't stop foreign intelligence, still can't get a sysadmin worth their salt to keep our future President's clandestine email server safe from the public's prying eyes!
That is...most jobs. Unless I'm not understanding your meaning. Many jobs will fire you for, say, getting a DUI or getting arrested for drug possession or something. And many employers do drug testing. But realistically, a "limited sub-selection of possible activities" is just about everything everyone else can do, except maybe go to Cuba or smoke pot, the latter of which will get one fired at many non-government places.
Is drug testing always legal in the US? Can drug use on weekends get you fired for any job?
I lived in several European countries, and as far as I know there's no way an employer can force you to take a drug test, with the exception of people handling heavy machinery. If you are on drugs while you're at your job this will of course have consequences, but a drug test that would reveal drug use in your free time is nothing the employer should be allowed to do or even care about.
As an example, the rules in the UK clearly say that you need consent and that it has to be required by the nature of the job, which should exclude most (if not all) office jobs. [1]
You can be fired for almost any reason in the US. There are only a few exceptions such as race, religion, sex or for reporting violations of some laws.
So, yes, in almost any job you could be required to take a drug test and be fired if you refuse.
Smoking weed is not going to get you fired anywhere, except for the worst jobs in the worst places. Any place that does regular drug testing and/or cares one teeny tiny itsy bitsy bit about employees smoking weed when they're not working is guaranteed to be a terrible place to work at, for that and any number of other reasons. There is not a single reputable company in the entire tech industry that does this.
> Smoking weed is not going to get you fired anywhere, except for the worst jobs in the worst places.
Do you have numbers to back this up? Because almost every place I've worked at has had this policy. I think it's a terrible policy and should be illegal, but it's been pretty standard everywhere. Maybe it's not common in SV, but everywhere else it seems to be the norm, IME.
It has never been an issue for me as I don't partake, but anecdotally, I've worked at 5-10 mid range (100-200) firms in Virginia and Illinois and while I have been provided many free beers at work, if anyone suggested a drug testing program they would have been laughed out of the building.
I even worked for a company that ran drug testing programs for the government and they did not drug test.
"Out of 617 ratings on glassdoor.com, fully 246 called themselves “very dissatisfied.” The number of employees considering themselves “very satisfied” was just 48. Some comments even suggest that executive meddling may be involved in the “very satisfied” scores, with one commenter saying “Joe Clayton (CEO) put us up to upgrading our score.” Another commenter called their time with Dish “...like a prison sentence.”
A few people are a little disgruntled that the government gave all of the information in our security clearance applications/re-ups to China. That's my beef, I'm sure other folks have other things to say.
How about open source code for some makes and models of voting machine?
The US Veterans' Administration health records software system is in the public domain. https://en.wikipedia.org/wiki/VistA#Licensing_and_disseminat... But it's not listed here. (It's also kind of complex. "wget; tar x; ./configure ; make" probably won't get you a running instance. Still.
Voting machines are products produced by private firms. Getting them to publish the source code is about as likely as getting Apple to publish the source for iOS.
I'm not sure if this it's a popular decision, but I don't really think the private sector should be innovating with our electoral system. Integrity is of utmost importance, and voting machines have been proven vulnerable in the past. I don't think the private sector has appropriate motivation to really get voting right.
The difficulty is the purse strings to pay for election infrastructure comes from politicians. Private firms have funds for product advocacy without advance funding from government, unlike open source solutions. So the entire field is distorted from day 0 when it comes to proprietary vs open source.
A smarter thing to do would be to create a selection pool defined as any registered voter over the age of 75; and from that pool they choose their election technology board. And that board procures the election system based on the budget the legislature gives them. That is, the budget is the budget only, it doesn't say anything about what technology or brand is used, the only attachment is that funds are used for the state election system. If someone over 75 can't understand how our election system works, we're screwed, and in fact we've seen how voting systems have failed, have altered the outcome of elections, and because they had no paper trail, and no meaningful way to do recounts or audits there's no recourse.
The question is about who you trust and how you make sure that all the actors are being fair.
Would you feel better if the government designed the machines? A lot of people would probably feel worse.
Right now, you have each state's board of elections making decisions - they get observed by both parties so that they can all sign off on what's happening. Idk. You can never eliminate the possibility of fraud. I just think the likelihood of all of these actors colluding is really low... I mean congress can't even pass a budget.
Integrity being of the utmost importance is probably the #1 reason why the people put in power by elections should not be in charge of running those elections.
I'm not saying I prefer Diebold, but I certainly prefer it if the only other option is the House/Senate $PARTY Caucus.
I, as a taxpayer/legislator, can't contract a private company to (for a token fee) write and support a codebase that they will (as part of the contract) open source? The people win, as we get open source and hopefully transparent code. The company wins, as they're paid for their work (as they should be).
I fully expect that this would be more expensive than closed source.
(I'm not convinced that we should be using electronic voting at all, but that's a different debate.)
Well, ostensibly, given enough time, we'd end up with a single code base, where the modular add-ons are really just about user interface, similar to how every county in the U.S. does their own ballot layout. In the old days, pen and paper could be considered to be a single code base. If you get the basic thing to do right, it's a choice, not a necessity, to have different code bases that do the same damn thing.
So eventually it would be less expensive than closed source, to implement and maintain. Basically wealthier and more ideologically committed states would subsidize that reference implementation, and the rest of the country, should they adopt that open source reference, would get to use the code more cheaply.
From a global perspective, there's a pretty decent chance a huge chunk of the code is already done - i.e. it could use the Linux kernel with a bunch of its more extraneous modules disabled. A handful of math libraries. A handful of UI libraries. And then a code signing mechanism from stem to stern. There's a lot more to it than that, but code wise, a bunch of it might already be written, the work is not innovating ducks but getting them in rows, and documenting it in a way that it's reproducible and ideally only boots up when it's exactly conforming to the spec.
Closed source with multiple competing companies nationally by definition means many different code bases all in different maintenance states. It really is wheel reinvention over and over again.
We should be using electronic voting to tabulate and aggregate the data, but at the most local level personally I think electronic but with two paper receipts (one to the voter with who they voted for, one to the state w/o) is the best option.
Hmm. No effort, ever again, in open source products? I wonder what RMS, Linus Torvalds, and the BSD folks have to say about that?
Seriously, the open source movement has generated some pretty good businesses. Maybe the case can be made that electoral equipment is an exception to that.
>No effort towards innovation would then go into that sector. Ever again.
Good. You're talking about something that needs to handle a small handful of registers perfectly and produce a paper audit trail. Innovation is not only not needed, it should be actively suppressed due to the risk to the security of the electoral process.
Don't let perfect be the enemy of good. Opening up code in voting machine would be leagues better than leaving it closed source. At least other experts in the field could verify it.
Its also not like every citizen today can or understand all of the vote counting procedures with paper ballots, even when just constrained to their ballot.
I think it's an electronic voting machine in general. There must be a paper ballot trail. The absolute best setup for voting is the scan-tron style paper ballot. Immediate confirmation, quick and continuous vote tallies, errors are very rare and it includes a paper trail.
Most people can't understand the code in openssl. So we shouldn't use it? Let's just make all cryptography closed source then. The average person doesn't even know what an elliptic function is.
The point is that there is a large amount of people that do. They check. Not every citizen needs to check, but it is harder for there to be an error or to hoodwink someone if there are more eyes on the code. Essentially why opensource cryptography is good too.
You took the time to go search through those repos, and then took the time to come here and point out your findings.
You could have easily just copy pasted what you wrote here into an Issue on those repos.
It would have helped out your government, made these projects a tad bit better, and taught a few developers something they'll remember in the future when they write more software for the government.
It's part of the "spirit" if you will. Let's support these efforts - they're long overdue and difficult enough to get done for those few trying.
Reporting issues to a vendor takes a hell of a lot of effort and time compared to idly posting them in a comment thread.
This is like those remarks along the lines of "it only takes 5 minutes to submit a patch to the documentation". No, it doesn't - it takes 5 minutes to write it and create a PR, and usually two hours to defend it from the maintainer's criticisms.
>usually two hours to defend it from the maintainer's criticisms
Isn't that what we want, though? Discussion and review, particularly with the people who are most familiar with the project (the maintainers) is what makes software better, not worse. If you're not willing to do that then why comment on issues you find in open source projects at all, other than to stroke your own ego?
No, it's not. Legitimate discussion and review, sure - but that doesn't take two hours. The problem is with maintainers arguing over small changes, trying to trivialize them, finding reasons to reject them, asking contributors to do more work first... It's pretty much impossible to just leave a contribution and move on.
This is something I encounter constantly. It's why I've stopped bothering submitting documentation fixes in most cases, and now resort to writing and publishing my own documentation. It makes one-off contributions to open-source unreasonably costly and non-viable. I know many others who feel the same.
If maintainers want more contributions to their open-source projects, they need to start recognizing that the contributor's time and skills are valuable as well - that means not second-guessing contributions unless there's a concrete reason to do so, and making an effort to write a comprehensive response to a contribution (especially if it's a rejection). And most of all, being clear about this policy.
It's easy to say "why don't you submit it to the project?" from the sidelines, but the reality is that 'formal' contributions nearly always come with an expectation of dedicating your time and attention towards them on a more-or-less ongoing basis. Many people cannot afford that, and as such publish them elsewhere.
Changing that starts with maintainers, not with contributors.
Generally. But on the other hand, no, this isn't a monoculture, where everyone wants the same exact thing.
> Discussion and review, particularly with the people who are most familiar with the project (the maintainers) is what makes software better, not worse.
Despite all of the time that industry experts have contributed towards identifying and discussing security problems, these projects have severe inertia in the security realm.
How much more time do you want us to carve out of our lives for projects that won't fix their problems?
> If you're not willing to do that then why comment on issues you find in open source projects at all, other than to stroke your own ego?
I don't like the way this is framed. It's a hidden false dichotomy.
I don't disagree with your or joepie91_'s points in that many (maybe most) open source project maintainers are perhaps more guarded and difficult in accepting contributions than they should be.
That said, even if you submit a pull request or an issue ticket and the maintainer doesn't follow up, or is making the issue unnecessarily complicated to resolve formally and merge into the main branch, at least the record of your findings is there attached to the repo as a rejected pull request or a closed issue ticket, and that can help someone who comes along after the fact who maybe experienced the same issue you did and can benefit from your comments. Maybe that same person who comes along later will have the time to fight for having the changes merged, and can use your comments as further evidence that the issue is worth reconsidering.
I stand by the idea that finding the issue, but posting it on an unrelated forum (like HN) instead of attaching it to the project itself serves no purpose than to tell others who are probably not even users of the open source code "look at this problem I found/solved, aren't I great?"
> That said, even if you submit a pull request or an issue ticket and the maintainer doesn't follow up, or is making the issue unnecessarily complicated to resolve formally and merge into the main branch, at least the record of your findings is there attached to the repo as a rejected pull request or a closed issue ticket, and that can help someone who comes along after the fact who maybe experienced the same issue you did and can benefit from your comments. Maybe that same person who comes along later will have the time to fight for having the changes merged, and can use your comments as further evidence that the issue is worth reconsidering.
In my experience, this is a continuous source of frustration and anxiety. Not responding doesn't stop the notifications, and on top of that it also makes the contributor look bad by ignoring follow-ups. It's much easier and less frustrating to just not file anything in the first place.
> I stand by the idea that finding the issue, but posting it on an unrelated forum (like HN) instead of attaching it to the project itself serves no purpose than to tell others who are probably not even users of the open source code "look at this problem I found/solved, aren't I great?"
This is just flat-out bullshit, and frankly says more about you and your assumptions than about CiPHPer (who, for the record, is somebody I know personally). Have you considered that maybe CiPHPer is just tired of arguing with vendors and getting nowhere, and has now resorted to just publicizing their fuckups in the hope that maybe that will change something?
To be clear: nobody owes any vendor anything. While I personally at least attempt to disclose a vulnerability privately before going public, there is absolutely no requirement to do this. It is first and foremost the vendor's responsibility to keep their software and its dependencies up-to-date, not that of the public. Be happy that it's being reported at all, rather than just quietly sold on the black market.
The reality is that vendors have been grossly negligent of security for the past several decades, and still continue to do so, using all kinds of shit excuses about how much time and money it costs to fix something they shouldn't have fucked up in the first place. Don't be surprised when security researchers get tired of that shit, and just start dumping vulnerabilities publicly when there are indications that the vendor is negligent (like is the case here).
Again: the change here is to come from the vendors, not from the security researchers.
> In this case: I did, and it was immediately closed without any change to the code, documentation, process, or culture.
You left out the part where they told you this perceived issue did not impact their codebase, due to how they were using the upstream project you cited.[1]
> While this is an issue it does not affect us as we aren't using session cookies with slim
You then started to rant about how they don't update their dependencies "quick enough".
> You left out the part where they told you this perceived issue did not impact their codebase, due to how they were using the upstream project you cited.
This is completely irrelevant. Vulnerable dependencies are vulnerable dependencies, and trying to avoid updates because you're "not affected" is a really good way to get owned.
This is for the exact same reason that "yes, there's an XSS vulnerability in the admin panel, but that doesn't matter because it requires an admin login" is invalid. At some point, somebody is going to combine multiple "unexploitable" vulnerabilities or exploitation paths that you overlooked, and successfully compromise your system.
A vulnerable dependency means that you need to update that dependency, full stop. No exceptions.
Whether it's relevant or not is up for the project maintainer to decide. All anyone can do is point out what they perceive as a potential problem, and then let the others take it from there.
CiPHPerCoder didn't exactly do that. He came out guns ablaze from the start, using bolded text, italics and inflammatory phrasing - basically just stopping short of calling the project maintainer a complete idiot.
It's not surprising CiPHPerCoder got the reaction he did - it is, however, surprising he decided to do all this under his company's name.
It's almost like CiPHPerCoder is personally offended that some joe random developer hasn't heard about some obscure CVE CiPHPerCoder was involved with, or that they didn't handle it like he would like. "Do you know who I am!?"
The initial issue was he came here and ranted about it, instead of pointing it out to the projects. Eventually he "gave in" and reported it to a single project, but took this holier-than-thou tone and was aggressive the entire time. That's ridiculous.
> It's almost like CiPHPerCoder is personally offended that some joe random developer hasn't heard about some obscure CVE CiPHPerCoder was involved with, or that they didn't handle it like he would like. "Do you know who I am!?"
Except this isn't "some joe random developer", this is software created by and for the US government, which is featured on code.gov.
I'd expect them to take security seriously and apply all upstream security patches immediately, not sit on them for years after they've been resolved.
Anything but that is sheer negligence. What else hasn't been updated which contains vulnerabilities that do affect them?
> The initial issue was he came here and ranted about it, instead of pointing it out to the projects.
That's what joepie91 was trying to explain to you.
> Eventually he "gave in" and reported it to a single project, but took this holier-than-thou tone and was aggressive the entire time. That's ridiculous.
Would you rather I do that or not report it to them at all? Choose only one. If I'm going to do it, I'm not going to do it your way. You can if you want.
Personally, I'd rather not report bugs at all. Until you've reported vulnerabilities to two or three dozen different projects, this might not mean much, but: It burns you out to keep reporting the same flaws to different projects.
Having developers respond to security risks with an air of entitlement just turns up the heat on the burn-out engine.
The first response to my comment here was
You should send in a pull request, or file a bug report in the repo.
Pay attention to the order of operations here. The "should" is immediately associated with a large amount of unpaid work, with an alternative that would also be a large amount of unpaid work disguised as a hypothetically smaller amount of effort. But as others have stated: It's not.
> It's not surprising CiPHPerCoder got the reaction he did - it is, however, surprising he decided to do all this under his company's name.
Even if I had remembered to switch Github accounts, people would still associate it with my employer anyway. Kind of a moot point, really.
I gave you what you asked for. Next time, maybe don't tell people what they should do? It's rude to bark orders like that, and it won't get the result you want.
Perhaps if you didn't act like a jackass when reporting bugs, you'd have better interactions, and get less of the "burn out" feeling you're describing.
And next time you decide to put on a show, consider not doing it under your company name.
You're forgetting this arrogant display is here for all to witness, including folks who may (or may not, now) want to contract your company in the future. You also seem to forget the very folks behind code.gov are the same ones that influence who gets contracted with the government...
The people working on code.gov and all of the repositories are truly doing something great. Code has been in the federal government for at least 60 years, probably longer - and this is the first time something like code.gov has been produced. It's an amazing effort, and it's surely not easy to effect change like this at the federal level.
The open source initiative will help increase code quality at the federal level, as well as encourage less duplication of efforts (different agencies likely solve similar or the same problems very often). It also encourages a baseline standard of code and organization. This is a fantastic beginning!
Next time, a simple "Hey, did you guys know about CVE-2015-2171? You may have some vulnerabilities." is all that's needed. Instead, you let everyone know you were in a fit of rage - how dare someone suggest you comment on an issue you brought up!
We need to encourage and support these efforts, not shit all over them.
> You're forgetting this arrogant display is here for all to witness, including folks who may (or may not, now) want to contract your company in the future. You also seem to forget the very folks behind code.gov are the same ones that influence who gets contracted with the government...
If you base your "security talent" hiring decisions the same way you approach "contract customer service representative decisions, you'll end up with very pleasant people who don't know jack shit about security. Which would explain a lot of the results we're seeing. So you might be right.
If anyone is reading this thread and wants their software to be actually secure-- no sugar-coating or letting bad decisions happen-- get in touch. :)
> The people working on code.gov and all of the repositories are truly doing something great. Code has been in the federal government for at least 60 years, probably longer - and this is the first time something like code.gov has been produced. It's an amazing effort, and it's surely not easy to effect change like this at the federal level.
For once, we are in agreement.
> Next time, a simple "Hey, did you guys know about CVE-2015-2171? You may have some vulnerabilities." is all that's needed.
OK, why didn't you do that then?
It's so easy to tell others what to do, when you have no skin in the game. What will you do next time?
- Tell the other person what to do.
- Do it yourself, because it clearly matters to you.
> We need to encourage and support these efforts, not shit all over them.
> In short, don't be an ass... please.
I won't be an ass if and only if folks aren't making demands of how I spend my leisure time.
I have no intention of getting into an argument with you but wow, you must be having a bad day. I wonder if you would have gotten a different response to your report if you'd taken a different tone with the people trying to understand the issue itself.
I've been thinking about this a lot lately....I think "why didn't you just use a different tone" is very similar to how people that grew up in environments where everything went right when they say to people struggling "why don't you just do x,y,z". The problem is, they don't know how.
I agree, but with a little twist: the problem is not that the latter party doesn't know how and the former party has the answers, maybe it's that the former party doesn't know how either.
In this particular instance, the actual problem is notification. OP said he fixed the bug in his original code, and one of his downstreams should fix the bug too.
Fair enough.
But then things took a turn towards "you should do the work", "civic duty", and what not. To which OP replied he's not going to fix it in a code base he's not compensated for.
The actual problem in this scenario here isn't OP's willingness to donate his time, it's that the right person at the downstream needs to be notified (and in turn acknowledge the problem). And in the world of open source, this is a common issue which is impacted by multiple notification avenues, developer continuity, and general infosec policy.
But instead of any one of us creating an issue saying "Hey dudes, this was fixed in the upstream -- here's a copy-and-paste pointing to the fixed and respective vulnerable code", it's a pile-on with holier-than-thou mantras regarding why OP should do the free work of reading the source code, testing a fix, and creating a patch.
This is a great idea. I feel like the big picture isn't open source code for government, but open source APIs for interacting with the government. In the medium-term future, I think we should push for a system where all government documents that ought to be a matter of public record are managed in a publicly readable versioned source repository like, like a federal git server.
Other things that I'd like to see in publicly visible git repositories:
- federal laws and regulations (particularly regulations, standards, etc.).
- procedures (rules for federal processes)
- all documents for making requests of the federal government, with mechanisms for getting those requests to the right place.
Also, requiring Javascript in a single page application is a terrible design for this kind of site. Almost all of this can be static pages or traditional web frameworks. Requiring Javascript made the download much larger than necessary, slowed down the page load time a lot, and forced the page to reflow multiple times as the data arrived. A wide variety of web frameworks could have rendered and cached static pages instead of massively over-engineering the site as an "app".
Just a wild guess here, but I bet they're using Google Analytics for the same reason everyone else uses Google Analytics--to see what their audience is looking at.
> Also, requiring Javascript in a single page application is a terrible design for this kind of site.
Good luck making a single page application without Javascript! It is all in 25 requests and 668KB, not the lightest site, but that's actually less data than the homepage of Google is today. It's also properly setting expires headers so future page loads are very quick.
I know why they want analytics; I'm asking why they are giving out live tracking data to a single private company. This free data is effectively a subsidy.
> Good luck making a single page application without Javascript
The bad design decision was making it a single page application in the first place. This should be a static site, or very lightweight (and easily cached) framework.
> 25 requests and 668KB
That's an obscene amount of bloat. This should be 3-5 requests not counting images, and 668kB is an order of magnitude too big.
> less data than the homepage of Google is today
Comparing against another massively over-engineered, bloated web page isn't particularly useful.
> expires headers
That's great, but it doesn't affect the initial download size or the memory/CPU usage in the user agent.
> I know why they want analytics; I'm asking why they are giving out live tracking data to a single private company. This free data is effectively a subsidy.
How is it a subsidy? They are getting the analytical data they want and are not having to pay anything for it. They could spend tons of time and money to build an analytics system, but that sounds like a waste of tax dollars. You'll also notice that their code is hosted on GitHub, is that a subsidy too?
Mobile browsers don't always keep things until the server says they expire. That being said a frequent user will probably end up saving data in the long run.
I hear they use Ford vehicles and Boeing airplanes as well. Highly inappropriate for the US Government to use products from some of the US's biggest companies!
Analytics makes websites better, and Piwik is not easy or cheap to use at scale. Sharing web traffic data with Google Analytics (especially with IPs anonymized) is a pretty small issue IMO, especially when the benefits are you get good data like what's on analytics.usa.gov.
While I'm sure you have good intentions in trying to anonymize the collected data, you should know that the claims made by Google at [1] are highly misleading. Their claim is that they strip the last octet of the IP address, but that isn't nearly enough cooking to call the data "anonymous".
Assuming Google actually does this and isn't simply lying about saving the full IP (which has to be sent to them, as it is obviously in the SRC field of the IP header), this means they are only binning IPs into groups of 256. They only need at most 8 bits of identifying data other than the IP to uniquely correlate individuals within each IP group.
More than 8 bits of unique ID are available, given that various types of data that being tracked[2] about the user agent and (presumably) IP geolocation.
Also, Google is keeping enough of the IP to lookup the ASN.
> I work on analytics.usa.gov.
So if you're actually interested in claiming that "The program does not track individuals, and anonymizes the IP addresses of visitors"[2], then you really shouldn't be using an analytics service that - in spite of their claims of anonymizing" some data - obvious is tracking individuals and saving the most interesting parts of their IP address.
> Sharing web traffic data with Google Analytics [...] is a pretty small issue IMO,
Building profiles of everything we read or do online is probably the largest problem we will have going forward into the future. Data doesn't go away, and the problem compounds when the data can be correlated with other types of data. If you think this isn't a problem, then you really need to study what is possible with pattern-of-life analysis[4].
"The Federal Source Code Policy is designed to support reuse and public access to custom-developed Federal source code. It requires new custom-developed source code developed specifically by or for the Federal Government to be made available for sharing and re-use across all Federal agencies. It also includes an Open Source Pilot Program that requires agencies to release at least 20% of new custom-developed Federal source code to the public."
That's really nice, but I think the 20% is too little. It should be at least 50%. They could still keep a critical low percentage secret.
They are not exercising a copyright claim when they don't release things, they're just not releasing them. Public domain only helps you if you have access to the work.
In addition to the point already made that the government has no obligation to release anything, code written by contractors is generally copyright by the contractors.
The 20% number is part of the initial 3-year pilot program, not necessarily the ultimate goal. That's also the mandatory minimum, with agencies "strongly encouraged" to release as much as possible.
See the policy [0], linked from the launch announcement [1].
The work to open source something is often extra work. So it would effectively increase the cost or, keeping costs fixed, reduce the impact of the software on it initial target...those who it was developed for.
I also imagine that in many cases the effort to open source some project would be much greater than the pay off. There are many custom software solutions developed that are of almost no value outside of the small domain specific target audience.
There is a lot of internal resistance to open sourcing code. Lots of people believe that their code is their secret sauce - and why would you have highly paid consultants create code just so the next department can take it?
There's also security concerns to take into account - which I know someone will say is a good thing, because of transparency and so forth. But just realize that it's not a trivial amount of work to clean up a code-base and display it to the public all of the time.
And if the policy were open-source by default, the unspoken policy for project managers would be "automatically submitting a request to be closed source"
It's not perfect, but I know that 20% is a lot, and I think anyone who's worked in the sector would agree.
Those reasons, which surely place some non-negligible burden on the teams involved, do not seem to be so overwhelming as to ultimately dictate that 80% of projects are not made open source.
I'm definitely aware of the difficulties involved in open sourcing significant amounts of your work. The bigger burden is less, in my opinion, about cleaning up your code base. If you're committing potential security vulnerabilities into your code that are then tracked by your version control system – and you're a Federal agency – that's already a problem that's just going to be exacerbated by making it public; making that code private doesn't make the problem disappear.
The real meaty problems that all open source projects share is people: people like me who come in and overwhelm the project with support requests. I just opened four issues tonight just for this website, in the span of several minutes. (Sorry team!) If you already have poor project management practices in place, or your team is too small, this can quickly overload you.
Of course, with a vibrant community around your project, even the social and management problems could become trivial with time. Look at especially great examples like Hoodie. Given the number of technical people who have left cushy, high-paying jobs to serve in 18F, USDS, and the other alphabet agencies of late, I have to imagine the rallying cry to support truly useful code by compassionate people will be significant enough to justify the upfront expense here.
Not only is 20% not perfect, I don't think it's enough. I want 100%, and I think it's a fair request as a taxpayer, even if there is some burden. This country has fought two world wars and gone to the moon. We can always do better.
Assuming that's true, private companies contracted by the government may not be under that same requirement (think military specs and designs). The output as a whole may be public domain (I know what an F22 is, and the parts to build one, but the specs to those parts are proprietary) but the operations, and binaries,(specs and requirements to build the individual pieces) to produce that output may remain private.
New projects with new companies is easy. The problem is open source isn't a requirement for old projects, retroactively adding it is hard.
The companies hired would need to be new companies to contacting, with higher costs. Don't forget, government contracts are a lowest bidder war. Going full open source immediately might expose code that built using proprietary technology from a different division of a company. Therefore open source from scratch will have a high startup cost.
The existing projects can be small, open source, but more importantly these projects begin to build a foundation others can build on. The library of code created to support the framework of these projects will eventually be used as the basis for larger projects.
Wrapping back to the 80% this allows the existing, proprietary code reuse. While new, it's useless without its is proprietary components. This code, if released with proprietary components (and that's a big if since the binary would then be openly distributed) may not qualify as open source. One step further, public free binaries can be reverse engineered (the legality of which isn't something I'm well versed in). Which may not be in a companies current best long-term interest.
Tldr; The largest effort to being 100% open source is the library of code built on must also be open source or commonly distributed. Starting at 0 the current goal is 20%.
Provide costs don't rise too much expect 30% within 2 to 3 years (I'm being optimistic) or the next major government project timeline.
We give away the laws we write. In fact, the government provides an awful lot of services. If the taxpayer paid for it, the taxpayer should certainly be able to own it. I've never personally found it difficult to explain that to non-technical people.
Doesn't 20% seem a bit pedantic? I feel like if you throw a requirement like that into a large bureaucracy only bull shit will come out. Like an "Open source web interface" where only the CSS is posted...
The Commerce.gov API is under active, but not public, development. As such, API code is not currently made available publically. This Github repository will be used to collect and respond to feedback regarding the API and engage with developers interested in using the API.
---
I don't believe that's the correct usage of GitHub. What's the point if you're not going to showcase the code?
You can't make an FOIA request for code. You can only make requests for agency records, and in the definitions section of the bill, computer code is explicitly listed as something that isn't an agency record.
"computer software, including source code, object code, and listings of source and object codes, regardless of medium are not agency records"
Works created by the federal government are in the public domain, per section 105 of the copyright code. So you'd be right I can't use a FOIA request to obtain the code, but if obtained through other methods, I believe I'm in the clear for hosting it on Github.
With the exception of the recent 18F/USDS/etc groups acting as contractors to the rest of the government, the USG doesn't produce much code. Code is produced for the government by contractors, and that's not in the public domain. Here's a decent discussion of why the government doesn't regularly produce its own code, or much of anything:
Basically, you're trying to figure out some way to piss off the techies and managers you want to convince to be more open? Even if FOIA worked on code, you really aren't helping your cause..
> you're trying to figure out some way to piss off the techies and managers
I'm not trying to piss anyone off, but to be honest, I don't really care if I do. I'm simply trying to pry government open into being more transparent through any means necessary.
> you want to convince to be more open
No. Not in the slightest. My opinion doesn't matter anywhere in government, so I'm left with the tools at my disposal.
> Even if FOIA worked on code, you really aren't helping your cause..
When the carrot doesn't work, one must use the stick.
I'd give them a pass since they do put a lot of other code up. It makes sense not to fragment the community, and more to the point I don't think keeping a precisely one-to-one relationship between repositories with code and issue trackers is that important as long as it leans heavily in the code direction.
Otherwise yeah, I'd say they should host their own damn issue tracker.
This is awesome! At work we're trying to help with this at the municiapl level, it's hard so it's really great to see it coming from the Fed. Shout out to Becky Sweger[1] from 18F who has been instrumental in helping move things forward, talking about open data and open code.
The one project that i would love to see opensource is the FBI Sentinel project.
A lot of countries have broken infrastructure for law enforcement. Just like USAID, it would probably be more efficient to give software that can be adopted.
The sentinel project captures millions of man hours of wasted... and ultimately successful product development - all focused towards law enforcement collaboration. It would be good to have that.
If there's one thing that deserves a federal massive budget, it's this. If laws and regulations can be understood easily by software, we would remove so many problems the government creates. Complexity would be easier to manage, but also transparency would inherently breed simplicity.
Just for an example, HNR Block and other companies built on doing your taxes for you lobby the government to prevent simplification of the tax code, because they profit off its complexity.
The same applies to almost all law. The complexity does not exist without a reason - there is almost always some entity actively funding the continued obfuscation of law for profit.
So don't expect government to ever actively get simpler, because that would only benefit the outsiders rather than incumbents in the industries such laws affect, and those outsiders never have the capital to compete in lobbying and bribery.
Exactly - there are tax code companies you've probably never even heard of like Avalara entering the Unicorn range. The kicker: all they do is calculate sales tax! Actually a cool company, but one that should never exist.
It's worth more like $30,000. You only save the tax on the deduction, not the whole thing.
(and some deductions will of course be worth less than the tax; a $100,000 donation to the Red Cross costs $100,000 and doesn't have any financial return, so the $30,000 tax benefit there costs $70,000)
Most of the code looks fairly dull and single-purpose. Except NASA's code, as you may expect. Trick and OpenMCT both look at least interesting, and OpenMCT in particular could be used as the basis for many an interesting web-based project.
I've already seen OpenMCT integrated with Kerbal Space Program. It looks like a great platform for real-time sensor data visualization. I'd love to hook it into a drone and violate some FAA regulations :)
In case any code.gov website people are looking: there's an unfortunate bug: when the site is first loading on a slow network, you can see all the agency names and click them, but they all show text like "No repositories found." This made me think that this was a brand-new project that hadn't done anything yet.
Also, the little department logo images shouldn't have alt-text. That alt text of the department name overlays the actual text of the department name and just makes it even less accessible when the image isn't there.
(I suppose the lesson for web development in general is to test on a slow connection.)
I wonder how they will determine which open source projects are included at code.gov. For instance, I contribute to a few projects that are used within government for determining the economic impacts of fiscal policy, and the code is in the public domain. I'd love for them to be included at code.gov, but I'm not sure whether they meet the criteria.
The general guidelines are in the policy [0], linked from the launch announcement [1], but each agency's CIO is responsible for the particulars at their org. If it's not a new project post-August it won't contribute to the 20% so it might not be reviewed for release without someone suggesting it specifically, I would guess.
Bro is actually already a widely used and incredibly popular piece of software. However, I don't think the main branch is developed by the US government, so this may be the DOE trying to dodge the law.
Just an honest question. Can I obtain some estimation of the capabilities of CS people working for the government by just reading all these code. Will I find here a glimpse of code beautifully designed and tailored to solve some really deep problem that require real intelligence?, can I use these code to learn how to code a big project?
Interesting that someone approved the communist reference (or that people didn't perceive that as a communist reference). It's not the 1980s anymore I guess.
Do you mean 'people's'? That would make 'the people's house' as a way to describe US executive mansions and legislature buildings communist references as well.
Under the Department of Commerce - unsurprisingly - there is nothing from IRS.
A bit more surprising and entirely missing from the list (no source code):
* Department of Education
* Department of Health and Human Services
* Department of Housing
* Department of Interior
* State Department
* Department of Transportation
And not really expecting anything from:
* Department of Defense
* Department of Homeland Security
Others are only semi-public; these generally consist forks of software they use or private repos used for external collaboration.
I'm working on getting some of our stuff open-sourced. When it's written by contractors, the intellectual property rules can get complicated. Convincing a contractor to publish code their consultants wrote for a government project, no matter how minor, can be a tricky bit of advocacy. Sometimes they have really skewed ideas surrounding IP that must be overcome before this can happen. (Best case, our code will likely end up hosted by Internet2 or a similar foundation.)
Code.gov is built on the latest web technology using Javascript. To use Code.gov, please enable Javascript so we can share all of the wonderful open source projects the Federal Government has to offer.
> the Federal Government of the United States is not "the People"
I disagree. It is indeed "government of the people, by the people, for the people". You may not like some particular people, and neither do I: Some are dumb, corrupt, or simply disagree with me.
But if that doesn't represent the people of the U.S., I don't know what else could. If they were non-corrupt, all smart and highly capable, and all agreed with me - that would be some other country (in a fairy tale).
It's not government of, by and for me. I want everyone else to have a vote too, even if I disagree with them.
> It is indeed "government of the people, by the people, for the people".
In theory at least, and it certainly should be. Often it looks more like a government by the millionaires, for the corporations. But there's increasing attention for turning that around, and this project looks like a small part of that.
And horrible domain name: How come `code.gov` is US-only? How come it's motto is "Help propel America’s next breakthrough in innovation"? If this in not imperialism, I don;t know what is.
And you think it is normal? Why not gov.us? The US are always giving lessons to the world, but do not even understand that we --the other parts of the world-- would like to have some minimal level of fairness.
.us is barely used. The three letter TLDs started out as US only because the whole internet started out as US only. The first country that got its own national TLD was Netherland (.nl). .us was introduced later, but by that time, the ship had sailed.
Civilisation, development of human ethics, laws, all this stuff is our way to adjust for the unfairness of life. Also, back people did not invent our create the buses, was it right to let them seat in the back, when they where even allow to get in?
Bearing in mind that I'm ecstatic to see this collection of projects and someone clearly put a lot of time and effort into this already, I do have couple points of hopefully constructive feedback here:
- Please list all of your repositories and feature ones that you think deserve special recognition. Seeing all your repos is a 4-step process otherwise: click on the project, click on the link, click on the acknowledgement that I clicked a link, click on the organization name on GitHub. [1]
- Some information on contribution policies, pulling in your README.md, etc would be helpful on this site. [2]
- When I click on the link to go to your repo, I know I'm clicking a link that leaves your site. You even say so. "But you probably knew that already." Don't hijack it, please. [3]
- The activity feed doesn't provide much value in its current form. Most of these GitHub events are completely without context. [4]
EDIT:
I've since opened the following issues on https://github.com/presidential-innovation-fellows/code-gov-.... Anyone who has feedback, definitely let them know (kindly!).
[1] https://github.com/presidential-innovation-fellows/code-gov-...
[2] https://github.com/presidential-innovation-fellows/code-gov-...
[3] https://github.com/presidential-innovation-fellows/code-gov-...
[4] https://github.com/presidential-innovation-fellows/code-gov-...