"There's this thing called an LFSR; it's math-y, but very cool" in a 2009 message from someone pretty well plugged-in to the vulnerability research scene tells you a lot about where we stand with crypto vuln research.
In reference to what he want's you can't compare a LFSR with a Mersenne twister. His goal is not primarily to produce a good pseudo random number (->Mersenne twister) but to enumerate enumerate all IPs of net range without repetition an. When you wand to be fast you want to avoid repetition.
I don't know the credentials of this 'Greg' in any field, but he comes across as an inexperienced blowhard, that makes me doubt he could possibly have any expertise in any area. A modicum of expertise in some area and experience in developing software should ensure someone has more respect for something like nmap. It's a nontrivial piece of software and this email comes across as "I can build Reddit in a weekend".
Let's definitely debate Greg Hoglund --- someone you admittedly don't know anything about, but are apparently happy to comment about --- rather than talking about the thing this comment thread is ostensibly about.
It is frustrating that, since Hoglund appeared in the Anonymous storyline, that's all we get to talk about --- because people, like (I don't know you, so don't take too much offense) you, come out from the woodwork and hang superficial barbs like so many ornaments on an ugly, boring Christmas tree.
You are commenting about this guy's personal, private email. You don't have to like or respect him. But your contribution to this thread --- "This guy sounds like a total asshole in his emails!" --- degrades the discussion. I'm calling you on it. Please, don't be such an asshole.
I think he has the arrogance of someone who is excellent in one area, but foolishly thinks it will qualify him to be excellent in other areas. Let's not forget we're building a huge psychoanalysis on a throwaway email here.
He was probably looking at a data structure white papers/research. LFSR is good to use as a primary key for DB's./Linked lists(sequential numbers, no so great) I was working on an API along while ago using C.
I think this could be a mission for Redis and Node!
Anyone want to hack together a little API/framework for LFSR Generation?
A question I posted on SO some time back may be of interest, it solves a similar problem using an LCG (generating a shuffled range of numbers with no repeats):
It's really simple to program in ASM or using ISE/Verilog. going to a low tech approach it can be done on graph paper.
No reason why it's specific..Simple, 'easy to port'. I was doing a little bit of research about a theory I had but when I figured out enough I never fully completed that little project.+2 years prior.
> I've already had my brain cranking on what
> elite networking code I could now write in
> the kernel and I've always wanted to write
> a badass portscanner too.
And now he can program those both over wifi with his laptop while in his tree fort with the "no parents allowed" sign hung off the side. Afterwards he can rescue his girlfriend from ninjas with is excellent karate skills...
Everything about it is mindboggling; if you're going to put effort into building something in a place as inhospitable as the kernel, a portscanner? Who's going to pay for that?
The harder it is to write an application, the better the application must be. Putting stuff in the kernel means it will be harder to get right, so clearly the app will be better.
A security hole in the kernel is a small price to pay for a 0% speed increase.
System calls are expensive for a variety of reasons (including, but not limited to, the cost of extra data copies and the cost of switching security "rings"). For system-call heavy applications (in particular, networked services), moving things into the kernel provide huge speed benefits. [0] [1]
You are right that it's generally a stupid choice to make, but you're dead wrong in assuming that it's got no benefit at all.
I am deeply sceptical of any technology where the builders think up the name first and then try to find a reason it should be called that. It just seems too much like stuff I did when I was 12.
My impression is that this part of the email sounds like some business/sales guy that has a great idea where the technical parts are 'easy' and can be 'implemented by monkeys.'
"I have a great idea, let's make a product that can make us a lot of money. We'll work out the details later, that's the easy part."
This characterizes his entire attitude better than anything I've seen so far: "too much like stuff I did when I was 12". That, and some of my early Web customers in the 90's. I cringe just to read this stuff.
GNU is not a technology, it's a movement. And to get all Marxist up on this joint, the building blocks for a new movement is created long before the leader is conscious of the need for one; the status-quo automatically provides an anti- checklist to motivate mobilization. One only has to scream "down with the government!" in a poor neighborhood to prove this; no further deliberation is needed, as the audience comes equipped with its own reasons to revolt.
Most programmers I know get the "I just discovered a silver bullet, damn I'm awesome" nonsense schooled out of them somewhere between entering college to a few years after, but apparently HBGary is an entire organization built on the exception.
Yeah, I remember the day when my default action on thinking up an "awesome game-changing idea" became immediately looking up who had already tried it and why it didn't work.
Older, sadder, wiser.
It gets better, eventually you think of an idea that some tried that did work and feel a little better.
Yeah so these are ideas that date back to the '90s, so it is genuinely weird to see Hoglund talking about them in 2009. But for whatever it's worth: if ever a tool needed its balls blown off, it's nmap. I have nothing but respect for Fyodor, but that space badly needs real competition.
There is, for what it's worth, very little reason why the best scanner need be written in C. Scanners aren't even I/O bound in their most common use case; they're timer-bound.
If you feel the need to justify nmap being writting in C, which you shouldn't, just remember: The people who wrote nmap know C, and managed to write fucking nmap in it.
If you want a better one written in something else, then get cracking...
He's just saying they used what they knew best. For example, there's no reason Git has to be written in C[1]. Mercurial, which is Python, is doing just fine. C is what Linus and the kernel hackers knew best.
[1] Yeah, I know it was mostly hacked together using Bash and Perl, but that was a long time ago. Now it's mostly C. I'm suspecting they rewrote the performance-critical stuff in C, and left everything else alone. Not sure, though; I don't follow Git development.
Not sure why I'm bothering to respond since you're being rather trolly.. but neither. I rarely program in C, and I rarely have a use for nmap.
Rather I just intensely dislike when people pick language choice to focus on. You pick your language based on what your team knows, and if you can ship with it. Picking it for any other reason just makes you come off in a PHB type of way. If you have real criticisms and suggestions to make, then do so and make the world a better place in the process.
I am saying something very simple that seems to have caused your head to explode. It is: you could write n+1map in Python using Twisted, or Ruby using EventMachine, or Erlang, and probably end up faster (not because of the language, but because you'd design for speed and the language doesn't matter) than nmap is in C.
This isn't a controversial statement. Nobody thinks nmap is the fastest possible scanner. To argue that n+1map should be written in C, you have to make a case for what aspect of a port scanner benefits from C. Which is why I pointed out: nmap doesn't even look I/O bound; it looks timer-bound. And in most cases, neither I/O-bound nor timer-bound programs benefit greatly from locality, cheaper copies, or (within reason) optimized memory allocation and layout.
I believe the problem is that you seem to be saying c should only be used when performance matters. They think the language choice is irrelevant and shouldn't have even been mentioned. Or so I believe.
For my part, I am an avid C programmer; it's my first language, and virtually the only one I used through the '90s. And I will say: you should avoid writing in C unless you need to. The cases I can come up with off the top of my head where it's necessary:
* When performance matters and can be gained through writing C code --- for instance, if you're compute-bound, or if you need fast access to data structures.
* When building incremental improvements to large C codebases --- ie, writing a loadable kernel module.
* When you're deploying in small-footprint environments.
What are the other cases where C makes sense?
I brought up the C thing not because I want to take potshots at what is probably my favorite language, but because a majority of the developers on HN don't write C, and you'd hate for them not to take a crack at competing with nmap because they had a faulty belief that they'd need to use C.
C is a bad choice for a new port scanner started in 2011. I've already written comments here justifying that statement. If you really think I'm wrong, it might be fun to hash out why. (I've got a fairly decent background in nmap's problem domain, for what it's worth.)
I don't have a problem with C, although if I had to do it I would use C++ for object-oriented support and more or less guaranteed portability. Possibly also to compile in a lua interpreter. That said, I'm not a domain expert and was responding to the question of whether nmap should have been coded in C rather than what the best language would be for something started NOW.
I don't know what the best alternatives would be and while I doubt I can contribute much to the discussion, I've read your other posts on security with interest and would be interested to hear which language you'd pick and why.
Personally, I think Lisp is a fantastic language to write a scanner in; you can express async state machines in it naturally, and most Lisps have very good foreign interfaces, which would allow you to just call into libpcap. Lisp is also good for DSLs --- look how Seibel did the little expression language for binary formats in his Lisp book; that's exactly the problem statement that drove me to make my own dumb language in '97 to express packet formats.
Virtually any language will work for this (which is why you should avoid C). About the only thing I'd steer you clear of is a JVM language. The JVM is ordinarily a win, but here, where you want the path-of-least-resistance to getting pcap into your program, it's a bit of a pain. So scratch Scala from the list. Other than that, have fun.
Nmap isn't fundamentally flawed. It's ambitious and sprawling in ways that detract from its basic purpose. It wants to be a general-purpose network surveying tool, but is constrained to the UX of a port scanner.
Someone else should write the best possible port scanner and free nmap up to be the giant network profiling system it clearly wants to be.
"It wants to be a general-purpose network surveying tool, but is constrained to the UX of a port scanner."
What exactly does that mean? Nmap certainly isn't constrained to the UX of a port scanner. In fact, their expansion wasn't even in the form of bundled build, you can get all the tools separated, like: nmap, nping, ncat, ncrack...
So actually, it really isn't detracting from its basic purpose, you can still run simple network scans. But, if you got the knowledge, skills and the other tools installed, you can then also do much than advanced network scans, like vulnerability discovery and exploitation, passwords bruteforcing, etc...
Your comment isn't making much sense to me, please help me understand.
It has N different kinds of "stealth" scans that all produce the same kernel of valuable information, it has an elaborate (and, against most corporate networks from external vantage points, not particularly accurate or helpful) OS detection scheme, it runs scripts, it produces stdout and file output in a variety of complicated formats (it really just wants to be a service hooked up to a database), it runs scripts, IT RUNS SCRIPTS, come on. This isn't a serious argument. You just like nmap. That's fine; I think nmap is impressive too.
Oh, and it takes _for_ _ever_ to run. I'll pastie the 50 line EventMachine scripts I race it with when people on my team get stuck waiting for 12-hour nmap runs.
Sure. You know how simple this task is when you have an event loop, right? Here, I'll write a port scanner right now:
class HostProbe
module Probe
attr_accessor :bp
def connection_completed
@win = true
end
def unbind
self.bp.closed(self, @win)
end
end
def closed(obj, won)
@inflight_now -= 1
port = @inflight_q.delete(obj)
if won
puts "%TARGET-LISTENING: #{ @target }:#{ port }"
end
fill()
end
def sweep
Generator.new do |g|
(1..65535).each {|port| g.yield port}
end
end
def fill
@inflight_now.upto(@inflight_max) {
@inflight_now += 1
if(port = @sweep.next)
EventMachine::connect(@target, port, Probe) do |c|
c.bp = self
@inflight_q[c] = port
end
end
}
end
def initialize(host, opts={})
@target = host
@sweep = sweep
@inflight_q = {}
@inflight_now = 0
@inflight_max = opts[:inflight_max] || 10
fill()
end
end
EventMachine::run {
HostProbe.new(ARGV[0])
}
I think that may be it? I'm not even going to bother seeing if it evaluates. I write that stupid script once a month or so. There is probably a set of nmap options that crushes it for speed, but with the default options and a firewalled target network (ie: most professional nmap targets), I lap nmap with it.
Bit twiddling is much easier in other languages than in C. And virtually every mainstream language has a pcap binding for the packet reading and injection (although a good straight-socket scanner could still give nmap a run for its money).
In all of these cases one of the main problems seems to be that non-technical people have too much decision making power in technical areas. (I'm not going to address this huge problem here.)
I've lost count of the number of times I've been in a meeting listening to someone talk about something like the "Linear Feedback Shift Register" mentioned in the article.
Unfortunately, technology seems like magic (or snake oil) to people who don't know anything about it...
While the FBI isn't known for being on the technical bleeding edge, and used to bleed a lot for technical screwups, the RCFL's - Regional Computer Forensics Labs - are very much out there on the leading edge of computer forensics technology.
---
There are some amazing technical innovations in EnCase and the products built on it. Unfortunately, the company has management issues that result in some very poor decisions in many areas, including deployment of software development resources.
I've been using EnCase for about a decade (including a large portion of that working with the FBI), and I honestly can't think of any amazing technical innovations.
The support for reading .pst files? The fact that it can finally handle filesystems besides FAT32 and NTFS? Certainly not it's stability.
EnCase is very good at what it was designed for, which is basically an idiot-proof way for an investigator to comb through a hard drive looking for search terms.
I would argue that technically EnCase isn't even as good as FTK, which is something that pretty much every third-party evaluation of the two products has shown.
I won't disagree that there are some smart folks working at the FBI, and that's not even counting the ones who spend all their time developing "zoom and enhance" software for photo analysis.
Greg and Aaron, two of the guys involved with HBGary, are incredibly smart and capable. They were very well respected in the industry for their intelligence, products, and services.
---
How well would you stand up in the court of public opinion if several years of the internal email messages for your startup were leaked?
I'm very concerned about the rise of the cybersecurity/infosec complex ala the military-industrial complex, but one of the reasons for my concern is that they have very bright, very motivated people working for them who know how to play the D.C. political games.
Well, that email is a piece of relatively ignorant crap that I could have written when I was 17. The last time I read something like that was a Phrack interview....
Apparently there are people I insulted that still use that terminology. It really makes you sound, umm, managerial and pointy haired when you use those.
/16 obviously. Say it aloud a few times and just feel how it flows off the tongue. "slash sixteen, slash sixteen." So much smoother than getting tongue tied saying "class B" out loud...
This coming out from someone as respected (at least in the past) and renowned researcher like Hoglund doesn't give a ringing endorsement to the industry. Was he that desperate for fame and fortune?!
It's lame that this is on HN. Basically it's just finger pointing and laughing at someone for being stupid on the internet (in a private mail, it seems).
http://seclists.org/nmap-dev/2011/q1/768
all these leaked emails just remind me why I left the netsec industry so long ago
Just reading this email and knowing these types of people I already know how this came about:
1. random surfing on the web and finds out about LFSR
2. thinks 'this would be awesome for a port scanner'
3. sends email with 'lets do something like nmap but better and faster using this thing i just read about on reddit'
4. 'i done my bit, GO TEAM!'
5. add 'software architect' to email signature