I also tried to crack exactly this program a while ago. The company (I believe it is MilSoft, one of the most reputable software companies in Turkey) sent this challenge to university students to hire a part-time CS student. Nevertheless, this was the first time I've ever attempted to crack something and while I had little to no idea what was going on, it was a very thrilling experience. I think I went on 14 hours without taking a break.
I began by trying to run the program in GDB, got SIGSEGV'd. Afterwards I inspected the faulty address and tried to avoid it by changing its value, instead it crashed at somewhere else. After trying this hopeless catch-and-run for several hours, I decided I needed a better disassembly tool and went on to IDA Pro.
This particular program contains a trick that intrigued me very much, and it is the reason why I was getting SIGSEGV'd at different locations when altering the program code.
The main payload of this program is simply XOR-encrypted by some key. The whole thing begins by decrypting the payload and then begins its execution as normal. The gist is, the particular key that encrypted the main payload is the decryption code itself (for the unacquainted, assembly code is also just a byte stream). Here, this exact part:
As far as I can remember, the key was a bit more than that, but I'm sure it was including this part.
At the end of every iteration (of something involving this loop which I can't precisely recall now) the program checks whether it is running under debug mode (essentially makes a PTRACE call and reads its output, the OP also talks about it) If this is the case, it makes a jump to random address, so even if you are just neatly watching the program run under debug mode, you weren't going to achieve anything.
The next thing that occured to me is to manipulate how PTRACE returns its value, but I thought it would involve some kernel code fiddling and running the program under the modified kernel, which is WAY beyond my ability for now. I didn't know how to do it, but later by some very stupid trick I managed to pass this decryption part and the program made a jump to something like "__glibc_start". I needed to save the altered program and run it under gdb again (I don't remember why), but I was using the trial version of IDA Pro which prohibits me of such a thing. After making a few more desperate attempts I gave up.
But this "using the code as the key".. I think spending 14 hours to see this done was well worth it.
At the end of every iteration (of something involving this loop which I can't precisely recall now) the program checks whether it is running under debug mode (essentially makes a PTRACE call and reads its output, the OP also talks about it) If this is the case, it makes a jump to random address, so even if you are just neatly watching the program run under debug mode, you weren't going to achieve anything.
Could you change the jmp into a nop? That should let you attach in debug mode.
Could you change the jmp into a nop, then xor every nth byte of the program with jmp xor nop, where n = whatever offset the jmp was at in the key? The result should be a valid decryption.
Nah, if instruction "foo" is xor'd with "jmp", then xoring foo with "jmp xor nop" will remove the jmp and add the nop. Then the nop is removed during decryption, because "foo xor nop" xor nop = foo.
The reason this is clean and convenient is because nop is a single byte (0x0f or 0x90), which lets you replace any instruction of any size with nops. But if you had to transform jmp into an instruction of a different size, things could get hairy if the byte sizes don't line up. But you could still replace several instructions with other code.
I began by trying to run the program in GDB, got SIGSEGV'd. Afterwards I inspected the faulty address and tried to avoid it by changing its value, instead it crashed at somewhere else. After trying this hopeless catch-and-run for several hours, I decided I needed a better disassembly tool and went on to IDA Pro.
This particular program contains a trick that intrigued me very much, and it is the reason why I was getting SIGSEGV'd at different locations when altering the program code.
The main payload of this program is simply XOR-encrypted by some key. The whole thing begins by decrypting the payload and then begins its execution as normal. The gist is, the particular key that encrypted the main payload is the decryption code itself (for the unacquainted, assembly code is also just a byte stream). Here, this exact part:
As far as I can remember, the key was a bit more than that, but I'm sure it was including this part.At the end of every iteration (of something involving this loop which I can't precisely recall now) the program checks whether it is running under debug mode (essentially makes a PTRACE call and reads its output, the OP also talks about it) If this is the case, it makes a jump to random address, so even if you are just neatly watching the program run under debug mode, you weren't going to achieve anything.
The next thing that occured to me is to manipulate how PTRACE returns its value, but I thought it would involve some kernel code fiddling and running the program under the modified kernel, which is WAY beyond my ability for now. I didn't know how to do it, but later by some very stupid trick I managed to pass this decryption part and the program made a jump to something like "__glibc_start". I needed to save the altered program and run it under gdb again (I don't remember why), but I was using the trial version of IDA Pro which prohibits me of such a thing. After making a few more desperate attempts I gave up.
But this "using the code as the key".. I think spending 14 hours to see this done was well worth it.