There are many great BitTorrent clients out there, yet I've made my own and willing to share and attract collaborators!

Guys, I've added a code screenshot above the gif, is it better now?

Just curious, how many hours do you think you've spent on this project?

I'd say something in the range of 500-1000 hours.

That's less than I was expecting. Did you study any existing library to get up to speed?

No, after I had studied BEP-3, wiki and and a few papers (most importantly "BitTorrent economics"), I had a very clear understanding of what needs to be done.

thank you!

Thank you, your work is terrific! Good docs, good code, and I especially like the ability to create custom code to write to alternative storage.

Thanks!! Yeah, I was pursuing such design that would allow to replace/customize most of the parts, including the internals. As a bonus, it makes it easier to continue working on the library as the codebase grows (now nearing 25K lines of code)

Man this looks slick, going to give it a shot. Love the ability to be able to download a magnet link from the CLI and not worry about what some fancy GUI is doing in the background cuz I am paranoid.

Thank you! Make sure to install http://www.oracle.com/technetwork/java/javase/downloads/jce8... to allow 160-bit cryptography and use -e flag to be completely safe!

I also have a simple shell script locally, that wraps the java command, so I have to just type `btmagnet AF0D9AA01A9AE123A73802CFA58CCAF355EB19F8` to download to a pre-determined location (hardcoded in the script). I wonder if I should share it on the CLI README page

> Make sure to install http://www.oracle.com/technetwork/java/javase/downloads/jce8.... to allow 160-bit cryptography

Or don't use the Oracle proprietary distribution but the OpenJDK of your operating system which normally doesn't exclude proper crypto algorithms.

(be aware that the JDKs downgrade silently if certain libraries are missing in your host)

That's some really useful information, thanks! Need to add this to the readme

do it! that'd be my use case more or less.

Done: https://github.com/atomashpolskiy/bt/tree/master/bt-cli#shel...

Or add something like this to your .bashrc

function bt() { java -jar /path/to/bt-launcher.jar -m "magnet:xt=urn:btih:$1" -d /$HOME/Downloads }

What things that makes you paranoid can a "fancy GUI" do in the background that an application with a CLI cannot also do in the background?

There’s less code for me to review in this project than say in utorrent or another application.

Talks about reviewing code, mentions a closed source application...

The gist being that GUI applications tend to have larger codebases but yes my example was wrong.

Credit to OP, this looks like a really solid project. The CLI aspect of it reminds me of peerflix (https://github.com/mafintosh/peerflix), which I've been using for a while and am pretty happy with (no affiliation, promise)

I use rtorrent for exactly this.

Sounds like deluge (deluged) would have suited you fine.

It looks great but the top GIF makes it seem like you are offering a command line BT client. If I were you I would have it slowly scroll through the code of the client in the GIF or put a few frames of code of the client and or some compiling and then show the current top GIF.

Oh, I didn't expect this :) Sounds good, but I'm not sure I have patience do this right, it's such a pain to record gifs - I always misspell or misclick something, and by take #47 go completely nuts

Try using asciinema. It records your terminal session and works so well. Of course typos and such are still an issue...

How about put a picture of source code above it then?

Nice work. I'm wondering, how would you test/benchmark such an application. Besides manual testing on a variety of torrents, how would you do regression test there? same question for benchmarking. How to compare this work with another client?

I have several hundred unit tests and also a bunch of integration tests. While UTs usually test API of individual classes, each of the ITs creates a swarm of local peers and launches a torrenting session with certain conditions: seeders to leechers ratio, downloading from .torrent file vs using a magnet link, PEX enabled/disabled, encrypted vs raw message streams, etc.

Benchmarks (like in libtorrent) is something I'm looking forward to, but the project is still in the early stage -- there's a lot to be done with regards to optimization, i.e. switching to using NIO selectors, adding a caching layer, etc. That's a lot of work, and that's why I'm looking for collaborators, before investing more time into performance-tuning.

I would advise against implementing a custom cache for piece data. Caching file data in-app is counterproductive because it wastes memory duplicating data which the OS already holds in its page cache. Instead memory mapped I/O should be preferred.

It makes sense, what would you advise in case of big files that don't fit into memory (or when it's undesirable to use big amounts of RAM, and speed can be sacrificed)? Maybe caching should be disabled by default but with the option to turn it on if the user wants it?

you can memory-map files in chunks and keep references to the mapped buffers or evict them once a certain limit is reached.

note that mapped buffers are not necessarily backed by RAM, they act more like swap space, except it goes straight to the original file instead of a swap partition. the downside is that anything reading from the buffer can unpredictably incur IO overhead if there's memory pressure and the OS decides to not back them with memory. java doesn't have an api to check residency of buffers. a possible workaround is to first queue them up on a separate thread and forcing them in before using. if they're already there it should have low latency. if they aren't then only the victim thread eats the IO penalty.

> java doesn't have an api to check residency of buffers.

Too late to edit now, but I misremembered that part. I think the issue was that isLoaded was not entirely reliable.

So, something like LRU cache of MappedByteBuffers, one buffer per chunk (i.e. "piece")? I wonder what queueing strategy would look like, just enqueue a couple of subsequent pieces when one is requested and loaded?

Start simple, then heap on optimizations if there are latency or throughput issues.

Really like the approach around using Guice as a central point for hooking your own code into the system.

Will look forward to reading through the code later!

Thanks! God bless IoC, it saves my ass each time I make some silly design choice

I'm curious, it looks like the screenshot shows the author is using a Mac. But Mac OS hasn't shipped with Java in several years. So why Java? Is it simply familiarity, or are there other reasons?

A mix of both, actually. I've been using uTorrent for years until 2016, when I've decided to switch to some new OS X release. After installing the system update, the old, ad-free version of uTorrent has stopped working. The new version seemed a bloated mess to me, so I've tried a few other clients, but was left unsatisfied. Finally, in despair I've decided to write my own client with simplistic command-line UI, with emphasis on streaming movies. The sheer amount of new information in BEPs was enough for me to decide to take the easiest route and use the language, that I'm most familiar with.

Thanks for the reply (it seems my question caused an uproar here, oops).

I definitely like the idea. My question was more aimed to understand if Java has some desirable features for writing a torrent client maybe as part of the SDK or other 3rd party libs. I didn't even look at your source code yet :P

Java stood out to me because I don't think I've seen other CLIs written in Java.

Turns out that the gif was kind of misleading, it's just a demo application for the library. I'm trying to use as little 3rd party dependencies as possible (core only uses slf4j and Guice)

This is a library for development. If you wanted to ship a client based on this, you could do that and package it with the Java runtime.

Most OS's don't ship with a C compiler, go, python 3, rust, arc, etc, what does that have to do with anything?

You're conflating developer tools with a language runtime. Every popular OS ships with everything necessary to run a C, Go, or Rust binary. Python 3 and Java both require a separate runtime to be installed, but I will say that every Java CLI I've used has had terrible startup performance.

I'm using Lanterna, a Java library for creating text-based terminal GUIs (https://github.com/mabe02/lanterna), and the startup is 1-2 seconds. Can't recommend it enough

Good to know, but 1-2 seconds is still very slow compared to a native app. I have an 8MB Go CLI application that can startup and print its help in .05 seconds, and even that is probably slow compared to a C or Rust equivalent.

Any savy Java developer knows how to AOT compile to native code, if that really matters.

There are plenty of JDKs that support it, and even Oracle is finally adding support for it with Java 9, initially only for Linux x64.

By Java 10 timeframe no one that only knows about OpenJDK and not the several other JDKs can state that Java starts slow vs C, because both will be AOT compiled to native code, loading native .so files.

From what I've seen Java/JVM AOT won't produce binary executables but shared libraries that will be loaded on startup by the JVM. When I first read about Java AOT-compilation I expected it to produce real executables that's why I think this is important to mention. But please correct me if I am wrong. So if this is true even with AOT compilation you still need the JVM as dependency (although I think you can get a minimal JVM in under 10MB). It also seems that the AOT compiled code needs to be recompiled on Java updates. Although I quite like Java and the JVM, I still wouldn't choose Java for writing simple command line tools.

Nevertheless it is quite useful for improving startup for tools like Gradle or server applications. But sure, I also don't expect Java startup performance to be a problem in the mid- to long-run.

Correct, but that is no different than having to ship glibc.so or MSVCRT.dll alongside your application, specially if you want to be certain which version gets used.

This is only relevant to Oracle's implementation though.

Other JDKs do support static linking, as long as you restrain yourself from features that could require dynamic compilation.

There was a talk at JVM Languages Summit about the current state of AOT compilation regarding Java 10.

https://www.youtube.com/watch?v=n5DCg6M2MDM&index=11&list=PL...

I would rather ship glibc (8MB) or musl (.5MB) than the JVM (100s of MBs?), but it's not necessary given that nearly every target already has a libc. Also, Go programs don't depend on libc, and C and Rust can probably statically link against some libc in a pinch.

Rust can use musl, yes.

Good point, but I wonder what size will the final binaries have, given that bytecode inflates significantly when translated to the machine code (there was a good discussion on HN recently about app sizes)

In any case smaller than Electron apps.

Electron apps aren't really my standard for "reasonably sized CLI applications"...

Well, printing stuff is easy. Just tried:

  $ time java HelloWorld 
  Hello, World

  real	0m0.088s
  user	0m0.088s
  sys	0m0.016s

That's presumably not loading any libraries at all though - the time to being interactive (or just doing something useful) will get worse as more bytecode has to be decompressed and loaded.

> That's presumably not loading any libraries at all though

Sure it is loading libraries, everything that System.out.println() depends on.

opening and quiting vim for a file of 2500 lines of C++ (so with all the plugins and stuff) takes 0.030seconds. I don't see how you can be satisfied with 2 seconds startup for a terminal application

Well the torrent download will take a lot more than 2 seconds so that time is totally amortized.

Besides this comparison looks like a total bikeshedding unless you plan on downloading torrents with vim.

By the way the project looks fantastic. I'll be giving it a try.

The alternative for just command line torrent on a *nix is aria2 [0]. Works exactly as advertised.

[0]: https://sourceforge.net/projects/aria2/

I completely agree that for a interactive tool like text editor startup time is crucial. But in my case being a bit on the slow side is probably fine, because the UI is for informational purposes only, and VLC takes around 5-10 seconds anyways to startup and begin playing the file, that is being downloaded.

I'm not conflating them and it's still a silly way to frame the question. If you want to ask someone why they wrote something in language X, just ask them that.

You're being pedantic; natural language is contextual for better or worse. Anyway, asking "Why did you use Java", is likely to be interpreted as a slight against the language and thus be downvoted (indeed, his politely phrased question was even downvoted when I saw it!).

Some apps just include JRE into the app package (+100MB), and just use it if it's not provided by OS.

But then you need to provide different app packages for different OSes.

C, Go, and Rust would require different app packages for different OSes--that's not a problem. The problem is bundling a 100MB runtime for a CLI.

Java 9 has a linker, you have remove everything that is not needed and have a application specific runtime.

On top of that, if lucky enough to be on an Linux x64 system, you can AOT compile to native code.

people bundle a browser for a tiny icon in your taskbar so 100MB for a runtime in a CLI isn't gonna prevent people from using it

Technically _most_ ship with a C compiler and a python compiler since pretty well every linux distribution & bsd distribution has those. But that really doesn't have anything to do with which language you should use.

Technically, that's not true.

Amazing work, how long did it take to build? Where you working on it full-time or part-time?

Thank you! Calendar year, in the evenings and on weekends, with sprints of several weeks and a month or two of rest/AFK after each sprint to keep sanity (I have a full-time job, a wife and a 3 year old kid). Happy that I had the persistence to keep working on the project

Very impressive! Thanks for sharing, it's encouraging to me and I hope to others to know that with proper time management we can put out amazing work.

I remember using Java based client Azureus and the memory usage was huge. Overnight it would leak to about 1Gb. Later uTorrent appeared and it used 2mb... Forever

I guess nothing to do with this but was triggered by two combined words: Torrent and Java

It's great to see projects like this on HN. Congrats man, well done!

In the older syntax, something like this:

  client.startAsync(new Consumer<TorrentSessionState> {
	void accept(TorrentSessionState state) {
		if (state.getPiecesRemaining() == 0) { 
			client.stop(); 
		}
	}
  }, 1000).join();

Thank you. Really useful.

Just read the Java 8 docs.

Some developers work on large code bases where adding lambdas or fresh new syntax features isn't really seen as an advantage for the rest of the teams, most often explicitly restricted.

I can only thank those who wrote such obfuscated code in previous years, that eventually forced everyone else afterwards to write code as simple and clear as reasonably possible. That's the reason why I've asked for the simplified code, Might as well save the time for other developers on similar situation.

No, the simplified code is the Java 8 code. You are just clinging to the archaic syntax that you are used to.

Of course you can choose to shoot yourself in the foot and just stick with Java 6, but don't try to force others to make the same mistake.

> don't try to force others to make the same mistake

Seems you prefer to put everyone on the same bag and speak for them too.

Wow thanks man! This can be useful in a variety of circumstances!

Not very familiar with the Java world yet. I've heard Android now supports at least some features from Java 8, but would this library work in an Android app?

I guess not. There is a ticket for backporting this project to Java 7 or RxJava (https://github.com/atomashpolskiy/bt/issues/7), but I personally think it's not worth it.

It's amazing how this made it to the front page of HN and I had no idea until now. I actually discovered this by googling a couple hours ago :)

Wonderful Lib!

It's possible in Azureus, but it might be a lot of effort to turn the functionality into a library.