How To Write Your Own BitTorrent Client By Using BT Library!

Hello everyone! In this article, we will talk about what the Bt library is and why it is worthwhile to have it in mind if BitTorrent is planned to solve some problems. Then, as a demonstration of the basic functions and the main API, we implement the simplest console torrent client.

Distinctive features of Bt

Bt is a modern full-featured implementation of the BitTorrent protocol on Java 8. Compared with existing open-source analogs, Bt has the following advantages:

• Modular design, which makes it easy to refine and expand. The architecture is based on the Guice IoC container and the standard java.util.ServiceLoader mechanism. Guice provides a transparent addition of new components and a redefinition of standard services, and ServiceLoader makes it extremely easy to assemble a client consisting of several modules, incl. located in a different jar’ah.
• The absence of binding to the mechanism of data storage. Standard storage assumes the existence of a file system, but, thanks to the use of the interface java.nio.file.Path supports including. and in-memory file systems, such as Jimfs.
• Full support for the mechanism for extending the standard protocol with your own message types, which can be useful in the case of developing a non-standard client for specific needs.
• Relatively low CPU and memory footprint and very fast performance, even with a large number of processed torrents and network connections, not least thanks to the use of NIO and a single-threaded kernel for sending and receiving messages (while alternative implementations for simplicity use a separate stream for each connection and blocking I/O).

The options required for the serious BitTorrent client options that are supported in Bt include:

• Extensive tuning and configuration options
• Integration with Mainline DHT
• Support for HTTP and UDP trackers, incl. multitrackers and private trackers
• Search for local peers through multicast
• Exchange of information about feasts with other participants in the distribution
• Obfuscation of traffic using session keys and asymmetric encryption
• Parallel upload/distribution of several torrents
• Selective downloading of individual files
• And finally, work with magnet-references. To download a torrent, just specify its unique identifier in the form of a link:

magnet:?xt=urn:btih:af0d9aa01a9ae123a73802cfa58ccaf355eb19f1

Creating the simplest command-line client

For the sake of minimizing body movements and avoiding unnecessary errors, we recommend not trying to reproduce the project code in the text of the article, but immediately download the finished project with GitHub.

Project Configuration

For convenience, the client executable file will be fat jar: the classes and resources of the application and its dependencies will be collected in a single archive. Create a new Maven project, in pom.XML we declare a class containing the main () method, the name of the executable file, external dependencies and a couple of plugins.

<? xml version = "1.0" encoding = "UTF-8"?>
<project xmlns = "http://maven.apache.org/POM/4.0.0"
xmlns: xsi = "http://www.w3.org/2001/XMLSchema-instance"
xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion> 4.0.0 </ modelVersion>

<groupId> com.github.atomashpolskiy </ groupId>
<artifactId> bt-cli-demo </ artifactId>
<version> 1.0-SNAPSHOT </ version>
<name> Bt CLI Launcher </ name>
<description> Command line BitTorrent client </ description>

<properties>
<project.build.sourceEncoding> UTF-8 </project.build.sourceEncoding>
<project.reporting.outputEncoding> UTF-8 </project.reporting.outputEncoding>
<compiler.source> 1.8 </compiler.source>
<compiler.target> 1.8 </compiler.target>

<main.class> bt.cli.CliClient </main.class>

<bt-version> 1.7 </ bt-version>
<jopts-version> 5.0.2 </ jopts-version>
<slf4j-version> 1.7.21 </ slf4j-version>
<log4j-version> 2.4.1 </ log4j-version>
</ properties>

<build>
<finalName> bt-launcher </ finalName>
...
</ build>

<dependencies>
<dependency>
<groupId> com.github.atomashpolskiy </ groupId>
<artifactId> bt-core </ artifactId>
<version> $ {bt-version} </ version>
</ dependency>
<dependency>
<groupId> com.github.atomashpolskiy </ groupId>
<artifactId> bt-http-tracker-client </ artifactId>
<version> $ {bt-version} </ version>
</ dependency>
<dependency>
<groupId> com.github.atomashpolskiy </ groupId>
<artifactId> bt-dht </ artifactId>
<version> $ {bt-version} </ version>
</ dependency>
<dependency>
<groupId> net.sf.jopt-simple </ groupId>
<artifactId> jopt-simple </ artifactId>
<version> $ {jopts-version} </ version>
</ dependency>
<dependency>
<groupId> org.apache.logging.log4j </ groupId>
<artifactId> log4j-core </ artifactId>
<version> $ {log4j-version} </ version>
</ dependency>
<dependency>
<groupId> org.apache.logging.log4j </ groupId>
<artifactId> log4j-slf4j-impl </ artifactId>
<version> $ {log4j-version} </ version>
</ dependency>
</ dependencies>

</ project>

As dependencies, we specified three standard Bt modules:

• bt-core: the “core” of the library; It contains basic functionality that does not use external dependencies
• bt-HTTP-tracker-client: module for integration with HTTP trackers
• bt-that: integration module with Mainline DHT

We also need Log4J (including the bridge to SLF4J) and the popular JOpt Simple library, which makes working with command line arguments simple and enjoyable.

Immediately add the configuration of the log4j2.XML log configuration, but here its text will not be given. Let’s just say that the application will log into two files: bt.log and bt-dht.log. The second file will contain events and messages related to the operation of DHT, which in most cases do not represent a great interest for the user.

Source Code

Well, with the project setup we are done, now we have to start writing the code. In the next section, we will devote some time to writing the necessary “binding” for processing the arguments of the program and setting up the JRE. The most impatient can skip this section and go straight to writing the torrent client code.

Options, modes, and parameters of JRE

Even such a small application as a torrent client can have a large number of configuration parameters, depending on the purposes of use and the current environment.

Let’s define the list of options that the user of our application would like to provide:

• Output help
• Specifying a .torrent file or magnet-links for downloading
• Specifying a directory to save files
• Activation of the forced traffic obfuscation mode (can be useful if the ISP user
• Cuts/shapes BitTorrent traffic and do not use advanced statistical analysis of traffic, which can determine its nature even when using obfuscation)
• Activation of the sequential download mode (for example, for playing media files in parallel with the download)
• Disable the interactive selection of files to download (to download all files that are contained in the torrent, this may be required if the list of files is large)
• Activation of the siting mode after the download is completed (by default, the client will shut down once all the files have been downloaded)
• An indication of the used IP address and ports (separately for BitTorrent and DHT connections)
• Request more detailed logging for debugging in case of problems

The list of options and arguments parsing is hidden in a separate Options class, but here we give only the original version of the main method of the program bt.cli.CLI client.main ().

package bt.cli;

import joptsimple.OptionException;

public class CliClient {
private static final Logger LOGGER = LoggerFactory.getLogger (CliClient.class);

public static void main (String [] args) {
Options options;
try {
options = Options.parse (args);
} catch (OptionException e) {
Options.printHelp (System.out);
return;
}
}
}

Now our application can display a nice help!

ption (* = required) Description
--------------------- -----------
- ?, -h, --help
-S, --sequential Download sequentially
-a, --all Download all files (file selection will be disabled)
* -d, --dir <File> Target download location
--dhtport <Integer> Listen on specific port for DHT messages
-e, --encrypted Enforce encryption for all connections
-f, --file <File> Torrent metainfo file
-i, --inetaddr Use specific network address (possible values include IP
address literal or hostname)
-m, --magnet Magnet URI
-p, --port <Integer> Listen on specific port for for connections
-s, --seed Continue to seed when download is complete
--trace Enable trace logging
-v, --verbose Enable more verbose logging

With the options processing almost everything, it remains only to configure Log4J and set the JRE parameters for the correct work of obfuscation. We add a call to several main methods in main ().

configureLogging(options.getLogLevel())
configureSecurity();
registerLog4jShutdownHook();

The methods associated with the configuration of Log4J are not of much interest to us now, you can see them here and here. The method configures security () will be covered in more detail.

The fact is that the obfuscation protocol uses encryption using session keys, the minimum recommended size is 160 bits. According to American laws that regulate the distribution of software (and therefore inevitably involve Oracle JDK), the maximum permissible size of keys for encryption by default cannot exceed 128 bits. It is not forbidden to use keys of a larger size, but the user must perform the necessary settings himself and “unlock” such a possibility. The Bt configuration allows you to set the key size from 128 to 4096 bits, but in this case, we would like to leave the optimal default value and adjust the JRE.

Up to Oracle JRE version 8u152 for this, it was necessary to download the jar file from the Oracle site and replace the file with the same name in the installed distribution. Starting with version 8u152, the same effect can be achieved by simply setting the environment variable crypto.policy = unlimited. This is exactly what the configured security () method does.

private static void configureSecurity () {
// Starting with JDK 8u152 this is a way
// to programmatically allow unlimited encryption
// See http://www.oracle.com/technetwork/java/javase/8u152-relnotes-3850503.html
String key = "crypto.policy";
String value = "unlimited";
try {
Security.setProperty (key, value);
} catch (Exception e) {
LOGGER.error (String.format (
"Failed to set security property '% s' to '% s'", key, value), e);
}
}

Thus, we choose a compromise option:

• If the user has a “fresh” JRE, then everything will work out of the box;
• If the user did not request compulsory obfuscation of all traffic, Bt will automatically disable the ability to establish connections with peers using obfuscation; while the plaintext connection will work in normal mode;
• If the JRE is old and not configured, and the user requests obfuscation, Bt will stop working with an error, asking the user to configure the JRE.

Integration with Bt

The torrent client code will consist of a constructor, several helper methods, and a method for starting. First, let’s look at the constructor.

private final Options options;
private final SessionStatePrinter printer;
private final BtClient client;

public CliClient (Options options) {
this.options = options;
this.printer = new SessionStatePrinter ();

Config config = buildConfig (options);

BtRuntime runtime = BtRuntime.builder (config)
.module (buildDHTModule (options))
.autoLoadModules ()
.build ();

Storage storage = new FileSystemStorage (options.getTargetDirectory (). ToPath ());
PieceSelector selector = options.downloadSequentially ()?
SequentialSelector.sequential (): RarestFirstSelector.randomizedRarest ();

BtClientBuilder clientBuilder = Bt.client (runtime)
.storage (storage)
.selector (selector);

if (! options.shouldDownloadAllFiles ()) {
CliFileSelector fileSelector = new CliFileSelector ();
clientBuilder.fileSelector (fileSelector);
runtime.service (IRuntimeLifecycleBinder.class)
.onShutdown (fileSelector :: shutdown);
}

clientBuilder.afterTorrentFetched (printer :: onTorrentFetched);
clientBuilder.afterFilesChosen (printer :: onFilesChosen);

if (options.getMetainfoFile ()! = null) {
clientBuilder = clientBuilder.torrent (toUrl (options.getMetainfoFile ())));
} else if (options.getMagnetUri ()! = null) {
clientBuilder = clientBuilder.magnet (options.getMagnetUri ());
} else {
throw new IllegalStateException ("Torrent file or magnet URI is required");
}

this.client = clientBuilder.build ();
}

Let’s go through the code, making the necessary explanations.

Config config = buildConfig (options);

The buildConfig () method creates a Bt runtime configuration. Runtime is a container for customers, each of which performs processing of its torrent. The main functions of the runtime are:

• Building modules into a single IoC container containing common services for all clients and extensions.
• Providing shared resources, such as connection pool, event center, components that interact with trackers and search for peers, DHT server, etc.
• Application lifecycle management: start and stop services, call custom callbacks.

A separate client is a small, lightweight wrapper over several objects specific to a particular torrent (context). Its task is to sequentially execute the stages of processing a particular type of torrent (.torrent file or magnet-link) and giving the user an API for starting and stopping the processing.

Accordingly, the Bt setting is performed at two levels:

• The configuration of the runtime, which contains parameters common to all clients: IP address, ports, start-up intervals for internal tasks, integration timeouts with external agents (trackers, peers), various limits and restrictions (on the number of connections, the size of the data block transmitted over the network, the depth queue I/O operations, etc., etc.
• The configuration of an individual client that includes torrent-specific parameters: a mode for selecting and loading blocks (sequential loading or random selection of the rarest blocks), a metadata source (.torrent file or magnet link), a directory for saving, callbacks that follow call after the completion of a certain stage of loading, etc.

Consider the code for creating a runtime configuration.

private static Config buildConfig (Options options) {
Optional <InetAddress> acceptorAddressOverride =
getAcceptorAddressOverride (options);
Optional <Integer> portOverride = tryGetPort (options.getPort ());

return new Config () {
@override
public InetAddress getAcceptorAddress () {
return acceptorAddressOverride.orElseGet (super :: getAcceptorAddress);
}

@override
public int getAcceptorPort () {
return portOverride.orElseGet (super :: getAcceptorPort);
}

@override
public int getNumOfHashingThreads () {
return Runtime.getRuntime (). availableProcessors ();
}

@override
public EncryptionPolicy getEncryptionPolicy () {
return options.enforceEncryption ()?
EncryptionPolicy.REQUIRE_ENCRYPTED
: EncryptionPolicy.PREFER_PLAINTEXT;
}
};};
}

private static Optional <Integer> tryGetPort (Integer port) {
if (port == null) {
return Optional.empty ();
} else if (port <1024 || port> 65535) {
throw new IllegalArgumentException ("Invalid port:" + port +
"; expected 1024..65535");
}
return Optional.of (port);
}

private static Optional <InetAddress> getAcceptorAddressOverride (Options options) {
String inetAddress = options.getInetAddress ();
if (inetAddress == null) {
return Optional.empty ();
}
try {
return Optional.of (InetAddress.getByName (inetAddress));
} catch (UnknownHostException e) {
throw new IllegalArgumentException (
"Failed to parse the acceptor's internet address", e);
}
}

Here, on the basis of the parameters specified by the user, we create a new instance of the class bt.runtime.Config, in which we redefine a number of methods so that they return the user-defined value if any, or the default value otherwise.

It is worth paying attention to the two parameters.

The first is numOfHashingThreads, or the number of threads that will perform the initial verification of the already downloaded data (“hashing” in common jargon, it is necessary when the client is restarted). By default, Bt uses only one thread, but the verification procedure is perfectly parallelizable, so it makes sense to use multiple threads. The optimal number of flows is in the interval number of cores; the number of cores 2; individual threads may be idle waiting for the completion of another I/O read operation.

The second parameter is the policy of using traffic obfuscation. Policies are used in the protocol for establishing connections with feasts, and there are only four of them:

1. Do not use obfuscation and do not establish connections with feasts, which require the use of obfuscation.
2. By default, offer a feast to use plain text, but agree to obfuscation if it is required by the feast.
3. By default, offer a feast to use obfuscation, but to agree to plaintext, if it is required by the feast.
4. Always use obfuscation and do not establish connections with peers that require plaintext.

In our application, we choose between the two most common policies: compulsory obfuscation or obfuscation at the request of a feast. The first policy is suitable for paranoid users (and unscrupulous ISP), and the second one allows you to establish connections with the maximum number of peers.

BtRuntime runtime = BtRuntime.builder (config)
.module (buildDHTModule (options))
.autoLoadModules ()
.build ();

Build runtime usually comes down to two things:

• Define the list of required modules (including disabling standard extensions, such as exchanging information about peers and searching for local peers).
• Enabling search and autoloading of modules present in the classpath of the application (in our case these are modules declared in pom.xml).

In addition to the main module, there are two extension modules in our application: an integration module with HTTP trackers and an integration module with Mainline DHT. The first module will be found and loaded automatically by calling autoLoadModules (), and for the second module we want to specify a non-standard configuration and therefore we override it manually.

private static Module buildDHTModule(Options options) {
Optional<Integer> dhtPortOverride = tryGetPort(options.getDhtPort());

return new DHTModule(new DHTConfig() {
@Override
public int getListeningPort() {
return dhtPortOverride.orElseGet(super::getListeningPort);
}

@Override
public boolean shouldUseRouterBootstrap() {
return true;
}
});
}

We redefine two parameters:

• The port on which the DHT server will listen for incoming messages.
• Permission to use the “public” DHT infrastructure (public bootstrap-nodes such as router.bittorrent.com); if you use BitTorrent for purposes other than downloading and distributing files on the Internet, you should leave this parameter set to false and specify a list of your own DHT nodes (in this role, any Bt instance can act).

Storage storage = new FileSystemStorage(options.getTargetDirectory().toPath());
PieceSelector selector = options.downloadSequentially() ?
SequentialSelector.sequential() : RarestFirstSelector.randomizedRarest();

BtClientBuilder clientBuilder = Bt.client(runtime)
.storage(storage)
.selector(selector);

The next step is to specify a directory to save the downloaded files. As mentioned above, bt.data.file.FileSystemStorage class constructor takes a parameter of type java.nio.file.Path, it can be used in combination with the in-memory file system, such as JimFS. In the library, such a possibility is used in integration tests (so as to save execution time on the file I/O), but hypothetically, there may be more exotic embodiments use, for example:

• Consecutive loading of text files directly into the console in the spirit of the more.
• A media player that plays files “on the fly” and does not use a storage device.

if (!options.shouldDownloadAllFiles()) {
CliFileSelector fileSelector = new CliFileSelector();
clientBuilder.fileSelector(fileSelector);
runtime.service(IRuntimeLifecycleBinder.class)
.onShutdown(fileSelector::shutdown);
}

We continue to collect the client. We have provided the user with the opportunity to skip the step of selecting files in case there are too many of them, or if the user wants to download the entire distribution. If this option is not specified, then we provide Bt own implementation of the file selector, oriented to work in the terminal interface.

public class CliFileSelector extends TorrentFileSelector {
private static final String PROMPT_MESSAGE_FORMAT =
"Download '% s'? (Hit <Enter> or type 'y' to confirm or type 'n' to skip)";
private static final String ILLEGAL_KEYPRESS_WARNING =
"*** Invalid key pressed. Please, use only <Enter>, 'y' or 'n' ***";

private AtomicReference <Thread> currentThread;
private AtomicBoolean shutdown;

public CliFileSelector () {
this.currentThread = new AtomicReference <> (null);
this.shutdown = new AtomicBoolean (false);
}

@override
protected SelectionResult select (TorrentFile file) {
while (! shutdown.get ()) {
System.out.println (getPromptMessage (file));

try {
switch (readNextCommand (new Scanner (System.in)))) {
case "":
case "y":
case "Y": {
return SelectionResult.select (). build ();
}
case "n":
case "N": {
System.out.println ("Skipping ...");
return SelectionResult.skip ();
}
default: {
System.out.println (ILLEGAL_KEYPRESS_WARNING);
}
}
} catch (IOException e) {
throw new RuntimeException (e);
}
}

throw new IllegalStateException ("Shutdown");
}

private static String getPromptMessage (TorrentFile file) {
return String.format (PROMPT_MESSAGE_FORMAT,
String.join ("/", file.getPathElements ()));
}

private String readNextCommand (Scanner scanner) throws IOException {
currentThread.set (Thread.currentThread ());
try {
return scanner.nextLine (). trim ();
} finally {
currentThread.set (null);
}
}

public void shutdown () {
this.shutdown.set (true);
Thread currentThread = this.currentThread.get ();
if (currentThread! = null) {
currentThread.interrupt ();
}
}
}

We are required to implement a class with one method, which for each individual file determines how to proceed: download or skip. In this case, we interactively request the user for the desired action. If the user presses Enter or enters “y”, the file will be downloaded, and if the user enters “n”, the file will be skipped.

clientBuilder.afterTorrentFetched(printer::onTorrentFetched);
clientBuilder.afterFilesChosen(printer::onFilesChosen);

As mentioned above, for each client, you can define a set of callbacks that will be called after the next phase of torrent processing:

• After the metadata (from the file or magnet link) is downloaded to display the information about the torrent on the screen (do not forget that initially, the program has only the path to the file or in general the 20-byte identifier in the form of a URI).
• Once the files for download are selected.

In our case, we notify the UI drawing component of the status change of the torrent so that it can output relevant information messages to the user.

There is no separate method for specifying the callback to be executed after the download is complete, but there is a convenient stopWhenDownloaded () method to automatically terminate the client. In our application, we do not use it. maybe we need to sit out the distribution.

if (options.getMetainfoFile() != null) {
clientBuilder = clientBuilder.torrent(toUrl(options.getMetainfoFile()));
} else if (options.getMagnetUri() != null) {
clientBuilder = clientBuilder.magnet(options.getMagnetUri());
} else {
throw new IllegalStateException("Torrent file or magnet URI is required");
}

this.client = clientBuilder.build()

We end the creation of the client by specifying the metadata source, in our case, it is either a .torrent file or a magnet link. In other cases, you can pass the abstract java.util.function.Supplier <Torrent>, in which you will receive metadata (for example, from the database).

The thing is for small: run the client to perform! To do this, add a couple of lines to main () and implement a method to run, which will also be the last one in our application.

public static void main (String [] args) throws IOException {
Options options;
try {
options = Options.parse (args);
} catch (OptionException e) {
Options.printHelp (System.out);
return;
}

configureLogging (options.getLogLevel ());
configureSecurity ();
registerLog4jShutdownHook ();

CliClient client = new CliClient (options);
client.start ();
}

// other code is omitted ...

private void start () {
printer.start ();
client.startAsync (state -> {
boolean complete = (state.getPiecesRemaining () == 0);
if (complete) {
if (options.shouldSeedAfterDownloaded ()) {
printer.onDownloadComplete ();
} else {
printer.stop ();
client.stop ();
}
}
printer.updateState (state);
}, 1000) .join ();
}

 

In the start () method, we see a new construction, namely the client method startAsync (). It is defined in two versions, one of which does not have parameters, and the other one – the one we use – takes on the input-listener function, which will be called by the client with a certain periodicity (here – every 1 second).

The tasks of the listener are very simple:

• Transfer the current information about the status of torrent processing to the UI drawing component (including the amount of transmitted data, connected peers, etc.)
• Monitor the end of the download.

In our application, we can not use automatic shutdown after loading, because the user can choose whether to remain on the distribution after downloading all the files, so we use a more complex approach with the listener function.

So, on this work on the application code is completed. Only the SessionStatePrinter class that manages UI drawing is left behind. It is quite simple and has a purely utilitarian nature, so we will not consider it, but we will proceed to test the developed application.

 

 

In the end, InterruptedException is logged in System.out, this limitation of the free version is a small roughness that does not affect the application in any way.

 

Conclusion

So, this concludes our first acquaintance with the Bt library. We reviewed the main features and the basic API and assembled our small BitTorrent client, albeit not as sophisticated as its older brethren, uTorrent, Transmission, Deluge, etc., but with enough features to use it on a regular basis to download files, watching movies, etc.

It can not be said that the project is at the beginning of the path: it will soon be 2 years since the first lines of code were written. At the same time, much remains to be realized, supplemented and corrected. The BitTorrent protocol itself does not stand still, the specification of the second version has recently been released, which support would be very desirable to include in Bt.

We are very pleased with the fact that, despite its age, BitTorrent still constitutes a significant part of the Internet traffic. This is a beautifully designed data synchronization protocol, which, unfortunately, in the view of many is synonymous with piracy and theft. And, we hope that the high-quality implementation of the Java protocol can spur interest in using BitTorrent for good purposes, including. within organizations, on mobile devices based on Android and for creating new products and ecosystems.