BitTorrent 4.20 Released

[SlyckTom]

Allegro! Typically used to describe a quick music tempo, in the BitTorrent context it describes the latest release of the Mainline client. The primary focus of this Mainline release is to address the issue of ISP bandwidth throttling, and more importantly, BitTorrent traffic caching.

It’s no secret that bandwidth concerns have been one of the more pressing issues surrounding the BitTorrent community. CacheLogic, which provides P2P caching solutions for ISP networks, has previously calculated that approximately 60% of a networks bandwidth is consumed by the BitTorrent protocol. This average varies according to the ISP, as some ISPs report less bandwidth consumption and other reporting more.

In response to this growing bandwidth consumption, ISPs generally have three choices; throttle bandwidth, cache traffic, or do nothing. Bandwidth throttling is becoming less of an attractive option, as ISP customers in Canada have becoming increasing vocal in their opposition. In addition, new end-to-end encryption technology developed by µTorrent and Azureus has been well received by these inhibited users.

A more advantageous solution which has grown in acceptance is traffic caching. A compromise between the ISP and customer, caching allows users to participate on BitTorrent without utilizing an ISP's external bandwidth. Caching servers, which are hosted by the ISP, maintains the most popular and most queried traffic. Instead of the BitTorrent client utilizing the ISP's external bandwidth to obtain the required file(s), traffic is contained between the cache server and the end user. In other words, the traffic is maintained within the ISPs internal network, which is considerably cheaper than handling external traffic.

“We spent an extraordinary amount of time face to face with the largest ISPs in the world who now see anywhere between a third and up to 70% of all their traffic in the BT protocol,” Ashwin Narvin, president of BitTorrent, Inc., told Slyck.com in May. “[We’re] trying to convince them there’s a better way to manage the BitTorrent protocol then to limit it and to shape it. We’ve been strong advocates of the caching solution in the next version of the (Mainline) BitTorrent client, what we are calling the Allegro release. There’ll be a protocol which allows ISPs to cache BitTorrent content which is a great thing for users. It improves the user experience for downloading with BitTorrent no matter what client it is as long as it’s implemented the Cache Discovery Protocol and it also offers ISPs a cost effective way to allow BT to exist on their networks.”

The latest release of BitTorrent, aka the “Allegro” release contains the anticipated Cache Discovery Protocol. Like its name suggests, the new release allows ISPs that implement caching technology to recognize and store BitTorrent traffic. Although in its infancy, the protocol will be available to any developer that wishes to implement it.

"Transparent caches are sophisticated pieces of hardware," Ashwin explained to Slyck.com. "They perform deep-packet inspection to detect the frequecy of certain files. If a file shows up on the network frequently, the cache stores that file so that its seeded in the network rather than by peers. ISPs appreciate this because their access networks are terribly congested with P2P traffic. Caches are legal and covered explicitly in the <a href=http://www4.law.cornell.edu/uscode/html/uscode17/usc_sec_17_00000512----000-.html target=_blank>DMCA</a>."

Specific details on Cache Discovery Protocol will be available relatively soon on BitTorrent.org.

BitTorrent, Inc. has been on a continuous goodwill campaign, attempting to convince ISPs there’s an alternative to blocking or throttling BitTorrent traffic. Now that the technology is readily available, it’ll be up to the ISP to determine which direction they wish to travel.

[SonyRootCanal]

This article just brings up so many questions....

With respect to caching, does the cache server participate in the torrent itself?

What impact could this have on private torrents and maintaining share ratios?

As I understand it, ISPs will be caching (retaining a copy of) the most popular torrents and saving bandwidth costs outside of its own network. How will the ISPs avoid being sued as aiding in an infringing activity with respect to current US policy? How much disk space do they need to provision?

Does an ISP want to help share porn?

[Allied]

"Allegro", I like the name. I'm getting kinda tired of fumbling with BitTorrent client names. "The Official", Bram's Client", "The Mainline", Allegro is alot simpler.

Caching servers, which are hosted by the ISP, maintain the most popular and queried search criteria. Instead of the BitTorrent client utilizing the ISPs bandwidth, traffic is contained between the server and the end user.

Does anyone have a Greek to English translator?

Let's start with "most popular and queried search criteria". What's that? Sounds like scraping the tracker.

Allegro uses its Cache Discovery Protocol(CDP) to look at an ISP's BitTorrent cache. What is in the cache?

Are ISP's holding the actual data that's being sent from 1 peer to another? So Rogers or Bell is going to be hosting the latest cam? It wouldn't take much for them to filter any traffic that has the name "X-Men 3" in it.
Or are the caches just holding IP lists? I can see that cutting down on traffic a litte. The end user could be confined to local peers, so there'd be no extra traffic along the international lines. Good for the ISP, but bad if the only seed is on another continent

If less bandwidth is being used, that means that either uploading or downloading is being slowed down. Or have the laws of physics changed on me?

Tommy, you'll be going to uTorrent and Azureus for a comment right? I'm not a programmer but I trust them more than Bram and is MPAA buddies.

[SlyckTom]

Sure, I can go talk to them.

And yes, the ISP cache server will hold a good portion of the actual content. So when you download...whatever...you'll be downloading from the cache server instead of from a peer. Torrents don't take up any real bandwidth - its the huge 5 gig files that do.

Edit: Allied, I'll clear up that paragraph a bit...

Edit2:

If less bandwidth is being used, that means that either uploading or downloading is being slowed down. Or have the laws of physics changed on me?

Less network bandwidth is being used, but I don't know how much that will affect u/d speeds. BitTorrent, Inc., Cachelogic and NTL (UK isp) just started using the technology, and supposedly the speeds are quite impressive....

[eMbry00s]

This is good news. Giving the ISPs a chance to do as their customers want, instead of only battleling them with encryption and futher obfuscation.

[Andu]

I wonder if the protocol is flexible enough to allow other networks to participate as well. If they are thrilled that the BT traffic can be cached then they'll probably be even happier if all p2p traffic can be cached.

[SlyckTom]

Cachelogic's solution caches all p2p traffic. You can use the concept for any P2P protocol, but this release is specific for BitTorrent.

[Fallingwater]

If less bandwidth is being used, that means that either uploading or downloading is being slowed down. Or have the laws of physics changed on me?

External bandwidth costs a lot more to an ISP than internal bandwidth.
When you download from BT, you get data from outside the ISP's own network. But if they were to cache actual data the path would be direct from their servers to you, and it would cost them much less.
Of course, when you face the truth, most of BT's bandwidth is used for illegal transfers. If ISPs only cached legal data it would benefit them, but not a lot (just compare the number of torrents for Linux ISOs, free movies and so on with those for the latest game, hollywood blockbuster or RIAA album). If they cached copyrighted stuff it'd benefit them a hell of a lot more, but this would open them to litigation from everybody in the *AA and their dogs.

On the other hand, they don't have much choice.
Raise prices to cover bandwidth costs, users leave in droves. Limit bandwidth, users leave in droves (and many of those who don't encrypt their transfers and say "in your face"). Either way you lose lots of money and possibly go bankrupt if you're not one of the big companies.

But since the only way they can cope with rising p2p bandwidth, at least in the near future, is to cache copyrighted data, they may prove to be an unlikely ally in changing current copyright laws. Either that or they'll just give the appropriate bribes to the appropriate people, come up with some unlikely legal loophole, and the *AA will see them as another target for their lawyers (thus weakening their efforts elsewhere).
In either case it's a win-win situation for Joe Average Filesharer.

Or at least I hope so.

[archy]

Well i don't see the server just holding the ip list in the cache that would make no discernible difference to the amount of traffic traversing the network. It would have to be caching the files being downloaded. This would obviously have legal consequences for the isp although if the isp filtered the cache it would severely limit the effectiveness of the caching system and so the traffic traversing the isp's network would increase again. Another legal point is many isp's run news servers with alt.binary.* newsgroups which hold copyrighted material and isp's don't get regularly sued over that.
As for private torrent sites i would imagine the likes of µtorrent and azureus implementing a flags system similar to that used for dht so trackers that don't want caching can turn it off for their torrents.
The way it will help the isp is that although the amount downloaded by a peer will stay the same if that peer is downloading it from the cache server on the same isp's network then its costing the isp less than if its coming from somewhere on the internet.
Since its in effect adding a extra seed to torrents it could speed up bittorrent downloads which cant be bad.

[Andu]

Hm I wonder if they take some kind of license fee from BT Inc. If they do it would be useless for networks which are run by open source communities.

[SlyckTom]

This would obviously have legal consequences for the isp although if the isp filtered the cache...

You bring up an interesting point about the newgroups, as ISP is protected via the safe harbor provision in the DMCA.

Also...since the traffic is maintained between the cache server and the client, and does not travel outside the network, how would any interdiction efforts ever realize such illegal behavior was taking place? It seems akin to the entertainment industry's inability to apply enforce their IP rights on campus LAN networks...

[wigwam]

My understanding is that Cachelogic believe they have the legal right in the US and EU to cache 'contraband' torrents so long as they don't speed them up or seed them i.e they can't improve the end user experience of the contraband downloader.

They can, however, speed up approved torrents, and that seems to be their ultimate goal. But the bandwidth throttling, IIRC, won't distinguish between the approved and the contraband.

[Allied]

Thanks Tom, Fallingwater.

So the BitTorrent Cache takes the most popular data from outside the ISP's zone. No meta data or bootstrapping data, but the actual file(s).

I guess they determine what's popular by signaling out the BitTorrent protocol. List all the data streams along with how many requests are made. Then take the ones with the highest percentage.
Right?
That's not going to help old torrents.

Rather than downloading from a peer, you're downloading from a cache server. These servers have got to have more up time than the average seeder.

I remember when I first started using BitTorrent. Those Blue lights in BitTornado never lasted very long. I guess when the tracker goes down now all you need is the cache server as 1 peer(or maybe even a seed) and you should be able to get the whole file.
Maybe these Caches should get their own DHT layer? Connect all the peer caches so they can become seeds.

What about all the encrypted traffic out there? I'm guessing it'll be left out of the popularity rankings. I wonder what Bram would think of a client using protocol encryption and at the same time using the CDP.

Data from a user, his ISP, to your ISP then to you is how it works now. But if your ISP already has the data, your download speeds could be faster.

But how easy would it be for the MPAA to step in and cause problems. The cache servers give the MPAA lists of all the BitTorrent info hashes. Then the MPAA decides which ones are allowed and which ones infringe on their copyrights. Then they just remove that torrent from the popularity rankings.

*puts on tin foil hat
Is it possible for the MPAA or the ISP to replace the cache data with null data? So you think you're downloading from a peer, but really the data is coming from the cache and it's all corrupt?

[blargh]

Is it only me whose alarm bells are tolling wildly?
Doesn't anyone realize that this opens up new vistas (the word well chosen in this instance) of censorship should the ISP's decide that hosting (yes, caching IS hosting) illegal content would be too risky of a buissness? I mean, if you can cache something, you can also see what you're caching aswell... so really, who out there thinks that the ISP's if faced with a choice to filter this cache or not, simply chooses to take the risk?

No, the further bittorrent "advances" it's protocol, the further from the consumer and more into "going legit" it becomes. It's going to be *very* interesting to see the replies of the developers of the major bittorrent clients regarding this change of protocol.

Edit: allied, interesting points! similar to my thoughts!

[archy]

if the isp's start posioning the cache then customers would just use a client that supports encryption and so would bypass the cache server since it wouldnt show up as bittorrent traffic. Also it would mean that bandwidth consumption would go up again and so its really in the isp's best interest to not corrupt the files held on the cache server.

[Red_Blue]

My understanding is that Cachelogic believe they have the legal right in the US and EU to cache 'contraband' torrents so long as they don't speed them up or seed them i.e they can't improve the end user experience of the contraband downloader.

EUCD (Article 5(1)) contains safe harbor provisions for transmission caches, except for computer programs, which are covered under the eCommerce Directive (Article 13). The same rules are in the DMCA (17 USC 512(b)).

Any strengthening of the legal precedent and applicability of the cache provisions for P2P traffic is in the end good for the P2P users, because that opens up the possibility to interpret individual P2P users being covered by the same safe harbors in P2P networks where peers route traffic for others.

However, I'm not entirely convinced that the implementation of this caching is really to best interest of both parties. In theory yes, it's an extremely good idea. But just like transparent HTTP proxies can fuck up web surfing if badly implemented, any cache is a complication to the system and does cause both transmission overhead and reduced overall reliability.

If I had to choose between fighting bandwidth stealing or bandwidth fraud (stealing or fraud are proper terms here, because the customer paid for that bandwidth and if he's not getting it, the ISP is either stealing it back or if they originally intended to deceive the customer out of his money without giving the promised service, then that's fraud) and caching, then caching would obviously be the better option as a compromise.

I haven't looked at the spesification of the "Cache Discovery Protocol" yet, mainly because it appears BitTorrent Inc. hasn't published any! There is also no mention about it in the outdated release notes. In general, bittorrent.com seems these days more like an ad site with no actual information about what the mainline BT downloadble from there actually does. One thing is quite certain however, this CDP won't work with PE/PHE/MSE encryption. There would be no point to amend these protocols to work with CDP, because that would make them vulnerable to bandwidth fraud and censorship. The only option would be for clients to drop encryption when interfacing with the cache.

Personally, I won't be enabling an option to use CDP instead of PE untill the client has an algorithm to decide which one will give better performance (or even to dynamically switch between them for a given peer based on speed measurements). CDP may help the users of ISPs which deploy it, but it will hurt those users who are customers of bandwidth fraud ISPs. And because the swarms are a mix of those, the clients need to be able to tell the peers apart (based on whether they announce PE and/or CDP flags, and handle them differently.

Basically any system that can identify protocols and content for which the communicating parties don't want identified (by using encryption), is fundamentally evil. Even if it's not used for censorship or bandwidth fraud initially, the same applies as with DRM, demand for such uses exist, so any "dual use technology" implementing involuntary content classification should be boycotted, banned and made illegal.

[Fallingwater]

My understanding is that Cachelogic believe they have the legal right in the US and EU to cache 'contraband' torrents so long as they don't speed them up or seed them i.e they can't improve the end user experience of the contraband downloader.

They can, however, speed up approved torrents, and that seems to be their ultimate goal

How, exactly, do you speed up a file without providing some of it by seeding?
The whole concept of the cache is that the ISP provides some or all of the file. Yeah, maybe they could only provide large fractions of files instead of complete ones, but I fail to see how that would change the situation.

So the BitTorrent Cache takes the most popular data from outside the ISP's zone. No meta data or bootstrapping data, but the actual file(s)

Well, I'd certainly think so. Torrent files themselves are a few kbytes each. Bootstrap data may be more, I don't really know, but compared to the actual data files I'm pretty sure their size (and consequent impact on bandwidth usage) is insignificant.

The cache servers give the MPAA lists of all the BitTorrent info hashes.

The *AA wouldn't be given access. The point is that the user must be allowed to download from the ISP's servers what he wants, not what the *AA want, or the whole cache system becomes worthless.

Is it possible for the MPAA or the ISP to replace the cache data with null data? So you think you're downloading from a peer, but really the data is coming from the cache and it's all corrupt?

I suppose it would be possible, but it'd be more trouble than it's worth.
They'd have to have someone in the ISP on their payroll to do that, and if it came out that they were doing it they'd become a perfect target for lawsuits.

so really, who out there thinks that the ISP's if faced with a choice to filter this cache or not, simply chooses to take the risk?

If p2p/torrent bandwidth usage keeps going up as it is, it may come down to either cache or die... and faced with that choice I think any ISP would at least try to fight and have it their way.

[LANjackal]

Good developments. Great article, Tom .

[Allied]

I suppose it would be possible, but it'd be more trouble than it's worth.
They'd have to have someone in the ISP on their payroll to do that, and if it came out that they were doing it they'd become a perfect target for lawsuits.

Well, BitTorrent Inc is making the software and giving it away to ISP's. The MPAA is working with BitTorrent. One could say BitTorrent is on the MPAA's payroll.

Bram has made his position clear. He's against piracy. He wants to use BitTorrent for money.

As for the ISP, I can't think of any rock solid motive for poisioning the cache.

But if I downloaded a 4gig DVD only to find out that the file(s) are corrupt, I'd certainly be weary of any other big torrent. I know I'd stop seeding it right away. Would there be any way to know if the bad data came from the cache or the real peers?

Poison the cache, give unlicensed torrents a bad reputation. MPAA, BitTorrent Inc and the ISP's all win.

As for the legal question. Did any of the FastTrack spoofers get sued by anyone? A few math equations made faking a UUhash a piece of cake. I'm sure it's not that easy for BitTorrent.

But if it's done, who would have the bank account to take on the MPAA, BitTorrent Inc and their ISP?

[Iron]

I wonder if the protocol is flexible enough to allow other networks to participate as well. If they are thrilled that the BT traffic can be cached then they'll probably be even happier if all p2p traffic can be cached.

Cachelogic's solution caches all p2p traffic. You can use the concept for any P2P protocol, but this release is specific for BitTorrent.

Other companies have tried caching in the past, one of them being Joltid, and a plugin was created for eDonkey and other programs were updated to work with it as well. The problem was that not many ISPs actually setup the caching server due to the possible liability of caching illegal material. The plugin would try to resolve 'cache.p2p' and if present then presumably your ISP had a cache server setup. You used to be able to find all this information on http://www.joltid.com but it's rather useless at the moment.

[eAi]

There would be no point destroying (nulling or otherwise) the cache. The BT client hashes the data as it comes in and most clients will ban a seed after a certain ammount of bad packets - it would just be self defeating if that had false caches.

The real issue here is that in effect this causes ISPs to be running their own "walled garden" download server for their customers. Unless the *AA get access to who downloads from the ISP, they have no way of knowing who is downloading what (assuming a client supports downloading purely from the cache)... Either way, it puts the ISP in an odd situation. Newsgroups have gone somewhat under the radar, partly due to their generally inefficiency but if the *AA decide to make an issue about this it could spill over to newsgroups as well.

[wolfgang128]

Eversince bram and the mpaa made peace i don't see any benefits to the p2p comunaty comming out of new inovations from them

[Moglar]

caching isn't limited to bottorrent OR other p2p. an ISP can cache anyting it wants, including much visited websites eg. the bbc frontpage or something. It could also use that as a legal defence, saying they just cahed must requested traffic, and couldn't know it was copyrighted.

[curzlgt]

Eversince bram and the mpaa made peace i don't see any benefits to the p2p comunaty comming out of new inovations from them

So what!

Has any harm come to the community from Bram's inovations? A man who has brought filesharing to at minimum 3 million more users....... ( though imo that estimation is on the grossly low side )

Bram gave the world BT, the fruits of his furtil mind and labor, for free and with little restrictions. FAR to many condem him when he tries to make a living and change the legitment digital distribution system for the benifit of the masses.

Even when BT's actions my help both "illicit" and "legit" users at the same time, we see condemnation and paranoia....

Where's the gratitude

[denominator]

has anyone considered that these ISP's might just cache content that gets the ok from the MPAA/online movie distributors/bittorrent, inc.?

there's just way too much speculation and no hard facts about the whole thing, and I think that bittorrent, inc. and their partners want to keep it that way for as long as they can. remember, this is a business, and as a business they're out to make profit. they're not out to improve filesharing among random user #1 and random user #2 who are transferring data for themselves, bittorrent is trying to secure a model of distribution which will gain them the most money out of the deal.

edit: ashwin reads these forums, and could respond to a number of the questions, however it's unlikely to see a meaningful response or explanation.