Slyck.com
 
Slyck Chatbox - And More

The Purpose of Splitting files

Discussion about the binary newsgroups and how to successfully participate in the file-exchange
Forum rules
PLEASE READ BEFORE POSTING: Slyck Forum Rules

The Purpose of Splitting files

Postby SlyckTom » Mon Jul 20, 2009 9:51 am

As I'm writing the never ending story, AKA my book on file-sharing, I'm coming back to a recurring issue: the purpose of archiving or splitting up a file before uploading to a news server. This seems archaic and no longer necessary to me.

Let's say we have a 700 MB file - typically that's broken down into 15 MB chunks, which is then broken down into small text messages.

I understand the purpose of breaking the parts down into small text messages - the news server won't accept the post otherwise. But if we're already breaking the file down into small text messages - why bother with the initial step of achieving?

I suppose it was more important back in the day when files would get damaged because of propagation - requesting a report of 1 part is easier then reposting an entire archive. But considering the vast improvement to Usenet architecture, is this really a problem any more? Even if it was damaged, couldn't PAR2 files take care of whatever blocks needed repair?
Follow us on Twitter @SlyckDotCom
Join our Facebook Fan page
SlyckTom
 
Posts: 5713
Joined: Fri Jul 26, 2002 7:22 pm
Location: New York City

Re: The Purpose of Splitting files

Postby Fartingbob » Mon Jul 20, 2009 10:35 am

The whole P2P scene has a fair few habits that seem very unnecessary these days. Splitting movies into CD sizes (who the hell stores movies on several CD's these days????), split rar files, i still see the occasional rar containing multiple zip files containing the actual files.

The P2P scene evolves much like everyone outside of Kansas evolved. Split rars dont really have any disadvantage, so sort of hang about until they condradict or prevent the next evolution. P2P clearly doesnt have an intelligent designer controlling everything.
User avatar
Fartingbob
P2P Trafficker
 
Posts: 13248
Joined: Sun Nov 20, 2005 4:18 pm
Location: Serenity

Re: The Purpose of Splitting files

Postby velocity37 » Mon Jul 20, 2009 5:05 pm

Perhaps for movies, but for things like PC games, sizes can vary greatly. I archive a lot of stuff to blank DVDs because it's the cheapest storage you can get. If I had only a 2.4GB file and 2.5GB RAR file left, I'd be pretty frustrated that I'd have to either repack or split (then combine again to unpack later) to not waste half the space of the blank DVD.

Compression is also a bonus for files that aren't already compressed.
velocity37
 
Posts: 40
Joined: Sat Feb 21, 2009 12:51 am

Re: The Purpose of Splitting files

Postby SleepyOne » Mon Jul 20, 2009 5:26 pm

"The scene" doesn't release stuff for distribution on usenet and/or p2p-networks.

The reason for the initial archiving is to easily and quickly be able to replace broken parts and/or check that they are valid.
By keeping the original archives along the way, if one file is corrupt or damaged from one source, you can get that one small file from a different source and still have a complete set without redownloading the whole set.

Par can in many cases fix "broken" parts. Except when the original file got screwed up at the senders computer etc, in which case everything will look fine, par will claim it's fine, but its not. That's why original archives are pref. by the "oldtimers".

The reason movies are split into 700 MB cd-sized parts is so people can burn them to CD's and use them in standalone players.

Also.. see here for detailed info on release-sizes etc.
http://www.aboutthescene.com/thescene/scenerules.html
SleepyOne
 
Posts: 129
Joined: Fri May 12, 2006 1:22 pm

Re: The Purpose of Splitting files

Postby zbeast » Mon Jul 20, 2009 5:37 pm

http://www.slyck.com/forums/viewtopic.php?f=31&t=39260&start=25

I would use an archive just to keep a group of files together thats it.
since I archive on DLT or DVD's I don't make 600meg archives.
I hate having to keep track of part 1 of 4's and that crap.

Unarchived any password protected files and re-archive them in a fresh no passworded rar.
zips are out of there. store everything in rar's. unless the installer itself is a zip.

This type of thing pays off in the long run it's make's it easy to find and delete and find and up date
one's horde.
Even with 1tb hd's as cheap as $70.00 each.. no use in wasting disk space with lot's of split and zips sitting around.
Knowledge is power. Power corrupts. Study hard. Be evil.
zbeast
 
Posts: 6783
Joined: Thu Apr 01, 2004 5:20 pm
Location: Behind that itch. yes, that one right there.

Re: The Purpose of Splitting files

Postby SlyckTom » Mon Jul 20, 2009 6:10 pm

Thanks Sleepyone...


The reason for the initial archiving is to easily and quickly be able to replace broken parts and/or check that they are valid.
By keeping the original archives along the way, if one file is corrupt or damaged from one source, you can get that one small file from a different source and still have a complete set without redownloading the whole set.

Par can in many cases fix "broken" parts. Except when the original file got screwed up at the senders computer etc, in which case everything will look fine, par will claim it's fine, but its not. That's why original archives are pref. by the "oldtimers".

I think that's the best explanation I've seen so far. I think this ties into my other question about PAR2/SFV, and the answerI received:

I've seen instances where a par2 set was made with a bad rar and quickpar was unable to fix it because it thinks the bad rar was a-ok. In this case you can use the sfv to ferret out the bad rar and either re-download it or have winrar try and fix it.


But I'm wondering if the .sfv file would be able to snuff out the bad section of a 700 MB part, or will it interpret the entire file as bad?
Follow us on Twitter @SlyckDotCom
Join our Facebook Fan page
SlyckTom
 
Posts: 5713
Joined: Fri Jul 26, 2002 7:22 pm
Location: New York City

Re: The Purpose of Splitting files

Postby SleepyOne » Mon Jul 20, 2009 6:30 pm

That depends on who in the chain of events makes the SFV.

If it was the original releaser that made the sfv and it has followed the release all the way to "end user" then it will be able to "snuff out bad sections".

If someone repackaged it along the way. Meaning, unpacking the rar's and making a new sfv for the unpacked material, then we would be back to the problem where you gotta just trust that the middle man didn't screw anything up.
And it will treat the whole thing as one part.
SleepyOne
 
Posts: 129
Joined: Fri May 12, 2006 1:22 pm

Re: The Purpose of Splitting files

Postby qwoo » Mon Jul 20, 2009 8:24 pm

There are two different questions here I think:
1. why would a non-scene person pack and post original content in parts?
2. why would a non-scene person re-post scene content in original packed parts?

Sleepyone answered 2. An additional reason for keeping scene releases intact is that it can help the end user a bit to avoid malware. For example a user that can get some intact scene release (e.g. application.rar & application.r01 & application.r02) from both torrents and usenet can then easily compare file hashes. If the hashes match the user can be SOMEWHAT more confident that the files were not injected with malware at some stage after the scene.

Note that none of this answers 1 however. It could perhaps make sense also for the person in case 1 to stick to a standard just for convenience since people are used to it. But that's not very convincing considering the major inconvenience for everyone in having to pack/unpack. I guess that is why some torrent sites do their own original non-scene releases without rar-packing, for example the edu/docu stuff from mvgroup.org.

BTW, the malware risks for us p2p/usenet end users could probably be diminished a lot more if some independent sources (private torrent sites or predb sites or whatever) started cross-posting secure file hashes for all releases to some online database. It wouldn't take more space than the average nfo file. Then the p2p and usenet applications could autocompare all downloads against that hashdb and remove all mismatch downloads.
qwoo
 
Posts: 8
Joined: Wed Mar 11, 2009 1:10 pm

Re: The Purpose of Splitting files

Postby SleepyOne » Mon Jul 20, 2009 9:04 pm

More info on the topic of keeping original rar-files:

http://whyrar.omfg.se/index_eng.php
SleepyOne
 
Posts: 129
Joined: Fri May 12, 2006 1:22 pm

Re: The Purpose of Splitting files

Postby SlyckTom » Mon Jul 20, 2009 9:24 pm

qwoo wrote:There are two different questions here I think:
1. why would a non-scene person pack and post original content in parts?
2. why would a non-scene person re-post scene content in original packed parts?


Well, those are good questions, but they aren't really my question ;). I am looking for the technical reason why its necessary to archive a large file into multiple parts in the first place...Let's say I wanted to make a video, which is 700 MB, then distribute it via Usenet. What technical justification do I have to split it up into parts? Since the news server will accept anything as long as its encoded into text and small enough, I haven't really seen a good reason why other than tradition and sleepyone's explanation.

edit: thanks for the link sleepyone :D
Follow us on Twitter @SlyckDotCom
Join our Facebook Fan page
SlyckTom
 
Posts: 5713
Joined: Fri Jul 26, 2002 7:22 pm
Location: New York City

Re: The Purpose of Splitting files

Postby friedcheese » Mon Jul 20, 2009 10:20 pm

It is also good for video files to grab a couple of rar files to sample the quality, again without downlolading the entire 700mb video.

also, i believe it would be easier and faster to recover a missing or broken rar using par files than to have to repair the entire 700mb all at once.

basically it seems like there are some downsides (albeit minor ones) to using the entire 700mbfile, even with par files, as a single upload; but really no reason to stop splitting.

another thought, is when i see a video at say 824MB or 760MB or what not, I know right away it is out of the 'standards' and I wonder what else they might have done out of standard (codec, ac3 audio [many standalones do not support direct ac3 i think], square pixels, custom matrix, etc...). This has become a problem more and more with the sourcing of a lot of material coming from p2p and such.

it used to be everything would play just fine on my standalone, but over the last couple years it's really just a crapshoot. i remember how nice it was to finally get cheap dvd players that would play DivX/XviD files, but about the same time the out of standard stuff started coming in waves. it was like "Hey, we tweaked the codec so it looks better!" But then most peeps could not play it on or stream it to their TV's without reencoding.

sorry to go off on a tangent.
User avatar
friedcheese
 
Posts: 79
Joined: Sat Jul 07, 2007 1:13 pm

Re: The Purpose of Splitting files

Postby IneptVagrant » Tue Jul 21, 2009 6:23 am

The reason for multiple parts is FTP, which doesn't have a significant file verification process. I don't use FTP very often myself, but I've still had times when a file transfers incorrectly. And most often the fastest way to fix a bad FTP transfer is to just re-get the file.

**

I'm sure if you have execution privileges on the FTP server you could script a better way, but too often you wouldn't in the environment this discussion is about.
IneptVagrant
 
Posts: 1247
Joined: Tue Nov 15, 2005 5:07 am
Location: close the world . . . . . . . . . . . . . . txEn eht nepO

Re: The Purpose of Splitting files

Postby HouseCrowd » Tue Jul 21, 2009 7:30 am

From my own experience, uploading to NG or transferring files via FTP, I'd say that the splitting isn't done to make things easier for downloaders, rather for the uploader's convenience and sanity. There's nothing worse (ok, well maybe a few things worse) than uploading a large file, only to have the connection fail at 90-odd percent and to have to start over again.
There are 10 types of people in the World; those who understand binary, and those who do not.
User avatar
HouseCrowd
 
Posts: 33862
Joined: Mon Oct 13, 2003 4:18 am
Location: UK

Re: The Purpose of Splitting files

Postby qwoo » Tue Jul 21, 2009 12:30 pm

SlyckTom wrote:Well, those are good questions, but they aren't really my question ;). I am looking for the technical reason why its necessary to archive a large file into multiple parts in the first place...Let's say I wanted to make a video, which is 700 MB, then distribute it via Usenet. What technical justification do I have to split it up into parts? Since the news server will accept anything as long as its encoded into text and small enough, I haven't really seen a good reason why other than tradition and sleepyone's explanation.


That sounds pretty much like my question 1 then. Unless you're asking what technical reasons the scene folks have but then usenet is out of the picture because that's not where they're at.

HouseCrowds suggestions sounds plausible. In addition some p2p sites that do their own releases seem to increasingly be using multiple release routes: torrents, usenet AND online file hosts (e.g. rapidshare). Such file hosts often have file limits so anyone uploading there who doesn't want to pack the same release twice might upload the split release everywhere.

And finally an alternative conspiracy explanation: the hard drive industry made'em do it! :D They have an interest in the continued massive disk thrashing that the endless raring/unraring requires. So they simply pay, coerce or hypnotize usenet uploaders to rar-pack! :shock: :tinfoilhat: :lol: This also explains those double packed warez releases (rar archive inside a zip archive) -- the releasers got unmarked bills from BOTH hitachi and samsung.
qwoo
 
Posts: 8
Joined: Wed Mar 11, 2009 1:10 pm

Re: The Purpose of Splitting files

Postby ntscuser » Sat Oct 03, 2009 9:38 am

SlyckTom wrote:Well, those are good questions, but they aren't really my question ;). I am looking for the technical reason why its necessary to archive a large file into multiple parts in the first place...Let's say I wanted to make a video, which is 700 MB, then distribute it via Usenet. What technical justification do I have to split it up into parts? Since the news server will accept anything as long as its encoded into text and small enough, I haven't really seen a good reason why other than tradition and sleepyone's explanation.

1) Few connections allow a single thread to be downloaded at full speed. Depending on the time of day I might get between one half and one-eighth of maximum speed for a single thread.

2)As already mentioned it is easier to use PAR2 files if the main file is already split but not absolutely essential.

3) Some downloaders complain if they're not able to decode small rar first to test the quality.

As for FTP I've had no problems restarting a single large file from where it left off after a connection was interrupted. It was incredibly slow though.
User avatar
ntscuser
 
Posts: 1968
Joined: Mon Feb 20, 2006 9:15 am
Location: United Kingdom but originally from Holland


Return to NewsGroups

Who is online

Users browsing this forum: No registered users and 4 guests

© 2001-2008 Slyck.com