SLSK File Transfer Files >2GiB Limit

Submitted by sammy123 on Mon, 09/07/2015 - 13:21
Forums: 
Soulseek General Discussion

Hi everyone;
I'm currently evaluating the sporadic file transfer errors with files bigger than 2GiB. Did any of you (and with what client/version) have had a full and successful transfer of files "big" files?
I tried a huge variety of file DL/UL (as sender and receiver) and really non worked or even start when bigger than 2GiB. Some big (16GB) files did start, but brake up when reaching a filesize minus multiples of 2GiB. thanks to the Admiral I'd plenty of time to test it.
I already tried Linux (32 + 64bit) and Windows (x86) with SLSK native and Nicotine+ (based on Linux64) and all showed the same problems with Files bigger 2GiB.
I created a file 2GiB minus 1 Byte (which works) and 2GiB plus 1 Bytes (which did not work (UL nor DL via user). I created files up to 32GiB+/- 1 Byte, but 2GiB was already the max. I'd not conclude why some bigger files start, but brake up (waiting-aborting, waiting-aborting,...). The (uploading) client shows "file not found" (share is of course updated before). Under cli Linux I get a message "file size <0 Byte" when DL a big file.

I checked different clients (feb+June 2015 releases), different Linux architectures and nicotine versions. So all shows a protocol (stack overflow) problem, and according to the cli message big files show a negative (<0Bytes) value. The filesystem (NTFS, ext4, btrfs) is never the limit.

As (C) Programmer I'm very interested and why it's (still) there?

Cheers
Sammy

Submitted by sammy123 on Thu, 09/10/2015 - 00:24.

Hi
I have the tested files online for testing. The name is quite detailed to explain the size/function:
working: Test2GiBminus1.bin
not working/Starting: Test2GiBplus1.bin
not working/Starting: Test4GiBminus1.bin

Bigger files up to 32GiB are available and were tested in my tests with two users, but as the 2GiB+1 already failed I see no meaning in wasting (expensive SSD) space :-)
Does nobody have had good/bad experience with file sizes?
Cheers

Submitted by sammy123 on Fri, 09/18/2015 - 10:46.

Hello Nir
is the problem affirmative or is my testsetup wrong in some way, so only I and the testers experience the "limit".
Cheers

Submitted by sammy123 on Sat, 10/24/2015 - 06:07.

OK, so far all my analyzes showed the protocol (handling) as limiter of Files bigger 2GB (2 GiB to be exact). As I have a lot of SACD and hiQ Vinyl rips (384k/24Bit) I'm already limited with certain songs with this limit. As nicotine shows the same limit it's a serious limitation for me. Serious enough to think about using it further.
For all music related (HiQ and SACDs are quite rare) it's maybe a wanted policy which hones the SLSK network after all. Anyway I hoped to get it solved in a way.
With the SPAMs of the last month it take considerable amount of my time to keep track of the forum.
@Nir: Hope you get it solved (the forum problem) somehow! A forum is the information and troubleshooting center for me; maybe the IRC in SLSK is more used as the forum, but important problems like the all-time-star "HOW DO I WORK BEHIND A FIREWALL AND STILL BE ABLE TO SHARE" should be addressed properly before someone get's pissed of and stop using SLSK. In the end only huge aggregated and well organized sharers stays in SLSK.
Cheers
Sammy

Submitted by sammy123 on Wed, 12/16/2015 - 10:46.

Hi all
it was a good time helping people here in the forum; hope someone will address the bugs and failed configs in the future:-)
For me it's time to move on. SLSK was a nice experience as the culture and people behind it are impressive. As mentioned before the 2GB Limit is too grave for me to build/rely on, but for most it will be OK.
Cheers
Sammy

Submitted by Nir on Wed, 12/16/2015 - 21:21.

Thanks for all your help with the forum Sammy. Sorry I haven't looked into the 2GB issue before, I've been absolutely throttled the last few months at work. I'll be sure to make some time for it soon to see if at the 2GB limit can at least be lifted between SoulseekQt clients.

Submitted by sammy123 on Thu, 12/17/2015 - 12:28.

Hello Nir
the 2GiB Limit is quite uninteresting for most of the SLSK Users, so there are surely more important issues (OS X...) to fix than sth far in the future. As prof. troubleshooter (for C programms) I do have to set priorities, and so do you :-)
The bug would have been lost if not SLSK users like the Admiral, RuntoMe or Renthorp and all the others helped me to localize it. As no one replied it was more hunting windmills.
As soon as you fixed your sourcecode (git/svn/cvs?) and provided the binaries it will spread itself for sure!
Cheers

Submitted by Nir on Wed, 12/16/2015 - 22:31.

*sigh* It ended up being a really dumb bug. I'll update the builds soon, but unfortunately the problem is on both the downloader and uploader's end so unless both upgrade it won't resolve the issue. Sorry for all the trouble.

Submitted by Nir on Wed, 12/30/2015 - 19:59.

Here we go Sammy, this should work with bigger files if both the uploader and downloader are using it:

https://www.dropbox.com/s/j3pf8kk8fc9uoh1/SoulseekQt-2015-12-30.exe?dl=0

I'll post it as a nightly build soon, need to look into an extra bit of functionality I might want to add before then.

Thanks, Nir

bincue's picture

Submitted by bincue on Thu, 01/07/2016 - 17:57.

a friend (father) and I are exclusive GNU/Linux users and love this software for transferring files... but there is that limit on what we can transfer so we resort to visiting each other with our laptops ready and we transfer via lan (it is fast but not convenient). For the files with a size larger than 2GiB there is only that way or parting the files and transferring via soulseek clients. is there a build ready for x86_64 linux?

Submitted by Nir on Thu, 01/07/2016 - 18:01.

I'll see if I can't get the Linux builds ready over the weekend. Next weekend hopefully at the latest.

Submitted by sammy123 on Thu, 12/31/2015 - 01:23.

Splendid news Nir!
I'll test it when I'm back at home next week.
I wish everyone here a happy new year 2016!
Cheers and Hooray!

Submitted by sammy123 on Mon, 01/04/2016 - 12:41.

Hello Nir
the Admiral and me tested a big 16GB file incl. a MD5 hash test afterwards, incl stops/starts:
"f2ebf4991f6ca6f19b765e26b1915e33 => perfect integrity. With the start/stops we proofed even that the SLSK internal engine works stable with it. Thanks Admiral; we are a big step into the future (were no SLSK user has gone before ;-) ) If you should have problems with big files remember that both sides need this version; the MAC is not so far and nicotine+ is in search for a programmer to fix it."
Hindsight tested to DL a 4GiB+1Byte file from me (took a few days) and the MD5 proofed OK; so it sounds good to me.
Cheers

Submitted by Nir on Mon, 01/04/2016 - 12:43.

Great news! Thanks for testing this Sammy, I really appreciate it.