Addressing memory use

For the last couple weeks I've been taking a closer look at the way SoulseekQt uses up memory. If you're a casual user, sharing no more than a couple of tens of thousands of files or less, and you don't go around leaving a whole lot of search and share browse windows open, chances are you haven't had too many problems with SoulseekQt's memory use. If you're one of the more heavy users out there, well, you should still be fine as long as your rig gives you enough memory to play around with. Whatever your situation may be, there's no denying that placed alongside Soulseek NS, SoulseekQt takes more memory to do pretty much anything. The reason for this is the data system I've been designing over the last few years and finally put into actual use with SoulseekQt, which has been a blessing and a curse. Certainly more blessing than curse, but the tradeoffs are there. It's a type of database that's extremely flexible and lets me implement relatively complex features reliably and painlessly, but that flexibility hasn't come cheap. Browsing another user's very large share that would take about 40 megabytes to load into memory under Soulseek NS could easily balloon up to 100-120 megabytes under SoulseekQt. And the situation is similar when you're sharing a very large number of files yourself. Without getting too technical, I've been going into the data system's internals and looking for any places where serious savings could be had. I tried a lot of different things, most of them made no difference, or made things worse to various degrees. The most successful experiments have been in the area of diminishing the system's reliance on complex data containers by replacing them with wrappers that only use them when they're absolutely needed. In the end, I arrived at something like a %15-20 percent savings in memory for your own share, and a more impressive %30 percent for browsed shares. Neither takes SoulseekQt out of memory-hungry territory, but it's certainly an improvement.

If you try the new build out, just remember that a lot has changed internally. We'd love to hear from you in the comments whether it works well for you or not! We want to make a big global push to advertise SoulseekQt to our users, but we'd like to know that we have a relatively stable build at the top of the download page before we do that.

Additionally:

  • Fixed a crash when downloading a folder containing other folders from a browsed share.
  • Multiple download entries can now be selected.
  • 'Stop Search' button for search results.
  • Searches only containing tokens that are one or two characters short are not longer processed by the client.
  • The client will look for a new parent in the search distribution network if no searches are received from the current parent for over a minute.
  • Dump Data option under Diagnostics tab. I can use this to diagnose very specific problems a user is having with their client if they dump their own data and send me the file.

Links on the download page!

Thanks, Nir

Comments

psynaturecybine's picture

This is getting better and better! Good work!
memory usage isn't an issue for me, however cpu usage and way it operates on hard drive is. with ~36.000 files it takes forever to start, making computer almost unresponsive. i can't even imagine how would it behave for people who shares 100.000+. UI could be better, for now it's still confusing and completely unconfigurable.

Thank you. Regarding the long start up time, I guess it depends on hard drive speed? I've tested with a share of about 75,000 files and it takes about 15-20 seconds to load. Might I ask what are your computer specs? I've been looking into re-scanning shared folders in the background when the client starts. Hopefully I can get something done by the next build. Which aspects of the UI would you like to be able to reconfigure?

edit: Try this one: SoulseekQt-9-16-2012.exe. I've moved the folder scanning code to a secondary thread, although you might still get a shorter pause a little bit after when the client's actually processing the results of the scan. Let me know how it works for you! If it works well I'll roll the changes into the next build.

Thanks, Nir

psynaturecybine's picture

xp sp3, atom 1600mhz, 1GB of RAM and internal sata interface hdd.

indexing takes ages! after 2hours and 10 minutes (to index 9,5k files (about 70GB)) my patience has ended and i turned it off. it didn't finished. isn't there a way to speed up this process? i assume it takes so long because it collects information about bitrate, size and duration? program was responding very slowly during this time for 10 minutes and then it actually stopped responding at all(for 2h). i will compare startup time between 9-15 and 9-16 once i index all my share again.

okay, so here is what i would do with the interface:
-remove empty tabs (like chat, browse) if not used
-allow user to close unused\any tab and eventually restore them from some place.
-add check-box (checked by default) in diagnostics background searches so i can turn it off and not see what words are used to search my share(is there any practical reason to see this?)

bug: "away" icon on user list will not appear. all are available or off-line. same goes for chat - icon on tab there. green text informs that "* example_user is away." but icon says different. not a big deal but it is confusing (i chat a lot)

bug: greek letters are not displayed is search results

That's a netbook CPU isn't it? If you can share that many files on the netbook with Soulseek NS I say stick with it. I tried SoulseekQt on a netbook once and I agree it is not an ideal experience.

As for these feature suggestions, they're interesting I'll admit, and I can see how you might want as little information as possible on a netbook screen at a time, but I don't know how I would prioritize them. You can always submit them individually as feature requests and have users vote on them.

The information in the diagnostics tab are mainly for my benefit, to see that things are working as they should. Not seeing anything in the search results tab can signify a lot of different problems, like your share not being indexed properly, or a disconnect from the search distribution network. It actually just helped me a few days ago to diagnose the latter problem. I left it for users to see in case they got curious.

As for greek letters, SoulseekQt uses Unicode to communicate different languages, so you won't see them coming from users of Soulseek NS by default. You can still set the right (greek) codepage under Options->General though to get the appropriate translation.

Sorry I can't be of any more help!

psynaturecybine's picture

thanks for reply
yes it's a netbook, but it's working pretty well and disk transfer is not slower than on normal notebook, and i believe thats most important when it's indexing files (since cpu is not used fully, but hd is). don't want to make a fuss here, so if there is no other reports of slow indexing i won't mention it again.

i have 1440x900 screen connected to this computer so it's not that i have not enough space, it's just that i never use "rooms", and "chat" and "browse" is empty until i (for example) double click user on completely different tab. it has no functionality until then. it's just confusing because at first i thought that i have to click something on chat tab to start conversation. perhaps at least let us change order of tabs?

about search in diagnostics - i can see now how that can be useful, but i still think small check box to turn if off would be good, sice regular user don't need this to be working in background all the time. (also there is invisible button next to "dumb data" on that tab)

sorry! i forgot other users must be on QT too!

psynaturecybine's picture

ok so it finished indexing. it took about 4h. nicotine+ is doing same job in 20min. anyway, i tried 9-16 and i've got to say that start up is super fast now, but does it really looking for changes in share? if yes, then it's perfect. thanks!
ps
do You intend to keep soulseekqt clean and simple? i'd suggest to take a closer look at some of nicotine+ options. there are really usefull things in there i'd love to see in new soulseek client. i will make a feature request at some point.

Yes, it does scan the folders for changes in a secondary thread, so the scan is effectively running side by side with the client. Once the scan is over the client does process the results in the main thread, which would probably manifest as a pause of some seconds when that happens. You can verify that it's scanning for changes by adding to or removing files from your shared folders and restarting the client. Just give it enough time to actually complete the scan before you check. You should see a new Profiling tab under Diagnostics->Logs when that happens measuring the time that processing the results is taking.

As for Nicotine, this comes up every once in a while. My general stance on Nicotine is that it's a fine client for someone who's more technically minded, and I wholeheartedly recommend that people try it out if they're dissatisfied with Soulseek NS or SoulseekQt. That said I'm not interested in replicating Nicotine's feature set. You're welcome to suggest individual features in the Feature Requests section, and I'll certainly consider them if they're useful enough and/or there is enough interest in them from other users, but if you need the more advanced options Nicotine offers, chances are SoulseekQt won't become a suitable alternative anytime soon.

many THX++!

Holy crap, this 9/16 update has finally made SoulseeQT usable for me! I have a little over 250,000 files, and it was practically impossible for me to share my entire library. The program would hang and hang and hang and hang and I'd pretty much just have to leave it the hell alone whenever I started it up for quite some time until it would be responsive again.

Now with this 9/16 build everything is loading up quite quickly. Please, PLEASE add this into the next build as a mainstay feature. I will be forever grateful.

Absolutely :) I'll be releasing this with a few other updates very soon. Thank you for your feedback!

TiraelSedai's picture

Hi there. Sorry for 1001'th question about performance :)
I am sharing ~215 gb of music, and mostly mp3's, so nearly 30k files.
40mb -> 130mb memory usage is sad, but not fatal.
But can you tell me, why SoulseekQT always eats about 8-11% of my CPU? It's just terrible, i am forced to switch back to NS version.
Using SoulseekQt build 11/7/2012. CPU i5-3570K, not overclocked.

Also sorry for my awful english.

I'm actually not sure. The prime suspect would be processing search requests, but I tried sharing something like 120k files and I'm still barely getting a reading in Task Manager... So I can't really say what might be causing it. That said, SoulseekQt using up less than half a core on a Quad-core CPU isn't too bad if you're sharing a lot of files. If there's a serious performance issue somewhere I hope I'll run into it eventually so I can fix it.