/t/ - Technology

Discussion of Technology

Index Catalog Archive Bottom Refresh
Mode: Reply

Max message length: 8000


Max file size: 32.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more


(used to delete files and postings)


Remember to follow the rules

The backup domain is located at 8chan.se. .cc is a third fallback. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 2.0.

Be aware of the Fallback Plan

8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.

(4.11 KB 300x100 simplebanner.png)
Hydrus Network General #2 Anonymous Board volunteer 04/20/2021 (Tue) 22:48:35 No. 3626
This is a thread for releases, bug reports, and other discussion for the hydrus network software. The hydrus network client is an application written for Anon and other internet-fluent media nerds who have large image/swf/webm collections. It browses with tags instead of folders, a little like a booru on your desktop. Advanced users can share tags and files anonymously through custom servers that any user may run. Everything is free, privacy is the first concern, and the source code is included with the release. Releases are available for Windows, Linux, and macOS. I am the hydrus developer. I am continually working on the software and try to put out a new release every Wednesday by 8pm EST. Past hydrus imageboard discussion, and these generals as they hit the post limit, are being archived at >>>/hydrus/ . If you would like to learn more, please check out the extensive help and getting started guide here: https://hydrusnetwork.github.io/hydrus/
>>4314 >>4315 >>4316 >>4317 Yeah I agree. It needs a hardcoded rule in the siblings system, but I'll have to write efficient mass-sibling logic first to get it to work in non-insane time, which will be a big step. Once that system is available though, we should also have 'rename all artist: tags to creator:' tech! >>4318 Yeah, do you mean like the translation text boxes some boorus support? I have it on my whiteboard to finally figure out 'note parsing' in the downloader system one of these 'medium size job' weeks (which I do once every four weeks). When we have that tech, I know some of the parser lads are going to try parsing that data in some standard json format or whatever, and then I can think and play around with actually displaying it. For 'collections' as I think you are talking about, please wait for later in the year, when I fully expect to flesh out the current stub 'file alternates' system. I want arbitrary file connections (e.g. 'this file is a costume change of this file', 'this file is the subsequent file in a photoshoot series to this file', and then proper thumbnail-bundling and media-viewer-browsing tech to employ that metadata. >>4323 Ah shit, I am sorry! It may have backed up a bunch of your old sessions to JSON in your 'db' directory, although I think in this special case since it was short disk space, that may even have failed. And your rsync would have wiped them, so not applicable. I am worried about your being reset to 'welcome to hydrus' like a first boot. That suggests your db files were actually wiped or truncated to zero bytes. I would obviously never do this intentionally, I have no code that deletes any db file. I think this must have been an unusual file truncation back to 0 bytes caused by the emergency situation of hot database files and no disk space. Well done for having the backup. I do the exact same as you, backup right before I update. Then you don't have to worry about anything. I have some 'check free disk space' tech in my old vacuum code. I will make a weekly job to see if I should do a pre-update check for certain amount of free space, and dump out of the attempt if it is too small. Sorry again for the trouble, and thank you for the report.
>>4326 >>4327 >>4328 Yeah, it means one of: - Someone went on an upload spree and spammed 150 pictures to that query in three days. - The site changed URL format, so the subscription thinks it is seeing a load of new stuff, when really it is the same content but with different URLs. The limit is to catch the latter situation, and the popup message is to let you check for the former. I'll update the text to be more human friendly. The remedy here atm is to run that query again in a downloader page to catch the gap you missed. I'd like to have a button that sets this up for you automatically, that would be most human friendly I think.
>>4335 It is a complicated subject. Briefly: Changing file content is a headache for various hash-based technical reasons that matter whenever you grab or communicate about files with other computers (e.g. boorus, the PTR). Compressing or converting files tends to not save all that much space overall. Normally 5-10%, and less for video, and can produce multiple conversations on what is the 'best' way to do it. (JPEGXL may be a silver bullet here, with ~50% lossless savings) I don't want to hardcode this stuff. So, what I plan to do is write an 'external executables' system for hydrus where you can set a variety of different pipeline algebras, for instance: waifu2x upscale | exe path | takes an image file using these launch parameters | produces a converted duplicate image file in this temp location ffmpeg to webm high quality | exe path | takes a video file using these launch parameters | produces a converted duplicate video in this temp location youtube-dl | exe path | takes a url using these launch parameters | produces a video for that URL in this temp location etc... Then, with that input/output algebra, I can plug those pipes into hydrus (e.g. right-click on image->convert using 'waifu2x upscale') and let users define and share all sorts of conversion and file/URL generation scenarios. Youtube-dl and Gallery-dl would obviously be great to plug in to the downloader system. Automated conversion to JPEGXL or whatever else you want would be a schedulable mass job I could pack into the existing file maintenance system, while preserving appropriate metadata across the dupes. This is a little way off, but that's my plan.
>>4352 >>4354 Thank you for the updates, I am sorry it has given you trouble. I agree that this technical dialog and the ugly help need a complete overhaul. I hope to make this all more user friendly in future. I think I should add some more red warning text to the top of this dialog in the meantime. Since you are several steps in, you can ignore my post I made earlier at >>4365 In your image in >>4354, I assume you want those media files to stay in "F:\Documents\Hydrus Network\Hydrus Network\db\client_files"? Is that the full path of that second location? It looks like hydrus has a memory (likely confused from an automatic healing routine) of them being in the "F:\Documents\Hydrus Network" location, and it thinks it should be moving them over there, so let's fix it: Click the good location with 0 weight. Click 'increase location weight' once Click the bad location with 'nothing' current usage Click 'remove location' The dialog should now have the one location, still 'portable' since it is inside your db directory, with current usage at 100% and ideal usage at 100%. Let me know if you still have any trouble.
Does that "related tags" suggestion feature in the "manage tags" area consider what tags are NOT often seen with each other? I see the related section suggestion a lot of tags that don't really make much sense together, and sometimes suggest tags that are mutually exclusive, like "solo" and a post that's already tagged "duo" for example. It would be much more useful if the statistical algorithm used here would consider tags NOT being seen often together in its suggestions as well.
(131.01 KB 1280x720 yay.jpg)
>>4366 >I want arbitrary file connections (e.g. 'this file is a costume change of this file', 'this file is the subsequent file in a photoshoot series to this file', and then proper thumbnail-bundling and media-viewer-browsing tech to employ that metadata.
Have there been any progress with ICC color profile management? I was searching for this issue, and found some posts from 2019, but after that I see nothing.
It would be cool if there was an option to have automatic search result fetching as you type, but disabling autocompletion, and instead having that be bound to a shortcut. For me, fetching exact results (ideal siblings, for example) is quick, but autocompletion is slow. I wish the was an option to enable the former, but not the latter, and have that be activated by a shortcut, like how you can do that fetching results using ctrl+space right now. This is probably easy to implement, and it would make adding tags to a bunch of files one after the other a lot easier for me, since I wouldn't have to deal with Hydrus constantly freezing for 30 seconds, but I would still automatic adding of an ideal sibling upon entering an un-ideal sibling, from the "exact" result fetching, and the shortcut would mean that I could still have actual autocompletion when I need it.
>>4172 >There is probably some proxy daemon you can run on a system that'll pipe certain requests to the TOR service that'd solve this better, given how jury rigged other solutions sound. There is already a Tor daemon that allows a SOCKS5 proxy, and I'm fairly certain you can route it through the Tor Browser's connection as well. I think the main problem is that when Hydrus makes a request, it looks a lot different from what the Tor Browser does to the server. I may be wrong, but in my understanding Hydrus just scrapes the HTML/API of the site, grabs the tags and image URL, and then downloads the image. Meanwhile, the Tor Browser would render the site in just about the same way for anyone using it, making it much more anonymous. If you use a Tor proxy in Hydrus, only your true IP is hidden from the server and your ISP cannot tell what server you connected to. They can still likely guess you're using Hydrus from HTTP headers or user agent and whatnot, along with selectively not downloading things that regular browsers do. This is better than nothing in my opinion but not nearly as anonymous as using the Tor Browser. Setting up some browser add-on to download through the browser and pipe to Hydrus would only really work for whatever image was on the page at the time. I'm not sure how subscriptions or download pages would work when called from within Hydrus without causing Hydrus dev a lot of work.
Reminder to check your backups. I had to restore from backup, and after a day of fucking around with various backups with corrupted databases the most recent working one I had was a month old.
>>4398 how do we check our backups to ensure they're okay? without restoring from it I mean. Also if there's not already, maybe it would be a good idea to implement a feature that compares the current db with the backup db to ensure it isn't corrupted and fully functional.
>>4401 It depends on how you back up. I've been making encrypted archives once a month instead of the built-in "copy everything to X folder" button, and the one I made on June 1st was corrupted. A separate daily automatic backup for some reason ignored my Hydrus folder most of the time and corrupted the databases when it didn't. I'll probably change to making a backup of the encrypted version of my Hydrus directory instead of an encrypted archive of the unencrypted data.
I had a great week working on small quality of life issues. A couple of bugs are fixed, some UI lag is reduced, and I worked on some layout too. Just a mix of cleanup before my vacation next week. I have some unavoidable IRL tomorrow, so the release may be a bit later than usual.
>>4398 >It depends on how you back up built in backup manager >>4406 >I had a great week working on small quality of life issues thanks devanon. >I have some unavoidable IRL tomorrow, so the release may be a bit later than usual. man live your life, no need to be online 24/7, unplug for a day or two. It's good for your health and feels good. we appreciate your work. just preferably let us know before you ever plan on abandoning the project :p thanks for your work!
(1.27 MB 3571x2678 aee7tg48.png)
>>4410 >just preferably let us know before you ever plan on abandoning the project :p Don't give him ideas.
(28.84 KB 658x901 photo_2021-06-09_14-33-52.jpg)
Is there a downloader for Endchan? I don't see it. So I can add an endchan catalog to my subscriptions.
>>4406 Could you consider adding a proper log/info area to see all errors from all the various things that are happening in hydrus? Like ti see subscription success and errors, watcher success and errors, any db issues, etc. Also does the backup database do any verification or anything? And can we get an option to run a manual verification on the backup to ensure it's proper even if it does? Relevant to >>4401 Thanks
>>4414 >Could you consider adding a proper log/info area to see all errors from all the various things that are happening in hydrus? Have you looked at db/client - <year>-<month>.log ? Because I think that's exactly what you're describing. Which reminds me – devanon, could you consider adding 0 padding to the month (stored in current_month, set with a tm_mon) for the log's filenames, please? My OCD gets sad when it sees 12 before 2. Thank you!
>>4416 I meant an internal viewer, in which the user could also easily filter/sort by type of error (for example, backup errors, PTR errors, etc.) Or sort/filter then export only those filtered lines to a temporary file to view it in whatever editor the user pleases.
>>4406 I'm that anon that was having the database migration troubles (E drive to F drive). All seemed to be working good, however currently I am trying to update to a newer version of Hydrus on Windows. >Update Issue/My Process Now, I used the installer as I have always done in the past, and I clicked install, and I clicked through and then it finished. I opened the client, and it was a fresh install of Hydrus. I went back into the installer, and think it tried to install it under the E drive and I didn't notice. So, I tried clicking browse in an attempt to change the install directory to the already installed one on F drive. I navigated to the install folder on the F drive, clicked it, and it prompted me with a message stating that "The folder: F:\Docments\Hydrus Network\Hydrus Network" already exists. Would you like to install to that folder anyways?; Yes; No". I selected no for now, and just canceled the install on that dir.. because I am unsure if this is going to overwrite that directory, or it's just stating it will update that directory (I would suggest clarifying this in the installer btw). >New Folder Also upon doing that first install, it created a Hydrus Folder in the E drive, I'm assuming this is safe to delete. It also changed my system shortcut (windows search) for hydrus to that E drive instead of the one on my F drive. If I manually go into my F drive at F:\Documents\Hydrus Network\Hydrus Network\client.exe and launch it, my original database is there and looks fine. So it seems like it created a new client and database on that E drive where Hydrus used to reside before I moved it to that F drive. >So I'm not sure what to do..? I just want to update my Hydrus on the F drive, and in the end the ONLY Hydrus install (and ALL the folders for Hydrus) that should reside on this system in that "F:\Documents\Hydrus Network\" folder.
A cool idea: fetch media from jellyfin and nextcloud and just parse the tags (don't download the files) So users can import their porn libraries but without having double of all their files (it basically streams right from Jellyfin through Hydrus). Hydrus would just store the url, and any file information such as the hash of the file and what not, then the user could apply any tags to it. It just wouldn't download the file.
https://www.youtube.com/watch?v=NgYIIPszZjA windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v443/Hydrus.Network.443.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v443/Hydrus.Network.443.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v443-macos/Hydrus.Network.443-macos.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v443/Hydrus.Network.443.-.Linux.-.Executable.tar.gz I had a great week doing nice cleanup and quality of life work. Hey, we had a problem getting the macOS release to build this week. The macOS link above goes to a build using a simpler and faster method. It should work fine, but please let me know if you have any trouble. As always, back up before you update! highlights Popup messages can now launch complex jobs from a button. The first I've added is when a subscription hits its 'periodic' file limit. The situation itself is now better explained, and a button on the popup will create a new downloader page with the specific query set up with an appropriate file limit to fill in the gap. The second is if you try to upload some content to a repository that your account does not have permission for (this is affecting sibling- and parent-uploading PTR users as the shared public account is changing), the popup message that talks about the issue now has a button that takes you straight to the manage services panel for the service and starts automatic account creation. Subs should now be more careful about determining when they have 'caught up' to a previous sync. Small initial file limits are respected more, and the 'caught up' check is now more precise with sites that can give multiple files per URL or very large gallery pages. I gave options->speed and memory a full pass. The layout is less crushed and has more explanation, the options all apply without needing a client restart, and the new, previously hardcoded cache/prefetch thresholds are now exposed and explained. There's a neat thing that gives an example resolution of what will be cached or prefetched, like 'about a 7,245x4,075 image', that changes as you fiddle with the controls. The client has recently had worse UI lag. After working with some users, the biggest problems seemed to come in a session with lots of downloaders. I traced the cause of the lag and believe I have eliminated it. If you have had lag recently, a second or two every now and then, please let me know how things are now. If you use the Client API a lot while the client is minimised, you can now have it explicitly prohibit 'idle mode' while it is working under options->maintenance and processing. full list - quality of life: - when subscriptions hit their 'periodic file limit', which has always been an overly technical term, the popup message now explains the situation in better language. it also now provides a button to automatically fill in the gap via a new gallery downloader page called 'subscription gap downloaders' that gets the query with a file limit five times the size of the sub's periodic download limit - I rewrote the logic behind the 'small initial sync, larger periodic sync' detection in subscription sync, improving url counting and reliability through the third, fourth, fifth etc... sync, and then generalised the test to also work without fixed file limits and for large-gallery sites like pixiv, and any site that has URLs that often produce multiple files per URL. essentially, subs now have a nice test for appropriate times to stop url-adding part way through a page (typically, a sub will otherwise always add everything up to the end of a page, in order to catch late-tagged files that have appeared out of order, but if this is done too eagerly, some types of subs perform inefficiently) - this matters for PTR accounts: if your repository account does not have permissions to upload something you have pending, the popup message talking about this now hangs around for longer (120 seconds), explains the issue better, and has a button that will take you directly to the _manage services_ panel for the service and will hit up 'check for auto-account creation' - in _manage services_, whenever you change the credentials (host, port, or access key) on a restricted service, that service now resets its account to unknown and flags for a swift account re-fetch. this should solve some annoying 'sorry, please hit refresh account in _review services_ to fix that manually' problems - a new option in maintenance and processing allows you to disable idle mode if the client api has had a request in the past x minutes. it defaults disabled - an important improvement to the main JobScheduler object, which farms out a variety of small fast jobs, now massively reduces Add-Job latency when the queue is very busy. when you have a bunch of downloaders working in the background, the UI should have much less lag now - the _options->speed and memory_ page has a full pass. the thumbnail, image, and image tile caches now have their own sections, there is some more help text, and the new but previously hardcoded 10%/25% cache and prefetch limits are now settable and have dynamic guidance text that says 'about a 7,245x4,075 image' as image cache options change - all the cache options on this page now apply instantly on dialog ok. no more client restart required!
- other stuff, mostly specific niche work: - last week's v441->442 update now has a pre-run check for free disk space. users with large sessions may need 10GB or more of free space to do the conversion, and this was not being checked. I will now try to integrate similar checks into all future large updates - fixed last week's yandere post parser link update--the post url class should move from legacy moebooru to the new yandere parser correctly - the big maintenance tasks of duplicate file potentials search and repository processing will now take longer breaks if the database is busy or their work is otherwise taking a long time. if the client is cluttered with work, they shouldn't accidentally lag out other areas of the program so much - label update on ipfs service management panel: the server now reports 'nocopy is available' rather than 'nocopy is enabled' - label update on shortcut: 'open a new page: search page' is now '...: choose a page' - fixed the little info message dialog when clicking on the page weight label menu item on the 'pages' menu - 'database is complicated' menu label is updated to 'database is stored in multiple locations' - _options->gui pages->controls_ now has a little explanatory text about autocomplete dropdowns and some tooltips - migrate database dialog has some red warning text up top and a small layout and label text pass. the 'portable?' is now 'beneath db?' - the repositery hash_id and tag_id normalisation routines have two improvements: the error now shows specific service_ids that failed to lookup, and the mass-service_hash_id lookup now handles the situation where a hash_id is mapped by more than one service_id - repository definition reprocessing now corrects bad service_id rows, which will better heal clients that previously processed bad data - the client api and server in general should be better about giving 404s on certain sorts of missing files (it could dump out with 500 in some cases before) - it isn't perfect by any means, but the autocomplete dropdown should be a _little_ better about hiding itself in float mode if the parent text input box is scrolled off screen - reduced some lag in image neighbour precache when the client is very busy - . - boring code cleanup: - removed old job status 'begin' handling, as it was never really used. jobs now start at creation - job titles, tracebacks, and network jobs are now get/set in a nicer way - jobs can now store arbitrary labelled callable commands, which in a popup message becomes a labelled button - added some user callable button tests to the 'make some popups' debug job - file import queues now have the ability to discern 'master' Post URLs from those that were created in multi-file parsing - wrote the behind the scenes guts to create a new downloader page programmatically and start a subscription 'gap' query download - cleaned up how different timestamps are tracked in the main controller next week I am now on vacation for a week. I'm going to play vidya, shitpost the limited E3, listen to some long music, and sort out some IRL stuff. v444 should therefore be on the 23rd. I'll do some more cleanup work and push on multiple local file services. Thank you for your support!
>>4421 >Thank you for your support! Best wishes OP and beware of the fucking normies.
(57.00 KB 174x1002 ClipboardImage.png)
>about 200 pages open >set all subscriptions to run in the background (which creates one page per subscription) >forget about it >once it gets over the page threshold it starts creating page warning for every page it tries to create over the threshold >creates too many warnings and client crashes Turns out python can't into too many windows.
I would love if the "pages" drop down menu actually listed all the open pages, with submenus of "pages of pages", so you could click one to have the main view switch to that page. It could be put under the top "X pages open" menu option, for example. If you have tons of pages open it's real slow and cumbersome to try to find a specific one by scrolling with the little arrows to the right of the tabs. In fact, that is probably the slowest and most user-unfriendly thing I do in Hydrus. I can't imagine this would be a ton of effort to add, and it would help so much with usability.
>>4423 solution: migrate off python ez
rule34.xxx now has an anti-ddos thing where it "checks your browser" before letting you visit the website. Hydrus fails to download from there now, calling it an "unsolvable cloudflare issue". I tried copying rule34.xxx cookies to hydrus, but it still didn't work. Is it truly unsolvable? If so, is it perhaps because I don't have a rule34.xxx account? Does anyone with an account know?
I had a twitter subscription that missed a number of files from certain days. Other files from tweets on those same days from other subscriptions were downloaded fine. Now I'm paranoid I missed a bunch of tweets from other accounts as well, or that I will miss a lot more tweets in the future. Is there an easy way to check this? Like, attempt to download all files for all subscriptions in the last 10 days? Another weird thing is that the twitter gallery dl for that account only goes back a few days, while others go back many more days.
(37.67 KB 854x694 Untitled.png)
The "# messages [dismiss all]" notification at the bottom right of my hydrus stays on top no matter what I do, unless I minimize hydrus entirely. It wasn't always like this, but it happened at some point today. The only thing I can think of my having done to trigger this was locking my computer (windows 7) as I went afk to take a piss while hydrus was importing. I'm not sure if that was actually triggered it, but it is the only idea I have. I imported many things since booting the client on this version (442). Pic related.
>>4435 > Like, attempt to download all files for all subscriptions in the last 10 days? This is what the gallery downloader does.
(218.09 KB 349x611 20210612.png)
It looks like "Review your fate" is getting incorrect file number in the inbox. Or maybe now it adds the files from the trash to it.
>>4440 Yes, but my problem is that it doesn't. I've checked it to show new files and files already in inbox and files already in archive but it doesn't. >>4442 I've noticed it doesn't always update immediately.
>>4435 I couldn't even get Twitter downloader to download anything, I had to go through the Nitter downloader.

Quick Reply

no cookies?