- This topic has 24 replies, 13 voices, and was last updated 4 years, 1 month ago by
.
-
Posts
-
Rank:
As storage is cheap (16TB HDD is about $250), and I have 1gb Internet access, downloading 'Mindlessly' isn't much of a problem, the organizational aspects are.
I'll admit, I often join bounty threads because, while I might not 'need' the particular asset, I feel that points fuel the system and this site, so if I can help a brother/sister by adding points to a bounty, it will make it better in the long run for everybody.I'm also working (at various stages) on 3 different VN's... All with different themes Sci-Fi, Fantasy, and Modern.
If I see something interesting, I grab it, and download it to a folder for later installation when I think I will need it... Basically, I've mimic'd my storage folder to be what my manual install folders are in my DAZ Library.
So I have E:DAZDOWNLOADS\CHARACTERS\GENESIS81\TEEN\SASE-CASEY-GF81.zip
Having a bit of OCD is a double edge sword... I need to spend more time creating, and less time Organizing, but sadly, they are not mutually exclusive.
@DazHakz that screenshot of your app looks amazing. Is it something you will release eventually? Need some beta testers? LOL
Thank you to all of you who took the time to answer my question. And sorry about the time it took me to answer, I rewrote this message countless times 🙂
To summarize we have (you are all named as s/he and her/his because I wanted to offense no one and, of course, the way I describe your doings is friendly and funny and not a all a criticism)
- @gts6 who uses an indexing tool to search her/his files but does not exclude building a tool with online parsing capabilities to query about assets s/he owns
- @Ethiopia who downloads faster than s/he can sort things which messed-up (for now) her/his filing which is organized by themes. S/he also mentioned a "work in progress" on something that I'm far from ready to forget 🙂
- @eelgoo who is a control freak and names her/his files and folders thoroughly, just like her/his library which s/he has extensively described in another post (thanks again)
- @Pooper who stuffs everything in her/his library and does not save a single archive but backups her/his library often and on 2 distinct mediums
- @Yngvarr who chooses what s/he downloads and uses online shops favorite feature to mark what s/he owns
- @Abad the philosopher who cites Diogenes and want us to believe that s/he has under a Terabyte of assets
- @VIAI who kinda adopted @Pooper's method of not archiving but on a less gigantic scale as I understand it, also splitting her/his library by generation
- @Frank22 who is also cautious about what s/he downloads and uses file naming to find her/his assets easily
- @Techsnoop who, I cite, is very lazy and stores everything in large drives and who started an Excel archive long ago but hasn't updated it in a long time
- @littlepleasures who only downloads assets as a test and buy them if they suit her/his needs. As such, and not being a collector s/he doesn't keep archives
- @hioushi who had a gluttony phase when s/he discovered DAZ Studio but who now knows better. S/he also built her/his own tool in an attempt to keep track of her/his assets with a visual interface displaying images (an application I'm am not ready to forget about either, like in @Ethiopia's case)
- @mariodaz who uses Microsoft Access to keep track of her/his DAZ assets and stores everything on large drives and online storage (I'd be curious to sneak in that database of yours to see how you built it)
- @Redflag who still is in her/his gluttony phase (like I am, no worries) and admits spending too much time organizing stuffs and not enough creating
- @randykoala who doesn't say much about her/his practices but judging by the interests s/he has in @hioushi's tool, has the same problem than most of us
My my, what a team we are 🙂
I, on my side, have almost 8 Terabytes of assets. A good half of the Synology home server is dedicated to that and running on RED 1 disk (mirroring setup, meaning 2 physically distinct drives have the same content at all time and you can loose one without trouble, provided you replace it fast) because I HATE LOOSING THINGS 🙂
I am only talking about what is identified because what isn't is stored elsewhere, awaiting a future identification (if it ever comes).As I already said on previous posts, I am more a collector than an artist. I know how to use DAZ and occasionally do use it, but nothing makes me happier than adding a new item to my collection (in my online life I mean). My pleasure is also in exhuming some assets that have not yet been made available here and sharing them with you. For the points ? Not really, I hardly download something unless it's inspiring. Please don't scrutinize my history to find out what I consider inspiring, that would be very embarrassing... 🙂
Back to the point....
My archived files and folders are organized by themes, pretty much like in the DAZ library.
In each theme folders are sub-folders named using always the same convention : 3 letters for a shortened vendor name, a dash followed without space by the SKU number, then a space, a dash, a space, and the full asset name at the end.
Each folder contains the files named without spaces : a least 1 image properly named and the archive(s). This looks something like this :WARDROBE\GUZ-1078 - Yennefer Outfit For G3F\GUZ-1078-YenneferOutfitForG3F-01.jpg
WARDROBE\GUZ-1078 - Yennefer Outfit For G3F\GUZ-1078-YenneferOutfitForG3F-02.jpg
WARDROBE\GUZ-1078 - Yennefer Outfit For G3F\GUZ-1078-YenneferOutfitForG3F-03.jpg
WARDROBE\GUZ-1078 - Yennefer Outfit For G3F\GUZ-1078-YenneferOutfitForG3F.zipI was tempted to had a layer of folders named by vendors, but even if the Synology system does not suffer from those limitations, I'm not too keen on long paths... bad souvenirs form passed eras in computer tech.
All this is very well, but that does not answer the question : How do I keep track of my current assets and future downloads ?
For years, I've been keeping a thorough list of my "belongings" making it easy to search and decide whether or not I wanted/needed to download an item that appeared online. This was a small home made application with a few forms to input asset details, artist names, vendors and compatibility details. All this was stored in an Sqlite database, so it was portable.
But I soon found cumbersome to type in manually every new addition to my collection. So I started scarping major stores to keep an updated list of their available articles (For those who don't know what it is, scraping is the action of getting data from web pages. Stores have a defined structures for their product pages and once identified, it's easy to parse and collect that data). That way, instead of typing manually what I had gathered, it was merely a case of searching the database and tagging the corresponding record as "owned". I could also tag them as "wanted" when I stumbled across an asset that I wanted to keep track of until made available, or "pending" meaning it was in download queue or already downloaded but awaiting sorting and filing. The only constraint was to regularly launch the scraping procedures to keep up to date with new additions to the stores. (that's also where I created the most bugs and I spare you the long hours searching for the function that was mixing Renderosity artists and linking them with DAZ assets...).
Side note : That is why, as I already said a few times on the forum, RDNA torrents (and assimilated) are a nightmare for me because... I have absolutely no reference in database and am unable to name and archive them properly...Anyway...
Just like @hioushi, this application is in perpetual beta state. This project started as a challenge to cover a personal need, and has now become a monster. And just like Doctor Frankenstein contemplating his creature I sometimes look at my application and wonder how in hell it ended-up that big and ugly... I started with a cheap and limited RAD development tool (that I paid for every year) and am now using a professional development environment (that we use at work), adding new features as I need them and tweaking existing ones in the same idea.
I released it some years ago (KAT was still running at that time) on an invite only forum but, despite the goodwill of a few people (some are registered on that very forum), what once was a hobby and a personal challenge soon became a full time job. As I said, the application has become such a monster that it's just impossible to even consider writing a user's manual. As for the code, I doubt anyone younger than 40 has ever heard about the language I use (despite it being very well documented and still widely used today : it's called Delphi and is an object oriented iteration of the Pascal language) and sharing the sources would do no good. Furthermore, after my first experience, I am not ready to retry such an adventure yet, partly because of the time needed, partly because... I turned kinda shy to be honest.
So why write this long post to conclude that I don't want to share my creation with others would you ask ? Well, maybe I could start by sharing the database. That could be useful to someone who wants to build his own application and/or feed his already existing one . As for the application itself, provided some people would be interested, I did not say never, I said...yet. Maybe I could release a "light" version. Not light in the sens of free by opposition to paid for extra features (all my applications have always been free), but light in the sens of containing only the main features and not all the "garbage" I added along the years.
As of now, the database contains 181185 references of assets (these are not my belongings, these are identified assets), mainly DAZ, Renderosity and Renderotica. This could be exported without too much trouble, be it in database format, or Excel or maybe JSON (not too sure for that last one).
Here is a screenshot of figures regarding it's actual content (alongside my "belongings") :
All assets have a least :
- A name
- A URL
- A SKU number (and old format SKU numbers for very old DAZ assets)
- A price
- A named vendor
Some have (almost all medium-old to recent DAZ assets do) :
- A descriptions
- A named artist (or several when multiple artists collaborated)
- Named compatible figure(s)
- Named compatible software(s)
- Named dependencies (references to other needed assets they come in addition to) when applicable
- Bundle content when they are a bundle
The Sqlite database is almost 170 MB in size but with a little cleanup and stripping of unnecessary tables, I'm pretty sure we can lower that to 100 MB or maybe less.
As implied but not written so far, this is of no use to someone who hasn't got a pronounced OCD syndrome doubled by a gluttony behavior regarding downloads. But if you feel you could use it, I can easily make it available for download in a reasonable amount of time 🙂
To conclude, I'm happy. Yeaaaa, sounds pretty dumb said like that, but that's true. I'm happy to be part of this community and happy to have shared this aspect of my life with you. I also feel like I know you all a little better now that I know how you "handle" your stuff. That kinda make you even more human than you already were to me. And who knows, maybe I'll finally "grow some balls" and resume coding because, after all, what holds me back is... fear to disappoint people. I see it now.
And because I saw your awesome screenshot @hioushi, I raise you those :
Cheers
Math (aka MeshaLee Adams)
@Mathmathou your custom app is pretty much the exact solution I had in mind to create myself lol. Even opting to use Delphi!! Delphi has become my go-to RAD development of choice. I used to love their C++Builder, never thought I'd want anything else, until I really started using Delphi more and more.
@mathmathou i am probably more a hoarder than creator at the moment but lost track of what assets i have. I was currently in the process of installing and curating (basicall saving the DIM files in categories like environment, characters etc).
However having read the various contributions this may not be the most sensible approach for 2 reasons: slowing down Daz if i install all the characters and also should i lose the content drive i would need to reinstall one by one. So i think i may revert to installing and backing up tha content directory.
I have run scrapers on other websites (eg IMDB for my kodi installation) but never managed to write my own. Would be interested in how easy it is to scrape Dazstore and Renderosity/Renderotica.
I did try various chrome extensions but none really worked as intendedMath said: How do I keep track of my current assets and future downloads ?
Despite the embarassingly large size of my archive, for me it is quite simple.
Due to the highly organised file structure & file nomenclature in my archive if a file isn't where it would be if I had it, or still in my recent downloads folder, I haven't got it.
🙂
@Mathmathou , with "181185 references of assets", now I understand your need to classify and your "Diogenes syndrome". LOL
And, I never lie. This folder is from years ago, and I have it disconnected from Daz. Everything is "minified" and the textures are compressed. Just 112Gb and 1339 assets. I hardly use it, maybe I connect it to some mophs or environment... I have a folder with only G8/8.1, without clothes or hair, etc... and scripts, which is what I use to pass to other software.
Scraping is not really complicated you just need the right tool and a good vision of the structure of the page you want to scrap (a little understanding of html is useful).
The mistake I made at the very beginning was using string operations on the source code of pages. In Delphi (and probably in most languages) you can perform operations on string like "search for X or Y", "copy what is between X and Y" and so on. This works but these are linear operations that start from the beginning of the source code each time and are not very optimal.
Another mistake I made was using regular expressions (I still do for some specific things but not on the whole code). Same problem : performances.Most programming languages have their own parser, some dedicated to html, others to json or even multipurpose ones. If you know a little about html, you probably noticed those "blocks" called div. By looking at the source code of the asset page (or the catalog page, sometimes you can scrap directly from those), you can easily identify which div contains the asset name, which one holds the sku number or the price and so on. Most of the time, those div have attributes that differentiate them. A div element with an attribute of "id= name of id=price will mots of the time contain what you are looking for. Html parsers, based on those landmarks you provide can extract the whole div block or only the content of the div. That way instead of searching each time through the whole source code of the page, you are left with much smaller blocks of text to work with.
You probably noticed that recently, Renderhub has change the presentation of the site (slightly). Had I based my scraper on fixed landmarks like "fourth line from the beginning, 25th character of that line", my scraper would have been screwed. But they did not touch the html div names and properties so it's still working 🙂
DAZ is kind enough to expose various URIs (API pages) you can query to obtain a complete list of SKU numbers currently being sold, large json files linking artists with assets or store categories, and even a clean summary of each asset page with a base64 encoded image included (for details and compatibility you need to scrap the full page though). Based on that sku list and after comparison with my local database, I only scrap pages for new assets and not all of them each time.
Renderosity also has an API but it's protected with a timestamp you have to include in you query or you will be denied access after a short period of time. Furthermore their catalog pages include all I need (asset name, url, artist(s) and price) so I usually do not go further.
Renderotica and CGBytes have exactly the same catalog and pages structures, once you have a scraper working for one, it works for the other.
Renderhub does not expose SKU numbers unless you are registered on the site and I was unable to go around the Cloudflare protection so I regularly open their catalog page, sort by newer and display 500 of them, and then copy/paste the source code in my application for processing. Regarding the structure of the data it's fairly straightforward.
I don't know how good and flexible the browser extensions you mention are, and I don't know if you are a programmer or not (I'm a hobbyist, it's not my actual job) but starting from scratch can be tough. If you are running Windows 11 or have an Office 365 account, I encourage you to have a look at PowerAutomate. That thing is nuts (and free) and you'll find numerous tutorials on Youtube.
This 13 minutes long video will give you a precise idea of what you can achieve with that tool of the devil 🙂
If you want to exchange more on the subject, I suggest we switch to private messages in order not to bother the others with technical details 🙂@gt6 So you are a Delphi coder too ? That's rare enough to be saluted 🙂 - I'm still using 10.3.3 because the 11th iteration is quite expensive and all my addons are not yet ported to that version. As for RAD Studio 10.4, it was not bringing anything significant enough to justify the upgrade. We can also exchange on the matter in private if you feel like it 🙂
@eelgoo - "Due to the highly organised file structure & file nomenclature in my archive if a file isn’t where it would be if I had it, or still in my recent downloads folder, I haven’t got it." What can I say other than you have a very good point here ? And also that I swear I LOLed because of the way you turned this sentence.
@abad - I am sure you don't lie and was not implying you were. I hope I did not offend you, I was just joking 🙂
As for the 181185 references, these are the assets I know about, not the ones I have on drive of course that would be enormous.Thanks @mathmathou - hadnt appreciated that Power Automate was part of Windows 11.
Just in case this is of interest to the others over the last couple of days (bored with work) i was playing around with two or three database utlities. I tried Data Crow (seems development has ceased) and GCStar (Flaky as it kept resetting the template i created.
Now playing around with Numento - unlike the others it isnt free (€19 so not exorbitant)but it does seem flexible and allows WYSIWYG Creation of the "Data Card"
Looks interesting - will report back
@mathmathou - i'd definitely like to see the DB sometime, if I ever do create my own version it would definitely help get launched faster. Or if you'd ever consider making public some of the functions, procedures, etc would also be great, no need to re-invent the wheel so to speak.
- You must be logged in to reply to this topic.
