- This topic has 16 replies, 7 voices, and was last updated 4 years, 3 months ago by
.
-
Posts
-
I must be sick in the head, but in a fit of research and experimentation, I found I can manually upgrade the PostgreSQL database... and run monitor and analysis on Studio and DIM traffic.
So Daz ships with PostgreSQL version 9.3 (although it says 9.5, it's not) and pgAdminIII. Both are horribly outdated by a decade.
While looking for a way to learn the SQL queries Daz and DIM use when you press buttons, I realized it has the admin panel included. It's old, and lacks nice analytical tools. So, I figured why not use the current pgAdmin4? Oh right, because it only supports pSQL v9.6 and up.
So a few hours later, and a few explosions and a bit of smoke (not the good kind either) I got the original cluster backed up to tar.gz, and the new 9.6 binaries and admin panel installed. A restore operation, and WHAM, Daz now talks to a more modern DB. AND, I now have realtime graphs and SQL logging so I can finally get my CMS tools off the ground. I think it's about time someone fixed all the broken bits of Studio, since DAZ isn't :p
Bonus: I haven't run benchmarks yet, but it feels like Studio runs a bit quicker.
Tomorrow I'm gonna try to push it to the wall; the current version of pSQL is 14.2 - there's been a TON of improvements since the stone ages =^-^=That's a darned skippy question... but then again, look at the cash cow they're milking. I'm guessing they figure on milking it until the barn collapses.
Hmm.. very interesting. So in essence, this would make finding content faster in search? Would this also make collecting all morphs used for a base figures faster too?
My genesis 8 female takes around 5 minutes to load these days. I would be ever so grateful if you were able to achieve some sort of update that could curb these load times.(I know that Riversoft Arts recently released a 'fix' buuut.. without losing the ability to have all morphs readily available at shorter load times would be much nicer)
Honestly, technical 'mumbo-jumbo' is beyond me. So, CMS could only mean for the search bar function.
Keep us updated, this is VERY interesting.
"CMS" is Content Management System, and it's how Daz makes heads or tails of all the bits and pieces that make up a scene. It's also the backbone of the Smart Content metadata system.
SO: I did in fact manage to upgrade the database all the way up to the current v 14.2 without any problems whatsoever. And yes, the newest version of the actual database IS faster than the dinosaur version... however, the tasks that Daz and Dim ask of the database are so short and sparse that you'd be hard pressed to notice any difference at all in performance, except maybe scrolling the Smart Content browser if you have thousands of products (like I do lol). Loading a scene, it still takes 8 minutes to load a pair of G8F together. The alarming part of it is 2 of those minutes are devoted to loading morph deltas and animations, the other 6 minutes are entirely devoted to throwing errors for each and every input and target formula for every single morph in the runtime.
6 minutes generating well over 15,000 lines of warnings...Formula target property not found: Genesis8Female:/data/DAZ%203D/Genesis%208/Female/Morphs/D.MAster/Tanany/Tanany.dsf#Tanany?value - in file: /data/DAZ%203D/Genesis%208/Female/Morphs/D.Master/Auto%20Face%20Enhancer%20Female/G8Female%20Character%20Links/AFEG8F_172.dsf
Every time THAT ^^^ happens, it searches every other of the 12 thousand dsf files for a valid reference and fails. IDK what happened, Studio never was this much epic fail before 4.20 dropped. It seemed only to give those warnings for assets actually used in the scene. It appears now Daz searches every asset compatible with a given compatibility base (like G8F) for a reference, instead of only searching assets connected to the scene that are compatible (the way it did/should).
Sadly, RiverSoft's tool tries and fails to reliably control the problem. While removing morphs from the runtime before a scene loads excludes them from this round-robin of failure on a per-run basis speeding up the load), doing so breaks some delicate balances overall (references and linkages go more than one level deep across support assets!) and the tool also has a few serious bugs.
Anyway, there it is, a now souped-up hotrod that still can't go faster than a bicycle.
If anyone is interested in trying it out for themselves, I can give a rundown how to upgrade, it's pretty simple."...it still takes 8 minutes to load a pair of G8F together."
A two G8F scene exactly 70 seconds for me. If inordinate load times are your main concern, apart from the general shittiness of CMS, I'd seriously think about spring cleaning my character library or separating my runtimes. I know you wouldn't have to if DS worked intelligently but it is what it is.
My videos on a HD (not SSD) take about 30 seconds to appear on the drive because there are so darn many of them. Should I blame Windows for that?
My fear is that if DS 5, or whatever future iteration, involves a serious code re-write then current content may be incompatible with it. Would Daz do that? Commercially it would make sense to bring out a whole new line of products to work with a new DS but it would certainly piss off a lot of current users. However if the new characters were far superior, like MetaHuman superior in realism and real-time animation, I'm sure they could attempt to justify it.I've also had problems with D.Master products in the past. Can't remember if it was AFE or auto muscle enhancer for G8F. Was so bad that it would actually crash DS or take over 1/2 hour to load. Had to completely remove it. Many of his products seem to conflict with other enhancement morphs, especially for G8F.
That or he is ingeniously releasing a faulty product to curb day one pirating???
As for the rundown on how to upgrade, I'm game. Sometimes search can take several minutes to complete. If it can help alleviate some of that down time, I'm all for it.
As for your runtime. Is it on a HDD or SDD / m2 storage? I read that it makes no difference but I wonder if it would if Daz was actually up to date on their PostgreSQL iteration.
Yeah, I tried RS arts script a couple times but it really didn't seem to lower the load times all that much. Though with RS arts, it's a learn experience for me every time.
Anyway, take care!
@frank22
I get what you're saying, but my point is that performance has dropped D R A M A T I C A L L Y in 4.20. I have a very old machine with Radeon gfx and a Ryzen 3 chip. Nobody expects it to keep up with a snail tbh.
Anyway, my runtime is on SSD and libraries are split up across several categories. I've thrown just about every optimization at the wall as one can, it is what it is.My original point of this whole post though wasn't about optimization, it was about discovering some secrets about the SQL heart of Daz and what can/might be done with it. Now I know just how Daz and DIM use the CMS, so now I can create a better tool to manage the unmanageable, outside of the severely limiting environments of DazScript or the SDK for plugins. As it stands, my dream of a centralized author's environment is coming together; I already have working improvements to the metadata creation and external information systems, now I can get cracking on managing the assets directly, whether they are in DIM installers or not, even replacing DIM *and* Content Package Assist with an authoring tool that builds complete installers with native Smart Content. Opening up the dadtabase opens up a whole realm of possibilities =^-^=
As for a Daz 5... I have my doubts it might even happen. Poser is already essentially a dead format, Daz incorporated its asset import years ago; Daz is gasping, and tools like Diffeomorphic already import Daz assets into Blender... it's only a matter of time before some smart cookie creates a Blender workspace that feels more like Daz/Poser, and renders in the engine of your choice natively. Daz will die on that day.
Yep, DS 5 is now 4 months late from what they said, "end of year 2021". And there has never been an update as to a new timeline since July 2021.
I think they're in trouble.
Ever look at the "new" Daz wiki? It's a snapshot of what it was in 2012. Literally not one "under construction" page has ever been updated since they moved from ArtZone wiki. Everything about Daz got more or less frozen a decade ago. It's time to let it go, Daz. Go into the light.
well, the reason why DAZ wanted to release DAZ Studio 5 in late 2021 was because of Mac M1 support, it would still have been extremally barebones (ie early alpha) and DAZ later got in touch with Apple developers and got DAZ Studio 4 to get M1 support. so those “late 2021” plans were abandoned. they are still developing DAZ Studio 5 and you can see progress with fixing stuff like the FBX importer/exporter (they added newer FBX versions, 2014 was the latest version that was compatible for YEARS) and adding more script commands for users to use, which means the SDK is being reworked in the background.
sometimes silence doesn’t always mean something is sketchy going on behind the scenes, it could be that they are silent because they are busy working on it and have little to really say.
spoke @amberkat:
"The alarming part of it is 2 of those minutes are devoted to loading morph deltas and animations, the other 6 minutes are entirely devoted to throwing errors for each and every input and target formula for every single morph in the runtime.
6 minutes generating well over 15,000 lines of warnings…"...not having too much time to drag into it, do they at least have an index DB in there? It seems like the presence of one would speed things up considerably. I guess I could go look, but I usually avoided digging too deep into my toys - it tends to maks things less fun and more like work 🙂
For what they do with it, I don't think (and if I read it right, you pretty much confirmed) that bumping the version would return too much of a benefit, so they likely shoved it to the bottom of the priority pile.
I think that, as an overall architectural thing, they'd probably get better performance from dumping postgres entirely and just going with something lighter/faster like a MariaDB rig, or even a small non-sharded MongoDB setup (MongoDB does document refs/handling/indexing a shitload better and faster, without all the weight and overhead that an RDB rig has hanging on it. I get the feeling that things would speed up a LOT more that way.) They may have tried a MariaDB (okay, MySQL back then) setup and found it wanting for some reason... likely a feature (indexing?) that was missing back then. Dunno - I'll have to dig around once time permits.
The db has 29 tables for Daz asset and user infos.
PostgreSQL is pretty fast, but again, the bottleneck is not the CMS.The whole thing about formula targets and inputs is what Daz does to connect morphs to scene objects.
Normally, only scene objects that have references to morphs or other delta objects would be tested for formula linkage. It seems now in 4.20+, literally everything named as compatible with a Genesis asset is being tested for such linkage regardless of whether or not it's actually in the scene graph. That's why all the errors. Stuff that isn't there isn't linked to anything. Huge amounts of time is wasted testing all that before Daz loads the actual deltas and unlocks the draw window.Formula target property not found: Genesis8Female:/data/DAZ%203D/Genesis%208/Female/Morphs/D.MAster/Tanany/Tanany.dsf#Tanany?value – in file: /data/DAZ%203D/Genesis%208/Female/Morphs/D.Master/Auto%20Face%20Enhancer%20Female/G8Female%20Character%20Links/AFEG8F_172.dsf
This is why I spend time paring down my morphs folder. Basically the only thing I bother to get figures for now is skins or the OSO/Other creature morphs. I have the basic daz and daz characters like Dain 8 and such and any pbm (Like claws. nails) or phms (ears, noses) I want or shaping morph packs. The rest I just toss. I never used full characters anyway, always been a "dial spinner."
@Oldwolf "... the only thing I bother to get figures for now is skins..."
Ditto. Unless there's a particularly interesting or unique character, which excludes about 99% of the Daz females.Interesting experiment I just tried... I removed all libraries, and set one up with ONLY Genesis 8 Female Starter Essentials. I compared both Studio 4.15.0.2 release, and 4.20.0.11 beta. I loaded G8F a number of times to get a baseline.
According to the log timestamps, the loading times for a single bare G8F are
v4.15 = 32 seconds avg.
v4.20 = 3 seconds avg. (!!!)
[~ 10x faster with essential assets]Shutting down Studio with the G8F loaded in scene (no save)
v4.15 = 10 seconds avg.
v4.20 = 10 seconds avg.Hmm. So far so good.
So then I restored all my usual libraries with everything available; roughly 13,000 support assets associated with G8F...Load time for single bare G8F:
v4.15 = 4 min 30 sec
v4.20 = 1 min 35 sec
[~ 3x faster with thousands of assets]Shutting down Studio with the G8F loaded in scene (no save)
v4.15 = 40 seconds
v4.20 = 40 seconds (interesting...)***
So we can see, all other things equal, there is generally a massive performance improvement in 4.20 over 4.15 (except for unloading the scene), although it scales with asset load.
***Next I repeated the benchmarks, this time with a prepared scene- single G8F with NGV8, Sloane Hair, and 7 other wearables:
Load time:
v4.15 = 1 min 40 sec
v4.20 = 2 min 4 sec (wait... what?!)Unload time:
v4.15 = 1 min 0 sec
v4.20 = 1 min 0 secSomething doesn't seem right... v4.20 took 24% longer to load a simply-dressed G8F! Let's try another one, this time 3 figures - 2 G8F and one G8M, all with hair and basic street clothes:
Load time:
v4.15 = 6 min 48 sec
v4.20 = 6 min 45 secUnload time:
v4.15 = 6 min 31 sec
v4.20 = 6 min 1 secThis one is strange, basically the same times. v4.20 has a very slight advantage.
Other combinations of models give different spreads, some better, some worse.
One thing I noticed is that debug output was added in 4.20 to indicate when a follower object stopped following during the unload cycle. This is what takes 90% of the time... kinda makes me think Daz is quite aware of an issue where dropping a link takes several seconds in many cases, where it should be just a few milliseconds.
Well, thanks for coming to my TED Talk, you can throw your tomatoes now. 🙂
- You must be logged in to reply to this topic.
