Content Library Rebuild -- Tips Tricks HELP ME lol

[Aftyrbyrn]

So, i've gotten to the point where 1TB content library is awfully slow. Takes about 5 to 10 minutes to load the G8 Female character. WTD. I can't figure out where the choke point is, so i gave up. This is my plan.

Two 1TB SSD drives, one for Runtime, since that seems to be about half the data and the other for the rest of the library, using a hard link to the runtime folder pointing the other drive.

Can't really copy 1 for 1 from the original library, since it's a mess, and i really dont' wnat to, but openening up all the zips to find the ones that actually have a good structure, or let alone an IMS file (1000s or zip/rars to parsse) i wrote a quicky that will scan each zip, find if it has an IMS file inside it, and extract it to a new folder. going to share that here. (At the end)

I'm going to use DIM to do the bulk of the installs, it's just easy, but if you have better ways to manage content, please share them. I'm all ears. I don't want to get back to 5 to 10 minute load times because of crap written morphs, etc...

If it comes down to it, i'm going to write something like nexusmods has with Vortex and have something that can manage hard links to files, with a simple check box, if it doesn't exist already.

Suggestions? Past Experiences and comments? Lemme know.

#SCRIPT#

# you have to update the paths in the code to reflect your storage etc...

$imlist=@()
$x=0
$contents = @()
$Zips = Get-ChildItem -filter "*.rar" -path G:\Incoming\new.ZoneGFX\RarsZipsEtc
$zips.count
$Destination = 'G:\Incoming\new.ZoneGFX\RarsZipsEtc\Extracted'
if(Test-Path $Destination ){}else{New-Item -ItemType Directory $Destination}
$WinRar = "C:\Program Files\WinRAR\winrar.exe"

foreach ($zip in $Zips)
{
$im=$false
$x++
$flist = &$rar lb $zip.FullName 
Write-Progress -Activity "$x : Getting data" -Status $zip.Name 
foreach($f in $flist)
    {
    if($f -like "*im00*")
        {
            $im=$true
            $row="" | select archive, path, filename
            $row.archive = $zip.name 
            $row.filename = $f.split("\")[-1]
            $row.path = $f
            $contents+=$row
        }
    }
    if($im)
    {
    &$Rar e -y- -r $zip.FullName IM0*.zip $Destination 
    $imlist += $zip.FullName
    }
#Get-Process $rar | Wait-Process >$Null
}
}

[eelgoo]

This is my take on it.

' https://zonegfx.com/forums/topic/manual-install-to-ds/ '

🙂

[Eidersson]

I'm in a similar situation, I have over 2 TB of installed assets. I have two main content directories, one for old runtime/V4 assets and one for the rest.

It got to the point where a base G8F character was taking roughly 5 minutes to load. I've now been manually removing them, as I installed all my assets manually.
After removing roughly 125 G8F characters, the load time is down to just over 2 minutes. I still prefer to install assets manually as I have my directories set as I like them.

Once I have removed the majority of characters, I will install them using DIM into a new directory, as and when I require them, then uninstall them when I have finished.
I've also begun using Content Package Assist to create DIM files for Renderosity characters, it will make the install/uninstall process so much easier.

I've only been using Daz roughly 18 months, and I'm now happy with my directory structure. From what I've read, it seems the main culprit is the installed character morphs that make load times longer.

[Aftyrbyrn]

Is there a script to create DIM packages? I could figure it out i guess, just didn't want to re-invent the wheel.

Yeah, fresh library, split on two ssds, g8F loads in ~10 seconds.

I was going to strip the drives, but i felt splitting runtime to one drive and everything else was essentially the same, and probably safer.

So far, runtime has averaged about 1.75 times larger than everything else. So it's a pretty close split.

[Aftyrbyrn]

@eelgoo, yeah, read your post, i'm pretty savvy with the structure, i just want an easier way to manage. LIke if i install a few morph packs, then i'm back to a huge load time, i want a app/option that can just uninstall them, even if they were DIM format. Which leads me back to writing something that can extract to a temp directory, possibly hardlink the files to the actual directory, do some testing, if everything is ok, commit the change and move the files to the final destinations, or if it goes tits up, undo the change.

Shouldn't be too hard, except many of the providers, like you said, use convoluted file structures in their zip/rar/7z files. I'm going to try and build something that looks for the base folders, and does the installs, like described above. I've probably bitched about this before, but it's almost annoying to use DAZ once it gets borked up. Just gotta grab Visual Studio and get cracking. Shit maybe i can even sell it, then one of y'all can buy it and post it here lol.

[eelgoo]

As I said in my linked thread, I create a custom Inst file & keep a copy of that.
If I need to uninstall I know exactly which files & where need to be removed.
Also should I ever decide to re-install, it is just a copy & paste job back into My Library.

🙂

[Eidersson]

@aftyrbyrn to create the DIM's

( https://zonegfx.com/content-package-assist/ )

[globalrecession2009]

I did't even understand my own reply post, it was all over the place, it was a mess so i deleted it.

[globalrecession2009]

lol i see that some replies had been done while i was typing....sorry if i repeated what others have said......i got side tracked before i posted.......seems we are saying the same kind of things....but some are better explaining than i am........like Eidersson 🙂

[ADAM]

Ive deleted and rebuild my Daz folder 3 times.

I keep promising myself to only add what I use but, I still install stuff anyway.

IMHO I would select a few characters you REALLY like skins, ect... Then a few morphs you REALLY like.

Delete your Daz file, install Daz and ONLY those characters you intend to use. This made things extremely easy and fast for me till I started breaking my rules again.

[Aftyrbyrn]

Once i get a good base of what i like, i'm backing it up intact, so i can just restore it later, to an empty folder if needed. 🙂 When i have some free time, i'll do the custom installer 'app' and a load time monitor, i can do something like that i'm sure. i just wish i hadn't lost my content drive... DO NOT use REFS as a file system. It's great, until it's not great. 🙂

[ADAM]

Ive no organization on my storage drives so its difficult to find stuff. I keep at least 2 external drives with the same content. This way if one craps out I should be ok.

I rarely revisited the same characters more than a few times until recently. Mostly for the textures. I plan on doing a delete again maybe middle of next year.

[Aftyrbyrn]

That was kind of the goal, a got a few 6 TB externals, used mini partition tool wizard and copied off all my SSD drives to them, the flipped them to active on the externals. Now I'm restructuring everything. And like you, i tell myself, oh yeah, this time it'll be different, i'll stay organized. Well, we'll see.

[Beezlebubb]

Wish someone would create a tool like Content Package Assist for the Mac ... sure could use it!

[Author]

Posts