log.txt cleanup -- wow so much faster

[Aftyrbyrn]

I had a crap ton of broke ass files showing up in log.txt log file in
%AppData%\DAZ 3D\Studio4 Public Build\log.txt

... hundreds upon hundreds of these:

2019-05-15 13:47:37.767,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/addy/Sienna/CTRLSienna.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.772,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/addy/Sienna/Sienna Body.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.772,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/addy/Sienna/Sienna Head.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.803,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/AliveSheCried/Aerilyn/Aerilyn Body.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.803,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/AliveSheCried/Aerilyn/Aerilyn Custom Head.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.807,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/AliveSheCried/Aerilyn/Aerilyn Head.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.807,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/AliveSheCried/Aerilyn/Aerilyn.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.808,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anagord/AnnikaG3F/Annika Nails.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.808,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anagord/AnnikaG3F/CTRL Annika.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.812,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anagord/AnnikaG3F/FBM Annika.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.812,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anagord/AnnikaG3F/FHM Annika.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.813,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anagord/BrisaG3F/Brisa Nails.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.813,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anagord/BrisaG3F/CTRL Brisa.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.817,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anagord/BrisaG3F/FBM Brisa.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.817,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anagord/BrisaG3F/FHM Brisa.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.818,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anain/Marie/Marie Body.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.819,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anain/Marie/Marie Head.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.823,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/Anain/Marie/MarieSisterBody.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.826,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/AQ Jenna/AQ Jenna Body.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.826,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/AQ Jenna/AQ Jenna Head.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.826,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/AQ Jenna/CTRLAQJenna.dsf, (1,1): Syntax error - expected '{' or '['
2019-05-15 13:47:37.830,A:/DAZ3d/StudioLibrary/data/DAZ 3D/Genesis 3/Female/Morphs/AQ Jenna/SCLPropagatingHead.dsf, (1,1): Syntax error - expected '{' or '['

and other issues, duplicate ids etc... i wrote a script the moved all the referenced files to a backup location, preserving the path just in case i want to find and drop back in, and now shit loads 100 times faster. wow. And the broke files didn't work anyway i guess, so nothing lost, and speed gained.

I used cmtrace.exe (microsoft system center log viewer, or get something that views text in real time)
Then watch the log.txt with it when you are loading files, if it seems to take a lot longer after a new package was installed (or just incredibly long), probably has some bad code in it?

Now i load a fully textured character in like 15 seconds, instead of 3 minutes.

[GrumpyOldMike]

Very nice 🙂 I have a pretty ugly log file which I'm sure is probably why my crap takes forever to load as well. Alas no scripting skills to perform the wizardry you did with your log file.

[John Lund]

Would you mind sharing that script? I would really need to clean up myself...

I believe I have seen a commercial tool doing just that, anyone that knows where, what it is called or maybe can upload it.....?

[Titan3D]

I second that request. My load times are horrendous.

[Anonymous]

I just got cmtrace, along with some other handy tools from this link...
https://www.microsoft.com/en-us/download/confirmation.aspx?id=50012

[Slackwave]

I'd be very interested in your script too if you agreed to share it.

[Aftyrbyrn]

ok @ultrabald, @kendrad, @titan3d, @chonakz
There were a few steps to this : one, parse the log, get the paths to the files out and put in a text file (see format at bottom), i used powershell .split() and excel text-to-columns to do that.

Once you get the list of files with path then i used the following: (you'll have to update the code to reflect your paths)

You'll need to tweak it as needed, i like keeping the original paths so i can drag and drop back if it causes issues with other settings. But now everything loads as it should, and the stuff that doesn't, i don't even think i miss. The code just takes the base path out of the move statement and replace with backup path, create directory structure (ommit already exist errors), move file. That's it. easy peasy.

$sourceDir = "A:/DAZ3d/StudioLibrary"
$backupDir = "D:/DAZ 3D/bak-failfiles"
$sourceFileListCSV = "A:/list.csv"

$files2Move = Import-Csv $sourceFileListCSV 
foreach($mFile in $files2Move)
{
    $dir = $($mFile.file.Replace($sourceDir,$backupDir)).replace($mFile.file.split("/")[-1], "")
    New-Item -ItemType Directory $dir -ErrorAction SilentlyContinue
    move-item $mFile.file $($mFile.file.Replace($sourceDir,$backupDir)).replace($mFile.file.split("/")[-1], "")
}

your input list.csv should look like:

data
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWArt\Natalie\CTRL-FWNatalie.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWArt\Natalie\FBM-FWNatalie.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWArt\Natalie\FHM-FWNatalie.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWArt\Rebekah\CTRL-FWRebekah.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWArt\Rebekah\FBM-FWRebekah.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWArt\Rebekah\FHM-FWRebekah.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWSA\Elizabeth\CTRL-FWSAElizabeth.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWSA\Elizabeth\FBM-FWSAElizabeth.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWSA\Elizabeth\FHM-FWSAElizabeth.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWSA\Katie\CTRL-FWSAKatie.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWSA\Katie\FBM-FWSAKatie.dsf
\DAZ3d\StudioLibrary\data\DAZ 3D\Genesis 8\Female\Morphs\FWSA\Katie\FHM-FWSAKatie.dsf

[Titan3D]

Thanks for this - I badly need to clean out those errors.

[John Lund]

Thanks a lot @Aftyrbyrn! I will look into it even if it looks like being above my pay level...

[Slackwave]

@aftyrbyrn thank you for sharing this. On my computer, your script gave an error in powershell but I managed to make a .bat file that does quite the same even if this method seems a bit radical to me.

We still really lack an efficient content cleaner/organizer tool. I've been using Daz for years, my library is 1.6TB large and it takes around 4 minutes to load G8F... 😀

[Aftyrbyrn]

well, the script was what left in my cached files, it was porbably a pre-final run, i just usually rewrite those on the fly as needed, but the structure should help others. 🙂 But i bet you get the idea, move the files that are identified, keeping the structure so they are easy to restore, you know just in case it crashes cuz you moved something that shouldn't

I can custom script anything, i just never save anything, unless it's monitoring a server or service and keeps me from having to do it manually, yet still bill for it 🙂

[Aftyrbyrn]

If i had a meta file for all my installs, i could use PS to read them all, count duplicate mentions, and keep track of what was installed, as long it was in the original locations. Then you could poll and see what is fully installed, what has partials etc... something i've been toying with, but my drive that held all my dls crapped. Effing REFS file system, no forgiveness if there is an error. I'm tempted to pay money for a disk recovery software and let it run on it. But for now, i guess i'm going to write something to scan all the dufs, and see what has missing references. Maybe something based on folder structure to determine what i have. I have big ideas on how to collect the data, and analyze it, just little motivation, as i have done it for millions of files on data servers where i have and had worked in the past. There really isn't anything great to manage it, unless you solely use the DIM and trust it. I didn't, i just extract copy paste. 🙂

[Aftyrbyrn]

I started writing it... and and decided, whoa, billions of dufs and folders and crap to parse... nah. Not for me at the moment lol

[Dragon]

Damn... too much for my ancient Dragon brain to wrap around. I'll just deal with my load times. LOL

[Aftyrbyrn]

lol, it's not too bad. there are lots of forums out there that explain cleanup in easier ways, or at least help identify the issues. I was so frustrated when it was taking 4 minutes to load a base g8 character. Gotta say though, after watching the logs play while loading, there is a lot of BS trying to load referenced in some of the files, that i'm guessing that the creator of package XYZ left some dangling entries w/o including them in their product.

[Author]

Posts