Fixing (or creating) Daz metadata, esp. Manifest.dsx?

Home Page Forums General Chat Fixing (or creating) Daz metadata, esp. Manifest.dsx?

Viewing 9 posts - 1 through 9 (of 9 total)
  • Author
    Posts
  • #1810807
    FleshKage
    Participant
    Transfer PointDaily VisitorArticle
    Rank:

    I noticed that SC Rebecca HD on the blog has incorrect metadata — Daz Install Manager thinks she’s the New York Bodega Liquor Store, of all things. Seems like she has the wrong Manifest.dsx and Supplement.dsx files included.

    Fixing Supplement.dsx by hand is easy enough, but the manifest file is a list of all filepaths in the package, and I rrrrreally don’t want to edit that by hand if that’s avoidable.

    Running into this reminded me of a general problem I have: I’ve got lots of stuff from renderosity, etc. that never had Manifest/Supplement files in the first place, and could only be installed manually. I’ve found I really don’t like installing stuff manually, and strongly prefer how DIM does things.

    (So please — don’t recommend I just install stuff manually! I’m familiar with that process already; I’m looking for a way to move away from that.)

    It would be nice if there were a tool or script or something that scanned a given directory and used it to generate a Daz Manifest file in the proper format.

    I did find this python script out that looked promising:
    https://github.com/esemwy/mkdim/blob/master/mkdim.py
    but I cannot for the life of me get it to work. Just get a bunch of mysterious errors from Python that I can’t seem to troubleshoot, even after changing around python versions and whatnot.

    Does anyone know of any other resources to generate a Manifest file that DIM can read?

    #1810922
    Amber
    Participant
    AnniversaryArticleDaily VisitorPeoples PersonTransfer Point
    Rank: Rank 5

    I’ve had a look at mkdim, I’m not so sure about it. Just looking at the source, there’s a lot of extraneous code to handle “special cases” when in fact there is a very simple and rigid orthography to DIM structure. Also there’s problems with incorrect regex in the string normalizer, the Daz CMS has formulas inside that define the actual regex Daz uses to clean strings.

    You can always try this (it’s what I currently use, with hand-editing):

    Content Package Assist

    This is the best tool currently out there to build a proper installer zip; it will build the manifest correctly. It does have some bugs though, like improper support file nomenclature (dsx, dsa, and image file will have to be renamed and reinstalled, and the manifest and dsx amended as well to capture those changes, in order to have a canonical installation that Daz likes). However, you can feed it an existing package to build, and get the manifest out of that to slipstream into the existing package.

    ONE THING: if you are rebuilding a product that you have already once installed in Daz, MAKE SURE NEVER TO ALTER THE PRODUCT GUID!!!! This might be problematic if you’ve installed a product with an incorrect manifest, as now Daz CMS has a reference to a product with an incorrect GUID.
    The big problem is, if you install any product with one GUID, uninstall it, and re-install it with a new GUID, it will work the first time, but if you ever try to update it, or uninstall and reinstall it again, it will fail since there are now 2 GUIDs associated with the product ref in the database. Daz takes a massive sh!t then, and DIM will refuse to install the content. I found this out the hard way 🙂
    The only cure is to use an admin panel to manually edit the CMS database ( a freakin nightmare, Daz db is overly-complex), OR, you can nuke the db and re-import all your products’ metadata (which can take a bit of time depending on how large your runtime is).

    I haven’t tested your scenario though, it seems that if Daz thinks your files are the Bodega product, you should be fine with a proper manifest that tells it the correct product details. If you have the Bodega, you should uninstall anything you currently have, and install that for correctness. Then the other question is what the real GUID is for your broken product… if you intend never to share it, you can make up any GUID, but if it may get out in the wild it must have either the original GUID it was published with, or none at all – and let it be someone else’s headache.

    I’ve spent a LOT of time mucking around under the tent flap with the CMS and DIM (currently working on my own uber-installer/CMS jockey), so if you need any very specific help, give a shout.

    <edit>
    I’ve just had a look at Rebecca (save the jokes!) and it turns out the support files are original. Therefore, we know that the proper GUID for this product is
    47baa836-7b75-414e-b11f-6517302caa9f
    So when you generate your manifest, be sure that gets edited in place, and be sure to include the support files if Content Package Assist did not write them in.
    Then you’ll have a perfect working product installer.
    pro-tip: you can get all the pertinent product data out of the dsx file in Runtime/Support, including GUID, SKU, author, store, etc. Very handy for hand-editing stuff.

    #1811038
    Amber
    Participant
    AnniversaryArticleDaily VisitorPeoples PersonTransfer Point
    Rank: Rank 5

    @fleshkage
    I’ve had a little time to work on this, so I’ll give a little walkthrough on how to tackle this one.

    1. PREPARE CONTENT
    I build packages right in the DIM Downloads folder, but you can do it anywhere. Within the Downloads folder, I’ve made an empty folder called FIX ME. This is where I prepare the raw content to be built, so as not to get it mixed up with other things. I’ve moved the broken product zip to there, and extracted the Daz folders from the Content folder in the zip.
    https://i.imgur.com/YqaYYFX.jpg

    2. CHECK WHAT NEEDS WORK
    I examine the \Runtime\Support folder, and find all 3 properly named support files as I would expect, so that’s good.
    https://i.imgur.com/NSqVRSa.jpg

    The image is a valid JPG, and the DSA file contains the same script no matter what (verify contents lol), so I open the DSX file in Notepad++ to check it for completeness, and to have it handy for the following steps. We’ll need some of the info at the top, and I need to check the support files references.
    The DSX file references itself in the <SupportAssets ...> tag, perfect.
    https://i.imgur.com/9CsdtXS.jpg

    However, the <SupportAsset .../> tags for the other two files are missing. This is easy to see, since Daz’s authoring tool always generates the XML tags in alphabetical order.
    https://i.imgur.com/hnhWoSo.jpg

    This is a common situation in many Daz products (why?!), but for the sake of completeness we can easily add them.
    https://i.imgur.com/qoCRld6.jpg

    Ctrl+S save the the dsx file (but leave the text editor open!), and we can now build the package.

    3. IMPORT CONTENT
    I launch Content Package Assist and enter Settings – make sure that you DO NOT CHECK “Auto Clean /Runtime/Support”, otherwise you will delete the support files when building the package, and you’ll have to start over from the beginning >.<
    Also check the default paths, at least the one to the DIM Downloads folder. They should already be correct but you can point them anywhere you like if you have a custom directory arrangement.
    I also make sure that Source Prefix is “IM” (you can also change this on the main window), and the rest can be left in default values. You can close Settings, and on the main window, make sure both “Generate Manifest.dsx” and “Generate Supplemental.dsx” are checked.
    https://i.imgur.com/mUgFJuv.jpg

    Now we can import the content – First, switch to the Files tab in CPA and enter the Content folder under the “Buildxxx” folder. Now select the \data, \People, and \Runtime folders in FIX ME, and COPY them to the blank window (the Content folder in CPA)
    Easiest way to do this is Ctrl+drag the files onto the CPA window, or you can Ctrl+C and Ctrl+V into the CPA window.
    NOTE: Due to a design flaw in CPA, there is no context menu in the file view, but there is one in the folder tree view. However it does behave like any other Explorer window as expected, you’re just running a bit “blind”. Also note that this is an actual temp folder, not virtual, so simply dragging and dropping will MOVE the files to the Build folder(if on the same drive), not copy them! Something to keep in mind.
    At this point your build should look like this:
    https://i.imgur.com/I0SGS9F.jpg

    Switch back to the Details tab in CPA, and have your Notepad handy with the dsx file open. We need to copy-paste some info to complete the build info.
    Source Prefix: should already be “IM”. Adjust if needed.
    Package SKU – ID: We will copy the value from the <ProductToken VALUE="85367"/> tag and paste it here.
    Product Name: We will copy the value from the <Product VALUE="SC Rebecca HD for Genesis 8.1 Female"> tag and paste it here. (note: for Sanity’s sake, if I ever see a word like “for” begin with a lower-case letter, I capitalize it. Daz CMS parses the dsx and gets the proper name, but other parts of Daz rely on regex to demangle the caps-case version, so it will always be wrong. Not really critical, but whatevs :p )
    -place a check for “Use Sanitized Product Name“, and simply leave the result in “Package Name“.
    Package GUID: !!! CRITICAL !!! Here we paste the canonical GUID found in the dsx <GlobalID VALUE="47baa836-7b75-414e-b11f-6517302caa9f"/>
    Compatibility Tags: this should already be set to the default “DAZStudio4_5“, adjust if necessary.
    Again, ensure that the Manifest and Supplemental checkboxes are filled, and with all thumbs-up, we are now ready to build the package. Your build info should look just like this:
    https://i.imgur.com/D0fBoeP.jpg

    4. BUILD PACKAGE
    In CPA, click the Make button, and you are given two choices;
    Make in Package Folder
    Make in DAZ IM Download folder

    Both of these paths are set in Settings as mentioned before, so you know where they go.
    Personally, I prefer to have the package appear directly in the Downloads folder, since that’s where DIM will eat it up, it saves a step or two. 🙂
    So, I choose Make in DAZ IM Download folder and let CPA do its work. The product image will appear in the window, and the progress bar will begin to advance. In a few moments, it is finished, and we can check our work.

    5. INSTALL PACKAGE
    Going back to the DIM Downloads folder, we see our new package, YAY!
    https://i.imgur.com/vnr1xHU.jpg

    Open the package in WinZIP, 7Zip, WinRAR, or whatever you use (I use Directory Opus, so I see all archives as just another folder 🙂 ) and verify you have the edited dsx file in \Runtime\Support (easy way is to see the very recent timestamp, you can also extract and examine it to see the entries you added). If perhaps you forgot to save the file before building, and built with the old version of the file, simply merge the new copy into the archive replacing the old one.

    NOW, we can go up and check the Manifest file.
    https://i.imgur.com/LSCrxp0.jpg

    https://i.imgur.com/maIeRom.jpg

    Success!
    At this point, you can go ahead and fire up DIM, you already know how to install. 🙂

    Say “HELLO”, Rebecca!
    https://i.imgur.com/ZZHOiaJ.jpg

    #1811114
    FleshKage
    Participant
    Transfer PointDaily VisitorArticle
    Rank:

    Wow Amber — this is a lot of good info. Thank you so much for the detailed instructions, that’s super helpful!
    I’ll try running through this on my end and see how it goes.

    #1811120
    Amber
    Participant
    AnniversaryArticleDaily VisitorPeoples PersonTransfer Point
    Rank: Rank 5

    Good luck! PM if you need to.:)

    #1811498
    yngvarr
    Participant
    Transfer PointArticleDaily VisitorPeoples Person
    Rank: Rank-1

    Amber, this is a really powerful instruction. But in my humble opinion, to solve the problem, it is enough to use Content Wizard

    #1811512
    Amber
    Participant
    AnniversaryArticleDaily VisitorPeoples PersonTransfer Point
    Rank: Rank 5

    Content Wizard is bloated, confused, and in a number of cases can wreck your runtime if you’re not careful. It makes a lot of incorrect assumptions about how Daz works under the hood, and is generally wacky. Also, it creates very broken packages that require much more editing to fix than what you see here.
    This above is a workflow distilled over several years and about 3000 packages.

    I’ve been through the CW source code, I think RS might have a drinking problem or something >.<
    Lots of superfluous stuff, duplicate routines and such.

    #1814432
    FleshKage
    Participant
    Transfer PointDaily VisitorArticle
    Rank:

    Finally had a chance to download Content Package Assist and look into this.

    Due to an abundance of caution, I ran the CPA’s exe through Virustotal and Manalyzer:
    https://www.virustotal.com/gui/file/38eebf6b21a91b211f7c000872a3f8407aa161c2d187bd3ba80d8cbee870212b
    https://manalyzer.org/report/b0e6ff607b7e214372f52a5d43177f99

    Virustotal says it’s clear, but Manalyzer reported the exe “contains functions mostly used by malware” which is making me a bit hesitant to try to run this.

    I realize that could be a false-positive type situation (especially given the clean Virustotal report), but I don’t know CPA well enough to know if the functions Manalyzer called out are things I should reasonably expect the application to use.

    (Also, there’s a weird “backwoods-country” url mentioned in the report that I wouldn’t have expected? Not sure what to make of that.)

    Just curious @amberkat, is your copy of Content Package Assist the same as the one here on the Zone? In your opinion, do the functions mentioned in the Manalyzer report sound like something the application would legitimately use and that I shouldn’t be worried about?

    Sorry if I’m being overly paranoid — just want to make sure I cover all my bases here!

    Blog post I got Content Package Assist from if it helps: https://zonegfx.com/content-package-assist-2/

    #1814455
    Amber
    Participant
    AnniversaryArticleDaily VisitorPeoples PersonTransfer Point
    Rank: Rank 5

    @fleshkage
    Yep, it’s a false positive. The tool is clean; I think it’s because of the compiler optimizer used in building it.
    My copy is the one from here.
    That manalyzer report is comedy gold. They point out every generic routine used in virtually all executables, and mention “malware uses this too”. Pretty much anything you feed it will return results like this.
    Then it says “VirusTotal score: 0/71 … All the AVs think this file is safe.”
    Cue the rimshot! *badum-TSSS!

    “backwoods-country.com” is the author’s website (under construction), it’s in the About button in-app… IIRC it was supposed to be an online store for that and other software, and has been in limbo since 2011. The progress bar has risen only slightly over the years, at this rate it should be online somewhere around 2089.
    Compare how it looked in 2014:
    https://web.archive.org/web/20140516233153/http://www.backwoods-country.com/
    with the same link today.
    Anyway, yeah safe, no worries.

Viewing 9 posts - 1 through 9 (of 9 total)
  • You must be logged in to reply to this topic.

 

Post You Might Like