- This topic has 44 replies, 10 voices, and was last updated 3 months ago by
.
-
Posts
-
For me, this has been more of a maintenance thing than a security issue.
After a while, the DIM dirs get pretty full, which causes DIM to have to sort through a lot of already installed stuff just to find a couple of new IMxxxx files you want to install.
So I clear it out every now and again. You could do this with BleachBit or CCleaner, but I just run a batch-file so I don't have to wait for a full system-cleaning process.
DIMCLEAN.bat: ------------------------------ @echo off :: showing standard install - change this path if you need to cd "C:\Users\Public\Documents\DAZ 3D\InstallManager\Downloads" :: uncomment preferred 'del' - comment unused 'del' :: everything del /q *.* :: just zips ::del /q *.zip :: just dsx ::del /q *.dsx :: showing standard install - change this path if you need to cd "C:\Users\Public\Documents\DAZ 3D\InstallManager\ManifestFiles" :: everything del /q *.* echo: echo **** DIM Cleaned %date% %time% **** pause exit ------------------------------Now, it seems clearing files in \downloads and \manifests does keep DIM from knowing what's already installed.
You can test this by clearing your DIM dirs and then trying to install a IMxxxx.zip you've already installed.
DIM will go right ahead and install it, overwriting what's already in there.
If you leave it alone, DIM will only install the new stuff, keeping the previous manifest and zips for reference.
At least that's been my experience.
So, I guess clearing the \downloads and \manifest dirs does give some measure of security, although I've only ever done this just to lighten the load on DIM so it gets to installing a little faster.
Or you can just select this option in DIM, for the downloads part:
I would NEVER recommend deleting the dsx files from the manifests folder. DIM and Connect are the only means to get product updates notifications, for purchased and installed products of course. Unlike Renderosity, Renderhub or Renderotica, Daz does not send any emails notifying product update roll out. Some of the plugins and products (like from Zev0, RSSY) are updated often with bug fixes and you wouldn't want to miss out on those, especially if you paid for them.
Edit: The manifests entries are also essential for cleanly and conveniently uninstalling products, especially for those who haven't grown bald from micro-managing their content folders.
@sandeep, This is indeed good advice and while my method works for my situation, that caution should be adhered to for most users.
I do, however, maintain backup archives of all IMxxxx files should any need to be regenerated to provide manifests.
But So far, that necessity hasn't arisen.
@jorgen74 In your method, how would DIM check for updates for legitimately bought products from Daz, when you are removing all dsx files from the manifest folder before going online?
I am just trying to understand what are you using DIM for?
a) If its just for product installation then you might as well manually download the archives from Daz or here (or from anywhere on the internet) and unzip them to content library folders. DIM isn't required.
b) If its for checking product updates then deleting all files from manifest folder, even if temporarily, before going online defeats the purpose, since DIM wouldn't find any product to update.
c) If its for clean install/uninstall/reinstall then you can safely do it in offline mode without ever going online (you would need to have the archives available locally in your DIM download folder for this).
Deleting files from the manifest folder helps none of the above scenarios. What am I missing?
In your method, how would DIM check for updates for legitimately...
My good sandeep, the term 'legitimate' has no place in reference to my content.
...you are removing all dsx files from the manifest before going online?
Exactly.
...unzip them to content library folders
Smart-Content?
I am just trying to understand what are you using DIM for?
I coded a C++ app to install the files I've 'collected' over time so I don't have to do anything manually.
I unpack and parse each download in a temp directory, process recognized files and folders, send them where they go and delete the contents of the temp directory before loading the next download in the loop. Non-destructive to the original downloads.
Non-Daz downloads usually consist of multiple folders that each have to be merged into MyDaz3DLibrary or it's equivalent content directory.
The IM zip files, however, don't need to be unpacked and parsed or merged, I can just send them, in-situ, over to InstallManager\downloads and let DIM handle those installs.
That's why I use DIM when I run across IM zip files in the parse, seems the reasonable thing to do.
One thing to bear in mind is that most vendors are inconsistent in file nomenclature.
Some of them seldom do things the same way twice.
This quickly leads to chaos, if you don't regulate it with a strict file hierarchy.
See the manual install sticky for more info.
🙂
Indeed they are inconsistent (putting it mildly). Not having a standard has been the bane of myself and others who have attempted this sort of thing along with 'creative' file structure and packaging, including archived files the same name as the parent archive, etc.
So I don't depend on structure at all in my parsing. I do depend on recognizable folder and file names, without which Daz wouldn't work at all. But there's always that 10%...
My good sandeep, the term 'legitimate' has no place in reference to my content.
If that sums up your usage of DIM then I see no reason for you to log in into DIM. DIM's offline mode should sufficient for your needs. You might as well lock down the DIM executable in your firewall to be extra safe. So you don't need to delete any of the dsx files in the DIM manifest folder.
The point I was trying to drive is, that level of manifest manipulation is only necessary if one has a mix of legitimate and illegitimate Daz content installed through DIM which I am assuming many (if not most) of us have, including myself.
In that case, they need to remove the dsx files ONLY for the illegitimate archive packages before logging in. Removing dsx files of legitimate packages from the DIM manifest folder prevents those packages from getting updates. This is easier said than done, considering that any such tool doing this filtering needs to be aware of the legitimate content that they have purchased from Daz store. This, I believe, is something that is too complex for a command-line Power Shell script to achieve.
This, I believe, is something that is too complex for a command-line Power Shell script to achieve
It'll involve some user interaction. You could do something as simple as adding a hyphen before the first char of bootleg manifests or removing the 'IM', or deleting the extension. DIM won't read a non-IM or non-manifest file so won't report it as installed, thus you have safety online and retain the manifest for later use by just changing the name.
To do this in a semi-automated way, you would copy the bootleg IM-files you want to install to an empty 'install' folder of your own, then run a program that iterates through the IM-files, saves their names to a file-list, then copies the bootleg IM to C:\Users\Public\Documents\DAZ 3D\InstallManager\downloads, until every file in the folder is processed and copied over.
You would then use DIM offline, as usual, to install all the files (wish DIM had a CLI).
After the install, all the manifests would be generated.With the manifests generated, you would run the program again to iterate through the IM-names saved earlier, match them with the filenames in the \manifests folder and rename them so that DIM won't read them.
The program would handle manifests and bootleg IM-zips in the \downloads folder the same way.
I'd like this to be a continuous 'one session' thing, but unless DIM has a CLI, it's gonna be a two part deal.
The file-list saved earlier would always open in 'append' mode, so you would have a 'live' list of every 'bootleg' file you have ever installed to refer to, like a log.
That's just off the top of my head, I'm sure it can be improved a bunch, have to build a prototype and see where it goes (and investigate the possibility of a DIM CLI).
But yeah, might be a thing.
The crux of my original question was that I never use DIM for anything but installing and updating the products downloaded form DAZ. Everything else is being installed manually. The addon I installed through it was the first time I broke my rule and motivated my post.
I am doing something similar to @jorgen74 but since I am not as skilled as him/her in programming I have to use a bit more convoluted method.
-First, I classify everything I download in separate temp folders (People, props, hair, shaders... etc)
-Then I rename each folder's files following a strict policy diffeerent for each group. For people, for instance is:<Generation1_Generation2_...GenerationN> <Age range> <Ethnic group> <Filename (sanitized)>
which leads to, for example:
G80F_G81F Adult EUR Yennefer of Vengerberg.zip
Age ranges (Teen, Adult Senior, Old..) and ethnic groups (EUR, AFR, ASI...) are just broad and fuzzy categories to help me locate more or less the kind of character I'm looking for. Hair is similar, but instead of age ranges and ethnic groups I have general styles (SHR for short, PON for ponytail, UPD for updo... etc), shaders have their own categories, and so on.
The renaming work is done via Excel and den4b's renamer. Excel to generate quickly the sanitized filenames with the help of a template I've built and Renamer to rename the files according to the generated names.
Once I've done this, I have created an VBscript file which recursively extracts all the files within these temp folder to its own subfolder, relocates the meaningful subfolders out of 'Content', 'My Library' or similar, moving them up to the base level if needed, deletes the manifest and supplement files (I don't use them, I install everything manually) and recompresses them all with 7zip, wich gives me significant smaller file sizes most of the times.
As you have said, the internal file organization in each file is downright chaotic, so it does not always do what I want, so I must quickly check the results an correct the rogue files' structure, but I'd say it works on over 90% of the cases.
Finally, I backup the recompressed 7z files and install everything manually. While I do this, I check where the hell the vendor has decided to place the files and navigate through the content manager to assign them one or several of my custom categories. Once I have done this, I can find almost anything righ away.
Which leads me to a new question: classifying everything by custom categories in content manager is really tedious when your library grows. If you use these custom categories ¿how do you backup them? I have googled this question, but I've found different answers and, should I have to migrate to a new computer in the future, I would like to know how can this be done, because I'd kill myself if I had to start over.
Have you read through this sticky?
OK, researched this last night. DIM's CLI just does filters, but doesn't let you do 'work offline' or 'start'.
But I think we can do a workaround for that. We'll spawn DIM ourselves with CreateProcess, so we control the thread. When we have a window up, we can use SendInput to simulate keypresses, in this case, tab.
Since we're in a child window that has a tab-order, we simulate tabs that get us to the 'work offline' box, then simulate a spacebar hit to check it, then tab down to 'start' and simulate another spacebar hit. That starts the cue. We just let it complete, CreatProcess releases the thread and we're done with DIM. At that point, were still in our parent process, so we continue with any renaming chores, etc.
So yeah, we might be able to pull off the whole thing without breaking stride to deal with DIM manually.
Another thing, instead of pulling copies of IMs intended for install into a user 'install' directory, we could just implement a list using a multiple file dialog, click on all the ones you want to install, it returns a list of file paths, we make our list and feed that to DIM.
As an aside, We could also 'mark' the original bootleg IMs as 'installed' by hyphenating the first byte of the filename, to preclude inadvertently installing them again.
In all, we could maybe get away with no extra-process user interaction at all.
Of course, this is all just mental gymnastics, we'll probably have to deal with some gotchas and limitations that inevitably arise when putting theory into practice, it's just the way.
A command line script would be useful only for windows users. Since you have already chosen windows, you might as well maintain all the legitimate and illegitimate product list in Excel, while doing any of the list generation using Excel VBA scripting. The limited pagination support in Excel could be frustrating while dealing with thousands of files. Also, the list of owned products could be dynamically fetched from the product library page of user's account instead of the user maintaining it manually, especially when they are bulk purchasing those bargain-bin items. You could then feed this list as a input for various batch commands to move/rename files as you planned. Last I checked, the DIM command line options are pretty limited with no support for batch installations. It is understandable since that would require passing not just the SKUs of the products but also the target content/library path shortcut entries as configured in DIM.
A few years back, while exploring Content Sets, I got the motivation to develop my own interface for managing products considering the various limitations of DIM such as:
a) Lack of grouping of products by library path. Unlike the Content Library in Studio there is no way to visualize what products are installed in any given library path in DIM. Since the library paths are key to composing content sets, their limited support makes it hard to visualize or compose content sets in DIM.
b) Moving assets across libraries. The only way to do it is to uninstall from one library location and then reinstall into another. But the lack of library path grouping makes this cumbersome.
When Daz Central was introduced, I recall there were some rumors of supporting plugins, as in Daz Central plugins not Studio plugins. But since that got discontinued, I guess we would have to live with DIM as is. There were a few bespoke installer products that were developed independently, but I couldn't find anything that could meet my requirements. Also, life had different priorities for me.
A command line script would be useful only for windows users
True, this is a Win10+, 64-bit executable. Are you on a Mac? Not sure refactoring in python, java or the like would succeed, limited in low-level and API, but could look into it. Or if you have apple development environment, you'll have Xcode and so, gcc and g++. I could 'sanitize' the code, remove all API stuff, so it's ISO compliant and you could give it a whirl.
...you might as well maintain all the legitimate and illegitimate product list in Excel
Excel might be a possibility if the primary objective was maintaining a product database, but here, we're dealing with product *recognition* within the DIM\downloads and \manifests directories.
As before, instead of 'moving out' illegit files and manifests before updating, we keep them in DIM's \downloads and \manifests folders and just 'mask' them so DIM won't read them. You can 'unmask' them too.
This isn't intended for general purpose installs, *only* use this for installing *illegit* IMxxx files. This isn't for installing anything else, legit or illegit, just IMxxx.zip files you come across on the 'High Seas'.
If you choose to unmask illegit files, you need a way to reinstate that mask when needed. A simple file-list does the job. There's no manual maintenance involved, it's handled transparently within the program. You don't even have to know the list exists.
A small utility like this is just to treat a single security concern, not to maintain overall product installation and for sure, doesn't justify Excel (I'd use sqlite, in a programming venue).
Also, life had different priorities for me.
Now that I'm past a certain 'actuarial age', I look back at what was important to me in decades long past. There were a lot of shifts, some mixed with what's important to me now, some that were laser-focused on achieving singular goals. It's been interesting, sometimes euphoric, sometimes harrowing, occasionally terrifying and wonder-inducing to look back on it all. As they say, 'It's the journey...not the destination'.
- You must be logged in to reply to this topic.
