stable diffusion

[Frank21]

I forgot to mention there is this extension that may help with SD identifying different characters.

[Ethiopia]

Hi Frank,
I've been through that site a bit and was wondering if you feel that it's worthwhile joining. I know that there's a HUGE amount of free tuts available but they tend to take a scattershot approach.
Thanks!

[Frank21]

I'm not a member myself but it does seem to have good tutorials. The beginners-guide was very helpful for me when I got started. You do need membership for some of them.

[gts6]

Thanks for that link, thats just what i've been looking for, perfect!
That site definitely is a great one, i've been learning everything so far from there, they do seem to have a pdf version for sale, not sure if it contains EVERYTHING from the site, or if its just the most crucial bits.

[Frank21]

Apologies for resuscitating this thread as I know that for some here mentioning SD is like bringing a ham sandwich at a Jewish (Daz) wedding... so I'll keep it short.
After 12 months of model training, some observations....
1. Training a "look-alike" model as a full model (Dreambooth) on a specific checkpoint produces far superior results and preserves subject identity* compared to training as a LoRa. Slow and produces a huge file but you can't have everything.
* As you're finetuning a base model, rendering it with other base models will generate poor results. Dreambooth works by altering the structure of the base model itself.
2. Kohya Training a 1.5 LoRa with a 768x768 dataset, compared to 512x512, will give much better results, but you're virtually doubling your training time in the process.
True commitment is a hard act to follow...
"Ramen lovers continue to eat noodles as restaurant goes up in flames around them" LOL ...
https://youtu.be/1FQLIEd-8J4?si=Ep6poNak9LknmHFX
Info might be useful for some, I'll go away now.

[gts6]

@frank22 Interesting, i've never tried dreambooth. In what way would you say it is superior to Lora training? More accurate depictions or just better quality color(skin tones,etc)?
I usually can get extremely accurate Loras with a variety of dataset images sizes, I never limit to just one (512x512, etc..) I just use any size and let them be sorted into buckets. And I also do pre-processing by cropping out anything that is not the intended output. For instance if I am doing a Lora of a particular human, I crop each image to show JUST that one person and get rid of all other people and things.

[Frank21]

I do the same with my dataset @Gts6. I used Dreambooth by accident because I forgot to change to the LoRa tab in the Kohya. I just think that the model looked more realistic and doesn't get that over-trained look that's hard to control with LoRas.
Yesterday I used the same datdaet to train a LoRa with the new CyberRealistic and IMO it was nowhere near as accurate. But that may be due to the base model itself.
I've tried using different size images with buckets but generally do the resizing and cropping myself.
I'd really like to try SDXL training but I'm limited to 12GB VRAM😭.

[gts6]

@frank22 the over-trained look could be due to the settings you're using (especially the ratio of repeats/# images/epochs/batches combo) these need to follow a specific way to work in unison with each other (and almost every single "guide" i've seen so far doesnt explain it correctly at all).
But when done correctly, every x number of passes will output both a lora file and a few sample renders. When completed, look at all the rendered images, find the one most to your liking that is not over or under done, and use the outputted lora file that generated those samples (there will be multiple lora files with sequential numbers).

I've oversimplified it here, but generally that gets rid of over/underdone loras problems

I can make more specific details if you dont already know the ins/outs of the settings

[Frank21]

Yes I agree that it's hard to find a clear explanation. With regard to yesterday's failure, I tried out each epoch (5) but none was any better than the final one. They gave my character massive dimples that couldn't be fixed with even strong negative prompting. I think I'll stick with my current base model as it works well for me.
If you're 1.5 training you could give me a link to your configuration file in a PM. That would be interesting to try out. Thanks for your help.

[gts6]

say for instance using 35 image dataset:
what I generally do is use low number of repeats (1-5, 5 is my usual go-to, usually works just fine)
then take that number of repeats (in this case, 5) then multiply that by the number of images (35 * 5 = 175)
Next to calculate the epochs.
The best quality loras usually come about 1500-3500 total steps. We will be using a config that is set to 2 batches, so take the number 7000 and divide by the total above (175)(and round UP if needed). (7000 / 175 = 40) No need to round up 40.
So set the epochs to *40*. Now if you are using a very small number of images that causes the total of epochs to be greater than the max number of epochs allowed, you will need to increase the repeats until the calculation works. I forget the max epochs but I think its like 70-something. You should never really have this problem most likely, unless using very small number of images.(should never really need to use large numbers of images, either. Loras were meant to work with small number of images, so id say no more than 100-150 max, is really pushing it)

Then set the config to generate sample images at every X lora. If it ends up a small number of epochs, id usually generate samples for every 1, but if theres a lot more, like more than 20 or so, then id use maybe every 3-5 epochs or so. Then when the process is finished look in the samples folder and see which looks best, and the epoch number will be in the sample images filename so you know which lora file it matches with.

Keep in mind this will use more drive space since multiple lora files will be created, but you can simply delete the unused lora files later when you pick the best one(s) to keep.

Side note: if you dont use epochs and just set the repeats to 100 with a batch of 1 the final result would be the same (35 images*100 repeats = 3500 steps). The difference is, by using epochs you can gauge progress and get multiple lora files output at x intervals and then use the best looking one. If you just use repeats only, you are stuck with one final output lora and if it sucks you are shit out of luck and need to experiment again and again.

Ill just paste the contents of the config file below, copy and paste that into a text file and save it with a .json extension. For the samples, I left a few prompts in there, you can modify as needed. I left it as a standard prompt for a girl in bikini from multiple sides. So each time a sample per epoch is generated you can get a few of them per lora and from different sides (sometimes rendered samples can be poor, so multiple are better, since if theres 3 angles, 2 might be perfect, 1 is poor screwed up render)

Youll also need to select your desired checkpoint, lora names and folders etc.
This config works fine with RTX3060 12gb.

=============================================================================================

{
"LoRA_type": "Standard",
"LyCORIS_preset": "full",
"adaptive_noise_scale": 0,
"additional_parameters": "",
"block_alphas": "",
"block_dims": "",
"block_lr_zero_threshold": "",
"bucket_no_upscale": true,
"bucket_reso_steps": 64,
"bypass_mode": false,
"cache_latents": true,
"cache_latents_to_disk": false,
"caption_dropout_every_n_epochs": 0.0,
"caption_dropout_rate": 0,
"caption_extension": ".txt",
"clip_skip": 2,
"color_aug": false,
"constrain": 0.0,
"conv_alpha": 1,
"conv_block_alphas": "",
"conv_block_dims": "",
"conv_dim": 1,
"dataset_config": "",
"debiased_estimation_loss": false,
"decompose_both": false,
"dim_from_weights": false,
"dora_wd": false,
"down_lr_weight": "",
"enable_bucket": true,
"epoch": 7,
"factor": -1,
"flip_aug": false,
"fp8_base": false,
"full_bf16": false,
"full_fp16": false,
"gpu_ids": "",
"gradient_accumulation_steps": 1,
"gradient_checkpointing": false,
"keep_tokens": "0",
"learning_rate": 0.0001,
"log_tracker_config": "",
"log_tracker_name": "",
"logging_dir": "",
"lora_network_weights": "",
"lr_scheduler": "constant",
"lr_scheduler_args": "",
"lr_scheduler_num_cycles": "",
"lr_scheduler_power": "",
"lr_warmup": 0,
"max_bucket_reso": 2048,
"max_data_loader_n_workers": "1",
"max_grad_norm": 1,
"max_resolution": "512,512",
"max_timestep": 1000,
"max_token_length": "75",
"max_train_epochs": "",
"max_train_steps": "",
"mem_eff_attn": false,
"mid_lr_weight": "",
"min_bucket_reso": 256,
"min_snr_gamma": 0,
"min_timestep": 0,
"mixed_precision": "bf16",
"model_list": "custom",
"module_dropout": 0,
"multi_gpu": false,
"multires_noise_discount": 0,
"multires_noise_iterations": 0,
"network_alpha": 128,
"network_dim": 128,
"network_dropout": 0,
"noise_offset": 0,
"noise_offset_type": "Original",
"num_cpu_threads_per_process": 2,
"num_machines": 1,
"num_processes": 1,
"optimizer": "AdamW",
"optimizer_args": "",
"output_dir": "",
"output_name": "",
"persistent_data_loader_workers": false,
"pretrained_model_name_or_path": "",
"prior_loss_weight": 1.0,
"random_crop": false,
"rank_dropout": 0,
"rank_dropout_scale": false,
"reg_data_dir": "",
"rescaled": false,
"resume": "",
"sample_every_n_epochs": 1,
"sample_every_n_steps": 0,
"sample_prompts": "masterpiece, best quality, 1girl, in white bikini,high heels, looking at viewer, sitting on bed --n low quality, worst quality, bad anatomy, bad composition, poor, low effort --w 512 --h 512 --d 1 --|7 --s 30|\nmasterpiece, best quality, 1girl, in white bikini, high heels, (full body), (rear view), looking at viewer --n low quality, worst quality, bad anatomy, bad composition, poor, low effort --w 512 --h 512 --d 1 --|7 --s 30|\nmasterpiece, best quality, 1girl, in white bikini, high heels, (full body), (side view), looking at viewer, --n low quality, worst quality, bad anatomy, bad composition, poor, low effort --w 512 --h 512 --d 1 --|7 --s 30|",
"sample_sampler": "euler_a",
"save_every_n_epochs": 1,
"save_every_n_steps": 0,
"save_last_n_steps": 0,
"save_last_n_steps_state": 0,
"save_model_as": "safetensors",
"save_precision": "fp16",
"save_state": false,
"scale_v_pred_loss_like_noise_pred": false,
"scale_weight_norms": 0,
"sdxl": false,
"sdxl_cache_text_encoder_outputs": false,
"sdxl_no_half_vae": true,
"seed": "1234",
"shuffle_caption": false,
"stop_text_encoder_training": 0,
"text_encoder_lr": 5e-05,
"train_batch_size": 2,
"train_data_dir": "",
"train_norm": false,
"train_on_input": true,
"training_comment": "",
"unet_lr": 0.0001,
"unit": 1,
"up_lr_weight": "",
"use_cp": false,
"use_scalar": false,
"use_tucker": false,
"use_wandb": false,
"v2": false,
"v_parameterization": false,
"v_pred_like_loss": 0,
"vae": "",
"vae_batch_size": 0,
"wandb_api_key": "",
"wandb_run_name": "",
"weighted_captions": false,
"xformers": "xformers"
}

[Frank21]

Thanks so much for this. I'll try it today. I generally use a 25-35 image dataset.
At these settings how long does the training usually take? I'm using a 12GB 4070 so I imagine the time would be similar to you.

[gts6]

Usually pretty quick, but it depends on the speed of your PC too, since only certain parts of the process utilize the GPU, the rest is all reliant on the RAM, CPU, etc.

ALSO: you might want to add more sample promtps, I only put 3 in there but generally I like to use around 5 or so, because as I said, sometimes renders come out deformed, etc.. or just generally not really representative of the lora. So a few extra and at least 1 of the samples should be accurate.

[gts6]

ALSO again: as you add more prompts, the time to train increases. Thats why if theres a lot of epochs I only generate samples every 3 or 5,7,10 epochs.

[Frank21]

Hi gts6. I haven't used sample prompts before in Kohya because I thought the image captioning was for that. Does sample prompting make it better? Do you use regularization images for training too?
I now have a 13700K and 32GB of DDR5 6000MHz so I guess it could be quite quick compared to my old PC.
EDIT - I'm getting hugely quicker training and much better it/s on your settings. I'll let you know how it turns out. Cheers.

Back again: great, very fast, good likeness, took 18 minutes (instead of my usual 90). A little less responsive to prompting but I can live with that. Do you think that by using sample prompts it's making it less flexible? Thanks again.

[gts6]

@frank22 No, I don't use regularization images. The sample prompts are the prompts used to generate the sample images that get rendered with every X epochs. So if the lora was of a dog, youd change the sample prompts to show a dog instead of a woman in a bikini. you could also change it to be the triggerword that you used or anything you find relevent to help you test the result of the lora.

As far as the captioning, I dont really know how it comes into play, im skeptical its very relevant at all, to be honest but I just use regular blip with min length of 25, and prefix to add to blip caption I set to the loraname/triggerword

[Author]

Posts