- This topic has 104 replies, 7 voices, and was last updated 2 years, 1 month ago by
.
-
Posts
-
Were you able to get good results in the end?
EDIT: I just noticed you edited and added to your post.
Not sure about your question about if prompts make it less flexible. I dont understand exactly what you mean. The prompts are what renders the sample images that are output in the samples folder. Those are the images that you are to be using to view to see which epoch has the best quality. The contents of those sample images are produced by the prompts you use. If the sample images that end in e25 look perfect,for instance, then you would use the lora that was generated for epoch 25, which would also be named as such.Sorry mate, I get it now that the sample images don't affect the training itself. I saw that there was a sub-folder with the sample images in the Model folder. None of those looked that good but the LoRa worked very well in SD.
Question - your config file shows 7 epochs but in the post above that you say 40 epochs for 35 images.
Because the settings worked so well and quickly I'll do more tests today (it's 7am here).
Really appreciate you sharing your knowledge and tips👍.Yes, the config file needs to be modified (those particular parameters: epochs, samples every x epoch, etc) each time you train a new lora model. So, yes if you are using 35 images *AND* set the repeats in the image folder to 5, then epochs should be 40. (35*5=175 and 7000/175 = 40)
Hi again.
I tried 5 repeats and 40 epochs and the best was #20 so I'm wondering if 40 was too much? The training time was about the same at around 18 minutes. I think the LoRa came out the best one so far.Edit- bloody hell, I wish they didn't make Dreambooth the default tab! Kept getting 15+ hours training time before I realised that I was on the Dreambooth tab ggggggggrrrrrrrr 😠.
Well, technically the process we are using (the calculations to use epochs & samples) will work with dreambooth as well, although it wont be utilizing the config file, it uses its own different setings and will have its own different results.
As for 40 epochs being too much, the answer is YES, sometimes it will be too much. Sometimes it will be just right. We never know, thats why we use a range of epochs. That is the reason for using this technique. Somewhere in the range of epoch 1-40 there will be a sample that looks the best. We never know where it will be. If you are sitting around while its running and see samples that look great and you feel they cannot improve any more, you can always cancel the training and use one that is already complete that looks good.
EDIT: I always usually just let it run to the end, and I do recommend doing that. Sometimes a sample can look great, but not have the look you really want when you start doing real true renders with it. And if you had cancelled the process, you will have to run the entire thing again. So its best to wait until you run real renders with real prompts the way you always intended the lora to look in a scene before you delete all the other lora epoch files. Make sure you got the real best one.
I do let them run to the end, and then I tried them in SD. I set it to only produce a prompt image every 5 epochs assuming there wouldn't be a huge difference in between them. Used 768x768 this time which took about 15 mins longer.
The one I chose was slightly overtrained but looked good when I reduced the LoRa weight a little bit. Deleted the rest as they were just taking up several GB of disk space.
I think I've now got a handle of what I'm doing thanks to you. Thanks, I'll post again if I have questions or run into any problems.@frank22 Sounds great. One last thing, it shouldn't be overtrained. Id try using a lower number epoched lora next time. Because keep in mind the samples are just "rough estimates", because the final renders you do with them will have not only much better prompts, but also different render settings (such as sampler, samples, refiners, CFG, etc.) The samples in the config file are set to just spit out quick and dirty basic Euler A samples. And of course just like any other "real" SD render that is not a sample, sometimes the same exact lora will look bad some times, and then other times it will come out perfect. So samples are the same way and should be just a rough estimate. So if its overtrained, try using a lesser epoch and render the shit out of it with batches of 8 and see if it holds up, and keep working backwards until you hit one that really looks good most of the time you render it.
I used the epiCRealism Natural Sin base model for training this time, and used epiCRealismHelper LoRa in SD at CFG 5. After about 50 test images I'm quite happy with the consistency and accuracy of the results.
I've seen some discussions about high epochs/low repeats and vice versa. Don't know which is the best method. It seems like it doesn't really matter. My dataset was all high quality pictures which I think matters the most in the scheme of things.Yes, the image quality is super important. Epochs do not affect the result AT ALL. Thats part of the misinformation floating around. Epochs are an iteration, basically. It is used to chunk the results and break them out into many results. You'd get the same exact result if you didnt use epochs at all and simply used 100 repeats. The reason to use epochs is if the best quality happens before you reached the 100th repeats. It would spit out 1 lora and you'd be stuck with it and never even realize you had already surpassed the best quality long ago. And with only 1 result lora, theres not a way to go back, unless you lessened the repeats and ran the whole thing again... and HOPED you picked the correct # of repeats.
So, the epochs do not effect the quality of the result at all, its just a technique to find the best number of steps to create the best lora. With just repeats, the steps will be exactly what the repeats were set at * the number of images. And usually that number of steps is not the ideal.
to simplify what I just said about epochs:
if 35 images * 100 repeats = 3500 steps when you finish training you have 1 lora with 3500 step outputif you used epochs, the FINAL lora would equal the same 3500 step output. BUT..
epochs are incrementally appended so if you were using say just 5 epochs
epoch 1 output lora would have 700 steps for instance
epoch 2 output lora would have 1400 steps
epoch 3 output lora would have 2100 steps
etc..so if 3500 steps was too much, if you used epochs you could go back and use an epoch that produced a lora with less steps that looks better.
Its a tool for testing basically, to simplify. They don't affect the creation of the lora details, they just help you find the sweet spot
The thing is, those generated preview images are so crap that I can't tell much about which indicate over training or not. I'm currently getting ones that look good following bad-looking ones which shouldn't happen. I think I'll just have to let it run until finished and dump some of the LoRas into SD to try them out.
Are you still using 768x768 samples? Maybe switch back to 512?
Could also try modifying the prompt. If im training a human im usually doing a full body train, not just face. What I normally do is look for when the body shape is just right in the samples, and then check the skin if its overtraining. Once the skin starts looking not so good, I work backwards from that point down to the range where the shape starts to look correct. Usually the sweet spot is within that range. The sample prompts will NEVER look good and not a true indicator of the final REAL render, they are just good for determining the sweet spot amongst themselves (the samples). Many (most?) times, I am surprised and impressed at a final real result when the sample/epoch I select as the best was underwhelming.Playing around today I made my SD images much better by just lowering the Sample Steps and CFG Scale. CFG from 7 to 5 and steps from 40 to 30. It really cleaned up my images which now look nearly perfect. Never knew it would make such a difference.
Yes, each sampling method seems to have their own sweet spot for the sampling steps. Changing the sampling step number is not a one-size-fits-all though, some sampling methods require higher/lower steps for optimal results. There was a list floating around somewhere of the optimal step settings for each sampling method. CFG also can be higher sometimes for some loras, some work better with cfg 3-5 some work best 7-9 and some work best 11-13. SD is so configurable, everything is really preference since each user knows what looks "best" to their own eyes
- You must be logged in to reply to this topic.
