Anilip2 singing?

[Abad]

For a change, another "weird" question...
I'm using ".vsqx" files from the Vocaloid4 software (a kind of synthetic singer...), which are incompatible with any software, except "MikuMikuDance" (converted to its old .vsq format, from Voaloid2) or a DAW like Reaper, Ablenton, Audition, etc... It has a virtue: It generates .wav files with the voice "a cappella".
If I use it in Anilip2, after "normalizing" the audio, apart from giving messages like "the sound is too loud, or too low to be recognized", it does not keep the morph of the phoneme/viseme when it is lengthened in time, for example , does not distinguish (sung aurally...), for example, the word "love" from the "LOVEEEEEeeeeeee...". It lacks expressiveness... There are some sustain and vibrato morphs, but they are "by hand" and I hardly see any difference if I manipulate them.
Anilip generates a .duf of "pose present", selecting only "visemes" in "pose controls" (Everything else is not important...) of about 200Kb for a 3 minute song, which is great for loading time.
On the other hand, the boring "PoseRecorder", which I have achieved very stable things, (never with a video recorded live, but with a face of a "Genesis 8 Female base", rendered in frontal view with Blender with the addons " diffeomorph daz importer" and one called "VMDRetargeter" which works very well for facials...), but do a quick render, (we're talking about 5000/7000 frames...), pass it to PoseRecorder, which takes an hour or more, and it generates a duf of 15/20Mb, which I have to "clean garbage" with MotionBuilder to leave it at 1Mb (uncompressed...) once passed to duf...
I know that Anilip is intended for spoken voice from a TSS engine for audio files.
Does anyone know how to fix this cryptic issue?

[Legolas18]

I don't think that anilip has been made for such use. As far as I understand, anilip is mostly for speech, and works better with some written text, which could be easier to manipulate so that you get the wanted result. Otherwise, you might need to manipulate the timeline manually adding keyframes with facial expressions when needed.

An easier process (and this is just an idea) would be to use an iphone with facial capture software such as Dynamixyz’ Performer2 (which you posted on the blog) or some other software such as Faceware, or iClone / Character creator 4 (with which it's easy to convert daz character into). However, once you go iClone, it's better to move to Blender or Unreal after.

[Abad]

@legolas18 I know... but all those softwares need a video (A real or virtual person...). Here there is only audio. Here I have uploaded a comparison video between the vmd of MikuMikuDance, blender, pose recorder from the blender video and anilip2 in daz... (The latter desynchronizes approximately at minute 0:54... I don't know why..., apart , I have only used one of the 4 voice channels of the vsqx..) If you search for "MMD Raspberry Monster" in video you will see the whole movement...

[Grouchy Old Fart]

Even with just the audio as a start, couldn't you use your own face, and with the selfie camera lipsync to the music to create the mocap?

[Legolas18]

Hmm, I think the best way to do this is to start an "automatic" lip sync, followed by some manual adjustments (if needed) of shape keys animation in Blender. I would not consider Daz for such things, I think it's very clunky for animations.

For "auto" lip synch, there is a free plugin called Rhubarb

For "manual" lip animation, Syncnix makes things a lot easier.

And, since you want to get vocaloid singing:

[Abad]

@pooper I sing badly, I don't know Japanese, and I have a beard...
To top it off, I haven't used a cell phone in about 10 years: I hate them!!. 😀

@legolas18 I'll try them, thanks. I was looking at one of Emotech, the LipsyncAI.

[Legolas18]

LipsyncAI seems interesting, although it seems to be geared more towards Unreal. Hopefully, in the future there will be more programs like this available for other platforms.

[Author]

Posts