Thimbleweed Park Italian fan dub Project - Official Thread(TM)

Sounds like Shane Black is writing for UD…

Real dictionaries do plenty of useless definition stuff too.

rijk → vermogend
vermogend → rijk

(Okay, I just made that up, but it could also be real! :wink: )

1 Like

And here we are, a month (and tons of attempts, at least ten I guess, following the suggestions of the TunaHeads dubbers) later. I uploaded my definitive version of the whole sheriff script. I don’t know if it’s good enough, for sure it’s the best I can get from my voice and acting skills. It’s quite decent, then if no one better shows up (let’s hope one of the seven doubles of the original voice actor is Italian and likes adventure games), it can be used without any concern (IMHO). Down here a sample:

Edit: just uploaded the last session of recording, concluding the whole script for Sceriff, Coroner, Hotel Manager. Good bye. Have a nice evening a-boo!

1 Like

With the Italian fan dub making such good progress, I’d love to give you guys the lip sync you deserve. At some point in the future, Rhubarb Lip Sync will have full support for other languages than English, including Italian. For now, however, I had an idea for a little hack that might improve the animation quality for Italian dialog.

@Guga, @ZakPhoenixMcKracken, @Gffp: If you’re interested, I could send you a preview version of Rhubarb with a rough implementation of my idea. I’d be glad to get feedback on whether it improves the overall animation qualiy for Italian dialog!

7 Likes

Wow, that would be great, thanks @DanielWolf ! :clap: Better to include also @Ema in the list: he worked way more than me behind the scenes to ensure such a good progress!

1 Like

Hi @DanielWolf !!
Thank you very much for your help!
We currently have implemented your lip sync routines, which works quite fine even with Italian sounds, but there’s room for improvements, of course!

You can send us your routines, we’ll be glad to test them and give you feedback and impressions or suggestions as well.

Please include also @Ema which is one of the project leaders. Thanks!!

1 Like

Please tell me you’re talking about adding phonemes to support Italian hand movements, right?
:joy: :wink:

6 Likes

200w

7 Likes

@Sushi: I see where you are going. The problem, however, is in the underlying feature representation. Basing the algorithm on a phonetic analysis falls short of providing the necessary features required to parse the contextual information and supply a reliable mapping to gestures. The only reliable approach is to use deep learning, training a neural network on a corpus of Italian quotes with corresponding hand movements. That istn’t too hard; here’s a rough sketch of the architecture:

After a few days of training, the resulting AI created this output:

135E47VKw6TM6A

It was a bit more work than expected, but worth it. It may be a while though until the new feature is ready to be realeased. Still some rough edges, you know.

6 Likes

I wanted to ask you for one of those, but then substituting Ransome’s head …

1 Like

Yeah… totally unconvincing! I think you’ll need to add three more layers. At least.

3 Likes

Alright, back to business. :grinning:

I’ve created a hacked version of Rhubarb that I’d like you to try. For now, I’m using this forum thread to discuss it. If anybody has a better suggestion, let me know!

First, download the .exe file and use it to replace the original file. If you’re on OS X or Linux, creating a native binary might take a bit.

The basic idea of my hack is this: Normally, Rhubarb tries to recognize whole words (and phrases). Since Rhubarb only knows English, it has a hard time finding English words and phrases that resemble the Italian dialog. That’s why the results are often rather inaccurate.

My hack simply lowers the granularity. Instead of looking for whole English words and phrases, it now looks for English phonemes and syllables. So the underlying language model is still English, but the chances that a given Italian phone is similar to an existing English phone are pretty good. And the chances that a given Italian syllable has a matching English syllable are still not bad.

There is, however, still some fine-tuning to be done. If Rhubarb only worked at the syllable level, there would still be many Italian syllables it couldn’t match. As a result, the animation would look wrong in those places. Worse, the mouth could even stop moving for a moment if Rhubarb really couldn’t find a suitable match.

The obvious solution would be not to work at the syllable level, but only at the phone level. Most Italian phones are also present in the English language. The problem here is fluttering. If the voice actor is saying a long phone that’s exactly between two known English phones, Rhubarb might first recognize phone A, then phone B, then A again and so on, while actually the speaker is still saying the same sound. As a result, the animated mouth might flutter between several shapes during a single phone, which looks quite bad.

The solution, then, is to blend the two approaches. I’ve temporarily added an additional (mandatory) command-line argument modelWeight. If you specify a high value (such as 2.0), Rhubarb will try to recognize whole syllables, leading to imprecise or freezing animation. If you specify a low value (such as 0.1), Rhubarb will try to recognize individual phones, leading to fluttering. I found that the value 1.0 seems to work well, balancing the advantages of both approaches. But I didn’t try any other values between the two extremes. So maybe something like 0.8 or 1.3 could work even better. Also, I only tried the new approach with a short one-minute dialog containing Italian, Spanish, French, and German. Trying it out on a larger body of recordings may give additional insights.

My plan is to settle for a fixed model weight before the release. Then I’ll add a new command-line option to switch between the original, word-based recognition (which looks best for English) and the phonetic recognition (which will hopefully work better for non-English dialog).

Let me know what you think! I’m grateful for any feedback. And if you found a modelWeight value that seems to work better than 1.0, let me know.

4 Likes

Update: The “important-looking pirates of the Italian dubbing project” (ZakPhoenixMcKracken) managed to create a private thread, where we’ll be discussing the technical details of lip sync.

1 Like

Can’t you make it public? I’m interested in the technical details of lip sync too.

1 Like

Wait what, you hacked your own software? :exploding_head:

1 Like

I once heard “rhubarb” is the word they say in the theater in crowd scenes or during whispering or any other situation where you can´t clearly understand what is being said.

1 Like

That’s right. I was actually looking for the clip of the Rubharb rubharb Monty Python cartoon, but eventually came ot with that one.

2 Likes

I end up always thinking of this:

2 Likes