Skip to content

March 10th through March 16th

The end of spring quarter! Finals are upon us this coming week, projects are due, and spring break is soon to come. Hopefully it will warm up a little and the snow will finally end, but we shall see…

This week, I needed to work on basic 1.5 – 2.0 second animations featuring Paula shifting from a neutral expression, to either surprise/yes or no/pah expression. I managed to create some basic first drafts, but the results were embarrassing; I have much work to do. I will post again once I have completed more satisfactory work. Thankfully, Dr. Wolfe supplied me with a chunk of better reference videos, and now that our ELAN corpus is nearly spotless, I can search that for great examples!

At our meeting, we discussed the importance of a realistic head turn, and a new script controller to work with in 3DS Max to test the dimensions of a good turn. Other members of the team and I will be testing it sporadically and report our results to Dr. Wolfe and Dr. McDonald as important research toward attaining natural movement.

The other section of our meeting concerned Dr. Schnepp discussing SignQuote and editing the test, important for any of us if we need to work on our upcoming survey to gauge the quality and effectiveness of our avatar. Great things are coming up in 2014!

Advertisements

March 2nd to March 9th

No new beautiful images to show, but I will recap what happened this week and what will be coming next week!

This week, we shot video footage of an interpreter for both a survey/research study we will be conducting with native ASL speakers, and for motion study. We hope to fluidly animate things such as fingerspelling in the future, along with role shift so that it no longer looks as stiff, robotic, and awkward. There are a lot of amazing things to look forward to with this research group!

I will be finishing some small animations of “pah”, which will show a *very* slight shift in the brow upward; a pleasant surprise; then finally, a quizzical “hmm?”. These animations won’t last longer than a second and a half at most, but will demonstrate how much the eyebrows are important for affect. It will show the important subtlety of using the brows as a way to impart inflection: both if the change is very slight, or if it is very obvious.

Clips should be posted in the next update! Cheers!

A New Year: New Discoveries, More Progress

Image

Hello, 2014! This is a long-belated post, but it will cover the past few weeks in review concerning progress with our favorite ASL avatar, Paula!

January 12th – 18th, 2014

The start of a new quarter with a new target: rather than pump out more unique expressions, let’s polish up the ones we currently have. The main focus on tidying up the existing expressions is to fix the eyebrows. With the way the brows are programmed, red spots “pop” suddenly through the skin texture when the brow assumes an extreme position. We will be exploring how exactly these brows are programmed later in the quarter. For now, it will be simple, manual tweaking to adjust issues, such as with this image (spot the reduced red spot by the brow!):

Image

 

January 19th – 25th, 2014:

Another new plot is in the works: working on visemes! While an ASL speaker does not literally speak words with their mouth, they use facial expressions that mimic lip, brow, eye gaze, etc positions as if they were speaking out loud. These visemes punctuate their sentences and add life to their words in a subtle way, while affect/expression adds it in a more obvious matter. As it stands, our current avatar has trouble with the viseme “oo” in particular. The way the lips pucker and wrinkle is not as optimal as we’d like to be. For the next few weeks, I hope to work on “oo” and other visemes. January 26th – February 1st: Firstly, happy Lunar New Year! My family celebrates Lunar New Year, as we have Chinese blood, so I wish everyone a prosperous Year of the Metal Horse. Onto the important ASL information – I took a crack at forming “oo” and “uh” this week. They aren’t that pretty, but I am still getting a grasp at forming visemes with the expression builder. Visemes are very subtle, for the most part, so accurately portraying them without going overboard is hard. And again, concerning “oo” – Paula has a difficult time making that look natural, and we hope to fix that in the future if possible.

Image“Oo”Image“Uh”

 

February 2nd – February 8th:

The work on visemes continues! I found a handy visual aide for common vowel-based visemes and have been using it to my advantage. This week, I worked on “AH”, “AW”, and three kinds of “OO” – soft, medium, and hard, since every time we say “OO” it isn’t like a cartoon trying to make a whistle or puckering up for a kiss. Forming OO in particular has continued to be a challenge, but I think I have gotten closer to our goal by breaking it into three different intensities. I have also begun to take renders in the side view to check how her mouth looks from other angles. Due to the way the “narrow”/pucker function of her lips works, she gains an overbite or underbite depending on how the toggles are pulled. I’ve been working on avoiding this in the future, mostly by manual manipulation. Here are the three “OO”s to demonstrate. Can you figure out which one is soft, which one is medium, and which one is hard?

ImageImageImageImageImageImage

 

February 9th – 15th:

More viseme work, and we have learned about the way brows are controlled. To control the brows, a gradient ramp is applied to a texture that controls the wrinkles. The gradient ramp works like a “mask” does in Photoshop – if the portion of the gradient is black, the texture underneath is 100% visible. If the gradient is white, the texture underneath is 100% invisible. By using this gradient, we can control what portion of the wrinkles will show at what time in a natural way. The controls are hitched onto the points of articulation on the brow bone, so depending on the position of those joints, more or less of the wrinkles will show. Genius, isn’t it? This is all done is MAXScript, too! We have code to check out. It’s a little mind-boggling to me, but I hope to look into it more in the future. For now, take a look at “AH” and “AW” tweaked!

ImageImageImageImage

 

February 16th – February 22nd:

A belated Happy Valentines Day to all! I hope everyone had a good holiday, or at least managed to snag some delicious discount chocolate after the fact. I have moved from vowels to consonants… almost. Currently, we have “EH” and “CH” on the roster. I soon hope to work on “W” and “PAH”, especially the latter because the slight shift of the eyebrows is incredibly important, regardless of how soft and subtle the motion is.

ImageImageImageImage

 

February 23rd – March 1st: It’s almost March! Let’s hope the snow will stop soon, or at least that we get a break from such bleak temperatures and awful wind. I took a short break from visemes to work on very small affects for Dr. Rosalee Wolfe’s upcoming work for a workshop on the first week of April. I still have to tweak the “surprised” eyebrows some, so for now, feel free to look at my puzzled brows (first image on this post at the top!) and irritated brows (right here).

Image

Stay tuned for further progress on visemes and brows! Hopefully we can begin to work on lip wrinkles and facial textures as well as the snow begins to melt.

Weeks 8 to 10, and into December 2013

Wow, long time, no post! Research has been very interesting – just lots of schoolwork and outside influences have me forget to actually upload my pre-written posts. I have edited them here to make them more concise.

Expression work has gone well. For some reason, one of the markers on the lips – well, two, one for the middle of the top and bottom lips – sticks in place whenever the lips move up and down. This was noticeable in older versions of expressions. Manual tweaking has fixed these issues, but hopefully cracking into the code can alleviate needing to manually tweak these spots.

Image

The teeth are also a very bright shade of white. For now, photoshop will have to do the work to hide that. Pearly whites are on the bottom of my to-do list, as…

Image

While it is fun to mess with the eyebrows and push them to extremes, this creates a huge stress on the texture map, revealing large blotches that do not move smoothly or look pretty when in actual use. I had to go back to some of my older expressions and loosen up the brows a bit – make them less theatrical – to eliminate a large amount of the glitchy map. I am learning where exactly the limits are on the bones to see when these maps falter. This affects both up/down motion and “scrunching” motion. If the brow is too close to the bridge of Paula’s nose, the red pops through the skin again. Small tweaks help avoid this.

Image

The focus has shifted from pure expression work to working on visemes. Visemes are the speech sounds that look the same when spoken, such as “pet”, “bell”, and “men”. These lip motions are difficult for lip readers to distinguish. There are subtleties with how the tongue may move in the mouth or how the teeth are utilized to make the actual sounds, but since the lips are in the way, there is no way to distinguish these mouth movements without being hearing capable.

ImageImage

(“UH” Original / “UH” tweaked)

From what I understand, visemes play a peculiar part in ASL: revealing sub-Sign units of activity. This means that these visemes can punctuate certain signs, perhaps inflect further types of stress or emotion, and add to the overall spirit of what is being signed.

Currently, our avatar has a hard time working with visemes that involve puckering the lips especially. This is because of how the joints are set up and how the texture map works on the lips. In real life, wrinkles appear on our skin as we pucker and purse our lips outward to make visemes like “oo”. Paula currently has a hard time with this, so my job right now is to attempt to tackle the dreaded “oo” to make it more naturalistic. So far, with our current set-up, this is quite a challenge. Hopefully in the future, I can work off of the crude creations I have made so far after (possibly) working on a new texture map for the lips to create better wrinkles.

ImageImage

(“OO” Original / “OO” Tweaking – note the poor wrinkling!)

I hope I can work on these visemes, create better maps, and try to fix these glitches to make Paula more naturalistic. 2014 will hopefully be very productive!

SLTAT 2013 – Weeks 5 through 7

This quarter has been a very exciting one, especially regarding research and development. I have thoroughly used and abused the program we used to create the expressions – a simple series of buttons and sliders to adjust the points of articulation and bones in the face. Unfortunately, this set up did not take into account the way that the human face does, in fact, push tendons and skin to the limit for even basic emotions.

Image 

For instance, when attempting to create a jaw drop, or in my case, a “happy” surprise versus a fearful one, Paula has issues baring her teeth – and if the jaw drops too low, the center of her upper and lower lip stick to her teeth. Her eyebrows also have a limit, as do the corners of her lips. Furthermore, her eyeballs are actually poking out of the sockets a bit – that’s what those dimples are near the puffy parts of her eyes! It sets a challenging standard for testing this software. I have pushed beyond its boundaries, and this is something that has been brought up in…

Image

SLTAT 2013! A neat symposium hosted this year at DePaul to check out the current work from many different groups around the world, all seeking to find a way to synthesize signed language.

This happened October 18th – 19th with about 30 or so people showing up to reveal their current work. To be honest, a lot of the technical details went over my head, but the depth of research is startling. Me, I just enjoy playing with the human face and finding what exactly creates uncanny valley vs. a naturalistic avatar. Discovering the nuances of coarticulation is a whole new realm of territory that I am nervous about, but excited to dip my toe into.

My partner, Marie Stumbo, put together a video and mounted her tablet onto our poster for our own presentation of our work – optimizing Paula, our avatar, for real-time rendering capabilities. I created a couple videos, which are posted below in three separate posts and linked on Vimeo, to demonstrate what I have done to help with that and now how making expressions without morph targets is absolutely possible.

The lighting rig was a piece of work I enjoyed playing with earlier in the summer. Simply put, lighting is taxing on the processor, and definitely forces rendering time to suffer. Combine this with a high polygon count and you may be sitting and waiting for something to render for hours or days! In our case, the original rig was a set of six symmetric lights that left Paula neutrally lit and with a shadow down the bridge of her nose, which almost created a line of symmetry.

I couldn’t do the traditional three point lighting scheme due to the lack of true ambient light in a computer setting; this caused Paula to have dark shadows on her face, which is too dramatic for our purposes. Instead, I created a four point lighting scheme and adjusted the glow and bloom of the one spot light I used in order to bring more saturation and color into her face, and also to remove the shadow going down the bridge of her nose. I’d like to think the results were pretty decent!

In my opinion, the expressions video was more interesting to look at. 3DS Max gave me issues with putting holds/pauses between most of the expressions until the last two or three at the end, but the results still surprised and delighted the colleagues who looked at the video. Each person was impressed by the results of a bone and articulation-based system instead of pure morph targets.

However, every single person brought up a point: the eyebrows. If you look at the video, you can notice how red/dark shading near her eye sockets and around her brows seems to jump sporadically. This is an issue with the texture map coded on the skin. As far as I understand, and pardon me if this explanation is lacking or a bit incorrect, but the map is based on opacity. It will show more or less depending on the push or pull of the brow bones.

Because I had pushed the expressions to the extreme (for the software), the texture map became confused and jumped, creating an ugly look that needs to be sorted out. That said, that is my next point of research and development: re-programming and playing with the texture map for the brows, so more fluid motion and transition can be achieved! I also hope to play with the eyes a bit to fix the issues of the corners popping out near her lower eyelids.

I am very excited for my upcoming work with the texture mapping. I hope this will allow for an even wider range of emotions that are more naturalistic – I love what I do, and I am always excited to improve upon what I have done.

Video

SLTAT 2013 – New Lighting Demo

https://vimeo.com/77993485

Here is a video I put together to demonstrate the new lighting rig I put together over the summer and into this current quarter. A more thorough explanation and report will be given in the post “SLTAT 2013 – Weeks 5 through 7”.

Video

SLTAT 2013 – Original Lighting Demo

https://vimeo.com/77993484

Here is a video I put together of the original lighting rig used for Paula that needed some updating. A more thorough explanation and report will be given in the post “SLTAT 2013 – Weeks 5 through 7”.