Back to Blog

The Power of Rotobrush 2 in After Effects

By Zeke French
After Effects

Worried about rotoscoping? Don’t even know what that means? Let’s go over the new Adobe update so you can level-up your vfx game.

If you’re looking at working on visual effects, you’ll need to learn how to separate and composite footage and images. One of the first steps to this is learning the time-consuming technique known as “rotoscoping”!
mkt3118-zeke french roto -article-20210202.jpg
The task of rotoscoping is fairly simple, but it takes some time. I’m Zeke French, a content creator, editor, and long-time After Effects user.
I’ll walk you through the basics of rotoscoping as well as some common mistakes you might make when first starting out. Then we're going to look at the powerful update to After Effects with Rotobrush 2. Here’s what you can expect from this tutorial:
  • A brief look at what rotoscoping is
  • Why you would want to use rotoscoping
  • How to use the rotoscoping tools After Effects provides
  • How to creatively use your rotoscoped assets

Project Files

Make sure to snag project files that will help you get the most out of this lesson.

Grab the project files so you can follow along!

Download Now

The Power of Rotobrush 2 in After Effects

What is Rotoscoping?

Rotoscoping began as a practice in the 1900s. Artists would trace real footage as a direct reference for their animation. It's how so many of the early animated shorts and features included such realistic movement for human and humanoid characters.
Animation so good, it's spooky. (Betty Boop: Snow White, 1933)
In modern times, rotoscoping is a tool for motion designers and VFX artists that covers a wide range of effects. In simple terms, rotoscoping isolates assets so they can be easily manipulated - it's like a manual green screen.
Artists are able to use a number of programs to achieve this effect, but we'll be focused on Adobe After Effects. Understanding this tool will allow you to properly isolate and composite images in order to enhance your videos, as well as opening up the options for a number of slick effects that help you stand out.

Why should you learn rotoscoping?

With rotoscoping, you can apply an effect only to one specific object, or to everything except a specific object. This allows us to draw the viewer's eye using blurs, glows, and a huge number of other adjustments... both simple and complex.
Once you isolate your asset, you can add all sorts of fun effects.
Rotoscoping is a tool you can use throughout your career. Whether you work with simple designs or do complex VFX for feature films, you will learn to love the rotobrush. New motioneers tend to be a little shy diving into this skill, as they've no doubt heard a few horror stories.
The truth is that it takes practice, but it is a superpower waiting to be unleashed. With a little effort, you can quickly:
  • Gain control of the composition's alpha layers and transparency
  • Isolate objects to then apply visual effects
  • Move objects within a scene, or remove them entirely
  • Place new items around or behind the key object
All of this allows you to deploy the principles of design to guide your audience's attention and focus wherever you would like them to go. So how exactly do you that?

How do you use the rotoscoping tools in After Effects?

In After Effects, there are a couple of ways to rotoscope. The most common is the tried and true method of applying a mask.

The Mask Tool

To begin, you simply grab your Mask tool, select the object, refine, and isolate. This works fine for simple objects (such as the ball above), but becomes far more complex and time-consuming with more detailed objects (such as the car we'll do next).
Once you keyframe the mask, you'll have to manually adjust to your object as it moves across the screen. The results will be good, but it will take more of your time and energy.
Up until the most recent updates, this was the primary way to rotoscope in After Effects. It was consistent and effective, but took patience. However, with the new update came the Rotobrush 2 tool...and it has completely changed my workflow for this task.

Rotobrush 2

The new Rotobrush 2 takes away a lot of the manual work, saving you a ton of time. However, it might not be as consistent and won't be great for every context. You'll have to experiment to find the best balance of works for you.
So how do we use it? First, select the Rotobrush Tool from the bar at the top of the screen. Also, make sure your composition frame rate is the same as your footage. That will save you a lot of frustration down the road.
Size your brush up or down so you can more effectively select the object.
Paint over the object and After Effects will automatically select it and highlight with a purple edge. Then you can hold SHIFT and continue to paint to refine the selection, or hold ALT and paint to remove areas you don't want.
Depending on how you'll be using the object, you can get more or less detailed. For our purposes here, I can feather the edges and achieve the desired effect.
Next you want to click the drop-down menu next to Quality and select Best. You'll now see a green frame at the bottom of the screen—your workspace for the clip. Press Spacebar and the program will propagate forward, tracking the object.
You can see an artifact on the left side of the ball, but that are easy to clean up.
The program tracks the ball with barely any input from you, using the guidance from the original selection to continue forward frame by frame. Now we click Freeze in the bottom right, which will cache our analyzed frames.
You'll notice that your timeline at the bottom has turned a purple color to indicate that those frames are cached. Now you can adjust your Matte however you need, to perfect the selection and dial in for the next steps.
Of course, if you nailed it on the first try, you can just bask in your excellence for this step.
With this element isolated, I can apply effects to only my selected layer to create a more dramatic image. For example, if I use Find Edges...
find edges.gif
Now let's take a look at a more complicated object. We want to select this car, so we can apply effects when it collides with another car in the video. A simple mask won't work here, so let's use a combination to achieve the desired effect.
selection car.gif
We select Rotobrush 2, paint the middle of the object, and then refine our selection until we're satisfied. Again, we change Quality to Best, press Spacebar, and watch After Effects take the wheel.
Uh oh, did an AI just blow your mind?
Click Freeze to cache your frames, and take a moment to marvel at how this easy that was. Anyone who's been in the industry has a knee-jerk reaction toward rotoscoping ... but it doesn't have to be a painful experience. In fact, with Rotobrush 2, it can be pretty fun.
Now, this isn't without drawbacks. With more complex objects, the edges can sometimes be a little janky, or the tool might pick up on objects in the background. Use Clear Chatter and manually drop unwanted areas and you'll be on your way.
So now that we have our car separated from the rest of the footage, what do we want to do?

Getting creative with Rotobrush 2 in After Effects

What you do next is up to you, and it couldn't be easier. I liked how Find Edges looked, so let's try that.
Add a glow, throw on some crazy colors, or drop a few effects between the car and the background. You can do anything now that you've isolated the object...and it took you, what, five minutes?
With this skill, you can add all kinds of amazing effects to your work (or your client's work) with ease.

Now you know the whole (roto) scope of this invaluable technique

There you have it, by understanding and implementing these pretty basic techniques, we’re given the ability to produce some pretty awesome things. We covered the function of rotoscoping, some practical ways of going about it using the new rotobrush tool, and how easy it is to apply some creative effects after we’ve isolated our layers. Now take what you've learned and bring your next project to a whole new level.

Put your visual effects in motion

Also, make sure to check out VFX for Motion from School of Motion. Instructor Mark Christiansen will teach you the art and science of compositing as it applies to Motion Design. Prepare to add keying, roto, tracking, match-moving, and more to your creative toolkit.