RotoBezier vertexes visually resemble Bezier vertexes because they are curved but RotoBezier vertexes differ in that they don't have handles you can access and manipulate. The RotoBezier option automatically calculates curved segments for you.
You may find it easier to draw a mask, and then convert it to RotoBezier afterward. There might be a shift in shape if you do so however. You can also change a RotoBezier mask into a Bezier mask, which will not cause any change to the mask shape. RotoBezier vertexes are easier to use while rotoscoping because you don't have to worry about direction handles.
To draw a RotoBezier mask, click the Pen Tool from the Tools Palette and click RotoBezier. Click and draw in the composition window.
To convert a mask to a RotoBezier Mask, select the mask in the layer or composition window and choose>Layer>Mask>RotoBezier. You could also select the mask and context click for the flyout menu; choose the RotoBezier option from there.