Every mask in After Effects has a mode. Select your layer and hit M on your keyboard to show masks. The Mask mode is visible in the switches column.
The default mask mode is Add, it means the new mask combines with the previous mask on the layer.
Masks work based on their placement in the timeline so the subtract will subtract the selected area from all masks above it.
Intersect adds the mask to all masks above it but will only display the area at which all masks intersect.
Difference adds a mask to all masks above it and displays all masked areas except the areas where masks intersect.
Use None when using a mask-based effect. A mask set to None will not affect the transparency of a layer.