Open masks are paths that can be used for several purposes.
They can be used for effects that employ masks like Path Text, Audio Waveform, Audio Spectrum, Stroke, or Vegas.Open masks can be used for effects that have positional attributes like Lens Flare, Spherize, most distort effects.
And open masks can also be used as motion paths.
Most of the time, you'll want to make sure your masks are closed.
To make sure your mask is closed, select a mask and choose Layer>Mask>Closed.
You can also select your mask in the composition window and Context click (Right click or Control click) to make sure "closed" has a checkmark by it.