To delete anything from a palette, without getting a dialog like this example asking you to confirm your choice, hold down the Option key (PC: Alt key) as you click on the palette's Trash icon. This applies to brushes, swatches, layers, etc.