You can easily remove colors from the User Swatches, for example colors which you will no longer use. Proceed as follows: