To remove rust this way, you'll need a clean cloth soaked with distilled white vinegar. Use the damp vinegar rag to rub and scrub away any rust or orange streaks too. Once you've sufficiently worked the lids and rings with the rag, use a sponge and warm, soapy water to wash the lids and rings by hand.