Side-channels enable attackers to break a cipher by exploiting observable information from the cipher program's execution to infer its secret key. While some defenses have been proposed to protect information leakage due to certain side channels, the effectiveness of these defenses have mostly been given only qualitative analysis by their authors. It is desirable to have a general quantitative method and metric to evaluate a system's vulnerability to side-channel attacks. In this paper, we define the features of a good side-channel leakage metric. We review a recently proposed metric called the Side-channel Vulnerability Factor (SVF) and discuss its merits and issues. We suggest the CSV metric, which tries to show how to overcome some of the shortcomings of the SVF metric, without completely changing its character. We use software cache side-channel attacks and defenses as an example to compare the metrics with known and measurable results on system leakiness.