Information leakage through cache side channels is a serious threat in computer systems. The leak of secret cryptographic keys voids the protections provided by strong cryptography and software virtualization. Past cache side channel defenses focused almost entirely on data caches. Recently, instruction cache based side-channel attacks have been demonstrated to be practical- even in a Cloud Computing environment across two virtual machines. Unlike data caches, instruction caches leak information through secret-dependent execution paths. In this paper, we propose to use a classification matrix to quantitatively characterize the vulnerability of an instruction cache to software side channel attacks. We use this quantitative analysis to answer the open question: can randomized mapping proposed for thwarting data cache side channel attacks secure instruction caches? We further study the performance impact of the randomized mapping approach for the instruction cache.