Cache side channel attacks are attacks that leak secret information through physical implementation of cryptographic operations, nullifying cryptographic protection. Recently, these attacks have received great interest. Previous research found that software countermeasures alone are not enough to defend against cache side channel attacks. Secure cache designs can thwart the root causes of cache side channels and are more efficient. For instance, Newcache is a cache design that can enhance security, performance and power efficiency simultaneously through dynamic memory-cache remapping and eviction randomization. However, these cache designs seldom had their security verified experimentally by mounting cache side channel attacks on them. In this paper, we test the security of Newcache using representative classes of cache side channel attacks proposed for conventional set-Associative caches. The results show that Newcache can defeat all these attacks. However, what if a very knowledgeable attacker crafted the attack strategy targeting the secure cache's design? We redesign the attacks specifically for Newcache. The results show that Newcache can defeat even crafted access-driven attacks specifically targeted at it but sometimes succumbs to the specifically crafted timing attacks, which is due to a very subtle vulnerability in its replacement algorithm. We further secure Newcache by modifying its replacement algorithm slightly, thus defeating these specifically crafted timing attacks. In addition, the improved Newcache simplifies the replacement algorithm in the original Newcache design.