Princeton University is building an integral field spectrograph (IFS), the Coronagraphic High Angular Resolution Imaging Spectrograph (CHARIS), for integration with the Subaru Coronagraphic Extreme Adaptive Optics (SCExAO) system and the AO188 adaptive optics system on the Subaru telescope. CHARIS and SCExAO will measure spectra of hot, young Jovian planets in a coronagraphic image across J, H, and K bands down to an 80 milliarcsecond inner working angle. SCExAO's coronagraphs and wavefront control system will make it possible to detect companions five orders of magnitude dimmer than their parent star. However, quasi-static speckles in the image contaminate the signal from the planet. In an IFS this also causes uncertainty in the spectra due to diffractive cross-contamination, commonly referred to as crosstalk. Post-processing techniques can subtract these speckles, but they can potentially skew spectral measurements, become less effective at small angular separation, and at best can only reduce the crosstalk down to the photon noise limit of the contaminating signal. CHARIS will address crosstalk effects of a high contrast image through hardware design, which drives the optical and mechanical design of the assembly. The work presented here sheds light on the optical and mechanical considerations taken in designing the IFS to provide high signal-to-noise spectra in a coronagraphic image from and extreme adaptive optics image. The design considerations and lessons learned are directly applicable to future exoplanet instrumentation for extremely large telescopes and space observatories capable of detecting rocky planets in the habitable zone.