Networks of spiking neurons in which spikes are fixed as a Poisson process are studied. The state of a cell is determined by the instantaneous firing rate, and in the limit of high firing rates the model reduces to that studied by Hopfield. The inclusion of spiking results in several new features, such as a noise-induced asymmetry between on and off states, and probability currents which destroy the usual description of network dynamics in terms of energy surfaces. Taking account of spikes also allows calibrating of network parameters such as synaptic weights against experiments on real synapses. Realistic forms of the post synaptic response alter the network dynamics, which suggests a novel dynamical learning mechanism.