Towards note-level prediction for networked music performance

The Internet allows musicians and other artists to collaborate remotely. However, network latency presents a fundamental challenge for remote collaborators who need to coordinate and respond to each other’s performance in real time. In this paper, we investigate the viability of predicting percussion hits before they have occurred, so that information about the predicted drum hit can be sent over a network, and the sound can be synthesized at a receiver’s location at approximately the same moment the hit occurs at the sender’s location. Such a system would allow two percussionists to play in perfect synchrony despite the delays caused by computer networks. To investigate the feasibility of such an approach, we record vibraphone mallet strikes with a high-speed camera and track the mallet head position. We show that 30 ms before the strike occurs, it is possible to predict strike time and velocity with acceptable accuracy. Our method fits a second-order polynomial to the data to produce a strike-time prediction that is within 10 ms of the actual strike, and a velocity estimate that will enable the sound pressure level of the synthesized strike to be accurate within 3 dB.