We present a rigorous physical description of an astrometric observation, deriving expansions for the observation vector in terms of defined small quantities. We use these derivations to formulate an extended Kalman filters for the purpose of characterizing planetary orbits from astrometric and radial velocity data. We show results from an initial implementation of this filters with simulated data sets of observations on single and multi-planet systems. It is found that the filters is always able to closely estimate the state of the largest planet in the system and accurately estimate the variance of the true error of the state estimate. We find that this technique can be used to constrain planetary masses to at least within ±50% of the true mass for Earth-mass planets.