We propose a method based on quadratic programming for learning control-oriented models of physical systems for which limited data from only one trajectory is available. To this end, we take advantage of the principle of least action from physics. We propose two methods based on quadratic programming to approximate either the Lagrangian or the Hamiltonian of the system from the data. We show how the learning methods based on convex optimization can accommodate symmetries about the underlying system if they are known a priori. Furthermore, we incorporate the error in the approximation to build a data-driven differential inclusion, that is suitable for control purposes. We illustrate the results by an example.