Many types of cells are able to accurately sense shallow gradients of chemicals across their diameters, allowing the cells to move toward or away from chemical sources. This chemotactic ability relies on the remarkable capacity of cells to infer gradients from particles randomly arriving at cell-surface receptors by diffusion. Whereas the physical limits of concentration sensing by cells have been explored, there is no theory for the physical limits of gradient sensing. Here, we derive such a theory, using as models a perfectly absorbing sphere and a perfectly monitoring sphere, which, respectively, infer gradients from the absorbed surface particle density or the positions of freely diffusing particles inside a spherical volume. We find that the perfectly absorbing sphere is superior to the perfectly monitoring sphere, both for concentration and gradient sensing, because previously observed particles are never remeasured. The superiority of the absorbing sphere helps explain the presence at the surfaces of cells of signal-degrading enzymes, such as PDE for cAMP in Dictyostelium discoideum (Dicty) and BAR1 for mating factor alpha in Saccharomyces cerevisiae (budding yeast). Quantitatively, our theory compares favorably with recent measurements of Dicty moving up a cAMP gradient, suggesting these cells operate near the physical limits of gradient detection.