Physics-based, atom-centered machine learning (ML) representations have transformed atomistic simulations. By incorporating physical priors (e.g. symmetry invariance and many-body interactions), these representations have been successfully used as a basis for sophisticated structure-property analyses and computationally inexpensive machine-learned potentials (MLPs). However, many of these representations are not suitable for coarse-graining, as they assume particles are spherical with isotropic interactions. To include the additional prior of nonspherical particle geometry, we extended one such representation, the Smooth Overlap of Atomic Positions (SOAP), to further incorporate known anisotropy into the representation. This representation, deemed AniSOAP, can then be used to coarse-grain molecules (or sections of molecules) as geometrically accurate beads. AniSOAP is useful for both supervised and unsupervised tasks, such as learning the Gay-Berne potential, learning the energetics of unstable benzene crystals, and delineating liquid crystal phases. We further show that AniSOAP can be used in determining optimal coarse-grained mappings, which coupled with an anisotropic potential, can lead to more accurate and physically-founded CG simulations.