The rapid growth of single-cell transcriptomic technology has produced an increasing number of datasets for both embryonic development and in vitro pluripotent stem cell derived models. This avalanche of data about pluripotency and the process of lineage specification has meant it has become increasingly difficult to define specific cell types or states and compare these to in vitro differentiation. Here we utilize a set of deep learning (DL) tools to integrate and classify multiple datasets. This allows for the definition of both mouse and human embryo cell types, lineages and states, thereby maximising the information one can garner from these precious experimental resources. Our approaches are built on recent initiatives for large scale human organ atlases, but here we focus on the difficult to obtain and process material that spans early mouse, and in particular, human development. Using publicly available data for these stages, we test different deep learning approaches and develop both a model to classify cell types in an unbiased fashion and define the set of genes required to identify lineages, cell types and states. We have used our predictions to probe pluripotent stem cell models for both mouse and human development, showcasing the importance of this resource as a dynamic reference for early embryogenesis.