We present a benchmark pilot study in which high-resolution Full-Field Optical Coherence Tomography (FF-OCT) is used to image human breast tissue and is evaluated to assess its ability to aid the pathologists management of intra-operative diagnoses. Our aim included evaluating the safety of FF-OCT on human tissue and determining the concordance between the images obtained with routinely prepared histopathological material. The compact device used for this study provides a 2 {mu}m-lateral and a 1 {mu}m-axial resolution, and is able to scan a 1.5cm^2 specimen in about 7 minutes. 75 breast specimens obtained from 22 patients have been imaged. Because the contrast in the images is generated by endogenous tissue components, no biological, contrast agents or specimen preparation is required. We characterized the major architectural features and tissue structures of benign breast tissue, including adipocytes, fibrous stroma, lobules and ducts. We subsequently characterized features resulting from their pathological modification and developed a decision tree for diagnosis. Two breast pathologists applied these criteria, resulting in a demonstrable ability to distinguish between normal or benign tissue, in situ and invasive carcinomas using FF-OCT images, with a sensitivity of 97% and 90%, respectively, and specificity of 74% and 77% respectfully. FF-OCT shows great potential for the evaluation of human tissue and its characterization as normal/benign vs. lesional, for numerous ex-vivo clinical use-cases. Its high imaging accuracy for in-situ and invasive carcinoma paves the way for applications where a fast architectural assessment could improve the core needle biopsy workflow, tumor margin assessments, and provides quality assurance for tissue acquired for clinical care and research.