© 2019 The Author(s). Background: The tumor immune microenvironment has become the focus of research in clear cell renal cell carcinoma (ccRCC) due to its important role in immune surveillance post nephrectomy. This study investigates the correlation of tumor infiltrating immune cell characteristics with rates of recurrence following surgery in localized ccRCC. Methods: We morphologically identified and scored tumor infiltrating lymphocytes (TILs) in hematoxylin and eosin (H&E) stained slides of patients with localized ccRCC (stage ≥T1b excluding stage IV). The University of Alabama at Birmingham (UAB) dataset (n = 159) was used to discover and the Fox Chase Cancer Center (FCCC) dataset (n = 198) was used to validate the results of morphologic immune cell analysis. We then performed gene expression analysis using the Immune Profile panel by NanoString in the UAB cohort and identified immune cells and pathways associated with recurrence, followed by validation in the Cancer Genome Atlas (TCGA) ccRCC dataset. Infiltrating immune cell types were identified by gene expression deconvolution. Results: The presence of TILs identified by morphology correlated with higher T cell, Th1, CD8+ T and Treg gene signatures. Recurrence was associated with lower T cells and higher neutrophils. Higher Teffector (Teff)/Treg ratio correlated with lower rate of recurrence and was validated in the TCGA dataset. Genes associated with adaptive immune response were downregulated in tumors that recurred. Unsupervised hierarchical clustering identified a subset of patients with over-expression of adaptive response genes including CD8, CD3, GZMA/B, PRF1, IDO1, CTLA4, PDL1, ICOS and TIGIT. These patients had higher morphologic lymphocyte infiltration and T cell gene expression. Higher levels of TILs identified by morphology correlated with higher rates of recurrence in our discovery dataset but not in our validation set. Conclusions: Recurrence of ccRCC following surgery was associated with lower T cell infiltrate, lower adaptive immune response and higher neutrophil gene expression. Presence of higher Teff/Treg ratio correlated with lower recurrence.