Diverse immune cells in the tumor microenvironment form a complex ecosystem, but our knowledge of their heterogeneity and dynamics within hepatocellular carcinoma (HCC) still remains limited. To assess the plasticity and phenotypes of immune cells within HBV/HCV-related HCC microenvironment at single-cell level, we performed single-cell RNA sequencing on 41,698 immune cells from seven pairs of HBV/HCV-related HCC tumors and non-tumor liver tissues. We combined bio-informatic analyses, flow cytometry, and multiplex immunohistochemistry to assess the heterogeneity of different immune cell subsets in functional characteristics, transcriptional regulation, phenotypic switching, and interactions. We identified 29 immune cell subsets of myeloid cells, NK cells, and lymphocytes with unique transcriptomic profiles in HCC. A highly complex immunological network was shaped by diverse immune cell subsets that can transit among different states and mutually interact. Notably, we identified a subset of M2 macrophage with high expression of CCL18 and transcription factor CREM that was enriched in advanced HCC patients, and potentially participated in tumor progression. We also detected a new subset of activated CD8+ T cells highly expressing XCL1 that correlated with better patient survival rates. Meanwhile, distinct transcriptomic signatures, cytotoxic phenotypes, and evolution trajectory of effector CD8+ T cells from early-stage to advanced HCC were also identified. Our study provides insight into the immune microenvironment in HBV/HCV-related HCC and highlights novel macrophage and T-cell subsets that could be further exploited in future immunotherapy.