Inhibitors of the PD-1-PD-L1 axis have been approved as therapy for many human cancers. In spite of the evidence for their widespread clinical activity, little is known about the immunological alterations that occur in human cancer tissue after PD-1 blockade. We developed and employed a patient-derived tumor fragment platform to dissect the early immunological response of human tumor tissue to ex vivo PD-1 blockade. We observed that the capacity of immune cells to be reactivated ex vivo was predictive of clinical response, and perturbation analyses identified tumor-resident T cells as a key component of this immunological response. In addition, through combined analysis of baseline properties and immune response capacity, we identified a new subgroup of infiltrated tumors that lacks the capacity to respond to PD-1 blockade. Finally, the baseline presence of tertiary lymphoid structures and their components correlated with the capacity of cancers to undergo intratumoral immune cell reactivation.