Immune checkpoint blockade (ICB) unleashes immune cells to attack tumors, thereby inducing durable clinical responses in many cancer types. The number of patients responding to ICB is modest, however, and combination treatments are likely needed to overcome the multifaceted suppressive pathways active in the tumor microenvironment (TME). The development of precision immuno-oncology (IO) strategies allowing to identify the optimal treatment of each patient upfront is therefore a pivotal question in the field of cancer immunotherapy. Although single-parameter biomarkers can enrich for response to ICB, their predictive capacity is far from perfect and their clinical utility is complicated by their continuous nature and the difficulty to determine cut-offs that reliably distinguish responding patients from those without clinical benefit. The antitumor immune response that is induced or reinvigorated by immunotherapy is a complex cascade of events requiring the interplay of multiple cell types. To move towards precision IO, it is therefore essential to understand for each individual patient at which level(s) the antitumor immune response failed and how it can be therapeutically restored. Holistic approaches to profile human tumor microenvironments and treatment-induced responses may help to identify critical rate-limiting factors of antitumor immunity. These factors need to be translated into clinically applicable multimodal predictors that allow for the selection of the best IO treatment. This review discusses strategies to (i) create such holistic views of antitumor immunity, (ii) identify measurable parameters capturing the complexity of a patient's immune status, and (iii) facilitate the incorporation of precision IO research in the clinic.