The genome consists of regions of transcriptionally active euchromatin and more silent heterochromatin. We reveal that the formation of heterochromatin domains requires cohesin turnover on DNA. Stabilization of cohesin on DNA through depletion of its release factor WAPL leads to a near-complete loss of heterochromatin domains. We observe the opposite phenotype in cells deficient for subunits of the Mediator-CDK module, with an almost binary partition of the genome into dense H3K9me3 domains, and regions devoid of H3K9me3 spanning the rest of the genome. We suggest that the Mediator-CDK module might contribute to gene expression by limiting the formation of dense heterochromatin domains. WAPL deficiency prevents the formation of heterochromatin domains, and allows for gene expression even in the absence of the Mediator-CDK subunit MED12. We propose that cohesin and Mediator affect heterochromatin in different ways to enable the correct distribution of epigenetic marks, and thus to ensure proper gene expression.