Abstract: Making decisions by aggregating the opinions of a collection of agents is a question of interest to a broad array of researchers, from ensemble learning theorists to political scientists designing democratic institutions. Herein, we investigate the optimal number of agents needed to make a decision over a binary issue under a majority rule. We take an \emph{epistemic} view where the group decides on a binary issue with one ground truth ``correct'' outcome. Each one of $n$ voters votes correctly with a fixed probability, known as their competence level, which is drawn from a distribution $\mathcal{D}$. After sampling the competence levels, we choose a number of agents to represent the group. After sampling their votes, the correct outcome is chosen as long as more than half of the representatives chose this option. Assuming that we can identify the best experts, i.e., those with the highest competence, to form an \emph{epistemic congress} we find that the optimal congress size should be linear in the population size. This result is striking because it holds even when allowing the top representatives to be accurate with arbitrarily high probabilities. We then analyze real world data, observing that the actual sizes of congresses are much smaller than the optimal size our theoretical results suggest. We conclude by investigating under what conditions congresses of sub-optimal sizes would still outperform direct democracy, in which all voters vote. We find that a small congress would outperform direct democracy if the rate at which the societal bias towards the ground truth decreases with the population size fast enough, and we quantify the speed needed for constant and polynomial congress sizes.