After the invention of lattice QCD in 1974 it quickly became clear that
- it had great potential to yield physically relevant non perturbative results and
- enormous computational re-sources would be required to obtain them.
Such resources were not readily available at the time.This led some of the physicists involved to design and build special supercomputers optimized for lattice QCD simulations. These activities started in the 1980s and are continuing until now. The resulting machines have been serving as work horses for large-scale lattice QCD simulations world-wide and were typically much more cost-efficient than commercially available machines. They also influenced the design of commercial supercomputers such as BlueGene. A probably incomplete list of such machines includes ACPMAPS, GF11, Fermi-256, QCDSP, QCDOC (all in the US), QCDPAX, CP-PACS, PACS-CS (Japan), as well as APE100, APEmille, apeNEXT, and QPACE 1 (Europe).
However, in the past several years the situation has changed. First, the development of specialized processor chips has become too expensive for academic projects. Second, commercial supercomputers such as BlueGene scale well for lattice QCD and are reasonably cost- and energy-effective (although the development of truly scalable machines such as BlueGene or the K computer relies on enormous government funding that is not always forthcoming). Finally, standard compute clusters (typically with accelerators) are now readily available and can be used for capacity-type problems where strong scalability is not a must.
Nevertheless, it still makes sense to pursue academic hardware-development activities by combining commercially available components in an innovative way, in particular regarding the communication between processors. This can result in machines that are more scalable and more cost-and energy-effective than commercial machines. The development activities typically involve industry partners that also benefit from the innovative concepts originating on the academic side ("co-design"). In our most recent projects these partners were IBM (QPACE 1) and Intel (QPACE 2) as well as Eurotech (QPACE 1 and 2). Note that the current "custom" designs for lattice QCD are suitable for a broader application portfolio than they used to be in the past. Note also that hardware development is only part of the story and that system software and high-performance application codes are equally essential (and in fact absorb a large fraction of the development activities).