Abstract Exquisite binding specificity is essential for many protein functions but is difficult to engineer. Many biotechnological or biomedical applications require the discrimination of very similar ligands, which poses the challenge of designing protein sequences with highly specific binding profiles. Current methods for generating specific binders rely on in vitro selection experiments, but these have limitations in terms of library size and control over specificity profiles. We present a multi-stage approach that overcomes these limitations by combining high-throughput sequencing of phage display experiments with machine learning and biophysical modeling. Our models predict the binding profiles of antibodies against multiple ligands and generate antibody sequences with desired specificity profiles. The approach involves the identification of different binding modes, each associated with a particular ligand against which the antibodies are either selected or not. We demonstrate that the model successfully disentangles these modes, even when they are associated with chemically very similar ligands. Additionally, we demonstrate and validate experimentally the computational design of antibodies with customized specificity profiles, either with specific high affinity for a particular target ligand, or with cross-specificity for multiple target ligands. Overall, our results showcase the potential of leveraging a biophysical model learned from selections against multiple ligands to design proteins with tailored specificity, with applications to protein engineering extending beyond the design of antibodies.