The control of quantum systems is of fundamental scientific interest and promises powerful applications and technologies. Impressive progress has been achieved in isolating the systems from the environment and coherently controlling their dynamics, as demonstrated by the creation and manipulation of entanglement in various physical systems. However, for open quantum systems, engineering the dynamics of many particles by a controlled coupling to an environment remains largely unexplored. Here we report the first realization of a toolbox for simulating an open quantum system with up to five qubits. Using a quantum computing architecture with trapped ions, we combine multi-qubit gates with optical pumping to implement coherent operations and dissipative processes. We illustrate this engineering by the dissipative preparation of entangled states, the simulation of coherent many-body spin interactions and the quantum non-demolition measurement of multi-qubit observables. By adding controlled dissipation to coherent operations, this work offers novel prospects for open-system quantum simulation and computation.