The combined finite–discrete-element method (FDEM) has made a groundbreaking progress in the computation of contact interaction. However, FDEM has a strict requirement on the element type, and the simulation result may be inconsistent due to a deficiency of physical meaning of the potential function. To address this problem, a new 3D discrete-element method based on a distance potential is proposed for a system consisting of a large number of arbitrary convex polyhedral elements. In this approach, a well-defined distance potential is proposed as a function of the penetration between the contact pairs. It exhibits a clear physical meaning and a precise measurement of the embedding between the elements in contact. The newly presented method provides a holonomic and accurate contact interaction without being influenced by the element shape. Therefore, the restraint of the element type in FDEM is removed and the proposed method can be used for arbitrary convex polyhedrons. In addition, an improved contact detection algorithm for non-uniform block discrete elements is provided to overcome the constraint of elements with the same size in the Munjiza-No Binary Search contact detection method. The new approach retains the merits of the FDEM and avoids its deficiencies. It is validated with well-known benchmark examples including an impact simulation, a friction experiment, a joint structure of a sliding rock mass, pillar impact, block accumulation, and analysis for the failure process of wedge slope. The results of this proposed method are in excellent agreement with the existing experimental measurements and analytical solutions.