Engine smoothness depends on a variety of factors. But the biggest determinant has to be how many cylinders it has.
So, how many cylinders would be the most optimal?
6 IS THE MAGIC NUMBER!
The basic dynamic problem of internal combustion engines is that in addition to rotating parts, there are also reciprocating parts. These pose a challenge for engine smoothness.
The piston, in particular, traverses a fixed vertical distance (both up and down). It thus experiences acceleration, deceleration, braking, and the same motions in the opposite direction. Although a piston doesn’t weigh a lot, at the speeds in question, momentum becomes an issue to reckon with.
This is why a piston’s change in direction or even change in velocity is accompanied by inertia forces that manifest themselves as vibrations.

The naturally aspirated 4-litre flat-6 engine from the Porsche 718 Cayman GT4.
HOW CAN ENGINE SMOOTHNESS BE ACHIEVED?
The oldest trick to quell these vibrations is to place a dead weight that generally moves in unison with the piston. However, it must be located such that its own momentum is opposite to the piston’s. A sort of “mirror-image force” if you like. A tyre shop balances wheels on very similar principles.
It’s possible to build a smooth engine by throwing in counterweights wherever necessary, but the “fat” is most undesirable. With multi-cylinders, neighbouring pistons also assist the counter-balance weights in cancelling the vibrations. This is why pairs of cylinders are easier to deal with.
When building a motor with engine smoothness in mind, if we add cylinders in a geometrically correct location on the crankshaft to offset dynamic imbalance, we would eventually have a 6-cylinder engine.
This would either be an inline-6 or flat-6 engine.

The turbocharged 3-litre inline-6 powering the BMW 6 Series Gran Turismo.