The answer will be different for every crew, but the issues are always the same:
Small motors are cheaper and use less fuel of course, but in addition they are lighter and easier to handle, and also make the dinghy easier to drag up a beach.
However, in a strong wind or current, it can be difficult to make way, and of course, every journey takes longer. Many people who start with a very small motor upgrade to a larger one, and many who start with a large motor wish they had a smaller one. Another issue is that the larger the motor, the more attractive it is to thieves. On Blackwattle, after traveling with two motors for a long time, we reckon that an 8 horsepower motor is about right for a 3-4 metre dinghy – you won’t win the dinghy Olympics, but neither will you be going backwards in a strong wind or current.