Saying someone has made a balls up is a rather rude of saying they’ve made a terrible mess of things. You might well have known that, but do you know the origin of the phrase?
When ships do pretty much anything, intentional or otherwise, they’re supposed to let other vessels know. Often this information is signalled through the ship’s horn and by displaying the appropriate flag(s). In the case of running aground, which really is messing things up, one signal is to display three balls in a vertical line.