The story goes that cannonballs used to be stored aboard ship in piles, on a brass frame or tray called a 'monkey'. In very cold weather the brass would contract, spilling the cannonballs: hence very cold weather is 'cold enough to freeze the balls off a brass monkey'.

The phrase is actually first recorded as 'freeze the tail off a brass monkey'.

It therefore seems most likely that the phrase is simply a humorous reference to the fact that metal figures will become very cold to the touch in cold weather.

More Info: www.oxforddictionaries.com