He was very tall and extremely pale, with an unshapely body, but very thin neck and legs.​ His eyes and temples were hollow, his forehead broad and grim, his hair thin and entirely gone on the top of his head, though his body was hairy. Because of this to look upon him from a higher place as he passed by, or for any reason whatever to mention a goat, was treated as a capital offence. While his face was naturally forbidding and ugly, he purposely made it even more savage, practising all kinds of terrible and fearsome expressions before a mirror.   Suetonius: Life of Caligula

It's probably the empty eye-sockets that do it but I have always felt that this cast bronze portrait captures the spirit of Caligula, the mad emperor as the stories paint him. I wouldn't say he was ugly though, Suetonius.