The Kasteli Hill, on the east side of the port, was an ideal setting for the establishment of a prehistoric hamlet (ancient Kidonia), as it bordered the sea and was surrounded by the rich land of the Chania plains.
The findings of ceramic objects of the Neolithic Transition (3.000 - 2.900 B.C.) provide the earliest proof of human activity on the Kasteli Hill.
At the site of Kidonia, the Byzantines later built a fortress, which, in several places sit on top of the ancient walls, and was made from building material of ancient Kidonia.
Most of the mansions of the old Venetian families were built at Kasteli. They were designed according to the trends of the time and were especially influenced by the Venetian Mannerism period of the 16th century. Here there was also a majestic Venetian cathedral, dedicated to the Virgin Mary which was destroyed during the bombings of the 2nd World War.
Many of the prevalent Turkish-Cretan families lived in this area where they added their own cultural elements to the area. There are three gates at Kasteli, one to the east, one to the west and a third with stairs that has maintained the name of "Ta Skalakia" – the little stairs.