The fort, originally built by the Portuguese in the 16th century to defend Galle, was an earthen structure with pallisades covering the northern inland side with a rampart and three bastions. They believed that the seaward side was impregnable and hence did not construct any fortifications on the sea side. The sea wall was an addition made in 1729 to make the city planning for defense purposes complete in all respects.
When the fort came under the control of the Dutch, they considered the old fortifications built by the Portuguese unsafe as they were made of earth and palisades. Hence, the Dutch decided to fully encircle the entire peninsula by building impregnable fortifications as defense against other colonial agencies in the region. They built some 14 bastions with coral and granite stones over an area of (52 hectares (130 acres). Many of the fortification walls were built in 1663. The city built within the fort was well-planned with a grid layout and the peripheral roads aligned parallel to fort's ramparts.
The fort has two gates. The two towering gates to the fort are termed “Portcullised gates” and the first gate of entry from the port is inscribed "ANNO MDCL XIX" which has depiction of Dutch Coat of Arms with the ubiquitous emblem of cock and an inset "VOC" inscribed in the centre.
The Main Gate is along the northern stretch of the fort on the land side and is heavily fortified. The Portuguese had built a moat here which was widened during the Dutch rule by breaking the fort wall in 1667 and building the Star, Moon and Sun bastions. Its construction is also dated to the British period from 1897 to facilitate easy flow of traffic to the old town. This gate was fortified with a draw bridge surrounded by a moat; inscriptions here indicate 1669 as the year of construction. Some of the other named bastions are: The Aurora Bastion, the Tremon Bastion, the Kleipenberg Bastion and the Emaloon Bastion, adding an element of grandeur to the fort as a whole.
Walking along the fort wall in a clockwise direction leads to the Old Gate where the British Coat of Arms is seen inscribed at the entrance at the top. In the inner part of the gate there is the 1668 dated inscription of the letters VOC, an abbreviation of Verenigde Oostindindische Compagnie, meaning Dutch East India Company with the insignia of a cock flanked by two lions. Further along the fort wall is the Portuguese-built oldest bastion, known as Zwart Bastion, meaning Black Bastion. The eastern section of the fort terminates in the Point Utrecht Bastion; the powder house is also seen here. The 18 metres (59 ft) high Galle Lighthouse was erected here in 1938. The next stretch of the fort wall is the location of Flag Rock Bastion, which was used as a signaling station to warn ships entering the port of the hazardous rocky stretches of the bay. Ships were warned by firing musket shots from the Pigeon Island near the Flag Rock. Further along the fort walls is the Trion Bastion where a windmill drew water from the sea to sprinkle the dusty roads of the town; it is also a view point to watch sunsets. Many more bastions are seen along the fortifications from this location up to the Main Gate.
The Fort really resembles a small laid out walled town, with a rectangular grid pattern of streets full of the low houses with gables and verandas in the Dutch colonial style. It has a well laid out road network. The fort area also has a number of historic churches, mosques, commercial and government buildings. Some of the locals stroll along the walls of the fort in the evenings.
After the fort came under the control of the British in 1796, it remained their southern headquarters. They made many modifications to the fort such as closing of the moat, building of houses, a lighthouse on the Utrecht Bastion, a gate between the Moon Bastion and the Sun Bastion. A tower was particularly erected in 1883 to commemorate the jubilee of Queen Victoria. The Second World War saw many more fortifications built to defend the fort. In spite of all the changes made over the years, since it was first built between the 16th and the 19th century, the Galle Fort still remains a unique monument complex said to be "the best example of a fortified city with a fusion of European architecture and South Asian traditions built by Europeans in South and Southeast Asia".
In the fort area, many buildings are of Dutch vintage with street names also in Dutch. The sewerage system built in the fort area ensured that the city sewerage was flushed into the sea during the tidal cycle. The Dutch exploited the musk rats in the sewers by exporting them to extract musk oil.