Since Roman times Swanage has been the centre of the local stone industry, It became a quarry town and harbour from which Purbeck Marble was shipped for use in many cathedrals in Europe. Purbeck Marble was used to build Westminster Abbey, and the Cathedrals of Lincoln, Salisbury and Exeter. Purbeck stone was used for the flagstone of the new Crown Jewels house at the Tower of London. The sea off the town was the site of one of the greatest sea battles of the Middle Ages, when King Alfred destroyed a Danish fleet of more than 100 vessels in 878.

In 1820 William Morton Pitt bought the estate and tried, with limited success, to develop Swanage as a seaside resort like the already successful Weymoth and Lyme Regis. It grew slowly during the 19th century, but after the railway arrived in 1885 the resort really expanded. So that today Swanage is full of buildings dating from the late 19th and early 20th century. The railway to Swanage closed in 1972 but has subsequently been re-opened by the Swanage Railway Company who now run steam trains on the line.

One of the towns most famous sons was John Mowlem, (1788-1868), a poor Swanage boy who left his home a went to London, where he eventually founded the great building company of Mowlem. In retirement he became a benefactor of this little seaside resort giving it probably its most photographed beauty spots. The Mill Pond, built into the side of a hill and surrounded by cottages of local stone.

Architecturally, Swanage is dominated by the scavengings of John Mowlem and his nephew and successor George Burt (1816-1894). The Wellington Clock Tower was first erected at one end of London Bridge in 1854 as a memorial to the Duke of Wellington. Other London relics include the front of the Town Hall, originally the facade of the Mercer's Hall in Cheapside, built in 1670 after the Great Fire of London. The cast iron lamp standards on The Parade and Beach Road came from Hanover Square. Most remarkable of all is Purbeck House, George Burts house in the High Street, which is built from chippings from the Albert Memorial, and incorporates a bollard from Millbank Prison, an archway from Hyde Park Corner, some cast iron columns from Billingsgate Market, as well as a mosaic and floor tiles from the Palace of Westminster.

The 40 ton Portland stone globe at Durlston Country Park owes its existence to the formidable and eccentric George Burt. He and his Weymouth architect, G.R. Crickmay, bult the nearby Castle in 1887-8 as a deliberate fake for use as a restaurant. All over the Country Park are stone benches and plaques set up by Burt giving information or improving quotations. The Globe is surrounded by slabs inscribed with Shakespearean or Biblical passages, whilst others remain blank for intending graffiti artists. The 260 acre park is now run by Dorset County Council, with a visitor centre on quarrying and wildlife.