In the eighteenth and nineteenth centuries the area was much more a hotel-quarter than is evident now. St. James's Street, Jermyn Street, and the side streets west of the square were particularly rich in hotels, some of which survived to appear on the 1869-74 Ordnance Survey. But they contributed little that was distinctive to the architecture of the area. The Cavendish Hotel preserves their quietness of aspect.

Compared with its present condition the area was less a place merely of daytime or weekday resort. In the early nineteenth century St. James's Square still provided a predominantly residential nucleus. This residential character is perhaps reflected in the fact that from 1828 to 1875 there were four places of worship in the area, although they doubtless attracted worshippers from other parts of London. Besides St. James's Church there were two churches of the Establishment, St. Philip's, Regent Street, and the proprietary chapel in York Street. There was also the Western Synagogue in St. Alban's Place.

In 1835 the St. James's Theatre in King Street made its contribution to the variegated public aspect of the area. When the establishment of the theatre was in prospect local residents complained of the undesirable clientele that would attend it. For one aspect of the area which had lasted into the nineteenth century was the 'rookeries' of disreputable courts and alleyways, and in the circumstances of early nineteenth-century theatre-going it is not surprising that the narrow ill-famed lanes on the south side of King Street should be thought a likely source of disorder. Pall Mall Place, which was already notorious in 1744, was described as a 'low den of infamy' a hundred years later and in 1868 was said to have enjoyed until recently 'no very delicate fame'. The market area, and Market Lane hard by the Opera House, were similarly notorious, sharing some of the ill-fame of the neighbouring Haymarket. In 1816 a resident in St. James's Square near Pall Mall had complained of the behaviour of street-walkers nearby and, on the other side of Pall Mall, Stone Cutter's Yard was of evil reputation in the early nineteenth century. Some of these disreputable enclaves were transformed by early nineteenth-century reconstructions of the areas adjoining Waterloo Place, around the Opera House, and in Little King Street where its widening into St. James's Street destroyed a 'rookery'.

The history of the area in the last hundred years or so has been such that what now survives of the older fabric often creates a piquant juxtaposition of unassuming and imposing building. The narrow approaches to Mason's Yard, and the tiny building at No. 7 Duke Street, contrast strikingly with the neighbouring square, and St. James's Street is singularly unlike Crown Passage on which it backs. This narrow arched passageway of small shops, occupying sites shown by Horwood in the 1790's, is the only one of the courts and alleys south of King Street to survive in recognizable form, and still retains a little of that outward appearance which now survives in the parish mainly northward of Piccadilly Circus.

Particularly since the 1930's, however, the conflation of sites and inflation of building has spread to the freehold areas, most notably in St. James's Square, which has in the process virtually ceased to be residential. There has, indeed, been a transformation in the use of the area, largely since the war of 1914-18. Until then the general character of St. James's had changed, in relation to London society, remarkably little in 250 years. In St. James's Square clubs and institutions had by the mid nineteenth century replaced some of the private families, and two of the business houses now in the square have been there since that period. But the general character of the area remained little altered. Then at the end of the nineteenth century business offices, as distinct from the traditional type of retail shops, moved more extensively into the area, and by 1936 this was recognized by the inclusion of the greater part of the area in 'zones' intended for 'special' or 'general' business use.

This change of use did not at first greatly affect the character of the street-architecture. The buildings which had been erected to accommodate residential apartments above shops were taken over for office use on some or all of the upper floors. Until after the war of 1914-18 there were probably very few buildings erected south of Piccadilly specifically to provide office accommodation. The alteration in the appearance of the area by buildings designed to serve the traditional use as private apartments, was until recently greater than any caused by the social transformation of the area towards one of office-buildings.