There is, however, no positive evidence of building before the seventeenth century. Until the 1530's the area was divided between various owners, mostly corporate, including the Abbey of Westminster, the Convent of Abingdon, and the Hospital of St. James, the last of which occupied, probably since the eleventh century, a position immediately adjacent, where St. James's Palace now stands. St. James's Fair, established in 1290 for the benefit of the hospital, was probably held nearby. Beyond this little is known of the medieval history of the area.

Between 1531 and 1536 almost all (perhaps quite all) the area of St. James's was surrendered to the Crown and formed into the Bailiwick of St. James. Part of it was granted away (mostly in 1665) but about half the area still remains Crown property. St. James's Field was enclosed by Henry VIII and turned to meadow-land. Nearly all the area westward of St. James's Street was held from at least the later sixteenth century by the Pulteney family as tenants of the Crown lessees.

In the sixteenth century the chief references to St. James's Field are as a mustering-ground for royal troops. The creation of St. James's Palace by Henry VIII did not bring about any significant growth of building in the area. Some may, however, have appeared in the course of the sixteenth century opposite the palace, where the partly timber houses of Sir Henry Henne and Sir William Pulteney survived until after the Restoration, and where, in Charles I's reign, Berkshire House was built.

In the 1620's some development was taking place near the palace. The most important was the large mansion built facing the palace in 1626-7 by the Earl of Berkshire, a former Master of the Horse to Charles I when Prince of Wales. The history of this notable courtier's house is obscure but some of its fabric probably endured into the nineteenth century and the site of the house and garden determined part of the intricate layout of this area. At about the same period St. James's Field on the other side of St. James's Street was, probably for the first time, being appropriated for royal recreation, with the establishment of a tennis court, southward of a physic garden, and of a stately pall mall alley on the line of the present Pall Mall. Post-Restoration changes swept away the tennis court and physic garden. Apart from these developments, St. James's Field seems to have been left unbuilt, and in the 1630's the King took measures to preserve its privacy.