Born in Tunapuna, Trinidad, then a British Crown colony, James was the first child of Elizabeth James and Robert Alexander James, a schoolteacher. In 1910 he won a scholarship to Queen's Royal College (QRC), the island's oldest non-Catholic secondary school, in Port of Spain, where he became a club cricketer and distinguished himself as an athlete (he would hold the Trinidad high-jump record at 5 feet 9 inches from 1918 to 1922), as well as beginning to write fiction. After graduating in 1918 from QRC, he worked there as a Teacher of English and History in the 1920s; among those he taught was the young Eric Williams, who would become the first Prime Minister of Trinidad and Tobago. Together with Ralph de Boissière, Albert Gomes and Alfred Mendes, James was a member of the anticolonialist "Beacon Group", a circle of Writers associated with The Beacon magazine, in which he published a series of short stories.