Physical economy research has, thus far, focused on the throughput of materials that underpin economic development. The role of stocks of buildings and infrastructure has remained underexplored, yet it is the physical stock that provides service to society. To fill this gap, this research investigates stock dynamics in Japan in relation to population and economic drivers using panel regression and IPAT analyses for the past five decades. We recognize characteristic changes in the strength and relative influence of the drivers throughout time, in different subnational regions, and on the dynamics of buildings compared to transportation infrastructure. We find that material stock accumulation mainly occurred due to growth in economic activity, specifically by tertiary sector demand. Apart from a period of government-driven stock accumulation in the 1990s to stimulate economic growth, as economic and population growth slowed stock accumulation dynamics also changed signaling a new stock saturation trend. Migration from rural to urban areas has recently become an influential driver, leaving behind underused buildings and roads. This analysis provides a case study on how socio-economic drivers and stock accumulation interacted and changed while the country matured, which may have implications for understanding stock dynamics in rapidly industrializing economies.