I think it’s relative to two factors: disposable income and product availability. Let’s start with disposable income.
In the very early days of the hobby most model railroads were #1 Gauge or larger clockwork trains running around on the floor with little to nothing more than a circle or oval. By 1900 electric trains were starting to become available but, only the well to do could afford to buy them for their children. (Adults didn’t play with model locomotives in the gilded age of the early 20th century except for the machinists who built scale replicas.)
When Lionel introduced Standard Gauge in 1906 it was only the rich who could afford them. And even though their houses were huge very few kids had a table to operate them on. Most of them could “play trains” in the daytime but everything absolutely had to be put away before their fathers came home from the office. (See the Banks family in Mary Poppins for an example of home life in 1910).
By the 1920s the concept of operating the ever larger and more magnificent trains being manufactured on a table in the basement started to take hold. But in Standard Gauge a layout large enough to accommodate a 3 car passenger set had to be at least 9 feet long and 4 feet wide. In modern terms, 4 by 9 could pack in a lot of N gauge, a reasonable amount of HO and a “nice” amount of O gauge. Mine is 5 by 9 with O and Standard Gauge.
I’ve never lived in a mansion and while I’d love to own and operate the kinds of top of the line Lionel Standard Gauge locomotives that were introduced around the time the great depression started, I wouldn’t be able to do them justice on a 5 by 9 oval. If I had disposable income, I’d upgrade a little, but I’d stay with equipment relative to the amount of operating space I have.
So I think it makes sense to measure your layout options with an eye towards keeping your spending limited to what can “effectively” be operated in the space you have available. I would love to have an O Gauge N&W Y6b pulling a scale mile of coal hoppers that could operate on O31 (15 1/2" radius) curves but smaller power and shorter trains are a better option.
Product availability as far as buildings and accessories are concerned really started in Europe around the turn of the century. IVES was the primary competitor in the United States to Mārklin trains imported from Germany, and Mārklin had stations, crossing gates, bridges and all manner of items that, as Ward Kimball said, "allowed the kids to really play trains ".
Today you can buy just about anything you could want to place on your layout. In the future I have no doubt that more of those items will come from a full color home printer with details we can scarcely imagine. I’ve seen many interviews and read many stories about model railroaders who wished they had a machine that could shrink down what they saw along the real railroads so that they could take it home and place it on their layouts. That is almost possible nowadays.
So, what is big? If you have the desire, the space, the money, the time and the ability you’re going to want to want to build the biggest and best layout possible. Those with the most money often create the space for the layout first, often in the form of an ancillary layout building on their property, and then pay someone to build the layout of their dreams for them. Others have the outbuildings but choose to do the work themselves.
I’ve built many layouts over the years but only two of them were constructed on tables. Both were flat tables too. At this point in my life I’m content to build a few structures by hand and place them on a rug for a couple of months before I change everything.