From about 1990 to the mid-2000s, the golf industry boomed, overbuilt and overpromised. Now it’s paying the price. By a couple of different reckonings, the game is losing one million golfers a year, net.

The promise of the boom was that golf, always a cultural litmus but in numbers never much more than a niche, could break out and become a sport for the masses.

The ramped-up industry still hopes it can. If that’s to happen, however, the game at its core may have to change, or at least accommodate some tradition-defying alternatives. The deep question golf is asking itself these days is wherein lies its soul: with the ancient game itself, played as it has been for hundreds of years, or with the modern industry that has grown up around it? Two million U.S. jobs are now tied to golf, according to an industry lobbying consortium called We Are Golf, and those businesses are desperate to reverse their losses and expand.

Golf’s leadership is responding to the situation with more urgency than ever. At golf’s big annual merchandise show in Orlando, Fla., last month, I sat through several state-of-the-industry hand-wringing sessions. Nobody in golf is complacent. The PGA of America is pushing a new, all-points initiative called Golf 2.0, whose goal is to make the game “more relevant” to lapsed golfers and others, especially women and minorities, it has identified as underserved. At last weekend’s annual meeting of the U.S. Golf Association in Houston, the incoming president, Glen Nager, sounded downright radical (by USGA standards) in urging golf to make itself more accessible.

From all this verbiage, one phrase from Nager’s speech stood out for me as best representing the predicament for golf’s traditionalists. At the end of a list of worthy goals—making golf more enjoyable to play, more affordable and more welcoming—he added that this must be done “without fundamentally changing the game itself.” The game itself, of course, is different than the business practices that support the game, many of which (like poor customer tracking and feckless rangers) are indeed hidebound and need to be revamped. But Nager also stressed that the USGA’s top priority is to protect golf’s core values—to preserve, as he put it, “the true spirit of the game as embodied in its ancient and honorable traditions.” But it’s unclear if that spirit is still viable for an industry that hopes to expand its customer base, as per the Golf 2.0 vision, to 40 million players by 2020 from 26 million players now? Never mind gaining those customers, especially young ones, in an ever more time-squeezed, electronically addicted culture.

Until perhaps 25 years ago, golf more or less contentedly filled its niche. Those who aspired to become golfers basically knew what they were getting in for and accepted the game’s demands. “When I was learning to love this game, it was never seen as too hard, or too time-consuming to play, or too expensive, or too frustrating,” said Susie Meyers, 51, who later played on the LPGA Tour and now teaches golf in Arizona. The course she played on with her family and friends was short and simple, but in her memories it was heaven.

In the intervening years, the character and challenges of the game changed, Meyers said. “Whose idea was it to make courses so difficult it takes 5½ hours to play?” she asks. “Whose idea was it to say there’s a perfect swing and if you come to me I’ll show you what’s wrong with it and fix it? Whose idea was it that you have to find the perfect club and the perfect ball and play on perfect grass?”

There are many interrelated factors behind golf’s transformation, but the housing bubble that caused so many troubles elsewhere played a lead role. It drove demand for thousands of new high-end courses to anchor developments, ratcheting up standards everywhere. The PGA Tour, fueled by big TV contracts and Tiger Woods, became much richer than it had been and more glamorous. Equipment makers poured millions into research and came up with high-price, technologically advanced clubs. Golfers always want the latest thing.

The thrust of the Golf 2.0 initiative is customer service—to give customers more of what they want, or would want if it were actually available: set-aside times and places for families and beginners to play, more leagues and competitions, more accessible instruction, friendlier staff. An allied initiative, begun last summer and co-sponsored (unusually) by the USGA and the PGA, is Tee It Forward, which encourages golfers to play from tees that make courses shorter and more fun. Most of these solutions, and others like them, are an attempt to return golf to simpler, less fancified times—the way Susie Meyers remembers the game.

Could these measures possibly be enough to give the golf industry the growth it desires? Maybe, given enough time and the rewiring of expectations in golfers’ brains. But for those who don’t think so, more extreme ideas have appeal. Among them: promulgating alternate sets of rules for players with less patience and fewer skills; legalizing clubs and balls that make the game easier; doubling the size of the hole. John Solheim, the chief executive of club maker Ping, recently floated the notion of adding two new types of balls: one that flies much farther for thrill seekers and short hitters, and a ball that is shorter for courses that are also short.

These are the kind of ideas that terrify traditionalists, even if they might introduce more people to the game. They like to think of golf as a single church, with one set of rules and beliefs for all. The analogy may be a stretch, but Christianity faced similar renegade elements in the 16th century, resulting in the Protestant Reformation. That was extremely messy, but in the end, if yielding more net Christians was the goal, all those new, competing denominations had a positive effect. The battle for golf’s soul may just be getting under way.

—Email John Paul at golfjournal@wsj.com.