Brendan McDermid | ReutersThe Dow Jones Industrial Average is only about 2% from a record high, but confidence is hard to come by when wealthy investors are asked about the future direction for the market. The percentage of the affluent investors who expect a stock market decline to be booked in the fourth quarter has doubled, according to a survey conducted by E-Trade Financial this month.The Dow, S&P 500 and Nasdaq 100 all hit record highs in July, and even as stocks remain in the range of those record levels as trade tensions show signs of easing and earnings start strong, the recent market action has been halting. The Dow has posted losses in four of the past five weeks, and the volatility that hit the market hard in successive bouts between August and September — as well as not-too-distant memories of the market plunge in fourth-quarter 2018 — are weighing on investors with $1 million or more in a brokerage account.The 40% of wealthy investors surveyed by E-Trade betting on a market drop in the year’s final quarter doubled from 19% in the Q3 survey.”The [third quarter] was a volatile quarter, and what’s remarkable about early Q4 is we’re knocking on the door of equity market records,” said Mike Loewengart, vice president of investment strategy at E-Trade. “When I think about where we are in the cycle, coupled with what transpired in Q3, it’s not surprising to see investors turning more bearish.”Most of the wealthier investors who have turned bearish on stocks (28%) are expecting a modest drop of no more than 5%. “But to me it is notable,” Loewengart said, as that percentage rose from 12% last quarter.The percentage of millionaires who think the market will end Q4 with a quarterly gain dropped from 56% to 42%.The E-Trade survey was conducted between Oct. 1 and Oct. 11 among 902 self-directed active investors with at least $10,000 in an online brokerage account. The millionaire data set provided exclusively to CNBC comprised of 178 investors with $1 million or more of investable assets.Economic confidence dropsA decline in economic sentiment matches the market caution.The percentage of millionaires who rates the U.S. economy at an A or B dropped from 74% to 59%.”Everyone has turned more bearish overall,” Loewengart said.But the declines don’t mean the majority are now bearish. Fifty eight percent of wealthy investors still describe themselves as bullish, though that’s down from 66% last quarter.”The economic picture is still not clear. We’ve seen a slowdown in manufacturing, and that is indicative of slowing growth, but we still see a tight labor market. Coupled with increasing consumer incomes and consumer spending, these economic readings remain at odds,” the E-Trade investing executive said.The percentage of millionaire investors who said recession was the biggest risk to their portfolio rose from 22% to 30% in the Q4 survey, while the percentage of the wealthy who said the economy currently is in recession increased from 9% to 19%. Those who think the economy is still expanding dropped from one third (33%) to 20%.”Sentiment has certainly moderated. And I keep coming back to just how volatile the last quarter was,” Loewengart said. He added that investors who have amassed $1 million or more in the market are also seasoned investors, often business owners too, who are “cognizant of the fact that the cycle cannot persist in one direction forever.”Defensive positioning, but no panicOn a sector-by-sector basis, these investors are showing a more defensive positioning. The biggest increases in confidence are in the utilities and consumer staples sectors, with a 10% increase in millionaire investors saying utilities offer the most potential in Q4, and an 11% increase in investors saying staples are the best bet right now. Overall, tech remains the most popular sector, though, with roughly half (52%) saying it offers the most potential among sectors in Q4. That’s virtually unchanged from Q3 (51%).”They are not pulling out of the market but repositioning into the historically strong defensive sectors,” Loewengart said. “That is in response to the heightened market volatility and concerns we are near the end of the economic cycle and a recession is inevitable.”The search for yield in stocks was back on as the Federal Reserve shifted to a rate-cutting stance and investors seeking income looked beyond fixed income, but Loewengart cautioned investors about placing too much faith in these sectors as fixed-income alternatives. The Federal Reserve has not indicated it will continue on a rate-cutting path for long. “These sectors are not without risk,” Loewengart said, noting that they saw significant declines the last time rates rose in 2016.The E-Trade investment official’s main takeaway from the survey was that even amid a decline in economic and market confidence, these investors are not making major changes to their asset allocations. In fact, the percentage of millionaires who said they were changing allocations declined from 27% to 21%, while those who indicated a move into cash (14%) were roughly equal to those who said they would be moving out of cash and into new positions (15%). The largest group of affluent investors (49%) are planning no changes, down slightly from Q3 (52%).”There’s always going to be something going on in the markets. Heightened market volatility is really not new for many of these investors. … It is part of the investor experience.”Subscribe to CNBC on YouTube.