Ken FisherCrystal Mercedes | CNBCFidelity has terminated a $500 million relationship with Fisher Investments, bringing the total yanked from the money manager to almost $1.8 billion.Officials at the giant Boston-based asset manager confirmed it would end its relationship with Fisher on Monday, in light of inappropriate comments founder Ken Fisher made at an investment conference on Oct. 8.Fisher managed $500 million in assets for Fidelity’s Strategic Advisers Small-Mid Cap Fund.”Fisher Investments does not provide investment advisory services for any portion of the assets of Strategic Advisers Small-Mid Cap Fund,” said Vin Loporchio, a spokesman for Fidelity. “Assets previously managed by Fisher Investments have been reallocated within the fund.”Fidelity is the fifth major institutional client to leave Camas, Washington-based Fisher since the billionaire made controversial remarks about sex at the Tiburon CEO Summit nearly two weeks ago. Previously, four government pensions pulled close to $1.3 billion from Fisher Investments.CNBC obtained an audio recording of Fisher’s comments at the Tiburon CEO Summit, as well as audio of him speaking at a previous conference. Clips from both were featured on CNBC Power Lunch. Combined, they show that the money manager made flippant remarks about sex.In the audio obtained by CNBC, Fisher says at the Tiburon conference: “Money, sex, those are the two most private things for most people,” so when trying to win new clients you need to be careful.He says: “It’s like going up to a girl in a bar … [inaudible] …going up to a woman in a bar and saying, hey, I want to talk about what’s in your pants.”Further, when Fisher was a speaker at the Evidence-Based Investing conference in 2018 he compared marketing mutual funds to propositioning a woman for sex at a bar.”I mean the, the most stupid thing you can do, which is what every mutual fund firm in the world always did, was to brag about performance, uh, in, in a direct mail piece, which is a little bit like walking into a bar if you’re a single guy and you want to get laid and walking up to some girl and saying, ‘Hey, you want to have sex?'” Fisher said, according to audio obtained by CNBC.Organizers of both conferences subsequently banned him from speaking again in the future.More from Personal Finance:What you need to know about required minimum distributionsHow to protect your side hustleFisher fallout: Who left? Who remains?