The man at the center of the alleged plot to extort $2 million from David Letterman was apparently not well-liked by some in the industry, a source close to a producer who worked with Robert Halderman tells FOX411.

The "48 Hours Mystery" producer pled not guilty to attempted grand larceny in Manhattan court on Friday, but an insider tells FOX411 that some who worked with him in the past were not surprised when they heard of his alleged involvement.

"People did not enjoy working with him," said the source. "He was a nightmare to work with. It was easier to avoid him. No one that has worked with him before is too shocked [with the allegations]."

Meanwhile, a source familiar with "The Late Show" says that while he is a good employer, David Letterman "has always been a big flirt with women."

"You can see how he is with the women guests on the show," the insider tells FOX411. "He is that way with most women, though he is also more serious off-camera. He gets along great with his team, and sometimes he develops closer relationships with the women on his team."

Indeed.

The source also says that being a father (Dave has a 5-year-old son Harry) has changed the late night funnyman.

"He learns things from Harry and talks about his family all the time," said the insider. "Dave is trying to do right by his wife and son."

Letterman admitted Thursday to having affairs with women on the staff of his talk show. He married his longtime girlfriend Regina Lasko, who also once worked for him, in March.