Mary-Kate Olsen and Olivier Sarkozy Clashed Over Having Kids: Source

"A few years ago, having a baby was not a priority for her. This has changed," an insider tells PEOPLE of tensions in Olsen and Sarkozy's marriage