Ok I'll answer to my own question for people who ask or have the same issue haha. If you use the iframe, using the vimeo api is, for the moment, the best way I found to solve that problem for Firefox and Chrome.
Insert the iframe :
<div class="iframeVimeo" style="padding:56.25% 0 0 0;position:relative;"><iframe src="yourvideocolor=ffffff&title=0&byline=0&portrait=0" style="position:absolute;top:0;left:0;width:100%;height:100%;" frameborder="0" allow="autoplay; fullscreen; picture-in-picture" allowfullscreen></iframe></div>
Add css to block the pointer event on that iframe :
.iframeVimeo iframe { pointer-events: none; }
And then add the JS for the api and the click on the play button :
<script src="https://player.vimeo.com/api/player.js"></script>
<script type="text/javascript">
var player = new Vimeo.Player(document.querySelector('.iframeVimeo iframe'));
var iframe = document.querySelector('.iframeVimeo').addEventListener('click', function() {
player.getPaused().then(function(paused) {
if (paused) {
player.play();
} else {
player.pause();
}
}).catch(function(error) {
});
});</script>
It should works ✌️