Armin, thank you very much! This is indeed what is happening.
The presence of a shortcode module (even with my shortcode plugin deactivated) will cause the page to refresh upon resizing the browser window to mobile/desktop width.
Could it be that there is some issue with the coded used? I managed to upload a file using the mpeg4 codec instead of the h.264 (x264) codec and had no issues.. the file was 21 mb in size.
"Fullscreen Slider will be active in the phone view if each of your rows only contains one element.
Also Fullscreen Slider can be active on the phone view if you use Custom Phone Layouts."
This is written in "Lay Options" -> "Fullscreen Slider Addon"