I ran into this nasty bug when all page content was loaded with AJAX (kind of mimicking an iframe because of various reasons)
What happened was as following. The video loads the first time the containing page was called (it didn't matter if it loaded via AJAX or directly from the browser). However when browsing through the website, randomly clicking from page to page, when loading the page with the video again, it sometimes wouldn't load.
After three days I finally found out what could solve it, and it's so simple I'm ashamed it took more than three minutes.
Simply adding a timestamp to the sources' url forced to browser to re-check the file. In the example I am using PHP.
<source src="/assets/video.mp4?<?php echo time(); ?>" />
<source src="/assets/video.ogv?<?php echo time(); ?>" />
<source src="/assets/video.webm?<?php echo time(); ?>" />
This way you force the browser to do a new look-up to the file, it will then recognise the file and still load it from its cache. (I think)
You might think this downloads the video everytime you load the page, however in Chrome it doesn't seem like it. The network monitor says it loads from cache, so let's just believe that. You might have to set proper expiration dates and caching information via .htaccess on your video files.