When open a Radwindow as modal, if modal size < 992px or 767px (default Bootstrap breakpoints), parent page will be apply CSS rules for that breakpoint too. That will make parent page drop to stack layout if you use col-md for grid system.
To solve this, simply add a version to your bootstrap css in page load as modal, like this:
<link href="bootstrap.min.css?v=modal" rel="stylesheet">
Hope that help :)