One of the common phenomena in ASP.NET MVC View Model architecture is using entity framework model as view model as well. This article does not discuss weather this is good or bad, but gives some useful tips on how you can improve performance (with JSON serialization) while taking this approach.
To make a long story short, the suggestions are as follows: Stop serializing child records to reduce response time (and it will also improve performance). The other is to use custom JSON converter (by Newtonsoft.Json.JsonConverter) to reduce response time even more.
If you use these tips, let us know if it helped you too.