This probably holds true for all ORMs but the specific case I dealt with today has to do with Linq to SQL. If you're querying data from a database and have no need for modifying any related data to what you query, turn off object tracking.
Our legacy app was pulling approximately 1.5mil objects back that it would iterate over and pass to a client application. This should have been about 90k but it was spiking memory by approximately 1.3gb!! Why so much memory you ask? Because object tracking was bringing back a whole lot of unneccessary objects.
Turned it off and what do you know, no memory spike!!