Assuming you know what plugins that you're using for the ajax calls, you could probably get away with something similar to this (see http://pastebin.com/13N9wh39) to exclude the loading of plugins unassociated with your ajax request.
Achievements
9
Karma
0
Total ProTip Views
Nephila Komaci
Have at least one original repos where PHP is the dominant language
Assuming you know what plugins that you're using for the ajax calls, you could probably get away with something similar to this (see http://pastebin.com/13N9wh39) to exclude the loading of plugins unassociated with your ajax request.