Excel report too slow when exported from SSRS

I have a SSRS reports that aggregates about 300K rows. It has drilldowns everywhere.When I export this to SSRS, the report is too slow. If I hit Crtl+A and then hit clear > Clear formats, it gets rid of the problem. All the drill downs work fine in excel then and report is super responsive.Any idea how to fix this in SSRS so the problem is not there during export.