That EHCache uses a lot of memory makes sense, since when caching is turned on, it will use the configured amount of memory to store items.
If you disable cache, be prepared to take a big performance hit. All models and items generated and fetched from the Tridion Broker database will always happen. Turning on the Tridion Object Cache will essentially move the memory problem to that area - you will see that in your monitoring / profiling the memory used there will then also start to spike.
The solution would be to limit the amount of items EHCache can store per server / JVM. You can add a maxBytesLocalHeap attribute to ehcache.xml to really limit on memory. More information can be found here. Note that you will have to remove the maxEntriesLocalHeap attribute for it to work.
An alternative can be to for instance move to Terracotta. DD4T can do this OOTB with some reconfiguration, but it would require major infrastructural changes.
Update
There appeared to be a bug in DD4T's EHCacheProvider after all , but it has nothing to do with a memory leak. When an item was put into cache with a null value and populated later, EHCache didnt recalculate the size if the item was not re-put in the cache.
Should be fixed in DD4T 2.0.9 with this commit.