That EHCache uses a lot of memory makes sense, since when caching is turned on, it will use the configured amount of memory to store items.
If you disable cache, be prepared to take a big performance hit. All models and items generated and fetched from the Tridion Broker database will always happen. Turning on the Tridion Object Cache will essentially move the memory problem to that area - you will see that in your monitoring / profiling the memory used there will then also start to spike.
The solution would be to limit the amount of items EHCache can store per server / JVM. You can add a maxBytesLocalHeap attribute to ehcache.xml to really limit on memory. More information can be found here.
An alternative can be to for instance move to Terracotta. DD4T can do this OOTB with some reconifguration, but it would require major infrastructural changes.