It should be possible to make it faster, but it all depends on implementation.
Notice ECL supports two ways of publishing. One is as you mention here - the image stays in the DAM, or is published by the DAM. But it does also support publishing the images through Tridion - and this would most likely be considerable slower than what you have now.
When you want the DAM to handle publishing, there are a few things you need to consider.
Obviously you need a reasonable fast connection - while the full image is not transferred as part of the publishing operation there are still read operations going to the DAM. If you have high latency it will be slow.
You need to deal with images being removed from the DAM (or prevent it form happening). ECL does not have a full solution built in to handle this but it does notify the provider when stubs are created/deleted - and which TCM URI is used. In some DAM systems it would be possible to call the Tridion core service to check if the item is in use before allowing removal.
If the DAM - like Tridion - has a publishing step where the image is made available (typically on a CDN) you need to design a way to handle publishing. One way would be to only display published images in Tridion. Another - more complicated - approach would be to perform the publishing on demand from Tridion. Again this is not something ECL supports straight out of the box, but it can be done either by using the Invoke method on the ECL item or passing information though the attributes available when getting the published link to the ECL item (both would be done from the Tridion templating code). But I suspect this approach would result in even longer publishing times as you now have to wait for the DAM.