There is no configuratio option available for you to remove the button. You could consider building a UI extension to remove the button, but it is not advisable to use UI extensions to change defauilt behaviour as it can lead to unexpected circumstances.
My main question on these type of requests are always, why exactly would you want to do this? Do you have a good explanation as to why you would want to take away functionality that is offered to the user by the product. I've often notices that we try to "help" the customer by removing functionality from them that they might not know how to use correctly, while in the end a bit of education would really help them in using it properly and enhance their experience.