Like DXA works with JSON, a database like MongoDB (non-relational) would give a better performance than a DB2 database.
Could we use a database non relational for the Broker?
SDL doesn't currently support any non-relational database for the broker, so the answer is "it depends".
If you want to you can use a different storage layer altogether by extending the deployer to interact with that database instead of Tridion's own Broker DB. You will lose the rest of the Tridion CD API (including dynamic linking, publication mapping, any sort of Component Presentation Assembly, etc) and you'll have to implement that yourself.
So... you're basically on your own to do it. Is it possible? Sure it is, I've demoed it with MongoDB back in 2013.
Oh yeah, and it is also not supported by SDL.
Putting my serious hat back on. It seems that your reasoning to use a different database system is performance related (based on your wording). A much easier way to improve performance is to implement Tridion and Application caching. DXA and Tridion CIL already come with client-side caching (time-based) and it is amazing how much turning this on can actually impact your system's performance. Depending on your editor profiles, you could probably get away with something like 15 minutes caching on the app (meaning editors would potentially have to wait up to 15 minutes to see a change they published to the live site).
Tridion Content Service Caching also makes a huge difference, and for this you need to make sure you implement invalidation correctly - see https://docs.sdl.com/LiveContent/content/en-US/SDL%20Web-v5/GUID-7E728735-073B-4827-AABE-B45592CFF36D for details.
Basically, you break the 'chain' from CMS->BROKER DB->ContentDelivery(DXA) when replacing the middle storage layer with a custom storage solution. NoSql may perform better on its own, but the total solution will not be faster than an in-memory cached website. See Nunos comment.
Technical consequences are:
Business consequences are (to name a few):
To summarize: will the (minimal) DB performance gain really deliver you better/faster sites and does it justifies the extra risks and costs involved?
DXA is just an implementation of Tridion and therefore is bound to the prerequisites of it, see https://docs.sdl.com/LiveContent/content/en-US/SDL%20DXA-v7.1.1/GUID-DD402008-D5C5-4DCA-A74F-893272D2B659
From there you can follow the prerequisites of the Tridion Content Delivery databases which are:
So to summarize, because Tridion currently does not support non relational databases for their Content Delivery environment, it is simply not possible (unless you want to proceed in an unsupported setup of course).