Hi Loes,
you are right. In the library the original files are piling up.
I would advise you to have a maximum size of 2500px and try to keep every image under 1 mb.
It will not really slow don your website though – even if you have big images online. But of course it can be more heavy for your server and to optimize the images upfront is for sure a good idea.
I hope I could help you!
Have a good day!
Marius