Thank you for writing in, It is not recommended to remove any core files from the theme because it will affect how your site renders. Fontawesome is the library of icons, removing that means all icons used in the theme will just display as weird squares.
I understand that you have optimize your site, but please check this article for how to improve more the site performance. (see the Gzip Compression and Content Delivery Network)
Hope this helps,