Actually its not a local cache. This is the whole collection of icons ever exported from the app.
Originally, we’ve been using the paths that you are suggesting. However, please note the issue that may arise from such a choice. The majority of users do not realize that by drag-and-dropping an icon to say AI, they actually transfer and save a link to the icon, but not the actual icon file. Now, guess what happens, if a user cleans up the system paths? Exactly, the user ends up with blank spaces in the Illustrator. This is why, by default, we are using the location that we are currently using.
If you prefer, you may hide the folder in Finder, from viewing it. Use the following command in terminal to achieve that: chflags hidden ~/Documents/com.visualpharm.Icons8/