Memory usage indicates memory size used by a PHP script in your WordPress site.
PHP scripts that takes too much memory may slow down your website, mostly, when you are using a shared hosting. However, a well-written theme or plugin usually do not consume too much memory.