Installation:

1. Download the ZIP
2. Go to Preferences
3. Add-ons
4. Install
5. Select the ZIP-File
6. Activate the Add-on

Blender usually has numpy already pre-installed.

Sometimes Blender does remove the Library Path.
When you are using a script and want to import scipy for example you can use this setup.

import site
import sys

sys.path.append(site.USER_SITE)

This is not a limitation to PyPip. Blender sometimes removes this directly from the sys.path.