How to install and create Python 3.6 virtualenv on HDP 3.0.1

Many times the default version bundled with HDP i.e 2.7 is not sufficient to explore a few libraries and it required python to add an additional version as in my case I need python 3.6+ to explore NLP libraries. I have done this many times with another version of HDP but this time I want to create a cheat sheet so that fire and forget would not be a problem in the future.

Centos 7 vanilla or HDP 3.0.1virtual box still does not have a package for Python version 3. So it’s EPEL to the rescue: (you need to be root for this).

*Be careful not to run the below command on dev/prod environment where many users are connected as it refreshes the packages/iptable with new system packages.

#yum -y install epel-release

In my case, it shows up to the latest version. Now let us first download rpm from Centos community using yum.

#yum install https://centos7.iuscommunity.org/ius-release.rpm

Examining /var/tmp/yum-root-SmdyGj/ius-release.rpm: ius-release-1.0-15.ius.centos7.noarch
Marking /var/tmp/yum-root-SmdyGj/ius-release.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package ius-release.noarch 0:1.0-15.ius.centos7 will be installed
--> Processing Dependency: epel-release = 7 for package: ius-release-1.0-15.ius.centos7.noarch
Loading mirror speeds from cached hostfile
 * base: centos.excellmedia.net
 * elrepo: mirrors.coreix.net
 * extras: centos.excellmedia.net
 * updates: centos.excellmedia.net
--> Running transaction check
---> Package epel-release.noarch 0:7-11 will be installed
--> Finished Dependency Resolution

Now install python3.6, devenv, and pip3.6 using yum.

#yum install python36u python36u-devel python36u-pip
..
..
.. 
Running transaction
  Updating   : 1:openssl-libs-1.0.2k-16.el7.x86_64                                                                                                                                       1/9
  Installing : python36u-3.6.7-1.ius.centos7.x86_64                                                                                                                                      2/9
  Installing : python36u-libs-3.6.7-1.ius.centos7.x86_64                                                                                                                                 3/9
  Installing : python36u-setuptools-39.0.1-1.ius.centos7.noarch                                                                                                                          4/9
  Installing : python36u-pip-9.0.1-1.ius.centos7.noarch                                                                                                                                  5/9
  Installing : python36u-devel-3.6.7-1.ius.centos7.x86_64                                                                                                                                6/9
  Updating   : 1:openssl-1.0.2k-16.el7.x86_64                                                                                                                                            7/9
  Cleanup    : 1:openssl-1.0.1e-42.el7.9.x86_64                                                                                                                                          8/9
  Cleanup    : 1:openssl-libs-1.0.1e-42.el7.9.x86_64                                                                                                                                     9/9
  Verifying  : 1:openssl-libs-1.0.2k-16.el7.x86_64                                                                                                                                       1/9
  Verifying  : python36u-setuptools-39.0.1-1.ius.centos7.noarch                                                                                                                          2/9
  Verifying  : python36u-devel-3.6.7-1.ius.centos7.x86_64                                                                                                                                3/9
  Verifying  : python36u-pip-9.0.1-1.ius.centos7.noarch                                                                                                                                  4/9
  Verifying  : 1:openssl-1.0.2k-16.el7.x86_64                                                                                                                                            5/9
  Verifying  : python36u-libs-3.6.7-1.ius.centos7.x86_64                                                                                                                                 6/9
  Verifying  : python36u-3.6.7-1.ius.centos7.x86_64                                                                                                                                      7/9
  Verifying  : 1:openssl-libs-1.0.1e-42.el7.9.x86_64                                                                                                                                     8/9
  Verifying  : 1:openssl-1.0.1e-42.el7.9.x86_64                                                                                                                                          9/9

Installed:
  python36u.x86_64 0:3.6.7-1.ius.centos7                     python36u-devel.x86_64 0:3.6.7-1.ius.centos7                     python36u-pip.noarch 0:9.0.1-1.ius.centos7

Dependency Installed:
  python36u-libs.x86_64 0:3.6.7-1.ius.centos7                                               python36u-setuptools.noarch 0:39.0.1-1.ius.centos7

Dependency Updated:
  openssl.x86_64 1:1.0.2k-16.el7                                                             openssl-libs.x86_64 1:1.0.2k-16.el7

Complete!

Let us verify the version installed.

# python --version
Python 2.7.5

# python3.6 --version
Python 3.6.7

# pip3.6 --version
pip 9.0.1 from /usr/lib/python3.6/site-packages (python 3.6)

Now create a virtual environment to run our python3.6 in an isolated environment.

# pip3.6 install virtualenv
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl (1.9MB)
    100% |████████████████████████████████| 1.9MB 805kB/s
Requirement already satisfied: setuptools>=18.0.0 in /usr/lib/python3.6/site-packages (from virtualenv)
Installing collected packages: virtualenv
Successfully installed virtualenv-16.2.0

Create a virtual environment for python3.6 like below:-

# virtualenv -p python3.6 mynlp
Already using interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /root/mynlp/bin/python3.6
Also creating executable in /root/mynlp/bin/python
Installing setuptools, pip, wheel...
done.

You need to activate the environment whenever next time you want to use python3.6 as your default python shell. Like below:-

cd /root/mynlp && source bin/activate
(mynlp) # python
Python 3.6.7 (default, Dec  5 2018, 15:02:05)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Hope this will help! Next, I’ll give you a demo of the NLP package using Python3.6+.

Happy Machine Learning!!!

Leave a comment

Your email address will not be published. Required fields are marked *