Hadoop

How to install and create Python 3.6 virtualenv on HDP 3.0.1

Many times the default version bundled with HDP i.e 2.7 is not sufficient to explore few libraries and it required python to add addition version as in my case I need python 3.6+ to explore NLP libraries. I have done this many times with another version of HDP but this time I want to create a cheat-sheet so that fire and forget would not be a problem in the future.

Centos 7 vanilla or HDP 3.0.1virtual box still does not have a package for Python version 3. So it’s EPEL to the rescue: (you need to be root for this).

*Be careful not run below command on dev/prod environment where many users are connected as it refreshes the packages/iptable with new system packages.

#yum -y install epel-release

In my case, it shows up to the latest version. Now let us first download rpm from Centos community using yum.

#yum install https://centos7.iuscommunity.org/ius-release.rpm

Examining /var/tmp/yum-root-SmdyGj/ius-release.rpm: ius-release-1.0-15.ius.centos7.noarch
Marking /var/tmp/yum-root-SmdyGj/ius-release.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package ius-release.noarch 0:1.0-15.ius.centos7 will be installed
--> Processing Dependency: epel-release = 7 for package: ius-release-1.0-15.ius.centos7.noarch
Loading mirror speeds from cached hostfile
 * base: centos.excellmedia.net
 * elrepo: mirrors.coreix.net
 * extras: centos.excellmedia.net
 * updates: centos.excellmedia.net
--> Running transaction check
---> Package epel-release.noarch 0:7-11 will be installed
--> Finished Dependency Resolution

Now install python3.6, devenv, and pip3.6 using yum.

#yum install python36u python36u-devel python36u-pip
..
..
.. 
Running transaction
  Updating   : 1:openssl-libs-1.0.2k-16.el7.x86_64                                                                                                                                       1/9
  Installing : python36u-3.6.7-1.ius.centos7.x86_64                                                                                                                                      2/9
  Installing : python36u-libs-3.6.7-1.ius.centos7.x86_64                                                                                                                                 3/9
  Installing : python36u-setuptools-39.0.1-1.ius.centos7.noarch                                                                                                                          4/9
  Installing : python36u-pip-9.0.1-1.ius.centos7.noarch                                                                                                                                  5/9
  Installing : python36u-devel-3.6.7-1.ius.centos7.x86_64                                                                                                                                6/9
  Updating   : 1:openssl-1.0.2k-16.el7.x86_64                                                                                                                                            7/9
  Cleanup    : 1:openssl-1.0.1e-42.el7.9.x86_64                                                                                                                                          8/9
  Cleanup    : 1:openssl-libs-1.0.1e-42.el7.9.x86_64                                                                                                                                     9/9
  Verifying  : 1:openssl-libs-1.0.2k-16.el7.x86_64                                                                                                                                       1/9
  Verifying  : python36u-setuptools-39.0.1-1.ius.centos7.noarch                                                                                                                          2/9
  Verifying  : python36u-devel-3.6.7-1.ius.centos7.x86_64                                                                                                                                3/9
  Verifying  : python36u-pip-9.0.1-1.ius.centos7.noarch                                                                                                                                  4/9
  Verifying  : 1:openssl-1.0.2k-16.el7.x86_64                                                                                                                                            5/9
  Verifying  : python36u-libs-3.6.7-1.ius.centos7.x86_64                                                                                                                                 6/9
  Verifying  : python36u-3.6.7-1.ius.centos7.x86_64                                                                                                                                      7/9
  Verifying  : 1:openssl-libs-1.0.1e-42.el7.9.x86_64                                                                                                                                     8/9
  Verifying  : 1:openssl-1.0.1e-42.el7.9.x86_64                                                                                                                                          9/9

Installed:
  python36u.x86_64 0:3.6.7-1.ius.centos7                     python36u-devel.x86_64 0:3.6.7-1.ius.centos7                     python36u-pip.noarch 0:9.0.1-1.ius.centos7

Dependency Installed:
  python36u-libs.x86_64 0:3.6.7-1.ius.centos7                                               python36u-setuptools.noarch 0:39.0.1-1.ius.centos7

Dependency Updated:
  openssl.x86_64 1:1.0.2k-16.el7                                                             openssl-libs.x86_64 1:1.0.2k-16.el7

Complete!

Let us verify the version installed.

# python --version
Python 2.7.5

# python3.6 --version
Python 3.6.7

# pip3.6 --version
pip 9.0.1 from /usr/lib/python3.6/site-packages (python 3.6)

Now create a virtual environment to run our python3.6 in an isolated environment.

# pip3.6 install virtualenv
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl (1.9MB)
    100% |████████████████████████████████| 1.9MB 805kB/s
Requirement already satisfied: setuptools>=18.0.0 in /usr/lib/python3.6/site-packages (from virtualenv)
Installing collected packages: virtualenv
Successfully installed virtualenv-16.2.0

Create a virtual environment for python3.6 like below:-

# virtualenv -p python3.6 mynlp
Already using interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /root/mynlp/bin/python3.6
Also creating executable in /root/mynlp/bin/python
Installing setuptools, pip, wheel...
done.

You need to activate the environment whenever next time you want to use the python3.6 as you default python shell. Like below:-

cd /root/mynlp && source bin/activate
(mynlp) # python
Python 3.6.7 (default, Dec  5 2018, 15:02:05)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Hope this will help! Next I’ll give you a demo on NLP package using Python3.6+.

Happy Machine Learning!!!

Leave a Reply

Your email address will not be published. Required fields are marked *