Many times the default version bundled with HDP i.e 2.7 is not sufficient to explore few libraries and it required python to add addition version as in my case I need python 3.6+ to explore NLP libraries. I have done this many times with another version of HDP but this time I want to create a cheat-sheet so that fire and forget would not be a problem in the future.
Centos 7 vanilla or HDP 3.0.1virtual box still does not have a package for Python version 3. So it’s EPEL to the rescue: (you need to be root for this).
*Be careful not run below command on dev/prod environment where many users are connected as it refreshes the packages/iptable with new system packages.
#yum -y install epel-releaseIn my case, it shows up to the latest version. Now let us first download rpm from Centos community using yum.
#yum install https://centos7.iuscommunity.org/ius-release.rpm
Examining /var/tmp/yum-root-SmdyGj/ius-release.rpm: ius-release-1.0-15.ius.centos7.noarch
Marking /var/tmp/yum-root-SmdyGj/ius-release.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package ius-release.noarch 0:1.0-15.ius.centos7 will be installed
--> Processing Dependency: epel-release = 7 for package: ius-release-1.0-15.ius.centos7.noarch
Loading mirror speeds from cached hostfile
* base: centos.excellmedia.net
* elrepo: mirrors.coreix.net
* extras: centos.excellmedia.net
* updates: centos.excellmedia.net
--> Running transaction check
---> Package epel-release.noarch 0:7-11 will be installed
--> Finished Dependency ResolutionNow install python3.6, devenv, and pip3.6 using yum.
#yum install python36u python36u-devel python36u-pip
..
..
..
Running transaction
Updating : 1:openssl-libs-1.0.2k-16.el7.x86_64 1/9
Installing : python36u-3.6.7-1.ius.centos7.x86_64 2/9
Installing : python36u-libs-3.6.7-1.ius.centos7.x86_64 3/9
Installing : python36u-setuptools-39.0.1-1.ius.centos7.noarch 4/9
Installing : python36u-pip-9.0.1-1.ius.centos7.noarch 5/9
Installing : python36u-devel-3.6.7-1.ius.centos7.x86_64 6/9
Updating : 1:openssl-1.0.2k-16.el7.x86_64 7/9
Cleanup : 1:openssl-1.0.1e-42.el7.9.x86_64 8/9
Cleanup : 1:openssl-libs-1.0.1e-42.el7.9.x86_64 9/9
Verifying : 1:openssl-libs-1.0.2k-16.el7.x86_64 1/9
Verifying : python36u-setuptools-39.0.1-1.ius.centos7.noarch 2/9
Verifying : python36u-devel-3.6.7-1.ius.centos7.x86_64 3/9
Verifying : python36u-pip-9.0.1-1.ius.centos7.noarch 4/9
Verifying : 1:openssl-1.0.2k-16.el7.x86_64 5/9
Verifying : python36u-libs-3.6.7-1.ius.centos7.x86_64 6/9
Verifying : python36u-3.6.7-1.ius.centos7.x86_64 7/9
Verifying : 1:openssl-libs-1.0.1e-42.el7.9.x86_64 8/9
Verifying : 1:openssl-1.0.1e-42.el7.9.x86_64 9/9
Installed:
python36u.x86_64 0:3.6.7-1.ius.centos7 python36u-devel.x86_64 0:3.6.7-1.ius.centos7 python36u-pip.noarch 0:9.0.1-1.ius.centos7
Dependency Installed:
python36u-libs.x86_64 0:3.6.7-1.ius.centos7 python36u-setuptools.noarch 0:39.0.1-1.ius.centos7
Dependency Updated:
openssl.x86_64 1:1.0.2k-16.el7 openssl-libs.x86_64 1:1.0.2k-16.el7
Complete!Let us verify the version installed.
# python --version
Python 2.7.5
# python3.6 --version
Python 3.6.7
# pip3.6 --version
pip 9.0.1 from /usr/lib/python3.6/site-packages (python 3.6)Now create a virtual environment to run our python3.6 in an isolated environment.
# pip3.6 install virtualenv
Collecting virtualenv
Downloading https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl (1.9MB)
100% |████████████████████████████████| 1.9MB 805kB/s
Requirement already satisfied: setuptools>=18.0.0 in /usr/lib/python3.6/site-packages (from virtualenv)
Installing collected packages: virtualenv
Successfully installed virtualenv-16.2.0Create a virtual environment for python3.6 like below:-
# virtualenv -p python3.6 mynlp
Already using interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /root/mynlp/bin/python3.6
Also creating executable in /root/mynlp/bin/python
Installing setuptools, pip, wheel...
done.You need to activate the environment whenever next time you want to use the python3.6 as you default python shell. Like below:-
cd /root/mynlp && source bin/activate
(mynlp) # python
Python 3.6.7 (default, Dec 5 2018, 15:02:05)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>Hope this will help! Next I’ll give you a demo on NLP package using Python3.6+.
Happy Machine Learning!!!