测试环境:Redhat 7
源码:gpdb-master(2020.08.20)
gpdb-6X_STABLE 源码编译安装请参考:Greenplum 6.X 从源码快速搭建demo环境
一、下载源码
二、依赖
假设:源码路径:/opt/gpdb/,bin安装到:/opt/gp_bin/。
依赖库
- snippet.bash
sudo yum install apr apr-util bash bzip2 curl krb5 libcurl libevent libxml2 libyaml zlib openldap openssh openssl openssl-libs perl readline rsync R sed tar zip
devel依赖库
- snippet.bash
libzstd、libzstd-devel、apr-devel、libcurl-devel、bzip2-devel、xerces-c-devel
三、升级gcc
gpdb master目前要求gcc支持C++14,但是Redhat 7自带的gcc版本是4.8,尚不支持”C++14“,需要至少升级到gcc 4.9.0。
如果不升级gcc,编译阶段会报以下错误:
- snippet.bash
checking whether g++ supports C++14 features with -std=c++14... no checking whether g++ supports C++14 features with +std=c++14... no checking whether g++ supports C++14 features with -h std=c++14... no checking whether g++ supports C++14 features with -std=c++1y... no checking whether g++ supports C++14 features with +std=c++1y... no checking whether g++ supports C++14 features with -h std=c++1y... no configure: error: *** A compiler with support for C++14 language features is required.
这里直接升级到gcc 9.3,参考:CentOS 7 升级gcc至9.3
四、安装pygresql5
gpdb-master 已经将PyGreSQL 从4.0.0 升级至 5.1.2,需要通过以下方式升级pygresql:
- snippet.bash
pip install pygresql
如果不升级,在后续的”make create-demo-cluster”阶段会报以下错误:
- snippet.bash
Error: unable to import module: No module named pg
五、配置demo
该demo会在本地服务器,部署GP集群,配置包括:
- 3个segment
- 3个mirror
- 数据目录在:/opt/gpdb/gpAux/gpdemo/datadirs/
如果需要定制demo集群,则需要修改:gpAux/gpdemo/Makefile。包括:端口号、segment个数、是否有mirror等。
- snippet.makefile
PORT_BASE ?= 6000 NUM_PRIMARY_MIRROR_PAIRS ?= 3 BLDWRAP_POSTGRES_CONF_ADDONS := fsync=off $(BLDWRAP_POSTGRES_CONF_ADDONS) ifeq ($(WITH_MIRRORS), ) WITH_MIRRORS = true endif ifeq ($(WITH_MIRRORS), true) WITH_STANDBY = true endif
修改demo默认端口号
默认的端口号6000与Xserver端口冲突,如果不修改,在有GUI的Linux环境中,几乎总会端口冲突。
- snippet.bash
vim gpAux/gpdemo/Makefile # PORT_BASE 修改为不会冲突的端口号 PORT_BASE ?= 6000
编译
- snippet.bash
cd /opt/gpdb/ # --prefix 指定安装路径 ./configure --with-perl --with-python --with-libxml --with-gssapi --prefix=/opt/gp_bin # 以上默认是release版本,如果需要debug,则: CFLAGS="-O0 -g3 -ggdb3" ./configure --with-perl --with-python --with-libxml --with-gssapi --enable-debug --enable-debug-extensions --prefix=/opt/gp_bin # 编译安装 make -j8 make install
注:编译debug版本时,如果不指定CFLAGS,仅使用enable-debug参数,编译出来的查询是加了优化参数的,不便于debug。
make clean
为本地用户配置互信
- snippet.bash
[gpadmin@test-4 ~]$ ssh-keygen [gpadmin@test-4 ~]$ ssh-copy-id gpadmin@localhost
六、部署demo
- snippet.bash
# GP环境变量 source /opt/gp_bin/greenplum_path.sh # 部署demo集群并启动服务 cd /opt/gpdb/ make create-demo-cluster # 集群demo环境变量 source gpAux/gpdemo/gpdemo-env.sh
问题1:No psutil
make create-demo-cluster阶段,如果报告以下错误:
- snippet.txt
ImportError: No module named psutil
则,需要安装python模块:
- snippet.bash
[gpadmin@bogon ~]$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py [gpadmin@bogon ~]$ python get-pip.py [gpadmin@bogon ~]$ pip install --upgrade psutil==5.7.0
注意:因为GP自带了Python,当“source greenplum_path.sh”后,实际上使用的是GP的Python的,需要在编译环境中安装psutil。
- snippet.bash
[gpadmin@bogon gpdb]$ which python /opt/gp/greenplum/greenplum-db-6.9.0/ext/python/bin/python
问题2:ping host FAILED
安装失败,报告以下warning:
- snippet.txt
20200601:11:06:44:029396 gpstart:bogon:gpadmin-[WARNING]:-Ping to host: 'bogon' FAILED 20200601:11:06:44:029396 gpstart:bogon:gpadmin-[WARNING]:-Skipping startup of segdb on bogon directory /opt/software/gpdb-6X_STABLE/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0 Ping Failed <<<<<<
怀疑GP脚本使用了ping6而不是ping,可以通过以下方法鉴别:
- snippet.bash
[gpadmin@bogon gpdb]$ cd /opt/gpdb/gpAux/gpdemo/ [gpadmin@bogon gpdb]$ rm -rf datadirs/* [gpadmin@bogon gpdb]$ gpinitsystem -D -a -c clusterConfigFile "" …… 20200601:11:11:35:033652 gpstart:bogon:gpadmin-[DEBUG]:-Running Command: /bin/ping6 -c 1 bogon 20200601:11:11:35:033652 gpstart:bogon:gpadmin-[DEBUG]:-[worker0] finished cmd: ping cmdStr='/bin/ping6 -c 1 bogon' had result: cmd had rc=2 completed=True halted=False
可以修改以下脚本,然后重新make install:
- snippet.python
[gpadmin@bogon gpdb]$ vim gpMgmt/bin/gppylib/commands/unix.py 296 def run(self, validateAfter=False): 297 if curr_platform == LINUX or curr_platform == DARWIN or curr_platform == OPENBSD: 298 # Get the family of the address we need to ping. If it's AF_INET6 299 # we must use ping6 to ping it. 300 301 try: 302 # addrinfo = socket.getaddrinfo(self.hostToPing, None) 303 addrinfo = socket.getaddrinfo(self.hostToPing, None, socket.AF_INET) # 修改后 304 if addrinfo and addrinfo[0] and addrinfo[0][0] == socket.AF_INET6:
问题3:Failed to load GLFW3 shared library
- snippet.txt
20210101:15:07:33:2262125 gpinitsystem:test-4:yz-[INFO]:-Restarting the Greenplum instance in production mode Error: unable to import module: Failed to load GLFW3 shared library. 20210101:15:07:34:2262125 gpinitsystem:test-4:yz-[FATAL]:-Error from Greenplum instance shutdown, check log files Script Exiting!
原因是安装python3环境安装了pg模块,需要卸载pg模块,使用pygresql5代替之:
- snippet.bash
sudo pip3 uninstall pg
该问题可以参考:Import error when trying to load import "pg" module in python3