FAIRYFAR-INTERNAL
 
  FAIRYFAR-INTERNAL  |  SITEMAP  |  ABOUT-ME  |  HOME  
Greenplum master 从源码快速搭建demo环境

测试环境:Redhat 7
源码:gpdb-master(2020.08.20)
gpdb-6X_STABLE 源码编译安装请参考:Greenplum 6.X 从源码快速搭建demo环境

一、下载源码

二、依赖

假设:源码路径:/opt/gpdb/,bin安装到:/opt/gp_bin/。

依赖库

snippet.bash
sudo yum install apr apr-util bash bzip2 curl krb5 libcurl libevent libxml2 libyaml zlib openldap openssh openssl openssl-libs perl readline rsync R sed tar zip

devel依赖库

snippet.bash
libzstd、libzstd-devel、apr-devel、libcurl-devel、bzip2-devel、xerces-c-devel

三、升级gcc

gpdb master目前要求gcc支持C++14,但是Redhat 7自带的gcc版本是4.8,尚不支持”C++14“,需要至少升级到gcc 4.9.0。

如果不升级gcc,编译阶段会报以下错误:

snippet.bash
checking whether g++ supports C++14 features with -std=c++14... no
checking whether g++ supports C++14 features with +std=c++14... no
checking whether g++ supports C++14 features with -h std=c++14... no
checking whether g++ supports C++14 features with -std=c++1y... no
checking whether g++ supports C++14 features with +std=c++1y... no
checking whether g++ supports C++14 features with -h std=c++1y... no
configure: error: *** A compiler with support for C++14 language features is required.

这里直接升级到gcc 9.3,参考:CentOS 7 升级gcc至9.3

四、安装pygresql5

gpdb-master 已经将PyGreSQL 从4.0.0 升级至 5.1.2,需要通过以下方式升级pygresql:

snippet.bash
pip install pygresql

如果不升级,在后续的”make create-demo-cluster”阶段会报以下错误:

snippet.bash
Error: unable to import module: No module named pg

参考:Update PyGreSQL from 4.0.0 to 5.1.2

五、配置demo

该demo会在本地服务器,部署GP集群,配置包括:

  • 3个segment
  • 3个mirror
  • 数据目录在:/opt/gpdb/gpAux/gpdemo/datadirs/

如果需要定制demo集群,则需要修改:gpAux/gpdemo/Makefile。包括:端口号、segment个数、是否有mirror等。

snippet.makefile
PORT_BASE ?= 6000
NUM_PRIMARY_MIRROR_PAIRS ?= 3
BLDWRAP_POSTGRES_CONF_ADDONS := fsync=off $(BLDWRAP_POSTGRES_CONF_ADDONS)
 
ifeq ($(WITH_MIRRORS), )
WITH_MIRRORS = true
endif
 
ifeq ($(WITH_MIRRORS), true)
WITH_STANDBY = true
endif

修改demo默认端口号

默认的端口号6000与Xserver端口冲突,如果不修改,在有GUI的Linux环境中,几乎总会端口冲突。

snippet.bash
vim gpAux/gpdemo/Makefile
# PORT_BASE 修改为不会冲突的端口号
PORT_BASE ?= 6000

编译

snippet.bash
cd /opt/gpdb/
 
# --prefix 指定安装路径
./configure --with-perl --with-python --with-libxml --with-gssapi --prefix=/opt/gp_bin
# 以上默认是release版本,如果需要debug,则:
CFLAGS="-O0 -g3 -ggdb3" ./configure --with-perl --with-python --with-libxml --with-gssapi --enable-debug --enable-debug-extensions --prefix=/opt/gp_bin
 
# 编译安装
make -j8
make install

注:编译debug版本时,如果不指定CFLAGS,仅使用enable-debug参数,编译出来的查询是加了优化参数的,不便于debug。

make clean

如果需要重新编译,重新编译前:

snippet.bash
make distclean

参考:「实战系列」Greenplum 编译、安装、调试

为本地用户配置互信

snippet.bash
[gpadmin@test-4 ~]$ ssh-keygen
[gpadmin@test-4 ~]$ ssh-copy-id gpadmin@localhost

六、部署demo

snippet.bash
# GP环境变量
source /opt/gp_bin/greenplum_path.sh
 
# 部署demo集群并启动服务
cd /opt/gpdb/
make create-demo-cluster
 
# 集群demo环境变量
source gpAux/gpdemo/gpdemo-env.sh

问题1:No psutil

make create-demo-cluster阶段,如果报告以下错误:

snippet.txt
ImportError: No module named psutil

则,需要安装python模块:

snippet.bash
[gpadmin@bogon ~]$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
[gpadmin@bogon ~]$ python get-pip.py
[gpadmin@bogon ~]$ pip install --upgrade psutil==5.7.0

注意:因为GP自带了Python,当“source greenplum_path.sh”后,实际上使用的是GP的Python的,需要在编译环境中安装psutil。

snippet.bash
[gpadmin@bogon gpdb]$ which python
/opt/gp/greenplum/greenplum-db-6.9.0/ext/python/bin/python

问题2:ping host FAILED

安装失败,报告以下warning:

snippet.txt
20200601:11:06:44:029396 gpstart:bogon:gpadmin-[WARNING]:-Ping to host: 'bogon' FAILED
20200601:11:06:44:029396 gpstart:bogon:gpadmin-[WARNING]:-Skipping startup of segdb on bogon directory /opt/software/gpdb-6X_STABLE/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0 Ping Failed <<<<<<

怀疑GP脚本使用了ping6而不是ping,可以通过以下方法鉴别:

snippet.bash
[gpadmin@bogon gpdb]$ cd /opt/gpdb/gpAux/gpdemo/
[gpadmin@bogon gpdb]$ rm -rf datadirs/*
 
[gpadmin@bogon gpdb]$ gpinitsystem -D -a -c clusterConfigFile ""
……
20200601:11:11:35:033652 gpstart:bogon:gpadmin-[DEBUG]:-Running Command: /bin/ping6 -c 1 bogon
20200601:11:11:35:033652 gpstart:bogon:gpadmin-[DEBUG]:-[worker0] finished cmd: ping cmdStr='/bin/ping6 -c 1 bogon'  had result: cmd had rc=2 completed=True halted=False

可以修改以下脚本,然后重新make install:

snippet.python
[gpadmin@bogon gpdb]$ vim gpMgmt/bin/gppylib/commands/unix.py
296     def run(self, validateAfter=False):
297         if curr_platform == LINUX or curr_platform == DARWIN or curr_platform == OPENBSD:
298             # Get the family of the address we need to ping.  If it's AF_INET6
299             # we must use ping6 to ping it.
300
301             try:
302                 # addrinfo = socket.getaddrinfo(self.hostToPing, None)
303                 addrinfo = socket.getaddrinfo(self.hostToPing, None, socket.AF_INET) # 修改后
304                 if addrinfo and addrinfo[0] and addrinfo[0][0] == socket.AF_INET6:

该问题可以参考:Missing steps for first installation #171

问题3:Failed to load GLFW3 shared library

snippet.txt
20210101:15:07:33:2262125 gpinitsystem:test-4:yz-[INFO]:-Restarting the Greenplum instance in production mode
Error: unable to import module: Failed to load GLFW3 shared library.
20210101:15:07:34:2262125 gpinitsystem:test-4:yz-[FATAL]:-Error from Greenplum instance shutdown, check log files Script Exiting!

原因是安装python3环境安装了pg模块,需要卸载pg模块,使用pygresql5代替之:

snippet.bash
sudo pip3 uninstall pg

该问题可以参考:Import error when trying to load import "pg" module in python3



打赏作者以资鼓励: