花灰资源网

花灰资源网

0722-6.2.0-如何在RedHat7.2使用rpm安装CDH(无CM)

yuhuahui 0 4 次浏览 未分类

3.4 Spark


1.安装Spark所需的包

yum -y install spark-core spark-master spark-worker spark-history-server spark-python




2.创建目录并修改属主和权限

sudo -u hdfs hadoop fs -mkdir /user/spark
sudo -u hdfs hadoop fs -mkdir /user/spark/applicationHistory
sudo -u hdfs hadoop fs -chown -R spark:spark /user/spark
sudo -u hdfs hadoop fs -chmod 1777 /user/spark/applicationHistory





3.修改配置文件
/etc/spark/conf/spark-defaults.conf

spark.eventLog.enabled=true
spark.eventLog.dir=hdfs://cdh178.macro.com:8020/user/spark/applicationHistory
spark.yarn.historyServer.address=http://cdh178.macro.com:18088





4.启动spark-history-server

systemctl start spark-history-server
systemctl status spark-history-server





访问Web UI




5.修改配置文件并同步到所有节点




6.测试Spark使用



至此Spark安装完成

3.5 Hive


1.安装Hive服务之前，先安装元数据库MySQL并创建好服务需要的库和用户如下：

create database metastore default character set utf8; 
CREATE USER 'hive'@'%' IDENTIFIED BY 'password'; 
GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%'; 
FLUSH PRIVILEGES;





2.安装Hive服务的包

在NameNode节点hive-metastore

yum -y install hive-metastore





在所有节点安装其他所需的包

yum -y install hive hive-server2 hive-jdbc hive-hbase




3.创建目录

在HDFS上创建目录并设置权限以及修改属主

sudo -u hdfs hadoop fs -mkdir /user/hive
sudo -u hdfs hadoop fs -chown hive:hive /user/hive
sudo -u hdfs hadoop fs -mkdir /user/hive/warehouse
sudo -u hdfs hadoop fs -chmod 1777 /user/hive/warehouse
sudo -u hdfs hadoop fs -chown hive:hive /user/hive/warehouse





4.修改配置文件

/etc/hive/conf/hive-site.xml

<configuration>
 <property>
 <name>javax.jdo.option.ConnectionURL</name>
 <value>jdbc:mysql://cdh178.macro.com:3306/metastore?useUnicode=true&characterEncoding=UTF-8</value>
 </property>
 <property>
 <name>javax.jdo.option.ConnectionDriverName</name>
 <value>com.mysql.jdbc.Driver</value>
 </property>
 <property>
 <name>javax.jdo.option.ConnectionUserName</name>
 <value>hive</value>
 </property>
 <property>
 <name>javax.jdo.option.ConnectionPassword</name>
 <value>password</value>
 </property>
 <property>
 <name>datanucleus.schema.autoCreateAll</name>
 <value>false</value>
 </property>
 <property>
 <name>yarn.resourcemanager.resource-tracker.address</name>
 <value>cdh178.macro.com:8031</value>
 </property>
 <property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
 </property>
 <property>
 <name>hive.exec.reducers.max</name>
 <value>1099</value>
 </property>
 <property>
 <name>hive.metastore.schema.verification</name>
 <value>true</value>
 </property>
 <property>
 <name>hive.metastore.warehouse.dir</name>
 <value>/user/hive/warehouse</value>
 </property>
 <property>
 <name>hive.warehouse.subdir.inherit.perms</name>
 <value>true</value>
 </property>
 <property>
 <name>hive.metastore.server.min.threads</name>
 <value>200</value>
 </property>
 <property>
 <name>hive.metastore.server.max.threads</name>
 <value>100000</value>
 </property>
 <property>
 <name>hive.metastore.client.socket.timeout</name>
 <value>3600</value>
 </property>
 <property>
 <name>hive.support.concurrency</name>
 <value>true</value>
 </property>
 <property>
 <name>hive.zookeeper.quorum</name>
 <value>cdh178.macro.com,cdh177.macro.com,cdh176.macro.com</value>
 </property>
 <property>
 <name>hive.zookeeper.client.port</name>
 <value>2181</value>
 </property>
</configuration>






/etc/hadoop/conf/core-site.xml，只贴出修改的部分

<property>
 <name>hadoop.proxyuser.hive.hosts</name>
 <value>*</value>
</property>
<property>
 <name>hadoop.proxyuser.hive.groups</name>
 <value>*</value>
</property>





5.将配置文件同步到所有节点




6.将MySQL驱动包在Hive服务的lib目录下设置软链




7.启动Hive服务

启动hive-metastore

systemctl start hive-metastore
systemctl status hive-metastore





启动hive-server2

systemctl start hive-server2
systemctl status hive-server2





8.测试Hive服务是否正常

连接Hive，建表正常




至此Hive安装完成

3.6 Oozie


1.在MySQL中创建Oozie服务所需要的库和用户

create database oozie default character set utf8; 
CREATE USER 'oozie'@'%' IDENTIFIED BY 'password'; 
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%'; 
FLUSH PRIVILEGES;





2.安装Oozie的包

yum -y install oozie oozie-client





3.配置Oozie

配置Oozie使用Yarn

alternatives --set oozie-tomcat-deployment /etc/oozie/tomcat-conf.http





修改
/etc/oozie/conf/oozie-site.xml配置文件

<property>
 <name>oozie.service.JPAService.jdbc.driver</name>
 <value>com.mysql.jdbc.Driver</value>
 </property>
 <property>
 <name>oozie.service.JPAService.jdbc.url</name>
 <value>jdbc:mysql://cdh178.macro.com:3306/oozie</value>
 </property>
 <property>
 <name>oozie.service.JPAService.jdbc.username</name>
 <value>oozie</value>
 </property>
 <property>
 <name>oozie.service.JPAService.jdbc.password</name>
 <value>password</value>
 </property>





将MySQL驱动包在Oozie目录下生成软链



4.运行Oozie数据库工具

sudo -u oozie /usr/lib/oozie/bin/ooziedb.sh create -run




5.配置Oozie的Web控制台

下载ExtJS library到服务器，地址如下：

https://archive.cloudera.com/gplextras/misc/ext-2.2.zip





将下载的包解压到/var/lib/oozie

unzip ext-2.2.zip -d /var/lib/oozie/




6.在HDFS中安装Oozie共享库

sudo -u hdfs hadoop fs -mkdir /user/oozie
sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie
sudo oozie-setup sharelib create -fs hdfs://cdh178.macro.com:8020 -locallib /usr/lib/oozie/oozie-sharelib-yarn





7.启动Oozie Server

systemctl start oozie
systemctl status oozie



8.访问Oozie服务的Web UI



至此Oozie服务安装完成

3.7 Impala


1.安装Impala的包

在一个节点上安装Impala Catalog Server和Impala StateStore

yum -y install impala-state-store impala-catalog




在所有节点安装其他的包

yum -y install impala impala-server




2.将Impala需要的配置文件拷贝到Impala的配置文件目录下




3.安装impala-shell

yum -y install impala-shell





4.安装完Impala后需要的配置

修改
/etc/hadoop/conf/hdfs-site.xml配置文件，启用块位置追踪和短路读取

<property>
 <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
 <value>true</value>
</property> 
 <property>
 <name>dfs.client.read.shortcircuit</name>
 <value>true</value>
 </property>
 <property>
 <name>dfs.domain.socket.path</name>
 <value>/var/run/hdfs-sockets/dn</value>
 </property>
 <property>
 <name>dfs.client.file-block-storage-locations.timeout.millis</name>
 <value>10000</value>
 </property>





将配置同步到所有节点




重启所有DataNode




将修改后的hdfs-site.xml复制到Impala的配置文件目录




5.启动Impala服务

启动Impala Catalog Server和Impala StateStore

systemctl start impala-state-store
systemctl status impala-state-store
systemctl start impala-catalog
systemctl status impala-catalog





所有节点启动impala-server

systemctl start impala-server
systemctl status impala-server





6.测试Impala使用

使用impala-shell连接Impala，进行插入、查询操作成功




至此Impala安装完成

3.8 Hue


1.安装Hue的包

yum -y install hue





2.为Hue配置CDH组件

  • 配置Hue访问HDFS

1）修改配置文件

/etc/hadoop/conf/hdfs-site.xml

<property>
 <name>dfs.webhdfs.enabled</name>
 <value>true</value>
 </property>





/etc/hadoop/conf/core-site.xml

 <property>
 <name>hadoop.proxyuser.hue.hosts</name>
 <value>*</value>
 </property>
 <property>
 <name>hadoop.proxyuser.hue.groups</name>
 <value>*</value>
 </property>





/etc/hue/conf/hue.ini




将修改的HDFS的配置文件同步到所有节点




2）重启HDFS服务

systemctl restart hadoop-hdfs-namenode
systemctl restart hadoop-hdfs-secondarynamenode
systemctl restart hadoop-hdfs-datanode





  • 配置Hue集成Hive

修改配置文件/etc/hue/conf/hue.ini




3.创建Hue服务所需的数据库和用户

create database hue default character set utf8; 
CREATE USER 'hue'@'%' IDENTIFIED BY 'password'; 
GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%'; 
FLUSH PRIVILEGES;





修改配置文件/etc/hue/conf/hue.ini数据库的部分




4.初始化数据库

/usr/lib/hue/build/env/bin/hue syncdb
/usr/lib/hue/build/env/bin/hue migrate





5.启动Hue服务

systemctl start hue
systemctl status hue





6.访问Hue服务的Web UI




至此Hue服务安装完成

总结

1.无CM使用rpm的方式安装CDH6.2.0与之前安装CDH5.10.0基本没有太大的区别。

2.此安装方式需要下载相关的所有rpm包到服务器，然后制作本地的yum源进行安装，下载的包的总大小在4.3G左右。

3.同样的在安装过程中需要最先安装Zookeeper。

原文链接：，转发请注明来源！
花灰资源网 › 未分类 › 0722-6.2.0-如何在RedHat7.2使用rpm安装CDH(无CM)