hive的安装和配置 1. download wget http://mirror.mel.bkb.net.au/pub/apache//hive/stable/hive-0.8.1.tar.gz tar zxf hive-0.8.1.tar.gz 只需要在一个节点上安装 2. 设置环境变量 vi .bash_profile export java_home=/usr/lib/jvm/java-1.6.0-openjdk-1.6
hive的安装和配置
1. download
wget http://mirror.mel.bkb.net.au/pub/apache//hive/stable/hive-0.8.1.tar.gz
tar zxf hive-0.8.1.tar.gz
只需要在一个节点上安装
2. 设置环境变量
vi .bash_profile
export java_home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
export hadoop_home=/home/hadoop/hadoop-1.0.0
export hive_home=/home/hadoop/hive-0.8.1
export hadoop_conf_dir=$home/conf
export hive_conf_dir=$home/hive-conf
export classpath=$hive_home/lib:$java_home/lib:$java_home/jre/lib:$hadoop_home
export path=$hive_home/bin:$hadoop_home/bin:$java_home/bin:/sbin/:/bin:$path
3. 配置hive
cp -r hive-0.8.1/conf $hive_conf_dir/
cd $hive_conf_dir/
cp hive-default.xml.template hive-default.xml
cat hive-env.sh
export hadoop_heapsize=512
export hive_conf_dir=/home/hadoop/hive-conf
3. 测试
$ hive
hive> show tables;
ok
time taken: 4.824 seconds
hive> create table hwz(id int, name string);
ok
time taken: 0.566 seconds
hive> select * from hwz;
ok
time taken: 0.361 seconds
$ hadoop dfs -lsr /user/hive
warning: $hadoop_home is deprecated.
drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse
drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse/hwz
4. 配置metastore用mysql数据库,这样才可以多用户同时访问
a. create user and database for hive in mysql
create database hive;
grant all on hive.* to hive@'%' identified by 'hivepass';
b. change metastore to use mysql
cat hive-site.xml
hive.metastore.local
true
javax.jdo.option.connectionurl
jdbc:mysql://slave1:3306/hive?createdatabaseifnotexist=true
javax.jdo.option.connectiondrivername
com.mysql.jdbc.driver
javax.jdo.option.connectionusername
hive
javax.jdo.option.connectionpassword
hivepass
c. 检查
$ hive
hive> use dw2;
ok
time taken: 3.43 seconds
hive> create table hwz2(id int, name string ) row format delimited fields terminated by ',';
ok
time taken: 2.519 seconds
hive> show tables;
ok
hwz2
time taken: 0.419 seconds
hive> load data local inpath 'demo.txt' overwrite into table hwz2;
copying data from file:/home/hadoop/demo.txt
copying file: file:/home/hadoop/demo.txt
loading data to table dw2.hwz2
deleted hdfs://master:9000/user/hive/warehouse/dw2.db/hwz2
ok
time taken: 0.557 seconds
hive> select * from hwz2;
ok
12 jack
12 jack
12 jack
12 jack
12 jack
12 jack
12 jack
12 jack
$ hadoop dfs -lsr /user/hive
warning: $hadoop_home is deprecated.
drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:36 /user/hive/warehouse
drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:48 /user/hive/warehouse/dw2.db
drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:48 /user/hive/warehouse/dw2.db/hwz2
-rw-r--r-- 2 hadoop supergroup 1201 2012-03-22 15:48 /user/hive/warehouse/dw2.db/hwz2/demo.txt
drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse/hwz
drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:36 /user/hive/warehouse/hwz2
-rw-r--r-- 2 hadoop supergroup 1201 2012-03-22 15:36 /user/hive/warehouse/hwz2/demo.txt
$ hadoop dfs -cat /user/hive/warehouse/dw2.db/hwz2/demo.txt |head
warning: $hadoop_home is deprecated.
12,jack
12,jack
12,jack
12,jack
12,jack
12,jack
12,jack
12,jack
12,jack
12,jack
d. 在mysql中验证建立的新表
mysql> use hive;
database changed
mysql> show tables;
+-----------------+
| tables_in_hive |
+-----------------+
| bucketing_cols |
| cds |
| columns_v2 |
| database_params |
| dbs |
| partition_keys |
| sds |
| sd_params |
| sequence_table |
| serdes |
| serde_params |
| sort_cols |
| table_params |
| tbls |
+-----------------+
14 rows in set (0.00 sec)
9. 常见错误
error 1:
-------------------------------------------------
hive> show tables;
failed: error in metadata: javax.jdo.jdofatalinternalexception: error creating transactional connection factory
solution:
hive不带mysql jdbc驱动,自己安装:
wget http://dev.mysql.com/get/downloads/connector-j/mysql-connector-java-5.1.18.tar.gz/from/http://mysql.mirror.kangaroot.net/
tar zxf mysql-connector-java-5.1.18.tar.gz
cd mysql-connector-java-5.1.18
cp mysql-connector*.jar $hive_home/lib
error 2:
-------------------------------------------------
hive> show tables;
failed: error in metadata: javax.jdo.jdoexception: couldnt obtain a new sequence (unique id) : cannot execute statement: impossible to write to binary log since binlog_format = statement and at least one table uses a storage engine limited to row-based logging. innodb is limited to row-logging when transaction isolation level is read committed or read uncommitted.
solution:
在mysql中设置 binlog_format='mixed'