`
bupt04406
  • 浏览: 343334 次
  • 性别: Icon_minigender_1
  • 来自: 杭州
社区版块
存档分类
最新评论

hive hiveconf 配置

    博客分类:
  • Hive
阅读更多
hadoop fs -count -q


配置环境变量:
# hadoop env
export HADOOP_HOME=
export HADOOP_CONF_DIR=
# hive env
export HIVE_HOME=
export HIVE_CONF_DIR=
export HIVE_AUX_JARS_PATH=
在HIVE_CONF_DIR下面需要有hive-default.xml

HiveConf    hive-default.xml  hive-site.xml

HiveConf:

hive.exec.mode.local.auto  true  决定 Hive 是否应该自动地根据输入文件大小,在本地运行(在GateWay运行)
hive.auto.convert.join false  是否根据输入小表的大小,自动将 Reduce 端的 Common Join 转化为 Map Join,从而加快大表关联小表的 Join 速度。

metastore相关配置:
hive.metastore.local  true or false   local or remote metastore 
HiveMetaStoreClient里面:
HiveMetaStoreClient(HiveConf conf, HiveMetaHookLoader hookLoader){
    boolean localMetaStore = conf.getBoolean("hive.metastore.local", false);
    if (localMetaStore) { //true,本地metastore
      // instantiate the metastore server handler directly instead of connecting
      // through the network
      client = new HiveMetaStore.HMSHandler("hive client", conf);
      open = true;
      return;
    }
}



Hive Configuration Variables
hive.exec.scratchdir This directory is used by hive to store the plans for different map/reduce stages for the query as well as to stored the intermediate outputs of these stages. /tmp/<user.name>/hive
hive.exec.compress.output Determines whether the output of the final map/reduce job in a query is compressed or not. false
hive.exec.compress.intermediate Determines whether the output of the intermediate map/reduce jobs in a query is compressed or not. false

mapred.reduce.tasks :reduce数目
The default number of reduce tasks per job.  Typically set to a prime close to the number of available hosts.  Ignored when  mapred.job.tracker is "local". Hadoop set this to 1 by default, whereas hive uses -1 as its default value.  By setting this property to -1, Hive will automatically figure out what should be the number of reducers.


hive.default.fileformat
<property>
  <name>hive.default.fileformat</name>
  <value>SequenceFile</value>
  <description>Default file format for CREATE TABLE statement. Options are TextFile and SequenceFile. Users can explicitly say CREAT
E TABLE ... STORED AS &lt;TEXTFILE|SEQUENCEFILE&gt; to override</description>
</property>
默认创建的文件是SequenceFile,因为textfile在一些压缩算法下文件不可切分。
分享到:
评论

相关推荐

Global site tag (gtag.js) - Google Analytics