GVKun编程网logo

编辑seaborn legend(编辑部的故事)

10

在这里,我们将给大家分享关于编辑seabornlegend的知识,让您更了解编辑部的故事的本质,同时也会涉及到如何更有效地4.集成学习(EnsembleLearning)Adaboost、com.su

在这里,我们将给大家分享关于编辑seaborn legend的知识,让您更了解编辑部的故事的本质,同时也会涉及到如何更有效地4. 集成学习(Ensemble Learning)Adaboost、com.sun.org.apache.bcel.internal.generic.LocalVariableGen的实例源码、ERROR org.hibernate.search.exception.impl.LogErrorHandler - HSEARCH000058: HSEARCH000183: Unable ...、HIVE出现Read past end of RLE integer from compressed stream Stream for column 1 kind LENGTH positio...的内容。

本文目录一览:

编辑seaborn legend(编辑部的故事)

编辑seaborn legend(编辑部的故事)

使用数据帧和Python中的代码,我可以创建一个绘图:

g = sns.lmplot(''credibility'', ''percentWatched'', data=data, hue = ''millennial'', markers = ["+", "."], x_jitter = True, y_jitter = True, size=5)g.set(xlabel = ''Credibility Ranking\n ← Low       High  →'', ylabel = ''Percent of Video Watched [%]'')

但是传说中说“+0”和“。1“对读者没什么帮助。如何编辑图例的标签?最好不要说“千禧年”吧会说“一代”和“+Millennial”。老一辈人”

答案1

小编典典

如果“legend_out”设置为“True”,则可以使用“g.\u legend”`
属性,它是图形的一部分。Seaborn legend是标准的matplotlib
图例对象。因此,您可以更改图例文本,例如:

import seaborn as snstips = sns.load_dataset("tips")g = sns.lmplot(x="total_bill", y="tip", hue="smoker", data=tips, markers=["o", "x"], legend_out = True)# titlenew_title = ''My title''g._legend.set_title(new_title)# replace labelsnew_labels = [''label 1'', ''label 2'']for t, l in zip(g._legend.texts, new_labels): t.set_text(l)sns.plt.show()

另一种情况是如果将“legend”设置为“False”。你必须定义
轴有一个图例(在下面的示例中,轴编号为0):

import seaborn as snstips = sns.load_dataset("tips")g = sns.lmplot(x="total_bill", y="tip", hue="smoker", data=tips, markers=["o", "x"], legend_out = False)# check axes and find which is have legendleg = g.axes.flat[0].get_legend()new_title = ''My title''leg.set_title(new_title)new_labels = [''label 1'', ''label 2'']for t, l in zip(leg.texts, new_labels): t.set_text(l)sns.plt.show()

此外,您可以结合两种情况使用以下代码:

import seaborn as snstips = sns.load_dataset("tips")g = sns.lmplot(x="total_bill", y="tip", hue="smoker", data=tips, markers=["o", "x"], legend_out = True)# check axes and find which is have legendfor ax in g.axes.flat:    leg = g.axes.flat[0].get_legend()    if not leg is None: break# or legend may be on a figureif leg is None: leg = g._legend# change legend textsnew_title = ''My title''leg.set_title(new_title)new_labels = [''label 1'', ''label 2'']for t, l in zip(leg.texts, new_labels): t.set_text(l)sns.plt.show()

This code works for any seaborn plot which is based on Grid
class.

4. 集成学习(Ensemble Learning)Adaboost

4. 集成学习(Ensemble Learning)Adaboost

1. 集成学习(Ensemble Learning)原理

2. 集成学习(Ensemble Learning)Bagging

3. 集成学习(Ensemble Learning)随机森林(Random Forest)

4. 集成学习(Ensemble Learning)Adaboost

5. 集成学习(Ensemble Learning)GBDT

6. 集成学习(Ensemble Learning)算法比较

7. 集成学习(Ensemble Learning)Stacking

1. 前言

前面我们已经详细的介绍了什么是集成学习,集成学习的种类有哪些,并且也介绍了集成学习里面相对比较简单的Bagging。今天我们开始涉及到集成学习里面比较难理解的Boosting中的Adaboost。

2. Boosting原理

Boosting算法的工作机制:

  1. 首先从训练集用初始权重训练出一个弱学习器1.
  2. 根据学习器1的学习误差率来更新训练样本的权重,使得之前弱学习器1学习误差率高的训练样本点的权重变高,使得这些误差率高的点在后面的弱学习器2中得到更多的重视。
  3. 然后基于调整权重后的训练集来训练弱学习器2。
  4. 如此重复进行,直到弱学习器数达到事先指定的数目$T$。
  5. 最终将这$T$个弱学习器通过集合策略进行整合,得到最终的强学习器。

image

这里有几个相关概念:

  1. 学习误差率$e$
  2. 学习器权重$\alpha$
  3. 更新下一次训练样本的权重$D$
  4. 最后如何结合所有的弱学习器

我们通过详细介绍Adaboost的分类算法,一步一步的讲解上面提到的几个过程

3. Adaboost

我们的训练样本是

$$ T={(x_1,y_1),(x_2,y_2), ...(x_m,y_m)} $$ 训练集的第$k$个弱学习器的输出权重为:

$$ D(k) = (w_{k1}, w_{k2}, ...w_{km}) ;;; w_{1i}=\frac{1}{m};;; i =1,2...m $$

  • 首先我们看下第$k$个弱学习器的错误率$e_k$,因为我们是二分类问题,所以我们的的错误率是所有误分类的样本的权重和如下,其中求和的是误分类的样本的集合。

$$ e_k = \sum\limits_{i=1}^mw_{ki} $$

  • 得到了$e_k$,我们就能去计算弱学习器在最后结合的时候的权重系数$\alpha_k$,根据下面的公式,我们发现,当$e_k$越大,则$\alpha_k$越小。这也符合大家的直观感受。

$$ \alpha_k = \frac{1}{2}log\frac{1-e_k}{e_k} $$

  • 更新下一个弱学习器的权重$D(k+1) = (w_{(k+1)1}, w_{(k+1)2}, ...w_{(k+1)m})$。

$$ w_{k+1,i} = \frac{w_{ki}}{Z_K}exp(-\alpha_ky_iG_k(x_i)) $$

$$ Z_k = \sum\limits_{i=1}^{m}w_{ki}exp(-\alpha_ky_iG_k(x_i)) $$ 上面公式中$G_k()$是第$k$个弱学习器,从$w_{(k+1)i}$计算公式可以看出,如果第$i$个样本分类错误,则$y_iG_k(x_i)<0$,导致样本的权重在第$k+1$个弱分类器中增大,如果分类正确,则权重在第$k+1$个弱分类器中减少.具体为什么采用样本权重更新公式,我们在讲Adaboost的损失函数优化时再讲。

  • 还剩下一个如何组合所有弱学习器的问题

$$ f(x) = sign(\sum\limits_{k=1}^{K}\alpha_kG_k(x)) $$

4. Adaboost分类损失函数

首先我们定义一个Adaboost的损失函数为:

$$ arg;min\sum\limits_{i=1}^{m}exp(-y_if_{k}(x)) $$ 其中

$$ f_{k-1}(x) = \sum\limits_{i=1}^{k-1}\alpha_iG_{i}(x) $$

$$ f_{k}(x) = f_{k-1}(x) + \alpha_kG_k(x) $$ 把$f_k(x)$带入损失函数得到

$$ arg;min\sum\limits_{i=1}^{m}exp[(-y_i) (f_{k-1}(x) + \alpha_k G_k(x))] $$

我们令$w_{ki} = exp(-y_if_{k-1}(x))$, 它的值不依赖于$\alpha, G$,因此与最小化无关,仅仅依赖于$f_{k-1}(x)$,随着每一轮迭代而改变。

将这个式子带入损失函数,损失函数转化为

$$ arg;min\sum\limits_{i=1}^{m}w_{ki}exp[-y_i\alpha G(x)] $$

又因为我们可以计算

$$ G_k(x) = arg;min\sum_{i=1}^mw_{ki} $$ 将$G_k(x)$带入损失函数,并对$\alpha$求导,使其等于0,则就得到了

$$ \alpha_k = \frac{1}{2}log\frac{1-e_k}{e_k} $$ 最后看样本权重的更新。利用$f_{k}(x) = f_{k-1}(x) + \alpha_kG_k(x)$和$w_{ki} = exp(-y_if_{k-1}(x))$,即可得:

$$ w_{k+1,i} = w_{ki}exp[-y_i\alpha_kG_k(x)] $$

5. Adaboost算法的正则化

为了防止Adaboost过拟合,我们通常也会加入正则化项,这个正则化项我们通常称为步长(learning rate)。定义为$\nu$,对于前面的弱学习器的迭代

$$ f_{k}(x) = f_{k-1}(x) + \alpha_kG_k(x) $$

如果我们加上了正则化项,则有

$$ f_{k}(x) = f_{k-1}(x) + \nu\alpha_kG_k(x) $$

$\nu$的取值范围为$0 < \nu \leq 1$。对于同样的训练集学习效果,较小的$\nu$意味着我们需要更多的弱学习器的迭代次数。通常我们用步长和迭代最大次数一起来决定算法的拟合效果。

6. 总结

到这里Adaboost就写完了,前面有一个没有提到,就是弱学习器的类型。理论上任何学习器都可以用于Adaboost.但一般来说,使用最广泛的Adaboost弱学习器是决策树和神经网络。对于决策树,Adaboost分类用了CART分类树,而Adaboost回归用了CART回归树。

这里对Adaboost算法的优缺点做一个总结。 Adaboost的主要优点有:

  1. Adaboost作为分类器时,分类精度很高
  2. 在Adaboost的框架下,可以使用各种回归分类模型来构建弱学习器,非常灵活。
  3. 作为简单的二元分类器时,构造简单,结果可理解。
  4. 不容易发生过拟合

Adaboost的主要缺点有:

  1. 对异常样本敏感,异常样本在迭代中可能会获得较高的权重,影响最终的强学习器的预测准确性。

com.sun.org.apache.bcel.internal.generic.LocalVariableGen的实例源码

com.sun.org.apache.bcel.internal.generic.LocalVariableGen的实例源码

项目:Openjsharp    文件:MethodGenerator.java   
/**
 * Remove the mapping from the name of the specified
 * {@link LocalVariableGen} to itself.
 * See also {@link #registerByName(LocalVariableGen)} and
 * {@link #lookUpByName(String)}
 * @param lvg a <code>LocalVariableGen</code>
 */
protected void removeByNameTracking(LocalVariableGen lvg) {
    Object duplicateNameEntry = _nametoLVGMap.get(lvg.getName());

    if (duplicateNameEntry instanceof ArrayList) {
        ArrayList sameNameList = (ArrayList) duplicateNameEntry;
        for (int i = 0; i < sameNameList.size(); i++) {
            if (sameNameList.get(i) == lvg) {
                sameNameList.remove(i);
                break;
            }
        }
    } else {
        _nametoLVGMap.remove(lvg);
    }
}
项目:Openjsharp    文件:MethodGenerator.java   
public void setMaxLocals() {

        // Get the current number of local variable slots
        int maxLocals = super.getMaxLocals();
        int prevLocals = maxLocals;

        // Get numer of actual variables
        final LocalVariableGen[] localVars = super.getLocalVariables();
        if (localVars != null) {
            if (localVars.length > maxLocals)
                maxLocals = localVars.length;
        }

        // We want at least 5 local variable slots (for parameters)
        if (maxLocals < 5) maxLocals = 5;

        super.setMaxLocals(maxLocals);
    }
项目:Openjsharp    文件:SlotAllocator.java   
public void releaseSlot(LocalVariableGen lvg) {
    final int size = lvg.getType().getSize();
    final int slot = lvg.getIndex();
    final int limit = _free;

    for (int i = 0; i < limit; i++) {
        if (_slotsTaken[i] == slot) {
            int j = i + size;
            while (j < limit) {
                _slotsTaken[i++] = _slotsTaken[j++];
            }
            _free -= size;
            return;
        }
    }
    String state = "Variable slot allocation error"+
                   "(size="+size+",slot="+slot+",limit="+limit+")";
    ErrorMsg err = new ErrorMsg(ErrorMsg.INTERNAL_ERR,state);
    throw new Error(err.toString());
}
项目:Openjsharp    文件:CompareGenerator.java   
public CompareGenerator(int access_flags,Type return_type,Type[] arg_types,String[] arg_names,String method_name,String class_name,InstructionList il,ConstantPoolGen cp) {
    super(access_flags,return_type,arg_types,arg_names,method_name,class_name,il,cp);

    _iloadCurrent = new ILOAD(CURRENT_INDEX);
    _istoreCurrent = new ISTORE(CURRENT_INDEX);
    _aloadDom = new ALOAD(DOM_INDEX);
    _iloadLast = new ILOAD(LAST_INDEX);

    LocalVariableGen iterator =
        addLocalVariable("iterator",Util.getJCRefType(Constants.NODE_IteraTOR_SIG),null,null);
    IteraTOR_INDEX = iterator.getIndex();
    _aloadIterator = new ALOAD(IteraTOR_INDEX);
    _astoreIterator = new ASTORE(IteraTOR_INDEX);
    il.append(new ACONST_NULL());
    il.append(storeIterator());
}
项目:openjdk-jdk10    文件:MethodGenerator.java   
/**
 * Remove the mapping from the name of the specified
 * {@link LocalVariableGen} to itself.
 * See also {@link #registerByName(LocalVariableGen)} and
 * {@link #lookUpByName(String)}
 * @param lvg a <code>LocalVariableGen</code>
 */
protected void removeByNameTracking(LocalVariableGen lvg) {
    Object duplicateNameEntry = _nametoLVGMap.get(lvg.getName());

    if (duplicateNameEntry instanceof ArrayList) {
        ArrayList sameNameList = (ArrayList) duplicateNameEntry;
        for (int i = 0; i < sameNameList.size(); i++) {
            if (sameNameList.get(i) == lvg) {
                sameNameList.remove(i);
                break;
            }
        }
    } else {
        _nametoLVGMap.remove(lvg);
    }
}
项目:openjdk-jdk10    文件:MethodGenerator.java   
public void setMaxLocals() {

        // Get the current number of local variable slots
        int maxLocals = super.getMaxLocals();
        int prevLocals = maxLocals;

        // Get numer of actual variables
        final LocalVariableGen[] localVars = super.getLocalVariables();
        if (localVars != null) {
            if (localVars.length > maxLocals)
                maxLocals = localVars.length;
        }

        // We want at least 5 local variable slots (for parameters)
        if (maxLocals < 5) maxLocals = 5;

        super.setMaxLocals(maxLocals);
    }
项目:openjdk-jdk10    文件:SlotAllocator.java   
public void releaseSlot(LocalVariableGen lvg) {
    final int size = lvg.getType().getSize();
    final int slot = lvg.getIndex();
    final int limit = _free;

    for (int i = 0; i < limit; i++) {
        if (_slotsTaken[i] == slot) {
            int j = i + size;
            while (j < limit) {
                _slotsTaken[i++] = _slotsTaken[j++];
            }
            _free -= size;
            return;
        }
    }
    String state = "Variable slot allocation error"+
                   "(size="+size+",state);
    throw new Error(err.toString());
}
项目:openjdk-jdk10    文件:CompareGenerator.java   
public CompareGenerator(int access_flags,null);
    IteraTOR_INDEX = iterator.getIndex();
    _aloadIterator = new ALOAD(IteraTOR_INDEX);
    _astoreIterator = new ASTORE(IteraTOR_INDEX);
    il.append(new ACONST_NULL());
    il.append(storeIterator());
}
项目:openjdk9    文件:MethodGenerator.java   
/**
 * <p>Set up a mapping of the name of the specified
 * {@link LocalVariableGen} object to the <code>LocalVariableGen</code>
 * itself.</p>
 * <p>This is a bit of a hack.  XSLTC is relying on the fact that the
 * name that is being looked up won't be duplicated,which isn't
 * guaranteed.  It replaces code which used to call
 * {@link MethodGen#getLocalVariables()} and looped through the
 * <code>LocalVariableGen</code> objects it contained to find the one
 * with the specified name.  However,<code>getLocalVariables()</code>
 * has the side effect of setting the start and end for any
 * <code>LocalVariableGen</code> which did not already have them
 * set,which causes problems for outlining..</p>
 * <p>See also {@link #lookUpByName(String)} and
 * {@link #removeByNameTracking(LocalVariableGen)}</P
 * @param lvg a <code>LocalVariableGen</code>
 */
protected void registerByName(LocalVariableGen lvg) {
    Object duplicateNameEntry = _nametoLVGMap.get(lvg.getName());

    if (duplicateNameEntry == null) {
        _nametoLVGMap.put(lvg.getName(),lvg);
    } else {
        ArrayList sameNameList;

        if (duplicateNameEntry instanceof ArrayList) {
            sameNameList = (ArrayList) duplicateNameEntry;
            sameNameList.add(lvg);
        } else {
            sameNameList = new ArrayList();
            sameNameList.add(duplicateNameEntry);
            sameNameList.add(lvg);
        }

        _nametoLVGMap.put(lvg.getName(),sameNameList);
    }
}
项目:openjdk9    文件:MethodGenerator.java   
/**
 * Remove the mapping from the name of the specified
 * {@link LocalVariableGen} to itself.
 * See also {@link #registerByName(LocalVariableGen)} and
 * {@link #lookUpByName(String)}
 * @param lvg a <code>LocalVariableGen</code>
 */
protected void removeByNameTracking(LocalVariableGen lvg) {
    Object duplicateNameEntry = _nametoLVGMap.get(lvg.getName());

    if (duplicateNameEntry instanceof ArrayList) {
        ArrayList sameNameList = (ArrayList) duplicateNameEntry;
        for (int i = 0; i < sameNameList.size(); i++) {
            if (sameNameList.get(i) == lvg) {
                sameNameList.remove(i);
                break;
            }
        }
    } else {
        _nametoLVGMap.remove(lvg);
    }
}
项目:openjdk9    文件:MethodGenerator.java   
public void setMaxLocals() {

        // Get the current number of local variable slots
        int maxLocals = super.getMaxLocals();
        int prevLocals = maxLocals;

        // Get numer of actual variables
        final LocalVariableGen[] localVars = super.getLocalVariables();
        if (localVars != null) {
            if (localVars.length > maxLocals)
                maxLocals = localVars.length;
        }

        // We want at least 5 local variable slots (for parameters)
        if (maxLocals < 5) maxLocals = 5;

        super.setMaxLocals(maxLocals);
    }
项目:openjdk9    文件:SlotAllocator.java   
public void releaseSlot(LocalVariableGen lvg) {
    final int size = lvg.getType().getSize();
    final int slot = lvg.getIndex();
    final int limit = _free;

    for (int i = 0; i < limit; i++) {
        if (_slotsTaken[i] == slot) {
            int j = i + size;
            while (j < limit) {
                _slotsTaken[i++] = _slotsTaken[j++];
            }
            _free -= size;
            return;
        }
    }
    String state = "Variable slot allocation error"+
                   "(size="+size+",state);
    throw new Error(err.toString());
}
项目:openjdk9    文件:CompareGenerator.java   
public CompareGenerator(int access_flags,null);
    IteraTOR_INDEX = iterator.getIndex();
    _aloadIterator = new ALOAD(IteraTOR_INDEX);
    _astoreIterator = new ASTORE(IteraTOR_INDEX);
    il.append(new ACONST_NULL());
    il.append(storeIterator());
}
项目:lookaside_java-1.8.0-openjdk    文件:MethodGenerator.java   
/**
 * <p>Set up a mapping of the name of the specified
 * {@link LocalVariableGen} object to the <code>LocalVariableGen</code>
 * itself.</p>
 * <p>This is a bit of a hack.  XSLTC is relying on the fact that the
 * name that is being looked up won't be duplicated,sameNameList);
    }
}
项目:lookaside_java-1.8.0-openjdk    文件:MethodGenerator.java   
/**
 * Remove the mapping from the name of the specified
 * {@link LocalVariableGen} to itself.
 * See also {@link #registerByName(LocalVariableGen)} and
 * {@link #lookUpByName(String)}
 * @param lvg a <code>LocalVariableGen</code>
 */
protected void removeByNameTracking(LocalVariableGen lvg) {
    Object duplicateNameEntry = _nametoLVGMap.get(lvg.getName());

    if (duplicateNameEntry instanceof ArrayList) {
        ArrayList sameNameList = (ArrayList) duplicateNameEntry;
        for (int i = 0; i < sameNameList.size(); i++) {
            if (sameNameList.get(i) == lvg) {
                sameNameList.remove(i);
                break;
            }
        }
    } else {
        _nametoLVGMap.remove(lvg);
    }
}
项目:lookaside_java-1.8.0-openjdk    文件:MethodGenerator.java   
public void setMaxLocals() {

        // Get the current number of local variable slots
        int maxLocals = super.getMaxLocals();
        int prevLocals = maxLocals;

        // Get numer of actual variables
        final LocalVariableGen[] localVars = super.getLocalVariables();
        if (localVars != null) {
            if (localVars.length > maxLocals)
                maxLocals = localVars.length;
        }

        // We want at least 5 local variable slots (for parameters)
        if (maxLocals < 5) maxLocals = 5;

        super.setMaxLocals(maxLocals);
    }
项目:lookaside_java-1.8.0-openjdk    文件:SlotAllocator.java   
public void releaseSlot(LocalVariableGen lvg) {
    final int size = lvg.getType().getSize();
    final int slot = lvg.getIndex();
    final int limit = _free;

    for (int i = 0; i < limit; i++) {
        if (_slotsTaken[i] == slot) {
            int j = i + size;
            while (j < limit) {
                _slotsTaken[i++] = _slotsTaken[j++];
            }
            _free -= size;
            return;
        }
    }
    String state = "Variable slot allocation error"+
                   "(size="+size+",state);
    throw new Error(err.toString());
}
项目:lookaside_java-1.8.0-openjdk    文件:CompareGenerator.java   
public CompareGenerator(int access_flags,null);
    IteraTOR_INDEX = iterator.getIndex();
    _aloadIterator = new ALOAD(IteraTOR_INDEX);
    _astoreIterator = new ASTORE(IteraTOR_INDEX);
    il.append(new ACONST_NULL());
    il.append(storeIterator());
}
项目:infobip-open-jdk-8    文件:MethodGenerator.java   
/**
 * <p>Set up a mapping of the name of the specified
 * {@link LocalVariableGen} object to the <code>LocalVariableGen</code>
 * itself.</p>
 * <p>This is a bit of a hack.  XSLTC is relying on the fact that the
 * name that is being looked up won't be duplicated,sameNameList);
    }
}
项目:infobip-open-jdk-8    文件:MethodGenerator.java   
/**
 * Remove the mapping from the name of the specified
 * {@link LocalVariableGen} to itself.
 * See also {@link #registerByName(LocalVariableGen)} and
 * {@link #lookUpByName(String)}
 * @param lvg a <code>LocalVariableGen</code>
 */
protected void removeByNameTracking(LocalVariableGen lvg) {
    Object duplicateNameEntry = _nametoLVGMap.get(lvg.getName());

    if (duplicateNameEntry instanceof ArrayList) {
        ArrayList sameNameList = (ArrayList) duplicateNameEntry;
        for (int i = 0; i < sameNameList.size(); i++) {
            if (sameNameList.get(i) == lvg) {
                sameNameList.remove(i);
                break;
            }
        }
    } else {
        _nametoLVGMap.remove(lvg);
    }
}
项目:infobip-open-jdk-8    文件:MethodGenerator.java   
public void setMaxLocals() {

        // Get the current number of local variable slots
        int maxLocals = super.getMaxLocals();
        int prevLocals = maxLocals;

        // Get numer of actual variables
        final LocalVariableGen[] localVars = super.getLocalVariables();
        if (localVars != null) {
            if (localVars.length > maxLocals)
                maxLocals = localVars.length;
        }

        // We want at least 5 local variable slots (for parameters)
        if (maxLocals < 5) maxLocals = 5;

        super.setMaxLocals(maxLocals);
    }
项目:infobip-open-jdk-8    文件:SlotAllocator.java   
public void releaseSlot(LocalVariableGen lvg) {
    final int size = lvg.getType().getSize();
    final int slot = lvg.getIndex();
    final int limit = _free;

    for (int i = 0; i < limit; i++) {
        if (_slotsTaken[i] == slot) {
            int j = i + size;
            while (j < limit) {
                _slotsTaken[i++] = _slotsTaken[j++];
            }
            _free -= size;
            return;
        }
    }
    String state = "Variable slot allocation error"+
                   "(size="+size+",state);
    throw new Error(err.toString());
}
项目:infobip-open-jdk-8    文件:CompareGenerator.java   
public CompareGenerator(int access_flags,null);
    IteraTOR_INDEX = iterator.getIndex();
    _aloadIterator = new ALOAD(IteraTOR_INDEX);
    _astoreIterator = new ASTORE(IteraTOR_INDEX);
    il.append(new ACONST_NULL());
    il.append(storeIterator());
}
项目:OLD-OpenJDK8    文件:MethodGenerator.java   
/**
 * <p>Set up a mapping of the name of the specified
 * {@link LocalVariableGen} object to the <code>LocalVariableGen</code>
 * itself.</p>
 * <p>This is a bit of a hack.  XSLTC is relying on the fact that the
 * name that is being looked up won't be duplicated,sameNameList);
    }
}
项目:OLD-OpenJDK8    文件:MethodGenerator.java   
/**
 * Remove the mapping from the name of the specified
 * {@link LocalVariableGen} to itself.
 * See also {@link #registerByName(LocalVariableGen)} and
 * {@link #lookUpByName(String)}
 * @param lvg a <code>LocalVariableGen</code>
 */
protected void removeByNameTracking(LocalVariableGen lvg) {
    Object duplicateNameEntry = _nametoLVGMap.get(lvg.getName());

    if (duplicateNameEntry instanceof ArrayList) {
        ArrayList sameNameList = (ArrayList) duplicateNameEntry;
        for (int i = 0; i < sameNameList.size(); i++) {
            if (sameNameList.get(i) == lvg) {
                sameNameList.remove(i);
                break;
            }
        }
    } else {
        _nametoLVGMap.remove(lvg);
    }
}
项目:OLD-OpenJDK8    文件:MethodGenerator.java   
public void setMaxLocals() {

        // Get the current number of local variable slots
        int maxLocals = super.getMaxLocals();
        int prevLocals = maxLocals;

        // Get numer of actual variables
        final LocalVariableGen[] localVars = super.getLocalVariables();
        if (localVars != null) {
            if (localVars.length > maxLocals)
                maxLocals = localVars.length;
        }

        // We want at least 5 local variable slots (for parameters)
        if (maxLocals < 5) maxLocals = 5;

        super.setMaxLocals(maxLocals);
    }
项目:OLD-OpenJDK8    文件:SlotAllocator.java   
public void releaseSlot(LocalVariableGen lvg) {
    final int size = lvg.getType().getSize();
    final int slot = lvg.getIndex();
    final int limit = _free;

    for (int i = 0; i < limit; i++) {
        if (_slotsTaken[i] == slot) {
            int j = i + size;
            while (j < limit) {
                _slotsTaken[i++] = _slotsTaken[j++];
            }
            _free -= size;
            return;
        }
    }
    String state = "Variable slot allocation error"+
                   "(size="+size+",state);
    throw new Error(err.toString());
}
项目:OLD-OpenJDK8    文件:CompareGenerator.java   
public CompareGenerator(int access_flags,null);
    IteraTOR_INDEX = iterator.getIndex();
    _aloadIterator = new ALOAD(IteraTOR_INDEX);
    _astoreIterator = new ASTORE(IteraTOR_INDEX);
    il.append(new ACONST_NULL());
    il.append(storeIterator());
}
项目:openjdk-icedtea7    文件:MethodGenerator.java   
/**
 * <p>Set up a mapping of the name of the specified
 * {@link LocalVariableGen} object to the <code>LocalVariableGen</code>
 * itself.</p>
 * <p>This is a bit of a hack.  XSLTC is relying on the fact that the
 * name that is being looked up won't be duplicated,sameNameList);
    }
}
项目:openjdk-icedtea7    文件:MethodGenerator.java   
/**
 * Remove the mapping from the name of the specified
 * {@link LocalVariableGen} to itself.
 * See also {@link #registerByName(LocalVariableGen)} and
 * {@link #lookUpByName(String)}
 * @param lvg a <code>LocalVariableGen</code>
 */
protected void removeByNameTracking(LocalVariableGen lvg) {
    Object duplicateNameEntry = _nametoLVGMap.get(lvg.getName());

    if (duplicateNameEntry instanceof ArrayList) {
        ArrayList sameNameList = (ArrayList) duplicateNameEntry;
        for (int i = 0; i < sameNameList.size(); i++) {
            if (sameNameList.get(i) == lvg) {
                sameNameList.remove(i);
                break;
            }
        }
    } else {
        _nametoLVGMap.remove(lvg);
    }
}
项目:openjdk-icedtea7    文件:MethodGenerator.java   
public void setMaxLocals() {

        // Get the current number of local variable slots
        int maxLocals = super.getMaxLocals();
        int prevLocals = maxLocals;

        // Get numer of actual variables
        final LocalVariableGen[] localVars = super.getLocalVariables();
        if (localVars != null) {
            if (localVars.length > maxLocals)
                maxLocals = localVars.length;
        }

        // We want at least 5 local variable slots (for parameters)
        if (maxLocals < 5) maxLocals = 5;

        super.setMaxLocals(maxLocals);
    }
项目:openjdk-icedtea7    文件:SlotAllocator.java   
public void releaseSlot(LocalVariableGen lvg) {
    final int size = lvg.getType().getSize();
    final int slot = lvg.getIndex();
    final int limit = _free;

    for (int i = 0; i < limit; i++) {
        if (_slotsTaken[i] == slot) {
            int j = i + size;
            while (j < limit) {
                _slotsTaken[i++] = _slotsTaken[j++];
            }
            _free -= size;
            return;
        }
    }
    String state = "Variable slot allocation error"+
                   "(size="+size+",state);
    throw new Error(err.toString());
}
项目:openjdk-icedtea7    文件:CompareGenerator.java   
public CompareGenerator(int access_flags,null);
    IteraTOR_INDEX = iterator.getIndex();
    _aloadIterator = new ALOAD(IteraTOR_INDEX);
    _astoreIterator = new ASTORE(IteraTOR_INDEX);
    il.append(new ACONST_NULL());
    il.append(storeIterator());
}
项目:Openjsharp    文件:RealType.java   
/**
 * Translates a real into a non-synthesized boolean. It does not push a
 * 0 or a 1 but instead returns branchhandle list to be appended to the
 * false list. A NaN must be converted to "false".
 *
 * @see     com.sun.org.apache.xalan.internal.xsltc.compiler.util.Type#translatetoDesynthesized
 */
public FlowList translatetoDesynthesized(ClassGenerator classGen,MethodGenerator methodGen,BooleanType type) {
    LocalVariableGen local;
    final FlowList flowlist = new FlowList();
    final ConstantPoolGen cpg = classGen.getConstantPool();
    final InstructionList il = methodGen.getInstructionList();

    // Store real into a local variable
    il.append(DUP2);
    local = methodGen.addLocalVariable("real_to_boolean_tmp",com.sun.org.apache.bcel.internal.generic.Type.DOUBLE,null);
    local.setStart(il.append(new DSTORE(local.getIndex())));

    // Compare it to 0.0
    il.append(DCONST_0);
    il.append(DCMPG);
    flowlist.add(il.append(new IFEQ(null)));

    //!!! call isNaN
    // Compare it to itself to see if NaN
    il.append(new DLOAD(local.getIndex()));
    local.setEnd(il.append(new DLOAD(local.getIndex())));
    il.append(DCMPG);
    flowlist.add(il.append(new IFNE(null)));        // NaN != NaN
    return flowlist;
}
项目:Openjsharp    文件:MethodGenerator.java   
/**
 * Allocates a local variable. If the slot allocator has already been
 * initialized,then call addLocalVariable2() so that the new variable
 * is kNown to the allocator. Failing to do this may cause the allocator
 * to return a slot that is already in use.
 */
public LocalVariableGen addLocalVariable(String name,Type type,InstructionHandle start,InstructionHandle end)
{
    LocalVariableGen lvg;

    if (_allocatorInit) {
        lvg = addLocalVariable2(name,type,start);
    } else {
        lvg = super.addLocalVariable(name,start,end);
        getLocalVariableRegistry().registerLocalVariable(lvg);
    }
    return lvg;
}
项目:Openjsharp    文件:MethodGenerator.java   
public LocalVariableGen addLocalVariable2(String name,InstructionHandle start)
{
    LocalVariableGen lvg = super.addLocalVariable(name,_slotAllocator.allocateSlot(type),null);
    getLocalVariableRegistry().registerLocalVariable(lvg);
    return lvg;
}
项目:Openjsharp    文件:MethodGenerator.java   
/**
 * Registers a {@link org.apache.bcel.generic.LocalVariableGen}
 * for this method.
 * <p><b>Preconditions:</b>
 * <ul>
 * <li>The range of instructions for <code>lvg</code> does not
 * overlap with the range of instructions for any
 * <code>LocalVariableGen</code> with the same slot index prevIoUsly
 * registered for this method.  <b><em>(Unchecked.)</em></b></li>
 * </ul></p>
 * @param lvg The variable to be registered
 */
protected void registerLocalVariable(LocalVariableGen lvg) {
    int slot = lvg.getIndex();

    int registrySize = _variables.size();

    // If the LocalVariableGen uses a slot index beyond any prevIoUsly
    // encountered,expand the _variables,padding with intervening null
    // entries as required.
    if (slot >= registrySize) {
        for (int i = registrySize; i < slot; i++) {
            _variables.add(null);
        }
        _variables.add(lvg);
    } else {
        // If the LocalVariableGen reuses a slot,make sure the entry
        // in _variables contains an ArrayList and add the newly
        // registered LocalVariableGen to the list.  If the entry in
        // _variables just contains null padding,store the
        // LocalVariableGen directly.
        Object localsInSlot = _variables.get(slot);
        if (localsInSlot != null) {
            if (localsInSlot instanceof LocalVariableGen) {
                ArrayList listofLocalsInSlot = new ArrayList();
                listofLocalsInSlot.add(localsInSlot);
                listofLocalsInSlot.add(lvg);
                _variables.set(slot,listofLocalsInSlot);
            } else {
                ((ArrayList) localsInSlot).add(lvg);
            }
        } else {
            _variables.set(slot,lvg);
        }
    }

    registerByName(lvg);
}
项目:Openjsharp    文件:MethodGenerator.java   
/**
 * Determines whether a particular variable is in use at a particular offset
 * in the byte code for this method.
 * <p><b>Preconditions:</b>
 * <ul>
 * <li>The {@link InstructionList#setPositions()} has been called for the
 * {@link InstructionList} associated with this {@link MethodGenerator}.
 * </li></ul></p>
 * @param lvg the {@link LocalVariableGen} for the variable
 * @param offset the position in the byte code
 * @return <code>true</code> if and only if the specified variable is in
 * use at the particular byte code offset.
 */
boolean offsetInLocalVariableGenRange(LocalVariableGen lvg,int offset) {
    InstructionHandle lvgStart = lvg.getStart();
    InstructionHandle lvgEnd = lvg.getEnd();

    // If no start handle is recorded for the LocalVariableGen,it is
    // assumed to be in use from the beginning of the method.
    if (lvgStart == null) {
        lvgStart = getInstructionList().getStart();
    }

    // If no end handle is recorded for the LocalVariableGen,it is assumed
    // to be in use to the end of the method.
    if (lvgEnd == null) {
        lvgEnd = getInstructionList().getEnd();
    }

    // Does the range of the instruction include the specified offset?
    // Note that the InstructionHandle.getPosition method returns the
    // offset of the beginning of an instruction.  A LocalVariableGen's
    // range includes the end instruction itself,so that instruction's
    // length must be taken into consideration in computing whether the
    // varible is in range at a particular offset.
    return ((lvgStart.getPosition() <= offset)
                && (lvgEnd.getPosition()
                        + lvgEnd.getInstruction().getLength() >= offset));
}
项目:Openjsharp    文件:SlotAllocator.java   
public void initialize(LocalVariableGen[] vars) {
    final int length = vars.length;
    int slot = 0,size,index;

    for (int i = 0; i < length; i++) {
        size  = vars[i].getType().getSize();
        index = vars[i].getIndex();
        slot  = Math.max(slot,index + size);
    }
    _firstAvailableSlot = slot;
}
项目:openjdk-jdk10    文件:RealType.java   
/**
 * Translates a real into a non-synthesized boolean. It does not push a
 * 0 or a 1 but instead returns branchhandle list to be appended to the
 * false list. A NaN must be converted to "false".
 *
 * @see     com.sun.org.apache.xalan.internal.xsltc.compiler.util.Type#translatetoDesynthesized
 */
public FlowList translatetoDesynthesized(ClassGenerator classGen,null);
    local.setStart(il.append(new DSTORE(local.getIndex())));

    // Compare it to 0.0
    il.append(DCONST_0);
    il.append(DCMPG);
    flowlist.add(il.append(new IFEQ(null)));

    //!!! call isNaN
    // Compare it to itself to see if NaN
    il.append(new DLOAD(local.getIndex()));
    local.setEnd(il.append(new DLOAD(local.getIndex())));
    il.append(DCMPG);
    flowlist.add(il.append(new IFNE(null)));        // NaN != NaN
    return flowlist;
}

ERROR org.hibernate.search.exception.impl.LogErrorHandler - HSEARCH000058: HSEARCH000183: Unable ...

ERROR org.hibernate.search.exception.impl.LogErrorHandler - HSEARCH000058: HSEARCH000183: Unable ...

2019-06-12 10:17:42 [Hibernate Search: entityloader-1] ERROR org.hibernate.search.exception.impl.LogErrorHandler - HSEARCH000058: HSEARCH000183: Unable to index instance of type net.shopxx.entity.Product while batch indexing: Entity of type net.shopxx.entity.Product with id: 11374
java.lang.IllegalArgumentException: position increments (and gaps) must be >= 0 (got 581043) for field ''introduction''
    at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:649)
    at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:365)
    at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:321)
    at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
    at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
    at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
    at org.hibernate.search.backend.impl.lucene.IndexWriterDelegate.updateDocument(IndexWriterDelegate.java:66)
    at org.hibernate.search.backend.impl.lucene.IndexWriterDelegate.addDocument(IndexWriterDelegate.java:55)
    at org.hibernate.search.backend.impl.lucene.works.AddWorkExecutor.performWork(AddWorkExecutor.java:54)
    at org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer.doWork(LuceneBackendTaskStreamer.java:52)
    at org.hibernate.search.backend.impl.lucene.WorkspaceHolder.applyStreamWork(WorkspaceHolder.java:74)
    at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performStreamOperation(DirectoryBasedIndexManager.java:103)
    at org.hibernate.search.backend.impl.StreamingOperationExecutorSelector$AddSelectionExecutor.performStreamOperation(StreamingOperationExecutorSelector.java:106)
    at org.hibernate.search.backend.impl.batch.DefaultBatchBackend.sendWorkToShards(DefaultBatchBackend.java:73)
    at org.hibernate.search.backend.impl.batch.DefaultBatchBackend.enqueueAsyncWork(DefaultBatchBackend.java:49)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.index(IdentifierConsumerDocumentProducer.java:297)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.indexAllQueue(IdentifierConsumerDocumentProducer.java:223)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.loadList(IdentifierConsumerDocumentProducer.java:177)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.loadAllFromQueue(IdentifierConsumerDocumentProducer.java:140)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.run(IdentifierConsumerDocumentProducer.java:117)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2019-06-12 10:17:42 [Hibernate Search: entityloader-4] WARN  org.hibernate.search.backend.impl.lucene.IndexWriterHolder - HSEARCH000052: Going to force release of the IndexWriter lock
2019-06-12 10:17:42 [Hibernate Search: entityloader-1] WARN  org.hibernate.search.backend.impl.lucene.IndexWriterHolder - HSEARCH000052: Going to force release of the IndexWriter lock
2019-06-12 10:17:42 [Hibernate Search: entityloader-6] WARN  org.hibernate.search.backend.impl.lucene.IndexWriterHolder - HSEARCH000052: Going to force release of the IndexWriter lock
2019-06-12 10:17:42 [Hibernate Search: entityloader-4] ERROR org.hibernate.search.exception.impl.LogErrorHandler - HSEARCH000058: HSEARCH000183: Unable to index instance of type net.shopxx.entity.Product while batch indexing: Entity of type net.shopxx.entity.Product with id: 12467
java.lang.IllegalArgumentException: position increments (and gaps) must be >= 0 (got 600772) for field ''introduction''
    at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:649)
    at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:365)
    at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:321)
    at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
    at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
    at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
    at org.hibernate.search.backend.impl.lucene.IndexWriterDelegate.updateDocument(IndexWriterDelegate.java:66)
    at org.hibernate.search.backend.impl.lucene.IndexWriterDelegate.addDocument(IndexWriterDelegate.java:55)
    at org.hibernate.search.backend.impl.lucene.works.AddWorkExecutor.performWork(AddWorkExecutor.java:54)
    at org.hibernate.search.backend.impl.lucene.LuceneBackendTaskStreamer.doWork(LuceneBackendTaskStreamer.java:52)
    at org.hibernate.search.backend.impl.lucene.WorkspaceHolder.applyStreamWork(WorkspaceHolder.java:74)
    at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performStreamOperation(DirectoryBasedIndexManager.java:103)
    at org.hibernate.search.backend.impl.StreamingOperationExecutorSelector$AddSelectionExecutor.performStreamOperation(StreamingOperationExecutorSelector.java:106)
    at org.hibernate.search.backend.impl.batch.DefaultBatchBackend.sendWorkToShards(DefaultBatchBackend.java:73)
    at org.hibernate.search.backend.impl.batch.DefaultBatchBackend.enqueueAsyncWork(DefaultBatchBackend.java:49)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.index(IdentifierConsumerDocumentProducer.java:297)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.indexAllQueue(IdentifierConsumerDocumentProducer.java:223)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.loadList(IdentifierConsumerDocumentProducer.java:177)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.loadAllFromQueue(IdentifierConsumerDocumentProducer.java:140)
    at org.hibernate.search.batchindexing.impl.IdentifierConsumerDocumentProducer.run(IdentifierConsumerDocumentProducer.java:117)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

在使用hibernate分析器的时候,如果分析器作用的字段过长的时候,就会出现以上的报错

解决方法:

/**
     * 介绍
     */
    @Field(store = Store.YES, index = Index.YES, analyze = Analyze.NO)
    @Lob
    @Basic(fetch = FetchType.LAZY)
    private String introduction;

analyze = Analyze.NO这句是关键,将YES改成NO

HIVE出现Read past end of RLE integer from compressed stream Stream for column 1 kind LENGTH positio...

HIVE出现Read past end of RLE integer from compressed stream Stream for column 1 kind LENGTH positio...

错误日志

Diagnostic Messages for this Task:
Error: java.io.IOException: java.io.IOException: java.io.EOFException: Read past end of RLE integer from compressed stream Stream for column 1 kind LENGTH position: 359 length: 359 range: 0 offset: 212849 limit: 212849 range 0 = 0 to 359 uncompressed: 4 to 4
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:226)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:136)
        at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
        at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: java.io.EOFException: Read past end of RLE integer from compressed stream Stream for column 1 kind LENGTH position: 359 length: 359 range: 0 offset: 212849 limit: 212849 range 0 = 0 to 359 uncompressed: 4 to 4
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
        at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
        at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:105)
        at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
        at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:224)
        ... 11 more
Caused by: java.io.EOFException: Read past end of RLE integer from compressed stream Stream for column 1 kind LENGTH position: 359 length: 359 range: 0 offset: 212849 limit: 212849 range 0 = 0 to 359 uncompressed: 4 to 4
        at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:56)
        at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:302)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDirectTreeReader.next(RecordReaderImpl.java:1480)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.next(RecordReaderImpl.java:1342)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1880)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:3133)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:164)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:138)
        at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
        ... 15 more

环境:

hive1.1.x ORC Snappy

 

原因:

使用

set hive.merge.mapfiles=true;
set hive.merge.mapredfiles=true;
set hive.merge.smallfiles.avgsize=256000000;

对小文件进行了合并。

 

经查询,是Hive的BUG,在hive 1.3.x以上版本修复。通常升级一下版本即可。

https://issues.apache.org/jira/browse/HIVE-10916

 

但是当前我们使用的是UCloud老版本集群,官方只能支持hive升级到1.2.x。

官方给出的方案是:非标升级,集群迁移到新版本集群。  风险和耗时太大。

目前我使用的方案是将所有的分区做一下输出,问题分区read时会报错, 通过别的数据源将问题分区修复即解决了问题。

 后续会考虑换文件类型。

 

关于编辑seaborn legend编辑部的故事的介绍现已完结,谢谢您的耐心阅读,如果想了解更多关于4. 集成学习(Ensemble Learning)Adaboost、com.sun.org.apache.bcel.internal.generic.LocalVariableGen的实例源码、ERROR org.hibernate.search.exception.impl.LogErrorHandler - HSEARCH000058: HSEARCH000183: Unable ...、HIVE出现Read past end of RLE integer from compressed stream Stream for column 1 kind LENGTH positio...的相关知识,请在本站寻找。

本文标签: