理论推导

参考我的博客(吴恩达）3.逻辑回归、正则化

Logistic回归的一般过程

1. 收集数据: 采用任意方法收集数据。
2. 准备数据: 由于需要进行距离计算, 因此要求数据类型为数值型。另外, 结构化数据 格式则最佳。
3. 分析数据: 采用任意方法对数据进行分析。
4. 训练算法: 大部分时间将用于训练, 训练的目的是为了找到最佳的分类回归系数。
5. 测试算法: 一旦训练步骤完成, 分类将会很快。
6. 使用算法: 首先, 我们需要输入一些数据, 并将其转换成对应的结构化数值;
   接着, 基于训练好的回归系数就可以对这些数值进行简单的回归计算, 判定它们属于 哪个类别; 在这之后, 我们就可以在输出的类别上做一些其他分析工作。

基于 Logistic 回归和 Sigmoid 函数的分类

Logistic回归

• 优点: 计算代价不高, 易于理解和实现。
• 缺点: 容易欠拟合, 分类精度可能不高。
• 适用数据类型: 数值型和标称型数据。

sigmoid 函数
$$\sigma (z)=\frac{1}{1+{\mathrm{e}}^{-z}}$$

训练算法：使用梯度上升找到最佳参数

梯度上升和梯度下降本质是一样的，只不过是多个负号，少个负号

每个回归系数初始化为 1

重复 $\mathrm{R}$ 次

计算整个数据集的梯度

使用alpha * gradient更新回归系数的向量 

返回回归系数

import numpy as np

#Logistic回归梯度上升优化算法
def loadDataSet():
    dataMat = []; labelMat = []
    fr = open('testSet.txt')
    for line in fr.readlines():
        lineArr = line.strip().split()
        dataMat.append([1.0,float(lineArr[0]),float(lineArr[1])])
        labelMat.append(int(lineArr[2]))
    return dataMat,labelMat

def sigmoid(inX):
    return 1.0/(1+np.exp(-inX))

def gradAscent(dataMatIn,classLabels):
    dataMatrix = np.mat(dataMatIn)
    labelMat = np.mat(classLabels).transpose()
    m,n = np.shape(dataMatrix)
    alpha = 0.001
    maxCycles = 500
    weights = np.ones((n,1))
    for k in range(maxCycles):
        h = sigmoid(dataMatrix*weights)
        error = (labelMat - h)
        weights = weights + alpha * dataMatrix.transpose() * error ③
    return weights

上面这块代码中梯度上升的③，其实是cross entropy损失函数对w求导得出来的，具体推导过程参见理论推导

dataArr,labelMat = loadDataSet()
weightsLogistic = gradAscent(dataArr,labelMat)
weightsLogistic

matrix([[ 4.12414349],
        [ 0.48007329],
        [-0.6168482 ]])

画出决策边界

设置sigmoid函数为0,0是两个分类的（类别1和类别0）的分界处

所以设定$0=w_0{x}_0+w_1{x}_1+$ $w_2{x}_2$

为什么是0，这要看sigmoid函数的性质，sigmoid（x），x>=0，划为正类，反之为负类
所以这设置sigmoid函数变量x取0时，求出x1，x2也就是这两个feature之间的关系式，来画出分类边界

def plotBestFit(weights):
    import matplotlib.pyplot as plt
    dataMat,labelMat = loadDataSet()
    dataArr = array(dataMat)
    n = np.shape(dataArr)[0]
    xcord1 = []; ycord1 = []
    xcord2 = []; ycord2 = []
    for i in range(n):
        if int(labelMat[i]) == 1:
            xcord1.append(dataArr[i,1]); ycord1.append(dataArr[i,2])
        else:
            xcord2.append(dataArr[i,1]); ycord2.append(dataArr[i,2])
    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.scatter(xcord1,ycord1,s=30,c='red',marker = 's')
    ax.scatter(xcord2,ycord2,s=30,c='green')
    x = np.arange(-3.0,3.0,0.1)
    y = (-weights[0]-weights[1]*x)/weights[2]
    y = y.transpose()
#     print(x.shape)
#     print(y.shape)
    ax.plot(x,y)
    plt.xlabel('X1');plt.ylabel('X2');
    plt.show

from numpy import *

plotBestFit(weightsLogistic)

训练算法：随机梯度上升

所有回归系数初始化为 1

对数据集中每个样本

计算该样本的梯度 
    
使用 alpha * gradient更新回归系数值 

返回回归系数值

#随机梯度上升算法
def stocGradAscent0(dataMatrix,classLabels):
    m,n = np.shape(dataMatrix)
    alpha = 0.1
    weights = np.ones(n)
    for i in range(m):
        h = sigmoid(sum(dataMatrix[i]*weights))
        error = classLabels[i] - h
        weights = weights + alpha * error * dataMatrix[i]
    return weights

from numpy import *

dataArr,labelMat = loadDataSet()

weights = stocGradAscent0(array(dataArr),labelMat)
plotBestFit(weights)

#改进的随机梯度上升算法
def stocGradAscent1(dataMatrix,classLabels,numIter=150):
    m,n = np.shape(dataMatrix)
    weights = np.ones(n)
    for j in range(numIter):
        dataIndex = list(range(m))
        for i in range(m):
            alpha = 4/(1.0+j+i)+0.01
            randIndex = int(random.uniform(0,len(dataIndex)))
            h = sigmoid(sum(dataMatrix[randIndex]*weights))
            error = classLabels[randIndex] - h
            weights = weights + alpha * error * dataMatrix[randIndex]
            del(dataIndex[randIndex])
    return weights

dataArr,labelMat = loadDataSet()
weights = stocGradAscent1(array(dataArr),labelMat)
plotBestFit(weights)

示例：从疝气病症预测病马的死亡率

数据集
训练集 密码:i53j
测试集 密码:2jvp

def classifyVector(inX,weights):
    prob = sigmoid(sum(inX*weights))
    if prob > 0.5 : return 1.0
    else:
        return 0.0

def colicTest():
    frTrain = open('horseColicTraining.txt')
    frTest = open('horseColicTest.txt')
    trainingSet = []; trainingLabels = []
    for line in frTrain.readlines():
        currLine = line.strip().split('\t')
        lineArr = []
        for i in range(21):
            lineArr.append(float(currLine[i]))
        trainingSet.append(lineArr)
        trainingLabels.append(float(currLine[21]))
    trainWeights = stocGradAscent1(array(trainingSet),trainingLabels,500)
    errorCount = 0; numTestVec = 0.0
    for line in frTest.readlines():
        numTestVec += 1.0
        currLine = line.strip().split('\t')
        lineArr = []
        for i in range(21):
            lineArr.append(float(currLine[i]))
        if int(classifyVector(array(lineArr),trainWeights)) != int(currLine[21]):
            errorCount += 1
    errorRate = (float(errorCount)/numTestVec)
    print("the error rate of this test is :%f" % errorRate)
    return errorRate
def multiTest():
    numTests = 1000; errorSum = 0.0
    for k in range(numTests):
        errorSum += colicTest()
    print("after %d iterations the average error rate is:%f" % (numTests,errorSum/float(numTests)))

multiTest()

<ipython-input-14-ee3abccbfdd7>:2: RuntimeWarning: overflow encountered in exp
  return 1.0/(1+np.exp(-inX))


the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.298507
the error rate of this test is :0.313433
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.313433
the error rate of this test is :0.507463
the error rate of this test is :0.313433
the error rate of this test is :0.477612
the error rate of this test is :0.328358
the error rate of this test is :0.388060
the error rate of this test is :0.268657
the error rate of this test is :0.298507
the error rate of this test is :0.358209
the error rate of this test is :0.507463
the error rate of this test is :0.373134
the error rate of this test is :0.283582
the error rate of this test is :0.328358
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.402985
the error rate of this test is :0.402985
the error rate of this test is :0.462687
the error rate of this test is :0.477612
the error rate of this test is :0.268657
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.402985
the error rate of this test is :0.373134
the error rate of this test is :0.313433
the error rate of this test is :0.402985
the error rate of this test is :0.298507
the error rate of this test is :0.432836
the error rate of this test is :0.283582
the error rate of this test is :0.283582
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.328358
the error rate of this test is :0.447761
the error rate of this test is :0.358209
the error rate of this test is :0.283582
the error rate of this test is :0.417910
the error rate of this test is :0.313433
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.298507
the error rate of this test is :0.298507
the error rate of this test is :0.313433
the error rate of this test is :0.402985
the error rate of this test is :0.313433
the error rate of this test is :0.388060
the error rate of this test is :0.313433
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.402985
the error rate of this test is :0.268657
the error rate of this test is :0.328358
the error rate of this test is :0.477612
the error rate of this test is :0.358209
the error rate of this test is :0.462687
the error rate of this test is :0.402985
the error rate of this test is :0.298507
the error rate of this test is :0.313433
the error rate of this test is :0.373134
the error rate of this test is :0.328358
the error rate of this test is :0.373134
the error rate of this test is :0.328358
the error rate of this test is :0.447761
the error rate of this test is :0.298507
the error rate of this test is :0.373134
the error rate of this test is :0.462687
the error rate of this test is :0.283582
the error rate of this test is :0.328358
the error rate of this test is :0.537313
the error rate of this test is :0.313433
the error rate of this test is :0.268657
the error rate of this test is :0.358209
the error rate of this test is :0.298507
the error rate of this test is :0.313433
the error rate of this test is :0.462687
the error rate of this test is :0.298507
the error rate of this test is :0.298507
the error rate of this test is :0.373134
the error rate of this test is :0.328358
the error rate of this test is :0.343284
the error rate of this test is :0.388060
the error rate of this test is :0.417910
the error rate of this test is :0.268657
the error rate of this test is :0.417910
the error rate of this test is :0.283582
the error rate of this test is :0.358209
the error rate of this test is :0.283582
the error rate of this test is :0.432836
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.328358
the error rate of this test is :0.253731
the error rate of this test is :0.417910
the error rate of this test is :0.313433
the error rate of this test is :0.388060
the error rate of this test is :0.417910
the error rate of this test is :0.402985
the error rate of this test is :0.358209
the error rate of this test is :0.268657
the error rate of this test is :0.283582
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.373134
the error rate of this test is :0.432836
the error rate of this test is :0.388060
the error rate of this test is :0.283582
the error rate of this test is :0.328358
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.417910
the error rate of this test is :0.402985
the error rate of this test is :0.298507
the error rate of this test is :0.402985
the error rate of this test is :0.552239
the error rate of this test is :0.373134
the error rate of this test is :0.343284
the error rate of this test is :0.283582
the error rate of this test is :0.343284
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.298507
the error rate of this test is :0.313433
the error rate of this test is :0.343284
the error rate of this test is :0.462687
the error rate of this test is :0.402985
the error rate of this test is :0.447761
the error rate of this test is :0.388060
the error rate of this test is :0.268657
the error rate of this test is :0.373134
the error rate of this test is :0.298507
the error rate of this test is :0.358209
the error rate of this test is :0.313433
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.417910
the error rate of this test is :0.268657
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.447761
the error rate of this test is :0.313433
the error rate of this test is :0.417910
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.313433
the error rate of this test is :0.447761
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.238806
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.343284
the error rate of this test is :0.447761
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.313433
the error rate of this test is :0.283582
the error rate of this test is :0.343284
the error rate of this test is :0.402985
the error rate of this test is :0.373134
the error rate of this test is :0.298507
the error rate of this test is :0.402985
the error rate of this test is :0.328358
the error rate of this test is :0.238806
the error rate of this test is :0.417910
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.373134
the error rate of this test is :0.283582
the error rate of this test is :0.298507
the error rate of this test is :0.402985
the error rate of this test is :0.417910
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.417910
the error rate of this test is :0.268657
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.283582
the error rate of this test is :0.328358
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.283582
the error rate of this test is :0.402985
the error rate of this test is :0.402985
the error rate of this test is :0.477612
the error rate of this test is :0.343284
the error rate of this test is :0.298507
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.447761
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.432836
the error rate of this test is :0.477612
the error rate of this test is :0.373134
the error rate of this test is :0.477612
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.373134
the error rate of this test is :0.462687
the error rate of this test is :0.402985
the error rate of this test is :0.313433
the error rate of this test is :0.402985
the error rate of this test is :0.358209
the error rate of this test is :0.298507
the error rate of this test is :0.298507
the error rate of this test is :0.328358
the error rate of this test is :0.298507
the error rate of this test is :0.373134
the error rate of this test is :0.298507
the error rate of this test is :0.388060
the error rate of this test is :0.388060
the error rate of this test is :0.358209
the error rate of this test is :0.268657
the error rate of this test is :0.388060
the error rate of this test is :0.328358
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.253731
the error rate of this test is :0.373134
the error rate of this test is :0.253731
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.298507
the error rate of this test is :0.388060
the error rate of this test is :0.417910
the error rate of this test is :0.343284
the error rate of this test is :0.208955
the error rate of this test is :0.268657
the error rate of this test is :0.313433
the error rate of this test is :0.373134
the error rate of this test is :0.432836
the error rate of this test is :0.298507
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.298507
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.417910
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.253731
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.388060
the error rate of this test is :0.432836
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.298507
the error rate of this test is :0.373134
the error rate of this test is :0.313433
the error rate of this test is :0.417910
the error rate of this test is :0.402985
the error rate of this test is :0.537313
the error rate of this test is :0.298507
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.343284
the error rate of this test is :0.402985
the error rate of this test is :0.298507
the error rate of this test is :0.447761
the error rate of this test is :0.402985
the error rate of this test is :0.328358
the error rate of this test is :0.417910
the error rate of this test is :0.268657
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.313433
the error rate of this test is :0.388060
the error rate of this test is :0.268657
the error rate of this test is :0.268657
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.417910
the error rate of this test is :0.328358
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.417910
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.268657
the error rate of this test is :0.388060
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.507463
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.283582
the error rate of this test is :0.462687
the error rate of this test is :0.447761
the error rate of this test is :0.283582
the error rate of this test is :0.238806
the error rate of this test is :0.388060
the error rate of this test is :0.447761
the error rate of this test is :0.432836
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.328358
the error rate of this test is :0.328358
the error rate of this test is :0.447761
the error rate of this test is :0.552239
the error rate of this test is :0.268657
the error rate of this test is :0.477612
the error rate of this test is :0.432836
the error rate of this test is :0.298507
the error rate of this test is :0.432836
the error rate of this test is :0.313433
the error rate of this test is :0.283582
the error rate of this test is :0.358209
the error rate of this test is :0.313433
the error rate of this test is :0.537313
the error rate of this test is :0.223881
the error rate of this test is :0.313433
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.283582
the error rate of this test is :0.343284
the error rate of this test is :0.388060
the error rate of this test is :0.447761
the error rate of this test is :0.388060
the error rate of this test is :0.343284
the error rate of this test is :0.417910
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.373134
the error rate of this test is :0.343284
the error rate of this test is :0.402985
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.417910
the error rate of this test is :0.388060
the error rate of this test is :0.268657
the error rate of this test is :0.328358
the error rate of this test is :0.373134
the error rate of this test is :0.313433
the error rate of this test is :0.417910
the error rate of this test is :0.358209
the error rate of this test is :0.223881
the error rate of this test is :0.343284
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.313433
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.298507
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.402985
the error rate of this test is :0.328358
the error rate of this test is :0.388060
the error rate of this test is :0.343284
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.462687
the error rate of this test is :0.328358
the error rate of this test is :0.298507
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.417910
the error rate of this test is :0.238806
the error rate of this test is :0.268657
the error rate of this test is :0.388060
the error rate of this test is :0.522388
the error rate of this test is :0.507463
the error rate of this test is :0.358209
the error rate of this test is :0.417910
the error rate of this test is :0.373134
the error rate of this test is :0.417910
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.417910
the error rate of this test is :0.343284
the error rate of this test is :0.388060
the error rate of this test is :0.268657
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.402985
the error rate of this test is :0.388060
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.298507
the error rate of this test is :0.343284
the error rate of this test is :0.477612
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.402985
the error rate of this test is :0.283582
the error rate of this test is :0.328358
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.283582
the error rate of this test is :0.388060
the error rate of this test is :0.402985
the error rate of this test is :0.373134
the error rate of this test is :0.462687
the error rate of this test is :0.447761
the error rate of this test is :0.283582
the error rate of this test is :0.313433
the error rate of this test is :0.417910
the error rate of this test is :0.462687
the error rate of this test is :0.417910
the error rate of this test is :0.432836
the error rate of this test is :0.432836
the error rate of this test is :0.522388
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.253731
the error rate of this test is :0.223881
the error rate of this test is :0.432836
the error rate of this test is :0.462687
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.238806
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.432836
the error rate of this test is :0.268657
the error rate of this test is :0.402985
the error rate of this test is :0.268657
the error rate of this test is :0.373134
the error rate of this test is :0.283582
the error rate of this test is :0.358209
the error rate of this test is :0.238806
the error rate of this test is :0.447761
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.417910
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.343284
the error rate of this test is :0.328358
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.298507
the error rate of this test is :0.253731
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.417910
the error rate of this test is :0.253731
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.238806
the error rate of this test is :0.328358
the error rate of this test is :0.328358
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.253731
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.328358
the error rate of this test is :0.298507
the error rate of this test is :0.223881
the error rate of this test is :0.388060
the error rate of this test is :0.298507
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.358209
the error rate of this test is :0.268657
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.417910
the error rate of this test is :0.298507
the error rate of this test is :0.343284
the error rate of this test is :0.283582
the error rate of this test is :0.626866
the error rate of this test is :0.402985
the error rate of this test is :0.432836
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.328358
the error rate of this test is :0.402985
the error rate of this test is :0.388060
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.432836
the error rate of this test is :0.432836
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.268657
the error rate of this test is :0.238806
the error rate of this test is :0.373134
the error rate of this test is :0.417910
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.447761
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.313433
the error rate of this test is :0.462687
the error rate of this test is :0.343284
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.268657
the error rate of this test is :0.313433
the error rate of this test is :0.402985
the error rate of this test is :0.417910
the error rate of this test is :0.373134
the error rate of this test is :0.313433
the error rate of this test is :0.432836
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.373134
the error rate of this test is :0.373134
the error rate of this test is :0.522388
the error rate of this test is :0.402985
the error rate of this test is :0.402985
the error rate of this test is :0.373134
the error rate of this test is :0.462687
the error rate of this test is :0.313433
the error rate of this test is :0.358209
the error rate of this test is :0.417910
the error rate of this test is :0.313433
the error rate of this test is :0.238806
the error rate of this test is :0.373134
the error rate of this test is :0.283582
the error rate of this test is :0.343284
the error rate of this test is :0.388060
the error rate of this test is :0.402985
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.238806
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.417910
the error rate of this test is :0.388060
the error rate of this test is :0.298507
the error rate of this test is :0.313433
the error rate of this test is :0.402985
the error rate of this test is :0.283582
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.268657
the error rate of this test is :0.432836
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.373134
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.462687
the error rate of this test is :0.298507
the error rate of this test is :0.597015
the error rate of this test is :0.522388
the error rate of this test is :0.343284
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.462687
the error rate of this test is :0.417910
the error rate of this test is :0.358209
the error rate of this test is :0.313433
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.298507
the error rate of this test is :0.417910
the error rate of this test is :0.343284
the error rate of this test is :0.417910
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.343284
the error rate of this test is :0.283582
the error rate of this test is :0.313433
the error rate of this test is :0.328358
the error rate of this test is :0.328358
the error rate of this test is :0.402985
the error rate of this test is :0.417910
the error rate of this test is :0.402985
the error rate of this test is :0.298507
the error rate of this test is :0.417910
the error rate of this test is :0.298507
the error rate of this test is :0.417910
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.343284
the error rate of this test is :0.388060
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.283582
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.462687
the error rate of this test is :0.462687
the error rate of this test is :0.388060
the error rate of this test is :0.313433
the error rate of this test is :0.268657
the error rate of this test is :0.283582
the error rate of this test is :0.343284
the error rate of this test is :0.298507
the error rate of this test is :0.388060
the error rate of this test is :0.238806
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.328358
the error rate of this test is :0.402985
the error rate of this test is :0.402985
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.462687
the error rate of this test is :0.388060
the error rate of this test is :0.268657
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.328358
the error rate of this test is :0.298507
the error rate of this test is :0.417910
the error rate of this test is :0.388060
the error rate of this test is :0.283582
the error rate of this test is :0.492537
the error rate of this test is :0.402985
the error rate of this test is :0.298507
the error rate of this test is :0.358209
the error rate of this test is :0.238806
the error rate of this test is :0.328358
the error rate of this test is :0.388060
the error rate of this test is :0.388060
the error rate of this test is :0.358209
the error rate of this test is :0.283582
the error rate of this test is :0.373134
the error rate of this test is :0.238806
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.238806
the error rate of this test is :0.343284
the error rate of this test is :0.388060
the error rate of this test is :0.462687
the error rate of this test is :0.313433
the error rate of this test is :0.417910
the error rate of this test is :0.268657
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.417910
the error rate of this test is :0.298507
the error rate of this test is :0.253731
the error rate of this test is :0.432836
the error rate of this test is :0.328358
the error rate of this test is :0.432836
the error rate of this test is :0.373134
the error rate of this test is :0.343284
the error rate of this test is :0.328358
the error rate of this test is :0.402985
the error rate of this test is :0.462687
the error rate of this test is :0.298507
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.432836
the error rate of this test is :0.328358
the error rate of this test is :0.283582
the error rate of this test is :0.462687
the error rate of this test is :0.283582
the error rate of this test is :0.238806
the error rate of this test is :0.388060
the error rate of this test is :0.388060
the error rate of this test is :0.492537
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.373134
the error rate of this test is :0.328358
the error rate of this test is :0.253731
the error rate of this test is :0.313433
the error rate of this test is :0.388060
the error rate of this test is :0.388060
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.238806
the error rate of this test is :0.388060
the error rate of this test is :0.388060
the error rate of this test is :0.447761
the error rate of this test is :0.298507
the error rate of this test is :0.328358
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.417910
the error rate of this test is :0.283582
the error rate of this test is :0.313433
the error rate of this test is :0.283582
the error rate of this test is :0.283582
the error rate of this test is :0.358209
the error rate of this test is :0.328358
the error rate of this test is :0.223881
the error rate of this test is :0.477612
the error rate of this test is :0.268657
the error rate of this test is :0.402985
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.388060
the error rate of this test is :0.417910
the error rate of this test is :0.492537
the error rate of this test is :0.298507
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.328358
the error rate of this test is :0.328358
the error rate of this test is :0.388060
the error rate of this test is :0.253731
the error rate of this test is :0.328358
the error rate of this test is :0.417910
the error rate of this test is :0.462687
the error rate of this test is :0.373134
the error rate of this test is :0.447761
the error rate of this test is :0.388060
the error rate of this test is :0.283582
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.388060
the error rate of this test is :0.298507
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.447761
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.343284
the error rate of this test is :0.402985
the error rate of this test is :0.417910
the error rate of this test is :0.373134
the error rate of this test is :0.358209
the error rate of this test is :0.477612
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.328358
the error rate of this test is :0.373134
the error rate of this test is :0.373134
the error rate of this test is :0.283582
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.402985
the error rate of this test is :0.328358
the error rate of this test is :0.373134
the error rate of this test is :0.402985
the error rate of this test is :0.298507
the error rate of this test is :0.417910
the error rate of this test is :0.358209
the error rate of this test is :0.283582
the error rate of this test is :0.313433
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.268657
the error rate of this test is :0.507463
the error rate of this test is :0.507463
the error rate of this test is :0.417910
the error rate of this test is :0.388060
the error rate of this test is :0.328358
the error rate of this test is :0.268657
the error rate of this test is :0.343284
the error rate of this test is :0.343284
the error rate of this test is :0.268657
the error rate of this test is :0.298507
the error rate of this test is :0.432836
the error rate of this test is :0.417910
the error rate of this test is :0.373134
the error rate of this test is :0.283582
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.343284
the error rate of this test is :0.343284
the error rate of this test is :0.388060
the error rate of this test is :0.447761
the error rate of this test is :0.417910
the error rate of this test is :0.343284
the error rate of this test is :0.238806
the error rate of this test is :0.328358
the error rate of this test is :0.298507
the error rate of this test is :0.388060
the error rate of this test is :0.268657
the error rate of this test is :0.417910
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.298507
the error rate of this test is :0.313433
the error rate of this test is :0.313433
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.373134
the error rate of this test is :0.432836
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.343284
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.358209
the error rate of this test is :0.298507
the error rate of this test is :0.298507
the error rate of this test is :0.343284
the error rate of this test is :0.388060
the error rate of this test is :0.343284
the error rate of this test is :0.417910
the error rate of this test is :0.283582
the error rate of this test is :0.298507
the error rate of this test is :0.298507
the error rate of this test is :0.298507
the error rate of this test is :0.417910
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.343284
the error rate of this test is :0.238806
the error rate of this test is :0.402985
the error rate of this test is :0.388060
the error rate of this test is :0.477612
the error rate of this test is :0.373134
the error rate of this test is :0.298507
the error rate of this test is :0.328358
the error rate of this test is :0.388060
the error rate of this test is :0.537313
the error rate of this test is :0.358209
the error rate of this test is :0.402985
the error rate of this test is :0.373134
the error rate of this test is :0.298507
the error rate of this test is :0.402985
the error rate of this test is :0.358209
the error rate of this test is :0.388060
the error rate of this test is :0.447761
the error rate of this test is :0.268657
the error rate of this test is :0.373134
the error rate of this test is :0.373134
the error rate of this test is :0.313433
the error rate of this test is :0.358209
the error rate of this test is :0.447761
the error rate of this test is :0.328358
the error rate of this test is :0.343284
the error rate of this test is :0.417910
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.388060
the error rate of this test is :0.313433
the error rate of this test is :0.298507
the error rate of this test is :0.328358
the error rate of this test is :0.402985
the error rate of this test is :0.343284
the error rate of this test is :0.417910
the error rate of this test is :0.432836
the error rate of this test is :0.417910
the error rate of this test is :0.298507
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.373134
the error rate of this test is :0.298507
the error rate of this test is :0.388060
the error rate of this test is :0.283582
the error rate of this test is :0.358209
the error rate of this test is :0.313433
the error rate of this test is :0.358209
the error rate of this test is :0.358209
the error rate of this test is :0.283582
the error rate of this test is :0.492537
the error rate of this test is :0.313433
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.328358
the error rate of this test is :0.417910
the error rate of this test is :0.343284
the error rate of this test is :0.358209
the error rate of this test is :0.417910
the error rate of this test is :0.268657
the error rate of this test is :0.268657
the error rate of this test is :0.492537
the error rate of this test is :0.328358
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.477612
the error rate of this test is :0.343284
the error rate of this test is :0.238806
the error rate of this test is :0.388060
the error rate of this test is :0.268657
the error rate of this test is :0.417910
the error rate of this test is :0.388060
the error rate of this test is :0.388060
the error rate of this test is :0.537313
the error rate of this test is :0.432836
the error rate of this test is :0.343284
the error rate of this test is :0.417910
the error rate of this test is :0.283582
the error rate of this test is :0.268657
the error rate of this test is :0.358209
the error rate of this test is :0.283582
the error rate of this test is :0.447761
the error rate of this test is :0.447761
the error rate of this test is :0.283582
the error rate of this test is :0.343284
the error rate of this test is :0.417910
the error rate of this test is :0.283582
the error rate of this test is :0.402985
the error rate of this test is :0.417910
the error rate of this test is :0.283582
the error rate of this test is :0.358209
the error rate of this test is :0.298507
the error rate of this test is :0.343284
the error rate of this test is :0.313433
the error rate of this test is :0.432836
the error rate of this test is :0.328358
the error rate of this test is :0.268657
the error rate of this test is :0.402985
the error rate of this test is :0.358209
the error rate of this test is :0.432836
the error rate of this test is :0.373134
the error rate of this test is :0.388060
the error rate of this test is :0.447761
the error rate of this test is :0.358209
the error rate of this test is :0.373134
the error rate of this test is :0.343284
the error rate of this test is :0.373134
the error rate of this test is :0.283582
the error rate of this test is :0.402985
the error rate of this test is :0.402985
the error rate of this test is :0.388060
the error rate of this test is :0.388060
the error rate of this test is :0.238806
the error rate of this test is :0.477612
the error rate of this test is :0.417910
the error rate of this test is :0.417910
after 1000 iterations the average error rate is:0.361343