2018-03-06

线性回归的核心代码

from sklearn.linear_model import LinearRegression
reg = LinearRegression()
reg.fit(ages_train,net_worths_train) # 训练回归直线

---------------------------------------------------------------------------

NameError                                 Traceback (most recent call last)

<ipython-input-3-331a5f2fde64> in <module>()
      1 from sklearn.linear_model import LinearRegression
      2 reg = LinearRegression()
----> 3 reg.fit(ages_train,net_worths_train) # 训练回归直线


NameError: name 'ages_train' is not defined

reg.coef_可以获取斜率
reg.intercept_可以获取截距
reg.score(target,data)可以获取r平方分数,这个是用来衡量这个拟合程度的一个变量.
这个数值介于0-1之间,如果是接近于1,意味着越好.

文本学习

# 提取词干的操作.
from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer("english")
for word in word_list:
    words = words + ' ' + stemmer.stem(word)

# 获取词袋的操作.
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
bags_of_words = vectorizer.fit(word_data)
bags_of_words = vectorizer.transform(word_data)

from sklearn.feature_extraction.text import CountVectorizer  
  
#语料  
corpus = [  
    'This is the first document.',  
    'This is the second second document.',  
    'And the third one.',  
    'Is this the first document?',  
]  
#将文本中的词语转换为词频矩阵  
vectorizer = CountVectorizer()  
#计算个词语出现的次数  
X = vectorizer.fit_transform(corpus)  
#获取词袋中所有文本关键词  
word = vectorizer.get_feature_names()  
print word  
#查看词频结果  
print X.toarray()  
  
from sklearn.feature_extraction.text import TfidfTransformer  
  
#类调用  
transformer = TfidfTransformer()  
print transformer  
#将词频矩阵X统计成TF-IDF值  
tfidf = transformer.fit_transform(X)  
#查看数据结构 tfidf[i][j]表示i类文本中的tf-idf权重  
print tfidf.toarray()  

[u'and', u'document', u'first', u'is', u'one', u'second', u'the', u'third', u'this']
[[0 1 1 1 0 0 1 0 1]
 [0 1 0 1 0 2 1 0 1]
 [1 0 0 0 1 0 1 1 0]
 [0 1 1 1 0 0 1 0 1]]
TfidfTransformer(norm=u'l2', smooth_idf=True, sublinear_tf=False,
         use_idf=True)
[[ 0.          0.43877674  0.54197657  0.43877674  0.          0.
   0.35872874  0.          0.43877674]
 [ 0.          0.27230147  0.          0.27230147  0.          0.85322574
   0.22262429  0.          0.27230147]
 [ 0.55280532  0.          0.          0.          0.55280532  0.
   0.28847675  0.55280532  0.        ]
 [ 0.          0.43877674  0.54197657  0.43877674  0.          0.
   0.35872874  0.          0.43877674]]

偏差,方差以及特征选择的状况

高偏差差对训练数据关系很少,是一种过度的简化.
高方差相反.它不能很好的推广到没有见过的情况.(往往会过拟合的状况.)