StandardScaler

class pyspark.mllib.feature.StandardScaler(withMean=False, withStd=True)

Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.

New in version 1.2.0.

Parameters

withMeanbool, optional

False by default. Centers the data with mean before scaling. It will build a dense output, so take care when applying to sparse input.

withStdbool, optional

True by default. Scales the data to unit standard deviation.

Examples

>>> vs = [Vectors.dense([-2.0, 2.3, 0]), Vectors.dense([3.8, 0.0, 1.9])]
>>> dataset = sc.parallelize(vs)
>>> standardizer = StandardScaler(True, True)
>>> model = standardizer.fit(dataset)
>>> result = model.transform(dataset)
>>> for r in result.collect(): r
DenseVector([-0.7071, 0.7071, -0.7071])
DenseVector([0.7071, -0.7071, 0.7071])
>>> int(model.std[0])
4
>>> int(model.mean[0]*10)
9
>>> model.withStd
True
>>> model.withMean
True

Methods

fit(dataset)

Computes the mean and variance and stores as a model to be used for later scaling.

Methods Documentation

fit(dataset)

Computes the mean and variance and stores as a model to be used for later scaling.

New in version 1.2.0.

Parameters

datasetpyspark.RDD

The data used to compute the mean and variance to build the transformation model.

Returns

StandardScalerModel