h2o-tutorials/tutorials/deeplearning/deeplearning.py at master · JavaMuller/h2o-tutorials

History

462 lines (329 loc) · 26.6 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

# coding: utf-8

# # Introduction

# This tutorial shows how a H2O [Deep Learning](http://en.wikipedia.org/wiki/Deep_learning) model can be used to do supervised classification and regression. This tutorial covers usage of H2O from Python. An R version of this tutorial will be available as well in a separate document. This file is available in plain R, R markdown, regular markdown, plain Python and iPython Notebook formats, and the plots are available as PDF files. All documents are available [on Github](https://github.com/h2oai/h2o-tutorials/tree/master/tutorials/deeplearning). More examples and explanations can be found in our [H2O Deep Learning booklet](http://h2o.ai/resources/) and on our [H2O Github Repository](http://github.com/h2oai/h2o-3/). The PDF slide deck can be found [on Github](https://github.com/h2oai/h2o-tutorials/tree/master/tutorials/deeplearning/H2ODeepLearning.pdf).

# ### H2O Python Module

# Load the H2O Python module.

# In[1]:

import h2o

# ### Start H2O

# Start up a 1-node H2O cloud on your local machine, and allow it to use all CPU cores and up to 2GB of memory:

# In[2]:

h2o.init(max_mem_size = 2) #uses all cores by default

h2o.remove_all() #clean slate, in case cluster was already running

# To learn more about the h2o package itself, we can use Python's builtin help() function.

# In[3]:

help(h2o)

# help() can be used on H2O functions and models. Jupyter's builtin shift-tab functionality also works

# In[4]:

from h2o.estimators.deeplearning import H2OAutoEncoderEstimator, H2ODeepLearningEstimator

help(H2ODeepLearningEstimator)

help(h2o.import_file)

# ##H2O Deep Learning

# While H2O Deep Learning has many parameters, it was designed to be just as easy to use as the other supervised training methods in H2O. Early stopping, automatic data standardization and handling of categorical variables and missing values and adaptive learning rates (per weight) reduce the amount of parameters the user has to specify. Often, it's just the number and sizes of hidden layers, the number of epochs and the activation function and maybe some regularization techniques.

# ### Let's have some fun first: Decision Boundaries

# We start with a small dataset representing red and black dots on a plane, arranged in the shape of two nested spirals. Then we task H2O's machine learning methods to separate the red and black dots, i.e., recognize each spiral as such by assigning each point in the plane to one of the two spirals.

# We visualize the nature of H2O Deep Learning (DL), H2O's tree methods (GBM/DRF) and H2O's generalized linear modeling (GLM) by plotting the decision boundary between the red and black spirals:

# Let's get our imports first.

# In[5]:

get_ipython().magic(u'matplotlib inline')

#IMPORT ALL THE THINGS

import matplotlib.pyplot as plt

import numpy as np

import pandas as pd

from h2o.estimators.deeplearning import H2OAutoEncoderEstimator, H2ODeepLearningEstimator

from h2o.estimators.gbm import H2OGradientBoostingEstimator

from h2o.estimators.glm import H2OGeneralizedLinearEstimator

from h2o.estimators.random_forest import H2ORandomForestEstimator

# First, we need to upload our datasets to the the H2O cluster. The data is imported into H2OFrames, which operate similarly in function to pandas DataFrames.

# In this case, the cluster is running on our laptops. Data files are imported by their relative locations to this notebook.

# In[6]:

import os

spiral = h2o.import_file(path = os.path.realpath("../data/spiral.csv"))

grid = h2o.import_file(path = os.path.realpath("../data/grid.csv"))

# Spiral is a simple data set consisting of two spirals of black and red dots.

# Grid is a 201 by 201 matrix with dimensions [-1.5, 1.5] by [-1.5, 1.5].

# To visualize these datasets, we can pull them from H2OFrames into pandas DataFrames for easier plotting.

# In[7]:

spiral_df = spiral.as_data_frame(use_pandas=True)

grid_df = grid.as_data_frame(use_pandas=True)

grid_x, grid_y = grid_df.x.reshape(201,201), grid_df.y.reshape(201,201)

spiral_r = spiral_df[spiral_df.color == "Red"]

spiral_k = spiral_df[spiral_df.color == "Black"]

spiral_xr, spiral_yr = spiral_r[spiral_r.columns[0]], spiral_r[spiral_r.columns[1]]

spiral_xk, spiral_yk = spiral_k[spiral_k.columns[0]], spiral_k[spiral_k.columns[1]]

markersize_ = 7**2

plt.figure(figsize = (5,5))

plt.scatter(spiral_xr, spiral_yr, c = 'r', s=markersize_)

plt.scatter(spiral_xk, spiral_yk, c = 'k', s=markersize_)

plt.axis([-1.5, 1.5, -1.5, 1.5])

plt.title("Spiral");

# ###Model Construction

# H2O in Python is designed to be very similar in look and feel to to scikit-learn. Models are initialized individually with desired or default parameters and then trained on data.

# Note that the below examples use model.train(), as opposed the traditional model.fit()

# This is because h2o-py takes the data frame AND column indices for the feature and response columns, while scikit-learn takes in feature frames.

# H2O supports model.fit() so that it can be incorporated into a scikit-learn pipeline, but we advise using train() in all other cases.

# In[8]:

X = spiral.col_names[0:2]

y = spiral.col_names[2]

dl_model = H2ODeepLearningEstimator(epochs=1000)

dl_model.train(X, y, spiral)

gbm_model = H2OGradientBoostingEstimator()

gbm_model.train(X, y, spiral)

drf_model = H2ORandomForestEstimator()

drf_model.train(X, y, spiral)

glm_model = H2OGeneralizedLinearEstimator(family="binomial")

glm_model.fit(spiral[X], spiral[y]) #model.fit() example

models = [dl_model, gbm_model, drf_model, glm_model]

m_names = ["Deep Learning", "Gradient Boosted Method", "Distributed Random Forest", "Generalized Linear Model"]

# Now that we've trained four models to classify points as red or black based on their (x,y) coordinates.

# To see how our models have performed, we ask them to predict the colors of the grid.

# Since we'll be doing a lot of spiral plotting, let's write a little helper function to keep things clean.

# In[9]:

def plot_spirals(models, model_names):

fig, ax = plt.subplots(2,2, figsize=(12,12))

for k, subplot in enumerate(ax.flatten()):

subplot.scatter(spiral_xr, spiral_yr, c = 'r', s=markersize_)

subplot.scatter(spiral_xk, spiral_yk, c = 'k', s=markersize_)

subplot.axis([-1.5, 1.5, -1.5, 1.5])

subplot.set_title(model_names[k])

subplot.set_xlabel('x')

subplot.set_ylabel('y')

pred_z = models[k].predict(grid).as_data_frame(True)

subplot.contour(grid_x, grid_y, (pred_z['predict'] == 'Black').astype(np.int).reshape(201,201), colors='b')

# Below are four graphs of the contour plots of the predictions, so that we can see how exactly the algorithms grouped the points into black and red.

# In[10]:

plot_spirals(models, m_names)

# ###A Deeper Dive into Deep Learning

# Now let's explore the evolution of our deep learning model over training time (number of passes over the data, aka epochs).

# We will use checkpointing to ensure that we continue training the same model

# In[11]:

dl_1 = H2ODeepLearningEstimator(epochs=1)

dl_1.train(X, y, spiral)

dl_250 = H2ODeepLearningEstimator(checkpoint=dl_1, epochs=250)

dl_250.train(X, y, spiral)

dl_500 = H2ODeepLearningEstimator(checkpoint=dl_250, epochs=500)

dl_500.train(X, y, spiral)

dl_750 = H2ODeepLearningEstimator(checkpoint=dl_500, epochs=750)

dl_750.train(X, y, spiral)

# You can see how the network learns the structure of the spirals with enough training time.

# In[12]:

models_dl = [dl_1, dl_250, dl_500, dl_750]

m_names_dl = ["DL " + str(int(model.get_params()['epochs']['actual_value'])) + " Epochs" for model in models_dl]

plot_spirals(models_dl, m_names_dl)

# ###Deep Learning Network Architecture

# Of course, there is far more to constructing Deep Learning models than simply having them run longer.

# Consider the four following setups.

# 1. Single layer, 1000 nodes

# 2. Two layers, 200 nodes each

# 3. Three layers, 42 nodes each

# 4. Four layers, 11 -> 13 -> 17 -> 19

# The H2O Architecture uses the hidden keyword to control model network architecture.

# Hidden takes a list of integers, representing the number of nodes in each layer.

# In[13]:

dl_1 = H2ODeepLearningEstimator(hidden=[1000], epochs=500)

dl_1.train(X, y, spiral)

dl_2 = H2ODeepLearningEstimator(hidden=[200,200], epochs=500)

dl_2.train(X, y, spiral)

dl_3 = H2ODeepLearningEstimator(hidden=[42,42,42], epochs=500)

dl_3.train(X, y, spiral)

dl_4 = H2ODeepLearningEstimator(hidden=[11,13,17,19], epochs = 1000)

dl_4.train(X, y, spiral)

# It is clear that different configurations can achieve similar performance, and that tuning will be required for optimal performance.

# In[14]:

models_network = [dl_1, dl_2, dl_3, dl_4]

m_names_network = ["1000", "200 x 200", "42 x 42 x 42", "11 x 13 x 17 x 19"]

plot_spirals(models_network, m_names_network)

# ###Activation Functions

# Next, we compare between different activation functions, including one with 50% dropout regularization in the hidden layers:

# In[15]:

models_act = []

m_names_act = []

for i,method in enumerate(["Tanh","Maxout","Rectifier","RectifierWithDropout"]):

models_act.append(H2ODeepLearningEstimator(activation=method, hidden=[100,100], epochs=1000))

models_act[i].train(X, y, spiral)

m_names_act.append("DL "+ method + " Activation")

# In[16]:

plot_spirals(models_act, m_names_act)

# Clearly, the dropout rate was too high or the number of epochs was too low for the last configuration, which often ends up performing the best on larger datasets where generalization is important.

# More information about the parameters can be found in the [H2O Deep Learning booklet](http://h2o.ai/resources/).

# ## Covertype Dataset

# The following examples use the Covertype dataset from UC Irvine, which concerns predicting forest cover based on cartographical data.

# We import the full covertype dataset (581k rows, 13 columns, 10 numerical, 3 categorical) and then split the data 3 ways:

# 60% for training

# 20% for validation (hyper parameter tuning)

# 20% for final testing

# We will train a data set on one set and use the others to test the validity of the model by ensuring that it can predict accurately on data the model has not been shown.

# The second set will be used for validation most of the time.

# The third set will be withheld until the end, to ensure that our validation accuracy is consistent with data we have never seen during the iterative process.

# In[17]:

covtype_df = h2o.import_file(path = os.path.realpath("../data/covtype.full.csv"))

#split the data as described above

train, valid, test = covtype_df.split_frame([0.6, 0.2], seed=1234)

#Prepare predictors and response columns

covtype_X = covtype_df.col_names[:-1] #last column is cover_type,

covtype_y = covtype_df.col_names[-1]

# ####First Impressions

# Let's run our first Deep Learning model on the covtype dataset.

# We want to predict the `Cover_Type` column, a categorical feature with 7 levels, and the Deep Learning model will be tasked to perform (multi-class) classification. It uses the other 12 predictors of the dataset, of which 10 are numerical, and 2 are categorical with a total of 44 levels.

# We can expect the Deep Learning model to have 56 input neurons (after automatic one-hot encoding). First run will be only one epoch to get a feel for the model construction.

# In[18]:

#set the model_id for easy lookup in Flow

covtype_model_v1 = H2ODeepLearningEstimator(model_id="covtype_v1", epochs=1, variable_importances=True)

covtype_model_v1.train(covtype_X, covtype_y, training_frame = train, validation_frame = valid)

print covtype_model_v1

# Inspect the model in [Flow](http://localhost:54321/) for more information about model building etc.

# Enter `getModel "covtype_v1"` into a cell and run in place pressing Ctrl-Enter. Alternatively, you can click on the Models tab, select List All Models, and click on the model named "covtype_v1" as specified in our model construction above.

# ### Variable Importances

# Variable importances for Neural Network models are notoriously difficult to compute, and there are many [pitfalls](ftp://ftp.sas.com/pub/neural/importance.html). H2O Deep Learning has implemented the method of [Gedeon](http://cs.anu.edu.au/~./Tom.Gedeon/pdfs/ContribDataMinv2.pdf), and returns relative variable importances in descending order of importance.

# Note that we have exactly 56 input neurons, as expected

# In[19]:

var_df = pd.DataFrame(covtype_model_v1.varimp(),

columns=["Variable", "Relative Importance", "Scaled Importance", "Percentage"])

print var_df.shape

var_df.head(10)

# ### Early Stopping

# Now we run another, smaller network and we let it stop automatically once the misclassification rate converges (specifically if the moving average of length 2 does not improve by at least 1% for 2 consecutive scoring events).

# We also sample the validation set to 10,000 rows for faster scoring.

# In[20]:

covtype_model_v2 = H2ODeepLearningEstimator(

model_id="covtype_v2",

hidden=[32,32,32], ## small network, runs faster

epochs=1000000, ## hopefully converges earlier...

score_validation_samples=10000, ## sample the validation dataset (faster)

stopping_rounds=2,

stopping_metric="misclassification", ## alternatives: "MSE","logloss","r2"

stopping_tolerance=0.01)

covtype_model_v2.train(covtype_X, covtype_y, training_frame=train, validation_frame=valid)

# To look at the scoring history, we can look up our new model in Flow.

# Alternatively, we can use the score_history method to retrieve the data as a pandas DataFrame.

# In[21]:

cov_v2_df = covtype_model_v2.score_history()

cov_v2_df

# In[22]:

plt.plot(cov_v2_df['training_classification_error'], label="training_classification_error")

plt.plot(cov_v2_df['validation_classification_error'], label="validation_classification_error")

plt.title("Covertype Deep Learner (Early Stop)")

plt.legend();

# ### Adaptive Learning Rate

# By default, H2O Deep Learning uses an adaptive learning rate ([ADADELTA](http://arxiv.org/pdf/1212.5701v1.pdf)) for its stochastic gradient descent optimization. There are only two tuning parameters for this method: `rho` and `epsilon`, which balance the global and local search efficiencies. `rho` is the similarity to prior weight updates (similar to momentum), and `epsilon` is a parameter that prevents the optimization to get stuck in local optima.

# Defaults are `rho=0.99` and `epsilon=1e-8`. For cases where convergence speed is very important, it might make sense to perform a few runs to optimize these two parameters (e.g., with `rho in c(0.9,0.95,0.99,0.999)` and `epsilon in c(1e-10,1e-8,1e-6,1e-4)`). Of course, as always with grid searches, caution has to be applied when extrapolating grid search results to a different parameter regime (e.g., for more epochs or different layer topologies or activation functions, etc.).

# If `adaptive_rate` is disabled, several manual learning rate parameters become important: `rate`, `rate_annealing`, `rate_decay`, `momentum_start`, `momentum_ramp`, `momentum_stable` and `nesterov_accelerated_gradient`, the discussion of which we leave to [H2O Deep Learning booklet](http://h2o.ai/resources/).

# ### Tuning

# With some tuning, it is possible to obtain less than 10% test set error rate in about one minute. Error rates of below 5% are possible with larger models. Deep tree methods are more effective for this dataset than Deep Learning, as the space needs to be simply be partitioned into the corresponding hyper-space corners to solve this problem.

# In[23]:

covtype_model_tuned = H2ODeepLearningEstimator(

model_id="covtype_tuned",

overwrite_with_best_model=False,

hidden=[128,128,128], # more hidden layers -> more complex interactions

epochs=10, # to keep it short enough

score_validation_samples=10000, # downsample validation set for faster scoring

score_duty_cycle=0.025, # don't score more than 2.5% of the wall time

adaptive_rate=False, # manually tuned learning rate

rate=0.01,

rate_annealing=0.000002,

momentum_start=0.2, # manually tuned momentum

momentum_stable=0.4,

momentum_ramp=10000000,

l1=0.00001, # add some L1/L2 regularization

l2=0.00001,

max_w2=10 # helps stability for Rectifier

)

covtype_model_tuned.train(covtype_X, covtype_y, training_frame=train, validation_frame=valid)

# Let's compare the training error with the validation and test set errors.

# In[24]:

cov_tuned_df = covtype_model_tuned.scoring_history()

cov_tuned_df

# In[25]:

plt.plot(cov_tuned_df['training_classification_error'], label="training_classification_error")

plt.plot(cov_tuned_df['validation_classification_error'], label="validation_classification_error")

plt.title("Covertype Deep Learner (Tuned)")

plt.legend();

# In[26]:

pred = covtype_model_tuned.predict(test[0:-1]).as_data_frame(use_pandas=True)

test_actual = test.as_data_frame(use_pandas=True)['Cover_Type']

(test_actual == pred['predict']).mean()

# As you can see, we've achieved approximately 90% accuracy on our test data, which is fairly in line with our validation results.

# ###Shutdown H2O Cluster

# This concludes the Python portion of the deeplearning demo.

# Shut down the cluster now that we are done using it.

# In[27]:

h2o.shutdown(prompt=False)

# ##Unsupervised Anomaly detection

# For instructions on how to build unsupervised models with H2O Deep Learning, we refer to our previous [Tutorial on Anomaly Detection with H2O Deep Learning](https://www.youtube.com/watch?v=fUSbljByXak) and our [MNIST Anomaly detection code example](https://github.com/h2oai/h2o-3/blob/master/h2o-r/tests/testdir_algos/deeplearning/runit_deeplearning_anomaly_large.R), as well as our [Stacked AutoEncoder R code example](https://github.com/h2oai/h2o-3/blob/master/h2o-r/tests/testdir_algos/deeplearning/runit_deeplearning_stacked_autoencoder_large.R).

# ##H2O Deep Learning Tips & Tricks

# ####Activation Functions

# While sigmoids have been used historically for neural networks, H2O Deep Learning implements `Tanh`, a scaled and shifted variant of the sigmoid which is symmetric around 0. Since its output values are bounded by -1..1, the stability of the neural network is rarely endangered. However, the derivative of the tanh function is always non-zero and back-propagation (training) of the weights is more computationally expensive than for rectified linear units, or `Rectifier`, which is `max(0,x)` and has vanishing gradient for `x<=0`, leading to much faster training speed for large networks and is often the fastest path to accuracy on larger problems. In case you encounter instabilities with the `Rectifier` (in which case model building is automatically aborted), try a limited value to re-scale the weights: `max_w2=10`. The `Maxout` activation function is computationally more expensive, but can lead to higher accuracy. It is a generalized version of the Rectifier with two non-zero channels. In practice, the `Rectifier` (and `RectifierWithDropout`, see below) is the most versatile and performant option for most problems.

# ####Generalization Techniques

# L1 and L2 penalties can be applied by specifying the `l1` and `l2` parameters. Intuition: L1 lets only strong weights survive (constant pulling force towards zero), while L2 prevents any single weight from getting too big. [Dropout](http://arxiv.org/pdf/1207.0580.pdf) has recently been introduced as a powerful generalization technique, and is available as a parameter per layer, including the input layer. `input_dropout_ratio` controls the amount of input layer neurons that are randomly dropped (set to zero), while `hidden_dropout_ratios` are specified for each hidden layer. The former controls overfitting with respect to the input data (useful for high-dimensional noisy data), while the latter controls overfitting of the learned features. Note that `hidden_dropout_ratios` require the activation function to end with `...WithDropout`.

# ####Early stopping and optimizing for lowest validation error

# By default, Deep Learning training stops when the `stopping_metric` does not improve by at least `stopping_tolerance` (0.01 means 1% improvement) for `stopping_rounds` consecutive scoring events on the training (or validation) data. By default, `overwrite_with_best_model` is enabled and the model returned after training for the specified number of epochs (or after stopping early due to convergence) is the model that has the best training set error (according to the metric specified by `stopping_metric`), or, if a validation set is provided, the lowest validation set error. Note that the training or validation set errors can be based on a subset of the training or validation data, depending on the values for `score_validation_samples` or `score_training_samples`, see below. For early stopping on a predefined error rate on the *training data* (accuracy for classification or MSE for regression), specify `classification_stop` or `regression_stop`.

# ####Training Samples per MapReduce Iteration

# The parameter `train_samples_per_iteration` matters especially in multi-node operation. It controls the number of rows trained on for each MapReduce iteration. Depending on the value selected, one MapReduce pass can sample observations, and multiple such passes are needed to train for one epoch. All H2O compute nodes then communicate to agree on the best model coefficients (weights/biases) so far, and the model may then be scored (controlled by other parameters below). The default value of `-2` indicates auto-tuning, which attemps to keep the communication overhead at 5% of the total runtime. The parameter `target_ratio_comm_to_comp` controls this ratio. This parameter is explained in more detail in the [H2O Deep Learning booklet](http://h2o.ai/resources/),

# ####Categorical Data

# For categorical data, a feature with K factor levels is automatically one-hot encoded (horizontalized) into K-1 input neurons. Hence, the input neuron layer can grow substantially for datasets with high factor counts. In these cases, it might make sense to reduce the number of hidden neurons in the first hidden layer, such that large numbers of factor levels can be handled. In the limit of 1 neuron in the first hidden layer, the resulting model is similar to logistic regression with stochastic gradient descent, except that for classification problems, there's still a softmax output layer, and that the activation function is not necessarily a sigmoid (`Tanh`). If variable importances are computed, it is recommended to turn on `use_all_factor_levels` (K input neurons for K levels). The experimental option `max_categorical_features` uses feature hashing to reduce the number of input neurons via the hash trick at the expense of hash collisions and reduced accuracy. Another way to reduce the dimensionality of the (categorical) features is to use `h2o.glrm()`, we refer to the GLRM tutorial for more details.

# ####Missing Values

# H2O Deep Learning automatically does mean imputation for missing values during training (leaving the input layer activation at 0 after standardizing the values). For testing, missing test set values are also treated the same way by default. See the `h2o.impute` function to do your own mean imputation.

# ####Loss functions, Distributions, Offsets, Observation Weights

# H2O Deep Learning supports advanced statistical features such as multiple loss functions, non-Gaussian distributions, per-row offsets and observation weights.

# In addition to `Gaussian` distributions and `Squared` loss, H2O Deep Learning supports `Poisson`, `Gamma`, `Tweedie` and `Laplace` distributions. It also supports `Absolute` and `Huber` loss and per-row offsets specified via an `offset_column`. Observation weights are supported via a user-specified `weights_column`.

# We refer to our [H2O Deep Learning R test code examples](https://github.com/h2oai/h2o-3/tree/master/h2o-r/tests/testdir_algos/deeplearning) for more information.

# a

# ####Reproducibility

# Every run of DeepLearning results in different results since multithreading is done via [Hogwild!](http://www.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf) that benefits from intentional lock-free race conditions between threads. To get reproducible results for small datasets and testing purposes, set `reproducible=T` and set `seed=1337` (pick any integer). This will not work for big data for technical reasons, and is probably also not desired because of the significant slowdown (runs on 1 core only).

# ####Scoring on Training/Validation Sets During Training

# The training and/or validation set errors *can* be based on a subset of the training or validation data, depending on the values for `score_validation_samples` (defaults to 0: all) or `score_training_samples` (defaults to 10,000 rows, since the training error is only used for early stopping and monitoring). For large datasets, Deep Learning can automatically sample the validation set to avoid spending too much time in scoring during training, especially since scoring results are not currently displayed in the model returned to R.

# Note that the default value of `score_duty_cycle=0.1` limits the amount of time spent in scoring to 10%, so a large number of scoring samples won't slow down overall training progress too much, but it will always score once after the first MapReduce iteration, and once at the end of training.

# Stratified sampling of the validation dataset can help with scoring on datasets with class imbalance. Note that this option also requires `balance_classes` to be enabled (used to over/under-sample the training dataset, based on the max. relative size of the resulting training dataset, `max_after_balance_size`):

# ### More information can be found in the [H2O Deep Learning booklet](http://h2o.ai/resources/), in our [H2O SlideShare Presentations](http://www.slideshare.net/0xdata/presentations), our [H2O YouTube channel](https://www.youtube.com/user/0xdata/), as well as on our [H2O Github Repository](https://github.com/h2oai/h2o-3/), especially in our [H2O Deep Learning R tests](https://github.com/h2oai/h2o-3/tree/master/h2o-r/tests/testdir_algos/deeplearning), and [H2O Deep Learning Python tests](https://github.com/h2oai/h2o-3/tree/master/h2o-py/tests/testdir_algos/deeplearning).

# ###Further Exploration

# Due to the limited scope of this talk, only a portion of the code has been ported to Python.

# Additional topics are covered in R: Please see the deeplearning.R file for code samples for the following:

# * Hyper-Parameter tuning with Grid Search

# * Random Hyper-Parameter Search

# * Checkpointing

# * Cross-Validation

# * Regression and Binary Classification

# * Exporting Weights and Biases

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

deeplearning.py

Latest commit

History

deeplearning.py

File metadata and controls