Specify options for final model only with caret

Context

I am using caret to fit and tune models. Typically, the best parameters are found using a resampling method such as cross-validation. Once the best parameters are chosen, a final model is fitted to the whole training data using the best set of parameters.

In addition to the parameters to tune (passed via tuneGrid), one can pass arguments to the underlying algorithm being called by passing them to train.

My question

Is there any way to specify model-specific options to be used for the final model only?

For extra clarity: I do want to fit all the intermediate models (to obtain a reliable performance estimate) but I want to fit the final model with different arguments (in addition to the best parameters).

Specific use case

Let's say I want to fit a bartMachine to some data and then use the final model in production. I would typically save the tuned model to disk and load it as needed. But I can only save/load a bartMachine model that has been serialized, i.e. I need to pass serialize=T to bartMachine via caret::train.

But that will serialize all the models which is very impractical. I really only need to serialize the final model. Is there any way to do that?

library("caret")

library("bartMachine")

tgrid <- expand.grid(num_trees = 100,

                       k = c(2, 3),

                       alpha = 0.95, 

                       beta = 2,

                       nu =  3)

# The printed log shows that all intermediate models are being serialized

fit <- train(hp ~ ., 

             data=mtcars, 

             method="bartMachine",

             serialize=T,

             tuneGrid=tgrid,

             trControl = trainControl(method="cv", 5, verboseIter=T))

edited Nov 20 '18 at 15:06

asked Nov 20 '18 at 12:31

antoine-sac

2,65621238

add a comment |

Context

In addition to the parameters to tune (passed via tuneGrid), one can pass arguments to the underlying algorithm being called by passing them to train.

My question

Is there any way to specify model-specific options to be used for the final model only?

Specific use case

But that will serialize all the models which is very impractical. I really only need to serialize the final model. Is there any way to do that?

library("caret")

library("bartMachine")

tgrid <- expand.grid(num_trees = 100,

                       k = c(2, 3),

                       alpha = 0.95, 

                       beta = 2,

                       nu =  3)

# The printed log shows that all intermediate models are being serialized

fit <- train(hp ~ ., 

             data=mtcars, 

             method="bartMachine",

             serialize=T,

             tuneGrid=tgrid,

             trControl = trainControl(method="cv", 5, verboseIter=T))

edited Nov 20 '18 at 15:06

asked Nov 20 '18 at 12:31

antoine-sac

2,65621238

add a comment |

Context

In addition to the parameters to tune (passed via tuneGrid), one can pass arguments to the underlying algorithm being called by passing them to train.

My question

Is there any way to specify model-specific options to be used for the final model only?

Specific use case

But that will serialize all the models which is very impractical. I really only need to serialize the final model. Is there any way to do that?

library("caret")

library("bartMachine")

tgrid <- expand.grid(num_trees = 100,

                       k = c(2, 3),

                       alpha = 0.95, 

                       beta = 2,

                       nu =  3)

# The printed log shows that all intermediate models are being serialized

fit <- train(hp ~ ., 

             data=mtcars, 

             method="bartMachine",

             serialize=T,

             tuneGrid=tgrid,

             trControl = trainControl(method="cv", 5, verboseIter=T))

edited Nov 20 '18 at 15:06

asked Nov 20 '18 at 12:31

antoine-sac

2,65621238

Context

In addition to the parameters to tune (passed via tuneGrid), one can pass arguments to the underlying algorithm being called by passing them to train.

My question

Is there any way to specify model-specific options to be used for the final model only?

Specific use case

But that will serialize all the models which is very impractical. I really only need to serialize the final model. Is there any way to do that?

library("caret")

library("bartMachine")

tgrid <- expand.grid(num_trees = 100,

                       k = c(2, 3),

                       alpha = 0.95, 

                       beta = 2,

                       nu =  3)

# The printed log shows that all intermediate models are being serialized

fit <- train(hp ~ ., 

             data=mtcars, 

             method="bartMachine",

             serialize=T,

             tuneGrid=tgrid,

             trControl = trainControl(method="cv", 5, verboseIter=T))

r r-caret bartmachine

edited Nov 20 '18 at 15:06

asked Nov 20 '18 at 12:31

antoine-sac

2,65621238

edited Nov 20 '18 at 15:06

asked Nov 20 '18 at 12:31

antoine-sac

2,65621238

edited Nov 20 '18 at 15:06

asked Nov 20 '18 at 12:31

antoine-sac

2,65621238

asked Nov 20 '18 at 12:31

antoine-sac

2,65621238

asked Nov 20 '18 at 12:31

antoine-sac

2,65621238

add a comment |

1 Answer
1

active

oldest

votes

To fit models to the entire data set without parameter tuning or resampling modify the train control method to none:

tgrid <- expand.grid(num_trees = 100,

                     k = 2,

                     alpha = 0.95, 

                     beta = 2,

                     nu =  3)

fit <- train(hp ~ ., 

             data=mtcars, 

             method="bartMachine",

             serialize=TRUE,

             tuneGrid=tgrid,

             trControl = trainControl(method="none"))

Note, that I have removed one of the two k values in the question code.
Otherwise there is an error: Only one model should be specified in tuneGrid with no resampling. I suggest building a separate model with the other k value.

The code above gives the following output:

bartMachine initializing with 100 trees...

bartMachine vars checked...

bartMachine java init...

bartMachine factors created...

bartMachine before preprocess...

bartMachine after preprocess... 11 total features...

bartMachine sigsq estimated...

bartMachine training data finalized...

Now building bartMachine for regression ...

building BART with mem-cache speedup...

Iteration 100/1250  mem: 17.6/477.1MB

Iteration 200/1250  mem: 25.1/477.1MB

Iteration 300/1250  mem: 30.8/477.1MB

Iteration 400/1250  mem: 39.9/477.1MB

Iteration 500/1250  mem: 19/477.1MB

Iteration 600/1250  mem: 59.6/477.1MB

Iteration 700/1250  mem: 39.6/477.1MB

Iteration 800/1250  mem: 79.8/477.1MB

Iteration 900/1250  mem: 119.9/477.1MB

Iteration 1000/1250  mem: 40.7/477.1MB

Iteration 1100/1250  mem: 80.8/477.1MB

Iteration 1200/1250  mem: 121/477.1MB

done building BART in 1.289 sec 



burning and aggregating chains from all threads... done

evaluating in sample data...done

serializing in order to be saved for future R sessions...done

The serialize parameter is set to TRUE in fit$finalModel:

fit$finalModel$serialize

[1] TRUE

For what it's worth, the bartMachine internal check_serialization function does not give any warnings or errors (or any other output):

bartMachine:::check_serialization(fit$finalModel)

It's not clear to me how to extract the serialized object from fit$finalModel.
I presume it is stored in fit$finalModel$java_bart_machine which contains an rJava pointer. It may be possible to gain further insight using the rJava package which bartMachine depends on.

Update:
@antoine-sac states in the comments below "serialize=T does not cause the model to be saved but serialises the samples into the model, which means they are saved when the model is written to disk".

edited Nov 20 '18 at 16:06

answered Nov 20 '18 at 12:54

makeyourownmaker

620521

That does not answer my question - I do want to tune and, even if not tuning, I want an out-of-sample performance estimate. What I want is to specify different options when fitting the final model.

– antoine-sac
Nov 20 '18 at 13:02

Which options do you want to specify in your final model?

– makeyourownmaker
Nov 20 '18 at 13:05

I want to pass serialize=T to the final model, but not the intermediate models.

– antoine-sac
Nov 20 '18 at 13:07

I cannot find a caret train serialize option. The "serialize=T" option is ignorded by the caret train function. Is serialize=T a bartMachine parameter?

– makeyourownmaker
Nov 20 '18 at 13:10

Thank you for your comments. I've made progress and updated my answer. However, I cannot find the serialized model. It is not in the working directory.

– makeyourownmaker
Nov 20 '18 at 13:44

|
show 3 more comments

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53393050%2fspecify-options-for-final-model-only-with-caret%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

To fit models to the entire data set without parameter tuning or resampling modify the train control method to none:

tgrid <- expand.grid(num_trees = 100,

                     k = 2,

                     alpha = 0.95, 

                     beta = 2,

                     nu =  3)

fit <- train(hp ~ ., 

             data=mtcars, 

             method="bartMachine",

             serialize=TRUE,

             tuneGrid=tgrid,

             trControl = trainControl(method="none"))

The code above gives the following output:

bartMachine initializing with 100 trees...

bartMachine vars checked...

bartMachine java init...

bartMachine factors created...

bartMachine before preprocess...

bartMachine after preprocess... 11 total features...

bartMachine sigsq estimated...

bartMachine training data finalized...

Now building bartMachine for regression ...

building BART with mem-cache speedup...

Iteration 100/1250  mem: 17.6/477.1MB

Iteration 200/1250  mem: 25.1/477.1MB

Iteration 300/1250  mem: 30.8/477.1MB

Iteration 400/1250  mem: 39.9/477.1MB

Iteration 500/1250  mem: 19/477.1MB

Iteration 600/1250  mem: 59.6/477.1MB

Iteration 700/1250  mem: 39.6/477.1MB

Iteration 800/1250  mem: 79.8/477.1MB

Iteration 900/1250  mem: 119.9/477.1MB

Iteration 1000/1250  mem: 40.7/477.1MB

Iteration 1100/1250  mem: 80.8/477.1MB

Iteration 1200/1250  mem: 121/477.1MB

done building BART in 1.289 sec 



burning and aggregating chains from all threads... done

evaluating in sample data...done

serializing in order to be saved for future R sessions...done

The serialize parameter is set to TRUE in fit$finalModel:

fit$finalModel$serialize

[1] TRUE

For what it's worth, the bartMachine internal check_serialization function does not give any warnings or errors (or any other output):

bartMachine:::check_serialization(fit$finalModel)

edited Nov 20 '18 at 16:06

answered Nov 20 '18 at 12:54

makeyourownmaker

620521

That does not answer my question - I do want to tune and, even if not tuning, I want an out-of-sample performance estimate. What I want is to specify different options when fitting the final model.

– antoine-sac
Nov 20 '18 at 13:02

Which options do you want to specify in your final model?

– makeyourownmaker
Nov 20 '18 at 13:05

I want to pass serialize=T to the final model, but not the intermediate models.

– antoine-sac
Nov 20 '18 at 13:07

I cannot find a caret train serialize option. The "serialize=T" option is ignorded by the caret train function. Is serialize=T a bartMachine parameter?

– makeyourownmaker
Nov 20 '18 at 13:10

Thank you for your comments. I've made progress and updated my answer. However, I cannot find the serialized model. It is not in the working directory.

– makeyourownmaker
Nov 20 '18 at 13:44

|
show 3 more comments

To fit models to the entire data set without parameter tuning or resampling modify the train control method to none:

tgrid <- expand.grid(num_trees = 100,

                     k = 2,

                     alpha = 0.95, 

                     beta = 2,

                     nu =  3)

fit <- train(hp ~ ., 

             data=mtcars, 

             method="bartMachine",

             serialize=TRUE,

             tuneGrid=tgrid,

             trControl = trainControl(method="none"))

The code above gives the following output:

bartMachine initializing with 100 trees...

bartMachine vars checked...

bartMachine java init...

bartMachine factors created...

bartMachine before preprocess...

bartMachine after preprocess... 11 total features...

bartMachine sigsq estimated...

bartMachine training data finalized...

Now building bartMachine for regression ...

building BART with mem-cache speedup...

Iteration 100/1250  mem: 17.6/477.1MB

Iteration 200/1250  mem: 25.1/477.1MB

Iteration 300/1250  mem: 30.8/477.1MB

Iteration 400/1250  mem: 39.9/477.1MB

Iteration 500/1250  mem: 19/477.1MB

Iteration 600/1250  mem: 59.6/477.1MB

Iteration 700/1250  mem: 39.6/477.1MB

Iteration 800/1250  mem: 79.8/477.1MB

Iteration 900/1250  mem: 119.9/477.1MB

Iteration 1000/1250  mem: 40.7/477.1MB

Iteration 1100/1250  mem: 80.8/477.1MB

Iteration 1200/1250  mem: 121/477.1MB

done building BART in 1.289 sec 



burning and aggregating chains from all threads... done

evaluating in sample data...done

serializing in order to be saved for future R sessions...done

The serialize parameter is set to TRUE in fit$finalModel:

fit$finalModel$serialize

[1] TRUE

For what it's worth, the bartMachine internal check_serialization function does not give any warnings or errors (or any other output):

bartMachine:::check_serialization(fit$finalModel)

edited Nov 20 '18 at 16:06

answered Nov 20 '18 at 12:54

makeyourownmaker

620521

That does not answer my question - I do want to tune and, even if not tuning, I want an out-of-sample performance estimate. What I want is to specify different options when fitting the final model.

– antoine-sac
Nov 20 '18 at 13:02

Which options do you want to specify in your final model?

– makeyourownmaker
Nov 20 '18 at 13:05

I want to pass serialize=T to the final model, but not the intermediate models.

– antoine-sac
Nov 20 '18 at 13:07

I cannot find a caret train serialize option. The "serialize=T" option is ignorded by the caret train function. Is serialize=T a bartMachine parameter?

– makeyourownmaker
Nov 20 '18 at 13:10

Thank you for your comments. I've made progress and updated my answer. However, I cannot find the serialized model. It is not in the working directory.

– makeyourownmaker
Nov 20 '18 at 13:44

|
show 3 more comments

To fit models to the entire data set without parameter tuning or resampling modify the train control method to none:

tgrid <- expand.grid(num_trees = 100,

                     k = 2,

                     alpha = 0.95, 

                     beta = 2,

                     nu =  3)

fit <- train(hp ~ ., 

             data=mtcars, 

             method="bartMachine",

             serialize=TRUE,

             tuneGrid=tgrid,

             trControl = trainControl(method="none"))

The code above gives the following output:

bartMachine initializing with 100 trees...

bartMachine vars checked...

bartMachine java init...

bartMachine factors created...

bartMachine before preprocess...

bartMachine after preprocess... 11 total features...

bartMachine sigsq estimated...

bartMachine training data finalized...

Now building bartMachine for regression ...

building BART with mem-cache speedup...

Iteration 100/1250  mem: 17.6/477.1MB

Iteration 200/1250  mem: 25.1/477.1MB

Iteration 300/1250  mem: 30.8/477.1MB

Iteration 400/1250  mem: 39.9/477.1MB

Iteration 500/1250  mem: 19/477.1MB

Iteration 600/1250  mem: 59.6/477.1MB

Iteration 700/1250  mem: 39.6/477.1MB

Iteration 800/1250  mem: 79.8/477.1MB

Iteration 900/1250  mem: 119.9/477.1MB

Iteration 1000/1250  mem: 40.7/477.1MB

Iteration 1100/1250  mem: 80.8/477.1MB

Iteration 1200/1250  mem: 121/477.1MB

done building BART in 1.289 sec 



burning and aggregating chains from all threads... done

evaluating in sample data...done

serializing in order to be saved for future R sessions...done

The serialize parameter is set to TRUE in fit$finalModel:

fit$finalModel$serialize

[1] TRUE

For what it's worth, the bartMachine internal check_serialization function does not give any warnings or errors (or any other output):

bartMachine:::check_serialization(fit$finalModel)

edited Nov 20 '18 at 16:06

answered Nov 20 '18 at 12:54

makeyourownmaker

620521

To fit models to the entire data set without parameter tuning or resampling modify the train control method to none:

tgrid <- expand.grid(num_trees = 100,

                     k = 2,

                     alpha = 0.95, 

                     beta = 2,

                     nu =  3)

fit <- train(hp ~ ., 

             data=mtcars, 

             method="bartMachine",

             serialize=TRUE,

             tuneGrid=tgrid,

             trControl = trainControl(method="none"))

The code above gives the following output:

bartMachine initializing with 100 trees...

bartMachine vars checked...

bartMachine java init...

bartMachine factors created...

bartMachine before preprocess...

bartMachine after preprocess... 11 total features...

bartMachine sigsq estimated...

bartMachine training data finalized...

Now building bartMachine for regression ...

building BART with mem-cache speedup...

Iteration 100/1250  mem: 17.6/477.1MB

Iteration 200/1250  mem: 25.1/477.1MB

Iteration 300/1250  mem: 30.8/477.1MB

Iteration 400/1250  mem: 39.9/477.1MB

Iteration 500/1250  mem: 19/477.1MB

Iteration 600/1250  mem: 59.6/477.1MB

Iteration 700/1250  mem: 39.6/477.1MB

Iteration 800/1250  mem: 79.8/477.1MB

Iteration 900/1250  mem: 119.9/477.1MB

Iteration 1000/1250  mem: 40.7/477.1MB

Iteration 1100/1250  mem: 80.8/477.1MB

Iteration 1200/1250  mem: 121/477.1MB

done building BART in 1.289 sec 



burning and aggregating chains from all threads... done

evaluating in sample data...done

serializing in order to be saved for future R sessions...done

The serialize parameter is set to TRUE in fit$finalModel:

fit$finalModel$serialize

[1] TRUE

For what it's worth, the bartMachine internal check_serialization function does not give any warnings or errors (or any other output):

bartMachine:::check_serialization(fit$finalModel)

edited Nov 20 '18 at 16:06

answered Nov 20 '18 at 12:54

makeyourownmaker

620521

edited Nov 20 '18 at 16:06

answered Nov 20 '18 at 12:54

makeyourownmaker

620521

answered Nov 20 '18 at 12:54

makeyourownmaker

620521

answered Nov 20 '18 at 12:54

makeyourownmaker

620521

That does not answer my question - I do want to tune and, even if not tuning, I want an out-of-sample performance estimate. What I want is to specify different options when fitting the final model.

– antoine-sac
Nov 20 '18 at 13:02

Which options do you want to specify in your final model?

– makeyourownmaker
Nov 20 '18 at 13:05

I want to pass serialize=T to the final model, but not the intermediate models.

– antoine-sac
Nov 20 '18 at 13:07

I cannot find a caret train serialize option. The "serialize=T" option is ignorded by the caret train function. Is serialize=T a bartMachine parameter?

– makeyourownmaker
Nov 20 '18 at 13:10

Thank you for your comments. I've made progress and updated my answer. However, I cannot find the serialized model. It is not in the working directory.

– makeyourownmaker
Nov 20 '18 at 13:44

|
show 3 more comments

That does not answer my question - I do want to tune and, even if not tuning, I want an out-of-sample performance estimate. What I want is to specify different options when fitting the final model.

– antoine-sac
Nov 20 '18 at 13:02

Which options do you want to specify in your final model?

– makeyourownmaker
Nov 20 '18 at 13:05

I want to pass serialize=T to the final model, but not the intermediate models.

– antoine-sac
Nov 20 '18 at 13:07

I cannot find a caret train serialize option. The "serialize=T" option is ignorded by the caret train function. Is serialize=T a bartMachine parameter?

– makeyourownmaker
Nov 20 '18 at 13:10

Thank you for your comments. I've made progress and updated my answer. However, I cannot find the serialized model. It is not in the working directory.

– makeyourownmaker
Nov 20 '18 at 13:44

That does not answer my question - I do want to tune and, even if not tuning, I want an out-of-sample performance estimate. What I want is to specify different options when fitting the final model.

– antoine-sac
Nov 20 '18 at 13:02

Which options do you want to specify in your final model?

– makeyourownmaker
Nov 20 '18 at 13:05

I want to pass serialize=T to the final model, but not the intermediate models.

– antoine-sac
Nov 20 '18 at 13:07

I cannot find a caret train serialize option. The "serialize=T" option is ignorded by the caret train function. Is serialize=T a bartMachine parameter?

– makeyourownmaker
Nov 20 '18 at 13:10

Thank you for your comments. I've made progress and updated my answer. However, I cannot find the serialized model. It is not in the working directory.

– makeyourownmaker
Nov 20 '18 at 13:44

|
show 3 more comments

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu