Package 'marshal'

Title: Framework to Marshal Objects to be Used in Another R Process
Description: Some types of R objects can be used only in the R session they were created. If used as-is in another R process, such objects often result in an immediate error or in obscure and hard-to-troubleshoot outcomes. Because of this, they cannot be saved to file and re-used at a later time. They can also not be exported to a worker in parallel processing. These objects are sometimes referred to as non-exportable or non-serializable objects. One solution to this problem is to use "marshalling" to encode the R object into an exportable representation that then can be used to re-create a copy of that object in another R process. This package provides a framework for marshalling and unmarshalling R objects such that they can be transferred using functions such as serialize() and unserialize() of base R.
Authors: Henrik Bengtsson [aut, cre, cph]
Maintainer: Henrik Bengtsson <[email protected]>
License: MIT + file LICENSE
Version: 0.0.1
Built: 2024-10-27 05:52:37 UTC
Source: https://github.com/futureverse/marshal

Help Index


Package marshal - Marshal Objects to be Used in Another R Process

Description

Some types of R objects can be used only in the R session they were created. If used as-is in another R process, such objects often result in an immediate error or in obscure and hard-to-troubleshoot outcomes. Because of this, they cannot be saved to file and re-used at a later time. They can also not be exported to a worker in parallel processing. These objects are sometimes referred to as non-exportable or non-serializable objects. One solution to this problem is to use "marshalling" to encode the R object into an exportable representation that then can be used to re-create a copy of that object in another R process. This package provides a framework for marshalling and unmarshalling R objects such that they can be transferred using functions such as serialize() and unserialize() of base R.


Marshalling of R objects

Description

Marshalling of R objects

Usage

marshal(...)

unmarshal(...)

Arguments

...

The object to be marshalled, or unmarshalled, followed by additional arguments passed to the specific S3 method.

Value

marshal() returns a marshalled object, which is a list with components:

  • marshalled: marshalled version of the original object

  • unmarshal: function that takes the marshalled object as input and returns an unmarshalled version of the original object.

unmarshal() returns an unmarshalled version of the original object.


Marshalling of 'inline' objects

Description

WARNING: This is very limited proof of concept!

Usage

## S3 method for class 'CFunc'
marshal(inline, ...)

## S3 method for class 'CFunc'
marshallable(...)

Arguments

inline

A CFunc function.

...

Not used.

Details

Currently, it is only possible to marshal a function:

  • of class CFunc that was created without "includes" or "otherdefs"

Value

A marshalled object as described in marshal().

Examples

if (requireNamespace("inline", quietly = TRUE)) {
  code <- "
    int i;  
    for (i = 0; i < *n; i++) x[0] = x[0] + (i+1);  
  "
  sum_1_to_n <- inline::cfunction(
    signature(n = "integer", x = "numeric"),
    code,
    language = "C", convention = ".C"
  )

  ## Marshal CFunc function
  sum_1_to_n_ <- marshal(sum_1_to_n)

  ## Unarshal CFunc function
  sum_1_to_n2 <- unmarshal(sum_1_to_n_)

  y <- sum_1_to_n(10, 0)$x
  print(y)

  y2 <- sum_1_to_n2(10, 0)$x
  print(y2)
}

Marshalling of R connections

Description

Marshalling of R connections

Usage

## S3 method for class 'connection'
marshal(con, ...)

## S3 method for class 'connection'
marshallable(con, ...)

Arguments

con

A connection.

...

Not used.

Value

A marshalled object as described in marshal().

Limitations

Not all connections can be marshalled, specifically we:

  • cannot marshal connections stdin (0), stdout (1), and stderr (2)

  • can only marshal read-only connections

  • can only marshal unopened or seekable connections

Examples

tf <- tempfile()
cat(file = tf, letters, sep = "")

## Read-only connection
con <- file(tf, open = "rb")

bfr <- readChar(con, nchars = 4L)
print(bfr)  ## "abcd"

## Marshal read-only connection, which records the
## current state, including the current file position.
con_ <- marshal(con)

## Unmarshal connection, which restores the state
## of the original connection, including the current
## file position
con2 <- unmarshal(con_)
stopifnot(
  all.equal(summary(con2), summary(con)),
  identical(seek(con2), seek(con))
)

bfr <- readChar(con, nchars = 4L)
print(bfr)
bfr2 <- readChar(con2, nchars = 4L)
print(bfr2)
stopifnot(identical(bfr2, bfr))

## Cleanup
close(con)
close(con2)
file.remove(tf)
con <- url("https://www.r-project.org")
print(con)

## Marshal read-only connection, which records the
## current state, including the current file position.
con_ <- marshal(con)

## Unmarshal connection, which restores the state
## of the original connection
con2 <- unmarshal(con_)
print(con2)

stopifnot(all.equal(summary(con2), summary(con)))

bfr <- readChar(con, nchars = 100L)
print(bfr)
bfr2 <- readChar(con2, nchars = 100L)
print(bfr2)
stopifnot(identical(bfr2, bfr))

## Cleanup
close(con)
close(con2)

Marshalling of 'data.table' objects

Description

Marshalling of 'data.table' objects

Usage

## S3 method for class 'data.table'
marshal(x, ...)

## S3 method for class 'data.table'
marshallable(...)

Arguments

x

A data.table::data.table object.

...

Not used.

Value

A marshalled object as described in marshal().

Examples

if (requireNamespace("data.table", quietly = TRUE)) {
  dt <- data.table::data.table(a = 1:3, b = letters[1:3])

  ## Marshal
  dt_ <- marshal(dt)
  
  ## Unmarshal
  dt2 <- unmarshal(dt_)
  
  stopifnot(identical(dt2, dt))

  ## Marshal and unmarshal again
  dt2_ <- marshal(dt2)
  dt3 <- unmarshal(dt2_)
  
  stopifnot(
    identical(dt2_, dt_),
    identical(dt3, dt2)
  )
}

Marshalling of 'DBI:DBIConnection' objects (not supported)

Description

Warning: Objects of this class are not possible to marshal. If attempted, an error is produced.

Usage

## S3 method for class 'DBIConnection'
marshal(x, ...)

## S3 method for class 'DBIConnection'
marshallable(...)

Arguments

x

A DBI::DBIConnection object.

...

Not used.

Value

A marshalled object as described in marshal().


Marshalling of 'h2o' objects

Description

Marshalling of 'h2o' objects

Usage

## S3 method for class 'H2OAutoML'
marshal(x, ...)

## S3 method for class 'H2OMultinomialModel'
marshal(x, ...)

## S3 method for class 'H2OBinomialModel'
marshal(x, ...)

## S3 method for class 'H2ORegressionModel'
marshal(x, ...)

## S3 method for class 'H2OAutoML'
marshallable(...)

## S3 method for class 'H2OMultinomialModel'
marshallable(...)

## S3 method for class 'H2OBinomialModel'
marshallable(...)

## S3 method for class 'H2ORegressionModel'
marshallable(...)

Arguments

x

An "h2o" object.

...

Not used.

Details

h2o::h2o.save_mojo() and h2o::h2o.saveModel() are used to produce a marshalled version of the original object. h2o::h2o.import_mojo() and h2o::h2o.loadModel() are used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().


Marshalling of 'rJava' objects (not supported)

Description

Warning: Objects of this class are not possible to marshal. If attempted, an error is produced.

Usage

## S3 method for class 'jclassName'
marshal(x, ...)

## S3 method for class 'jobjRef'
marshal(x, ...)

## S3 method for class 'jclassName'
marshallable(...)

## S3 method for class 'jobjRef'
marshallable(...)

Arguments

x

"rJava" object.

...

Not used.

Value

A marshalled object as described in marshal().


Marshalling of 'keras' objects

Description

Marshalling of 'keras' objects

Usage

## S3 method for class 'keras.engine.base_layer.Layer'
marshal(model, ...)

## S3 method for class 'keras.engine.base_layer.Layer'
marshallable(...)

Arguments

model

A keras:keras.engine.base_layer.Layer object.

...

Not used.

Details

keras::serialize_model() is used to produce a marshalled version of the original object. keras::unserialize_model() is used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().

Examples

## Run only in interactive mode, because example takes > 5 seconds,
## which is longer than what is allowed on CRAN
if (interactive() && requireNamespace("keras", quietly = TRUE)) {
  library(keras)
  
  ## Create a keras model (adopted from {keras} vignette)
  inputs <- layer_input(shape = shape(32))
  outputs <- layer_dense(inputs, units = 1L)
  model <- keras_model(inputs, outputs)
  model <- compile(model, optimizer = "adam", loss = "mean_squared_error")
  print(model)

  ## Not needed anymore
  rm(list = c("inputs", "outputs"))

  ## Marshal
  model_ <- marshal(model)
  
  ## Unmarshal
  model2 <- unmarshal(model_)
  
  stopifnot(
    identical(summary(model2), summary(model))
  )


  ## Fitted keras model (adopted from {keras} vignette)
  test_input <- array(runif(128 * 32), dim = c(128, 32))
  test_target <- array(runif(128), dim = c(128, 1))
  hist <- fit(model, test_input, test_target)
  print(hist)
  print(model)
  
  ## Not needed anymore
  rm(list = "test_target")
  
  ## Marshal
  model_ <- marshal(model)
  
  ## Unmarshal
  model2 <- unmarshal(model_)
  
  stopifnot(
    identical(summary(model2), summary(model)),
    identical(stats::predict(model2, test_input), stats::predict(model, test_input))
  )
}

Marshalling of 'torch' objects

Description

Marshalling of 'torch' objects

Usage

## S3 method for class 'luz_module_fitted'
marshal(model, ...)

## S3 method for class 'luz_module_fitted'
marshallable(...)

Arguments

model

A luz_module_fitted object.

...

Not used.

Details

luz::luz_save() is used to produce a marshalled version of the original object. luz::luz_load() is used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().


Marshalling of 'magick' objects (not supported)

Description

Warning: Objects of this class are not possible to marshal. If attempted, an error is produced.

Usage

## S3 method for class ''magick-image''
marshal(x, ...)

## S3 method for class ''magick-image''
marshallable(...)

Arguments

x

A magick:magick-image object.

...

Not used.

Value

A marshalled object as described in marshal().


Marshalling of 'parsnip' objects

Description

Marshalling of 'parsnip' objects

Usage

## S3 method for class 'model_fit'
marshal(x, ...)

## S3 method for class 'model_fit'
marshallable(...)

Arguments

x

A parnsip:model_fit object.

...

Not used.

Details

The fit element of the model_fit object is marshalled.

Value

A marshalled object as described in marshal().

Examples

if (requireNamespace("parsnip", quietly = TRUE) && requireNamespace("xgboost", quietly = TRUE)) {
  library(parsnip)
  
  ## Adopted from example("boost_tree", package = "parsnip")
  model <- boost_tree(mode = "classification", trees = 20L, engine = "xgboost")  
  model <- set_mode(model, "regression")
  fit <- fit(model, mpg ~ ., data = datasets::mtcars)
  
  ## Marshal
  fit_ <- marshal(fit)

  ## Unmarshal
  fit2 <- unmarshal(fit_)

  ## Marshal again
  fit2_ <- marshal(fit2)

  ## Assert identity
  stopifnot(
    all.equal(fit2_, fit_)
  )
  
  fit3 <- unmarshal(fit2_)

  ## Assert identity
  stopifnot(
    all.equal(fit3, fit2)
  )
}

Marshalling of 'ncdf4' objects

Description

Marshalling of 'ncdf4' objects

Usage

## S3 method for class 'ncdf4'
marshal(x, ...)

## S3 method for class 'ncdf4'
marshallable(...)

Arguments

x

A ncdf4:ncdf4 object.

...

Not used.

Details

base::readBin() is used to produce a marshalled version of the original object. base::writeBin() and ncdf4::nc_open() are used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().

Examples

if (requireNamespace("ncdf4", quietly = TRUE)) {
  library(ncdf4)
  
  ## Create a temporary NCDF file
  tf <- tempfile(fileext = ".nc")
  
  x <- ncvar_def("x", units = "count", dim = list())
  nf <- nc_create(tf, x)
  ncvar_put(nf, x, 42)
  print(nf)
  nc_close(nf)
  rm(list = c("nf", "x"))

  ## Open NCDF file
  nf <- nc_open(tf)
  y <- ncvar_get(nf, "x")

  ## Marshal
  nf_ <- marshal(nf)

  ## Unmarshal
  nf2 <- unmarshal(nf_)

  y2 <- ncvar_get(nf2, "x")
  stopifnot(identical(y2, y))
}

Marshalling of 'reticulate' objects (not supported)

Description

Warning: Objects of this class are not possible to marshal. If attempted, an error is produced.

Usage

## S3 method for class 'python.builtin.object'
marshal(x, ...)

## S3 method for class 'python.builtin.object'
marshallable(...)

Arguments

x

"reticulate" object.

...

Not used.

Value

A marshalled object as described in marshal().


Marshalling of 'raster' objects

Description

Marshalling of 'raster' objects

Usage

## S3 method for class 'RasterLayer'
marshal(x, ...)

## S3 method for class 'RasterLayer'
marshallable(x, ...)

Arguments

x

A raster:RasterLayer object.

...

Not used.

Details

raster::writeRaster() is used to produce a marshalled version of the original object. raster::raster() is used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().

WARNING - Numerical identical results cannot be guaranteed

Marshalling of RasterLayer objects is leaky. More precisely, the method cannot guarantee that multiple rounds of marshalling and unmarshalling produce numerically identical objects.

Examples

if (requireNamespace("raster", quietly = TRUE)) {
  r <- raster::raster(system.file("external/test.grd", package = "raster"))
  print(r)
  
  ## Marshal
  r_ <- marshal(r)

  ## Unmarshal
  r2 <- unmarshal(r_)
  print(r2)
}

Marshalling of 'parallel' objects (not supported)

Description

Warning: Objects of this class are not possible to marshal. If attempted, an error is produced.

Usage

## S3 method for class 'SOCKcluster'
marshal(x, ...)

## S3 method for class 'SOCKnode'
marshal(x, ...)

## S3 method for class 'SOCK0node'
marshal(x, ...)

## S3 method for class 'SOCKcluster'
marshallable(...)

## S3 method for class 'SOCKnode'
marshallable(...)

## S3 method for class 'SOCK0node'
marshallable(...)

Arguments

x

A magick:magick-image object.

...

Not used.

Value

A marshalled object as described in marshal().


Marshalling of 'terra' objects

Description

Marshalling of 'terra' objects

Usage

## S3 method for class 'SpatVector'
marshal(terra, ...)

## S3 method for class 'SpatVector'
marshallable(...)

Arguments

terra

An terra::SpatVector.

...

Not used.

Details

terra::wrap() is used to produce a marshalled version of the original object. terra::vect() is used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().

Examples

if (requireNamespace("terra", quietly = TRUE)) {
  file <- system.file("ex/lux.shp", package = "terra")
  v <- terra::vect(file)
  
  ## Marshal SpatVector object
  v_ <- marshal(v)
  
  ## Unmarshal SpatVector object
  v2 <- unmarshal(v_)
  
  stopifnot(all.equal(v2, v, check.attributes = FALSE))
}

Marshalling of 'rstan' objects

Description

Marshalling of 'rstan' objects

Usage

## S3 method for class 'stanfit'
marshal(x, ...)

## S3 method for class 'stanfit'
marshallable(...)

Arguments

x

A rstan:stanfit object.

...

Not used.

Value

A marshalled object as described in marshal().


Marshalling of 'sparklyr' objects (not supported)

Description

Warning: Objects of this class are not possible to marshal. If attempted, an error is produced.

Usage

## S3 method for class 'tbl_spark'
marshal(x, ...)

## S3 method for class 'tbl_spark'
marshallable(...)

Arguments

x

"sparklyr" object.

...

Not used.

Value

A marshalled object as described in marshal().


Marshalling of 'caret:train' objects

Description

Marshalling of 'caret:train' objects

Usage

## S3 method for class 'train'
marshal(train, ...)

## S3 method for class 'train'
marshallable(...)

Arguments

train

A caret::train object.

...

Not used.

Details

bundle::bundle() is used to produce a marshalled version of the original object. bundle::unbundle() is used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().


Marshalling of 'udpipe' objects (not supported)

Description

Warning: Objects of this class are not possible to marshal. If attempted, an error is produced.

Usage

## S3 method for class 'udpipe_model'
marshal(x, ...)

## S3 method for class 'udpipe_model'
marshallable(...)

Arguments

x

"udpipe" object.

...

Not used.

Value

A marshalled object as described in marshal().


Marshalling of 'xgboost' objects

Description

Marshalling of 'xgboost' objects

Usage

## S3 method for class 'xgb.DMatrix'
marshal(x, ...)

## S3 method for class 'xgb.Booster'
marshal(x, ...)

## S3 method for class 'xgb.Booster'
marshallable(...)

## S3 method for class 'xgb.DMatrix'
marshallable(...)

Arguments

x

An xgboost::xgb.DMatrix or an xgboost:xgb.Booster object.

...

Not used.

Details

xgboost::xgb.DMatrix.save() is used to produce a marshalled version of the original object. xgboost::xgb.DMatrix() is used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().


Marshalling of 'xml2' objects

Description

Marshalling of 'xml2' objects

Usage

## S3 method for class 'xml_document'
marshal(xml2, ...)

## S3 method for class 'xml_nodeset'
marshal(xml2, ...)

## S3 method for class 'xml_document'
marshallable(...)

## S3 method for class 'xml_nodeset'
marshallable(...)

Arguments

xml2

A xml2::xml_document or similar.

...

Not used.

Details

xml2::xml_serialize() is used to produce a marshalled version of the original object. xml2::xml_unserialize() is used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().

Examples

if (requireNamespace("xml2", quietly = TRUE)) {
  doc <- xml2::read_xml("<body></body>")

  ## Marshal 'xml_document' object
  doc_ <- marshal(doc)

  ## Unmarshal 'xml_document' object
  doc2 <- unmarshal(doc_)

  stopifnot(all.equal(doc2, doc))
}

Marshalling of 'XML' objects

Description

Marshalling of 'XML' objects

Usage

## S3 method for class 'XMLAbstractNode'
marshal(xml, ...)

## S3 method for class 'XMLAbstractDocument'
marshal(xml, ...)

## S3 method for class 'XMLAbstractDocument'
marshallable(...)

## S3 method for class 'XMLAbstractNode'
marshallable(...)

Arguments

xml

An XML::XMLAbstractNode or XML::XMLAbstractDocument.

...

Not used.

Details

XML::xmlSerializeHook() is used to produce a marshalled version of the original object. XML::xmlDeserializeHook() is used to reconstruct a version of the original object from the marshalled object.

Value

A marshalled object as described in marshal().

Examples

if (requireNamespace("XML", quietly = TRUE)) {
  node <- XML::xmlParseString("<a><b>c</b></a>")
  print(node)
  
  ## Marshal XMLAbstractNode object
  node_ <- marshal(node)
  ## Unmarshal XMLAbstractNode object
  node2 <- unmarshal(node_)
  print(node2)
  
  stopifnot(all.equal(node2, node))
  
  
  doc <- XML::xmlParse(system.file("exampleData", "tides.xml", package = "XML"))
  
  ## Marshal XMLAbstractDocument object
  doc_ <- marshal(doc)
  ## Unmarshal XMLAbstractDocument object
  doc2 <- unmarshal(doc_)
  
  stopifnot(all.equal(doc2, doc))
}

Check if R object can be marshalled

Description

Check if R object can be marshalled

Usage

marshallable(...)

Arguments

...

The object to be checked, followed by additional arguments passed to the specific S3 method.

Value

TRUE if the object can be marshalled, FALSE if it cannot be marshalled, and NA if it is not known whether the object can be marshalled.