R packages are an effective way to distribute research data. A package carries a version, documentation for each dataset, and a defined installation location, and R already provides mature ways to reach the contents: data(), lazy loading of package data, and namespace-qualified access such as pkg::object. For a single data package these tools are usually all that is needed, and projectData is designed to work alongside them, not to replace them.
Difficulties tend to appear when an analysis depends on several independently distributed collections. Data may be divided between a primary project package and supplementary or reference packages, and different collections may define objects with the same name. Attaching packages or calling data() for each one leaves the choice of source to attachment order or to repeated manual qualification, which is hard to keep consistent across a project. Some resources are also large or stored in formats that ordinary deserialization cannot restore, such as disk-backed matrices, so they need a format-specific restoration step and should stay unopened until they are used. Finally, it is often useful to ask what a project has available, and which installed package version and library each object came from, without reading any data. This matters most when several versions or libraries are present. These are workflow and coordination problems. They do not reflect a limitation of base R, and many projects never meet them.
projectData is a small coordination layer over installed data packages. Calling data_attach() on a package registers it as a named collection and records an inventory of its resources, along with the package name, version, and the installation root that was resolved. Registering reads package metadata only; no data values are read. Objects are then read on first access, and data_get() can return an object from a named collection or package without attaching anything. When collections share a name, projectData does not pick a winner by attachment order. You can designate a primary collection, state a preference for individual objects, or add suffix aliases. An unresolved conflict produces an error that lists the candidates. Attachment is optional and happens only when you request it, so objects can also be reached without adding names to the search path. data_catalog() and data_info() report provenance from the recorded metadata alone. Ordinary data packages need no changes to work this way. An optional, validated manifest can describe special files or object mappings, and a versioned loader interface lets an adapter supply the discovery and opening steps for a specialized storage format.
This division keeps the core package light. The core handles registration, resolution, caching, and provenance, and it reads ordinary datasets and explicitly described RDS files. Format-specific code lives in separate adapter packages that you choose to install. For example, the separate projectData.monocle3 package restores Monocle 3 cell_data_set objects backed by BPCells matrices, so that projectData does not depend on monocle3 or BPCells. See the adapter vignette for how adapters work. Missing or unavailable adapters affect only the resources that need them.
projectData is not a package manager. It never installs or updates data packages, and it does not replace distribution through CRAN, Bioconductor, Zenodo, or similar channels. It cannot make compatible software versions or external backing files available, and it does not by itself make an analysis reproducible. It is also not a data-processing framework. If data() and namespace-qualified access already meet your needs, you can keep using them. projectData is intended for projects that need consistent access to several collections, deferred opening of large or specialized resources, and a record of where each object came from.
The normal workflow has three steps:
Need another data package later? Attach it too. If the packages define the same name, projectData asks you to decide which one you mean (see Conflicts).
Installing is ordinary R package installation and is not done by projectData. A data package distributed as a tarball (for example from Zenodo) is installed like any source package:
install.packages("path/to/rnaseqData_1.0.0.tar.gz", repos = NULL, type = "source")To keep this vignette offline, the next chunk builds two tiny throwaway data packages in a temporary library. In real use, skip this chunk and omit the lib.loc argument below.
library(projectData)
source(system.file("examples", "fixtures.R", package = "projectData"))
lib <- fixture_library()
fixture_package(lib, "rnaseqData", "1.0.0", lazy = TRUE, datasets = list(
counts_rna = list(counts_rna = matrix(1:6, 2)),
meta_rna = list(meta_rna = data.frame(sample = c("a", "b")))
))
fixture_package(lib, "proteinData", "2.1.0", lazy = FALSE, datasets = list(
counts_prot = list(counts_prot = matrix(7:12, 2)),
meta_prot = list(meta_prot = data.frame(sample = c("a", "b")))
))data_attach() takes the name of an installed data package. It records what the package contains and puts its objects on the search path under their own names. This reads package metadata only: no data values are read, and nothing is installed. Nothing is attached unless you call this function.
data_attach("rnaseqData", lib.loc = lib)Use the objects by their ordinary names. The first use of an object reads it.
data_config(messages = TRUE)
counts_rna
#> projectData: opening resource 'counts_rna' of collection 'rnaseqData' (rnaseqData 1.0.0) with object(s) counts_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 1 3 5
#> [2,] 2 4 6The message names the resource and says it is opened from the installed package; projectData never installs packages. The same code works in a script after the data_attach() call.
Attach another package whenever you need it. Objects already in use are not affected.
data_attach("proteinData", lib.loc = lib)
counts_prot
#> projectData: opening resource 'counts_prot' of collection 'proteinData' (proteinData 2.1.0) with object(s) counts_prot [109 bytes on disk: data/counts_prot.rda]. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 7 9 11
#> [2,] 8 10 12These read only recorded metadata. They never open any data.
data_catalog()[, c("collection", "exposed_name", "resource", "state")]
#> <projectData catalog>: 4 row(s); metadata only, no data opened.
#> collection exposed_name resource state
#> rnaseqData counts_rna counts_rna ready
#> rnaseqData meta_rna meta_rna unopened
#> proteinData counts_prot counts_prot ready
#> proteinData meta_prot meta_prot unopenedstate is unopened until the object has been used, then ready. The resource column is the loadable unit, which can differ from the object name. A resource that produces several objects is opened once for all of them.
data_info("counts_rna") returns the full provenance row for a name: package, version, installed location, source file, where it was discovered (origin), and whether its object names have been checked (mapping). data_help("counts_rna") opens the help page of the original topic in the source package. It is separate from inspection and never runs implicitly.
Bare names are convenient, but data_get() returns an object without relying on what is on the search path, so it is the more robust choice inside functions and scripts. (Attached bindings can also be evaluated by IDE completion or previews; see system.file("ide-repro", package = "projectData").)
data_get("meta_prot")
#> projectData: opening resource 'meta_prot' of collection 'proteinData' (proteinData 2.1.0) with object(s) meta_prot [145 bytes on disk: data/meta_prot.rda]. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> sample
#> 1 a
#> 2 b
data_get("meta_rna", collection = "rnaseqData")
#> projectData: opening resource 'meta_rna' of collection 'rnaseqData' (rnaseqData 1.0.0) with object(s) meta_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> sample
#> 1 a
#> 2 bGive several names to open several objects in one call. This is a good way to open large objects at the start of a batch job: the first-access messages appear together, before the rest of the analysis runs.
str(data_get(c("meta_rna", "meta_prot")))
#> List of 2
#> $ meta_rna :'data.frame': 2 obs. of 1 variable:
#> ..$ sample: chr [1:2] "a" "b"
#> $ meta_prot:'data.frame': 2 obs. of 1 variable:
#> ..$ sample: chr [1:2] "a" "b"Sometimes you want a single object and no new names on the search path, much as you would write package::object for an exported object. Give data_get() the package:
fixture_package(lib, "imagingData", "0.3.0", lazy = TRUE, datasets = list(
counts_img = list(counts_img = matrix(13:18, 2))
))
counts_img <- data_get("counts_img", package = "imagingData", lib.loc = lib)
#> projectData: opening resource 'counts_img' of collection 'imagingData' (imagingData 0.3.0) with object(s) counts_img. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
counts_img
#> [,1] [,2] [,3]
#> [1,] 13 15 17
#> [2,] 14 16 18
"imagingData" %in% data_collections()$package[data_collections()$attached]
#> [1] FALSEThere is nothing to register first. data_get() inventories the package if necessary (metadata only), then returns the object. It does not attach anything and does not change collections that are already attached. The name is looked up in that package itself, so preferences, primary collections and suffixes you set for attached access do not affect it.
data_get() still opens the object you ask for; it is not a metadata query. Use data_catalog() or data_query() to look without opening. If you attach the package later, the object is not read again.
Before each resource is read for the first time, projectData prints a message naming the collection, the objects, the size of the source file when it is a single file, and the loader. For large data this is the moment to notice what is about to load. Turn the messages off with data_config(messages = FALSE); the setting lasts for the session.
data_query() shows what a package contains, and which names would collide with the packages you list or with what is already attached, without attaching anything or remembering anything. Use it to look before you attach.
data_detach()
fixture_package(lib, "rnaseqBatch2", "1.0.0", lazy = TRUE, datasets = list(
counts_rna = list(counts_rna = matrix(101:106, 2)),
meta_rna = list(meta_rna = data.frame(sample = c("c", "d")))
))
data_query(c("rnaseqData", "rnaseqBatch2"), lib.loc = lib,
id = c("batch1", "batch2"))[, c("collection", "exposed_name", "binding")]
#> <projectData catalog>: 4 row(s); metadata only, no data opened.
#> collection exposed_name binding
#> batch1 counts_rna ambiguous
#> batch1 meta_rna ambiguous
#> batch2 counts_rna ambiguous
#> batch2 meta_rna ambiguous
nrow(data_collections())
#> [1] 0The query already shows the problem: counts_rna and meta_rna exist in both packages, so their binding is ambiguous. The query changed nothing: no collections are attached.
When two attached collections define the same name, projectData never resolves it by attachment order. Attaching still succeeds, and the catalog flags the conflict. Using the bare name fails with an error that lists the candidates. Use id to give each collection a name of your own:
data_attach(c("rnaseqData", "rnaseqBatch2"), lib.loc = lib, id = c("batch1", "batch2"))
counts_rna
#> Error: 'counts_rna' is ambiguous: it is provided by collections 'batch1', 'batch2'. Resolve with data_prefer("counts_rna", "batch1"), designate a primary collection with data_primary(), use a suffix alias if one was registered, or access it explicitly with data_get("counts_rna", collection = "batch1").There are three ways to resolve conflicts, and they can be combined.
Per-object preference. Choose the collection for a particular name. This is the right tool when an object from one package is always preferred over an object with the same name in another package.
data_prefer("counts_rna", "batch1")
counts_rna
#> projectData: opening resource 'counts_rna' of collection 'batch1' (rnaseqData 1.0.0) with object(s) counts_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 1 3 5
#> [2,] 2 4 6Other contested names stay unresolved until you decide them:
meta_rna
#> Error: 'meta_rna' is ambiguous: it is provided by collections 'batch1', 'batch2'. Resolve with data_prefer("meta_rna", "batch1"), designate a primary collection with data_primary(), use a suffix alias if one was registered, or access it explicitly with data_get("meta_rna", collection = "batch1").A primary collection. If one collection should win all its contested names, designate it. A per-object preference for a non-primary collection overrides the primary designation for that name. A primary collection can also be set when attaching: data_attach("rnaseqData", id = "batch1", primary = "batch1").
data_primary("batch1")
meta_rna
#> projectData: opening resource 'meta_rna' of collection 'batch1' (rnaseqData 1.0.0) with object(s) meta_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> sample
#> 1 a
#> 2 bSuffix aliases and qualified access. Qualified access, data_get(name, collection = ), is the most robust way to access objects within scripts or functions. It returns the named object and also reaches the copy that lost a conflict.
data_get("counts_rna", collection = "batch2")
#> projectData: opening resource 'counts_rna' of collection 'batch2' (rnaseqBatch2 1.0.0) with object(s) counts_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 101 103 105
#> [2,] 102 104 106A suffix exposes every object in a collection as an alias of the form <name><suffix>. Set it when you attach.
data_attach("rnaseqBatch2", lib.loc = lib, id = "batch2", suffix = ".batch2",
replace = TRUE)
counts_rna.batch2
#> projectData: opening resource 'counts_rna' of collection 'batch2' (rnaseqBatch2 1.0.0) with object(s) counts_rna. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [,1] [,2] [,3]
#> [1,] 101 103 105
#> [2,] 102 104 106Suffix aliases must be unique. A clash is reported when you attach, before anything is changed.
To clear decisions, use data_prefer("counts_rna", NULL) and data_primary(NULL).
A package that ships non-lazy data scripts (data/*.R) is opened by running the script, only on first access. The script sees the functions and data of the standard startup packages (stats, graphics, grDevices, utils, datasets, methods and base, in the order of an ordinary R session), so head() or model.matrix() work unqualified. That set is fixed; it does not follow your defaultPackages option or your search path. Your workspace and any other attached package are not visible, so qualify calls to other packages (otherpkg::fn()). This isolates names; it is not a security sandbox. See the “Script datasets” section of ?data_attach.
data_detach()data_detach() removes the attached collections and the environment projectData created, and forgets them. It does not delete files or modify objects you already hold, so an object kept in a variable can keep reading its backing files after detachment. That is not a promise about replacing or removing the installed package: a disk-backed object that references installed files may stop working then. See ?data_detach. data_detach("batch2") removes just one collection. Packages you only reached through data_get(package = ) are forgotten the same way (they were never on the search path); a later data_get() simply opens them again.
conflicted resolves conflicts between package exports and library(include.only = ) selects exports. They work with exported data but do not provide collection inventory, deferred access beyond lazy data, format-specific restoration, or provenance.