Import and download data

Select before importing

Inspect datasets, file groups, and variables first. This avoids transferring unneeded data and gives you the stable ids and source names used by the read selectors.

glc_datasets(iztech)
glc_files(iztech, dataset_id = iztech_dataset)
glc_variables(iztech, dataset_id = iztech_dataset)

dataset_id is required by glc_read(). Use dataset_id = "all" only when you deliberately want every locally available dataset. A dataset can contain questionnaires, diaries, and several sensor streams that should be processed separately, so selecting an entire dataset does not imply that all its file groups can be collected into one table.

Narrow an import with any combination of file-group ids or indices, file paths or basenames, source variable names, and semantic terms. Here, the file inventory identifies the S001 chest-sensor file:

file_inventory <- glc_files(
  iztech,
  dataset_id = iztech_dataset
)
sensor_file <- file_inventory[
  file_inventory$file_group_id == iztech_chest_light,
]
sensor_file[, c(
  "file_group_id", "path", "device_id", "expected_bytes"
)]

source_file <- basename(sensor_file$path[[1]])
source_file

Read only its declared melanopic EDI variable and parse at most 10,000 rows:

light_collection <- glc_read(
  iztech,
  dataset_id = iztech_dataset,
  file_group = iztech_chest_light,
  files = source_file,
  variables = "MEDI",
  n_max = 10000
)
light_collection

When variables or terms are selected, the imported tables contain only the matching source variables. glcdp still uses required date or time columns internally to construct .glc_datetime, but does not retain them unless the filters select them. Set primary_only = TRUE to select declared primary variables:

primary_light <- glc_read(
  iztech,
  dataset_id = iztech_dataset,
  file_group = iztech_chest_light,
  primary_only = TRUE,
  n_max = 10000
)
primary_data <- primary_light$data[[1]]
primary_data[stats::complete.cases(primary_data), ]

Interactive calls show progress across the selected files by default. Use progress = FALSE to suppress the indicator when a script or application provides its own progress reporting. n_max limits rows parsed after a remote file is available; it does not reduce the whole-file transfer.

Import schema-defined R types and factor levels

Schema 3.0.2 declares a type for each source column. Factor declarations also contain their levels in schema-declared order. These declarations are exposed by the variable inventory:

demographic_variables <- glc_variables(
  iztech,
  file_group = iztech_demographics
)
demographic_variables[, c("name", "type", "factor_values")]

glc_read() uses those declarations rather than guessing from file contents. The IZTECH demographics file therefore yields numeric, logical, and factor columns with the declared levels:

demographics <- glc_read(
  iztech,
  dataset_id = iztech_dataset,
  file_group = iztech_demographics
)
demographic_data <- demographics$data[[1]]

levels(demographic_data$sex)
levels(demographic_data$employment_status)

The same import metadata controls headers, datetime construction, decimal marks, encodings, and time zones. By default, undeclared extra columns, values that cannot be parsed to the declared type, and values outside declared factor levels are errors. During exploratory work, use problems = "warn" to retain problematic data with warnings; inspect the result before analysis.

exploratory_demographics <- glc_read(
  iztech,
  dataset_id = iztech_dataset,
  file_group = iztech_demographics,
  problems = "warn"
)

Understand the imported collection

A glc_data_collection has one row per file group. Descriptive columns record the dataset, participant, device, modality, role, data state, time zone, datetime specification, and source files. The imported table is in the data list-column.

light_collection[, setdiff(names(light_collection), "data")]
names(light_collection$data[[1]])

Each imported table also contains .glc_* provenance columns such as the dataset id, participant id, source file, and constructed datetime. These make row origins explicit after tables are combined.

Collect analysis-ready data

Combine file groups only when their structures and meanings are compatible:

light_data <- glc_collect(light_collection)
head(light_data[!is.na(light_data$MEDI), ])

The default standardize = "lightlogr" follows the data conventions used by LightLogR and adds:

• Id, derived from the dataset id;
• file_group_id, the stable dataset file-group id used by the metadata helpers;
• participant_Id, derived from the participant id;
• Datetime, constructed from the metadata-defined datetime specification; and
• file.name, retained from a declared source column when present or derived from the package path otherwise.

The standardized result contains no internal .glc_* provenance columns. It is ordered by Id and Datetime and grouped by Id. Use standardize = "none" if you want an ungrouped tibble whose source and provenance columns are left unchanged.

source_data <- glc_collect(
  light_collection,
  standardize = "none"
)
head(source_data)

glc_collect() refuses to combine groups that differ in columns, types, time zones, modalities, roles, data states, or datetime specifications. It also rejects contradictory file-group relationships and multiple device links within one dataset. Keep those groups separate or select a compatible subset with glc_read().

Collected data can be used directly with LightLogR’s data-quality and insight functions, visualization guide, and metrics guide.

Extract or add metadata

Use extract_metadata() when you want one concise row per imported file group, and add_metadata() when the same fields should be available on every observation. Both functions require an explicit metadata source and field selection. The default key is file_group_id, which glc_collect() adds to standardized data. If the input is grouped, the extracted tibble retains that grouping and includes its grouping columns before file_group_id.

analysis_metadata <- tibble::tibble(
  file_group_id = unique(as.character(light_data$file_group_id)),
  analysis_set = "chest sensor"
)

metadata_summary <- extract_metadata(
  light_data,
  analysis_metadata,
  fields = "analysis_set"
)
metadata_summary

enriched_data <- add_metadata(
  light_data,
  analysis_metadata,
  fields = "analysis_set"
)

enriched_data |>
  head() |>
  dplyr::select(-file.name)

The metadata source can also be a local CSV/TSV path or the package handle. For a package handle, glcdp follows each file group to its dataset and device, and follows the dataset to its participant and study. It can therefore assemble fields such as participant_age without manually naming the participant resource. Dataset-, participant-, and study-level values repeat across file groups. Use by = "Id" when you explicitly want one row per dataset instead. Field and relationship resolution must remain unambiguous; use resource to restrict field discovery when the same field occurs in multiple connected resources.

glc_search_metadata(
  iztech,
  "participant_age",
  resources = "participants",
  search_in = "fields"
)

dataset_metadata <- extract_metadata(
  light_data,
  iztech,
  fields = c(
    "dataset_timezone",
    "dataset_location",
    "participant_age",
    "study_title",
    "device_model"
  )
)
dataset_metadata

add_metadata(
  light_data,
  iztech,
  fields = c(
    "dataset_timezone",
    "dataset_location",
    "participant_age",
    "study_title",
    "device_model"
  )
) |>
  head() |>
  dplyr::select(
    Id,
    dataset_timezone,
    dataset_location,
    participant_age,
    study_title,
    device_model
  )

When adding project-specific metadata to a data package, store it under a stable package-relative path such as data/metadata.csv and declare it as a resource in datapackage.json. The functions never guess from the working directory or neighboring files. They error when no identifiers or fields match, and warn while retaining useful results for partial matches.

Download a reproducible subset

glc_download() creates a persistent directory while preserving package-relative paths. Its safe default downloads only the descriptor, core metadata, and required schemas:

For public packages, ordinary Git files are transferred from immutable raw URLs at the selected commit, avoiding per-file GitHub API requests. A token supplied to glc_open() continues to use the authenticated API transport, including for private repositories.

metadata_dir <- tempfile("iztech-metadata-")
glc_download(iztech, metadata_dir)

Request data explicitly and apply the same selectors used during inspection. This compact example downloads the S001 demographics group:

data_dir <- tempfile("iztech-s001-demographics-")
downloads <- glc_download(
  iztech,
  data_dir,
  include = "data",
  dataset_id = iztech_dataset,
  file_group = iztech_demographics
)
downloads

Use include = "all" only when you intend to mirror every declared resource. The resources and files arguments can further narrow a download. Existing files are protected unless overwrite = TRUE is set explicitly.

Every download writes glcdp-manifest.json, recording the source repository, exact commit, registry verification state, schema version, selection, hashes, storage types, sizes, and Git LFS object ids. Reopen the directory to use the same inspection and import API without fetching the package again:

local <- glc_open(data_dir)
glc_summary(local)
glc_files(local, dataset_id = iztech_dataset, available = TRUE)

local_collection <- glc_read(
  local,
  dataset_id = iztech_dataset,
  file_group = iztech_demographics
)
local_data <- glc_collect(local_collection)
local_data

The descriptor and core metadata retain the records declared by the source package. For a local subset, glc_summary() distinguishes locally available datasets, file groups, and files from those declared records. When the package is incomplete, glc_read() also reports how many declared datasets and files are locally available, then skips absent files. Thus, dataset_id = "all" reads all data included in the subset. Use glc_files(local, available = FALSE) to inspect omitted file records.

Temporary reads versus persistent storage

Remote glc_read() calls use session-temporary storage by default. This is a good fit for one-off analysis and leaves no persistent files behind.

Pass cache_dir to glc_open() when you want remote files reused across calls, or use glc_download() when you want an explicit, portable package subset with a manifest:

cached <- glc_open(
  iztech_repository,
  cache_dir = file.path(tempdir(), "glcdp-iztech-cache")
)

Choose the cache for performance; choose a downloaded subset for a durable, inspectable analysis input.