read_rcdf() checks each Parquet file against the md5
checksum stored in the archive and stops if it differs. This detects
corruption, not deliberate tampering. Files the checksum doesn’t list
are skipped, so older archives still read.read_rcdf(ignore_duplicates = TRUE) now warns about
duplicate primary keys, as documented (it previously did nothing).read_rcdf() checks primary keys stored in every archive
it reads, not only the last one.merge_rcdf() keeps summary_statistics and
the earliest source created_at; the merge time is recorded
as updated_at.write_rcdf() accepts metadata$created_at
to keep an original creation time; the write time is then stored as
updated_at.merge_rcdf() now returns
invisible(list(path, password)) instead of
NULL. When it generates a new key pair, the private-key
password is printed only in interactive sessions, so it no longer lands
in job logs; scripts should read it from the return value.write_rcdf_csv() and write_rcdf_tsv() no
longer write a row-name column. TSV headers now line up with the data in
non-R tools.merge_rcdf() /
read_rcdf(return_meta = TRUE): metadata from several
archives is combined without dropping values. Data frames are stacked,
named lists merge key by key, and plain values are combined (repeats
removed), so a value that differs between files now becomes a vector
instead of keeping only the last file’s value.rcdf.options option set at package load has
been removed.Measured on 2.5M rows across two records (2M-row person table), with a dictionary and primary keys:
write_rcdf(): about 4x faster (3.4s -> 0.8-0.95s)
and archives about 45% smaller (39.8 MB -> 21.9 MB). Archives are now
stored without zip compression (the encrypted data does not compress,
and deflate made zipping about 10x slower), encrypted Parquet now uses
zstd, and the write-time duplicate key check runs in DuckDB instead of
pasting keys together in R.write_parquet(): compression now applies
to encrypted files too (default "zstd").read_rcdf(): normal (non-lazy) reads go straight from
Parquet into R through DuckDB views instead of copying into a DuckDB
table first, lowering peak memory by about 10%. Read time is unchanged;
this was measured with primary keys stored, so it includes the read-time
duplicate check, which reads only the key columns.read_rcdf(lazy = TRUE) no longer fails on archives that
store a data dictionary.read_rcdf() combines a record from several files by
column name. Before, columns were matched by position, so files with a
different column order put values under the wrong columns.read_rcdf() / merge_rcdf(): area names,
data dictionaries and metadata data frames whose columns differ between
files are combined instead of failing.add_metadata() (and so read_rcdf() with a
dictionary): code columns are converted to integer only when every value
is a plain integer. Before, codes like "012" lost their
leading zero, values like "9X" became NA, and
decimals were truncated.write_rcdf_xlsx(as_single_file = TRUE): record names
longer than 31 characters are shortened to unique sheet names instead of
failing.write_parquet() / read_parquet(): 48- and
64-character hex keys now work as documented.read_rcdf(): records with zero rows are kept (they were
dropped unless lazy = TRUE), so merge_rcdf()
no longer loses them.read_dot_env(): spaces around = and
trailing spaces are ignored, so PASSWORD = 1234 reads as
PASSWORD = "1234".read_rcdf(): legacy archives whose metadata has
key_app but no key_admin are read correctly
again (R’s partial $ matching picked up
key_app as the current-format key).primary_key can name a record as "d" or
"d.parquet". Both forms are now checked at write time and
saved into the archive for read-time checks; before, each form silently
skipped one of the two.write_rcdf(): stages files in a private temp folder, so
leftover Parquet files from an earlier failed write are no longer packed
into a new archive.write_rcdf(): a data dictionary passed as a file path
(metadata$dictionary) is no longer silently dropped.read_rcdf(): an RSA private key object (from
openssl::read_key()) now works as
decryption_key; unsupported key types give a clear
error.read_rcdf(ignore_duplicates = FALSE): now actually
checks primary keys, including duplicates across several files.read_rcdf(): errors when the number of keys or
passwords matches neither 1 nor the number of files.read_rcdf(): no longer uses %||%, which
needs R >= 4.4.write_parquet(): the encryption key and path are now
quoted safely in SQL.write_rcdf_sqlite(): the connection is closed even if a
write fails.write_rcdf_csv() etc.) skip the
__data_dictionary element, as
write_rcdf_parquet() already did.write_rcdf_parquet() returns only the files it
wrote.get_attr(): dotted paths work at any depth
("meta.source.note"); before, anything past the second
level was ignored.generate_pw(): removed modulo bias, so every character
in the pool is equally likely per draw.read_rcdf(lazy = TRUE): docs now show how to close the
DuckDB connection, which holds the decrypted data until closed.write_parquet(): docs no longer claim arrow writes
unencrypted files or that ... is passed to it.write_rcdf(): documents the recognised
metadata fields.generate_pw(): switched from Mersenne Twister to a
cryptographically secure RNG (openssl::rand_bytes).write_parquet(): DuckDB errors no longer leak raw
encryption key material in messages.write_rcdf_parquet(): plaintext temp table is now
always removed after a shared-connection write.write_rcdf_as(): replaced eval(parse())
dispatch with a safe named-list of function references.hex_to_raw(): added strict input validation (type,
length, character set).write_parquet() / write_rcdf_parquet():
removed httpfs overhead; shared DuckDB connection across
writes; default compression changed to zstd.read_rcdf(): new lazy = TRUE option for
DuckDB-backed lazy reads; n_threads for parallel I/O;
duplicate PRAGMA calls deduplicated per connection.add_metadata(): label assignment vectorised with batch
update.open_duckdb_connection(): no longer forces
single-threaded mode.collect.rcdf_tbl_db(): fixed infinite-dispatch
recursion; NULL metadata no longer causes an error.read_parquet(): unencrypted early-return path now
correctly applies metadata and
as_arrow_table.normalize_key_value(): hex keys no longer misidentified
as base-64.encrypt_value() and decrypt_value()
functions, respectively, to encrypt and decrypt string using RSA..env) reader from
read_env() to read_dot_env().read_rcdf() and get_rcdf_metadata().get_attr() to retrieve attribute/s from RCDF
object.read_parquet_tbl() to read encrypted Parquet file
and return as lazy data from DuckDB connection.pre_collect argument in
read_rcdf()merge_rcdf() function to combine multiple RCDF
files with different encryption keys into one.read_rcdf() to allow reading of multiple RCDF
file with different encryption keys.pre_collect argument to read_rcdf()
to allow lazy loading the RCDF object.NEWS.md file to track changes to the
package.