StatCan being unavailable no longer aborts with an error.
Timeouts, connection failures and error responses are now reported with
a loud warning and the function returns NULL, so that a
script or a document can decide for itself what to do when the servers
are down. This applies to every function that talks to StatCan, and it
also covers the two calls that previously bypassed the retry helper,
get_cansim_table_last_release_date() and
get_cansim_series_info_cube_coord(). Set
options(cansim.error_on_unavailable=TRUE) to get the
previous behaviour of raising an error
examples that make a single lightweight API call are now
\donttest{} rather than \dontrun{}, so they
are checked rather than merely displayed. Examples that download a full
table or the cube list stay \dontrun{} because of their run
time, and cansim_old_to_new() needs no network at all so
its example now always runs
data retrieved by vector or by table/coordinate now carries
UOM and UOM_ID columns, taken from the cube
metadata. StatCan flags a single dimension of each cube as carrying the
unit of measure and the unit varies by member of that dimension, so the
unit is resolved per coordinate. Tables that have no unit of measure,
for example census tables, get no unit columns, matching the full table
download (#170). With the unit known, percentage values retrieved by
vector or by coordinate are now normalized the same way as full table
downloads: val_norm carries the value divided by 100 and
the unit of measure is relabelled to Rate
(Taux in French). Previously the same series normalized
differently depending on whether it was retrieved as a full table or by
vector, so scripts that fetch percentage vectors will see
val_norm change by a factor of 100
non-breaking spaces and control characters in names returned by
StatCan are now replaced with regular spaces. These characters render as
an ordinary space or as nothing at all, so a column whose name contained
one could not be reached by typing or copy-pasting what the console
displayed. The repair covers table downloads, vector and coordinate
calls, cube metadata, table templates and the cube list, and emits a
warning that shows the offending characters by code point, for example
Performance<U+00A0> strategy, together with a count
of how many names were repaired. Set
options(cansim.suppress_repair_warnings=TRUE) to silence
the warning. The warning also says that the characters are in the data
StatCan publishes rather than anything the user did, and that it will
disappear on its own once StatCan stops sending them, pointing at the
issue where that is tracked. Column names of tables cached before this
release keep the original characters until the table is downloaded
again, get_cansim_connection() warns when it finds such a
cache (#169)
the same repair now also covers the member labels in the data
itself, not just the names of the columns holding them. These characters
turn out to be more common in member labels than in dimension names, 53
of 500 sampled tables carry at least one. Repairing only the metadata
side would have left the labels in the data unable to match their own
factor levels, so every row carrying an affected label would have become
NA. Labels are now also identical whichever way the data is
retrieved, so a table can be joined to template, vector or coordinate
data on its dimension columns (#169)
an internal scan_statcan_character_problems() reads
the cube metadata straight off the API, without the repair applied, and
reports every title, dimension name and member name StatCan publishes
with a non-breaking space or a control character in it, by table, level
and language. summarize_statcan_character_problems()
aggregates that by survey. Neither is exported, they exist to track
whether the upstream problem is shrinking, and both go away along with
the repair once it has (#169)
cached tables now record the package version they were parsed
under alongside the download timestamp, in a single
.Rda_info file that replaces the .Rda_time
file the timestamp used to have to itself. The timestamp says whether
StatCan has newer data, the version says whether this release still
reads those files the same way. list_cansim_cached_tables()
reports it in a new cansimVersion column, empty for
anything cached before this release. The old timestamp file is still
read, so existing caches keep their download date, and is replaced when
a table is refreshed. get_cansim_connection() uses the
version to check whether a cache predates the repair of non-breaking
spaces and control characters, and if so reads the metadata cached
alongside the table to see whether its dimension names or member labels
actually carry any. Only then does it warn, naming the offending label
and pointing at refresh=TRUE (#169)
every call that sends StatCan a list of vectors, coordinates or
tables is now split into batches of at most 300 items. StatCan refuses a
longer list outright with an HTTP 416, which had gone unnoticed because
most of these calls already batched.
get_cansim_vector_info() and the cube metadata download did
not, so asking either for more than 300 items at a time failed rather
than returning data
a vector or coordinate StatCan cannot answer for is no longer
passed off as data. StatCan signals a bad item two different ways
depending on the method, either marking the record FAILED
or answering SUCCESS and putting the reason in
responseStatusCode, and the package only checked the first.
That let an invalid vector through get_cansim_vector_info()
as a row of NAs indistinguishable from real metadata. Both
are now checked everywhere, and the items that carry no data are dropped
and reported by reason, naming the vectors or coordinates
concerned
vector calls that come back with nothing now warn and return an
empty table. Previously the empty answer travelled on to the metadata
join and surfaced there as
Column 'cansimTableNumber' doesn't exist, which said
nothing about what had happened. The warning names all three things that
produce it: vectors that do not exist, vectors with no data in the
requested time frame, and the daily window from midnight to 8:30am
Eastern in which StatCan does not serve vector data
when StatCan refuses a request it explains why in the response body, and that explanation is now shown alongside the status code instead of being discarded. An HTTP 409 says whether the product is simply not released yet, and an HTTP 416 names the limit the request went past. Those two status codes also got the plain-language translation the other codes already had. The HTTP 504 message now says that StatCan builds a whole response before sending any of it, so the way past a gateway timeout is to ask for less at once rather than to retry the same request
three new functions expose StatCan’s changed series methods,
which report what changed at a finer grain than
get_cansim_changed_tables() does.
get_cansim_changed_series_list() lists the series StatCan
changed today as vectors, with the table and coordinate each belongs to,
and get_cansim_changed_series_data_for_vectors() and
get_cansim_changed_series_data_for_coordinates() retrieve
the changed data points themselves, in the same shape and with the same
metadata as the corresponding get_cansim_vector() and
coordinate calls. Series that did not change simply contribute no rows,
and if none of the ones asked about changed the answer is an empty table
rather than an error. Like the other vector methods the two data ones
batch requests of more than 300 items. How long the list method takes is
entirely a matter of how much StatCan released that morning. It takes no
parameters, so a busy day cannot be asked about in smaller pieces, and
on one heavy enough the request has been seen to outlive StatCan’s own
gateway and come back as an HTTP 504 after some nine minutes of silence.
Its timeout defaults high to let that answer arrive as
StatCan’s own rather than as a vaguer local abort, but the limit is at
StatCan’s end and raising it further will not help;
get_cansim_changed_tables() is the question to ask on such
a day
the package now talks to StatCan through httr2
rather than httr. Requests that fail on a status StatCan
recovers from within seconds, an HTTP 429, 500, 502 or 504, are now
retried with exponential backoff and jitter and honour a
Retry-After header, where the previous retries went out
back to back and stood a good chance of arriving while the server was
still busy. Requests are also throttled to the 25 per second StatCan
documents as its per-IP limit, so a script asking for many tables or
vectors no longer risks being turned away for asking too quickly, and
they now identify themselves with a cansim/<version>
user agent. Statuses that will not improve on a retry are deliberately
not retried: an HTTP 416 carries more items than StatCan accepts however
often it is sent, an HTTP 409 is the nightly update window, and an HTTP
503 is StatCan being down for maintenance or an outage, which lasts far
longer than any retry budget worth spending. The 503 case says so, and
says to try again later, rather than appearing to hang while retries run
down. Users behind a proxy should note that httr2 reads the
standard http_proxy and https_proxy
environment variables instead of taking an
httr::set_config() call
the timeout argument now bounds how long StatCan may
go without sending anything, rather than how long the whole transfer may
take. As a cap on the total it could not tell a connection StatCan had
stopped answering on from a large table that was simply taking a while,
and cut both off alike, so a slow download could fail after two hundred
seconds with most of the data already in hand. A transfer that keeps
delivering is now left alone however long it runs, and one that goes
quiet for timeout seconds is dropped, which is the
distinction the argument was always described as making. Note that
StatCan works out a whole response before sending any of it, taking
roughly a tenth of a second per vector, so a request for a full batch of
300 is silent for something like thirty five seconds before the first
byte arrives; the default of two hundred seconds leaves ample room for
that, but a much smaller value passed by hand will cut off large
requests. Establishing the connection is bounded separately and briefly,
so an unreachable host now fails in ten seconds instead of waiting out
the full timeout
get_cansim() and
get_cansim_connection() now ask StatCan where a table lives
instead of assembling the download address from the table number. The
address the package built was a guess at a layout StatCan is free to
change, and the extra call that replaces it is small next to the table
download it precedes
get_cansim_sqlite(),
list_cansim_sqlite_cached_tables() and
remove_cansim_sqlite_cached_table() are now also documented
as deprecated, matching the deprecation warnings they already emit. Use
get_cansim_connection(..., format="sqlite"),
list_cansim_cached_tables() and
remove_cansim_cached_tables(..., format="sqlite")
insteaddisconnect_cansim_sqlite() is deprecated in favour of
the new disconnect_cansim_connection(), which does the same
thing under a name that does not claim a format. It closes a sqlite
connection and leaves parquet and feather connections alone, so a
connection can be closed without knowing which format it came from. It
was the last function still named for sqlite that was not itself
deprecated, and its own example demonstrated the deprecated
get_cansim_sqlite()get_cansim_sqlite(),
list_cansim_sqlite_cached_tables(),
remove_cansim_sqlite_cached_table(),
disconnect_cansim_sqlite(),
list_cansim_tables() and
search_cansim_tables() are scheduled for removal in a
future releaseget_cansim_vector_for_latest_periods() or
get_cansim_data_for_table_coord_periods() for all periods
no longer sends an arbitrary round number as the period count. StatCan
takes latestN as a signed 32-bit integer, rejecting zero or
less and anything past 2147483647, and quietly clamps a count longer
than the series to the whole series, so the default is now that bound as
the API itself enforces it, rather than a guess that happened to exceed
the longest series. A period count larger than the bound, or an infinite
one, is capped instead of being silently coerced to NA and
sent to StatCan as "latestN":NA, and a count below one now
fails immediately with a message instead of earning an HTTP 406. This
also applies to the per-coordinate periods column of a
table templatelanguage argument is now an error
naming what was passed, instead of an NA that travelled on
into a cache directory name or the tail of a StatCan URL and surfaced
later as a download failure or a missing column. Either language can be
named in either language, so "english", "en",
"eng" and "anglais" all select English and
"french", "fr", "fra" and
"français" all select French, along with their longer and
shorter forms; case, surrounding whitespace and accents are ignored.
get_cansim_table_url() and
get_cansim_table_notes() now default to
"english" like every other function that takes a language,
which selects the same language their previous "en" default
did (#152)if (TRUE) ... else ... in metadata
parsing. The else branch held the readr::read_delim()
implementation that utils::read.delim() replaced in
February 2025 and had since fallen behind the live branch, so it was no
longer a working fallback (#151)case_when() deprecation warning emitted by dplyr
1.2.0 on every table readget_cansim_changed_tables() passing “days” to
difftime() as a time zone instead of a unitget_cansim_changed_tables() now takes both the current
date and the cutoff after which the day’s changes are available in
Eastern time. It used to compare against 9am, half an hour after StatCan
actually closes its nightly update window, and to take “today” from the
local clock, so a machine set west of Eastern could ask StatCan about a
day that had not started there yetget_cansim_connection() no longer fails when the
release date of a table cannot be determined, the staleness check is
skipped with a message insteadget_cansim_cube_metadata() and
get_cansim_table_template() for vectors of table numbers,
metadata for all tables is still retrieved in a single API call and
cached per tableget_cansim_cube_metadata() adds a
cansimTableNumber column for the “members”, “notes” and
“corrections” typesRate, or Taux in French tables, as the
documentation always described; the comparison doing the relabelling
could never match before, so the unit columns used to keep their
original Percent... labels after the values had been
divided by 100add_cansim_vectors_to_template() now finds vectors for
coordinates whose member ids end in a zero, trimming of trailing
.0 positions used to eat into member ids like
10 and the affected rows came back with an NA
vectorcollect_and_normalize(), an internal language
comparison never matched the French settingget_cansim_connection() now falls back to the previously
cached version with a warning instead of returning NULL.
Refreshing cube metadata degrades the same way, so the notes, column,
overview and template functions keep working from previously seen
metadata when the servers are downadd_provincial_abbreviations that
could lead to mislabelling of provinces in some casesview_cansim_webpage to view
vector information from statcan in the browserget_cansim_vector_info()list_cansim_tables and
serach_cansim_tables and fallback to corresponding “_cube”
methods as Open Data Canada API has changed and similar functionality is
available through the “_cube” methods that tie directly into StatCan
APIS ## Minor changeslist_cansim_sqlite_cached_tablesadjust_cansim_values_by_variable
functionnormalize_cansim_values into the default
table and vector output, in particular always add a scaled variable
column called val_norm and an imputed Date
column and covert categories to factors by default.get_cansim_sqlite function that stores tables in an
SQLite database and facilitates access and management of data.getOption("unzip") is set to enable
unzip for files larger than 4GB on unix-like systemsget_cansim_table_notes() functionality