Query Clients#

The basic query clients, with some simple embellishments.

rubin_nights.connections.get_access_token(tokenfile=None)[source]#

Retrieve RSP access token.

Parameters:

tokenfile (str | None, default: None) –

Path to the RSP token file. See documentation on RSP tokens at https://rsp.lsst.io/guides/auth/creating-user-tokens.html The token will be read from the tokenfile if available. If tokenfile is None, then further attempts will be made to access the token value from:

  1. lsst.rsp.get_access_token

  2. the environment variable “ACCESS_TOKEN”

  3. the environment variable “ACCESS_TOKEN_FILE”

  4. the home directory + ‘.lsst’ + DEFAULT_TOKENFILE

If no RSP token is available, access to most services will not be available.

Returns:

token – Token value. A zero-length token will not be valid for use.

Return type:

str

Notes

RSP access tokens are unique to the different RSP sites, and the services which run on a particular site must receive tokens from the same site.

rubin_nights.connections.get_clients(tokenfile=None, site=None, auth_token=None)[source]#

Return a wide set of site-specific client connections.

Parameters:

  • tokenfile (str | None, default: None) – Path to the RSP tokenfile. See also get_access_token. Can be None if one of other methods to set token will be successful.

  • site (str | None, default: None) – Override site location to a preferred site. Most likely to be used to specify usdf-dev vs usdf. Note that this should be equivalent to the repertoire discovery site.

  • auth_token (str | None, default: None) – The bare authentication token string. If not None, this will override any tokenfile argument. Useful in services running behind Gafaelfawr authentication.

Returns:

endpoints – Dictionary with efd, obsenv, sasquatch, narrative_log, exposure_log, night_log, and consdb connection information.

Return type:

dict

Note

The authentication token required to access the log services is an RSP token, and is RSP site-specific (including usdf vs usdf-dev). For users outside the RSP, a token can be created as described in https://rsp.lsst.io/v/usdfprod/guides/auth/creating-user-tokens.html

rubin_nights.connections.usdf_lfa(uri, bucket='s3://lfa@')[source]#

Convert LFA uri recorded in the EFD to a version accessible at USDF.

Parameters:

  • uri (str) – The URI written into the EFD from the summit.

  • bucket (str) – The bucket access at the USDF.

Returns:

uri – The LFA uri at USDF.

Return type:

str

class rubin_nights.consdb_query.ConsDbFastAPI(api_base, auth, query_timeout=600)[source]#

Bases: ConsDb

Query the ConsDB through the REST API / FastAPI interface.

Parameters:

  • api_base (str) – Base API for services. e.g. https://usdf-rsp.slac.stanford.edu

  • auth (tuple) – The username and password for authentication.

  • query_timeout (float, default: 600) – Seconds to wait for the query to return data.

async async_query(query)[source]#

Execute asynchronous FastAPI ConsDB query.

Parameters:

query (str) – SQL query.

Returns:

results

Return type:

pd.DataFrame

query(query)[source]#

Execute synchronous FastAPI ConsDB query.

Parameters:

query (str) – SQL query.

Returns:

results

Return type:

pd.DataFrame

class rubin_nights.consdb_query.ConsDbSql(site='usdf', connection_string=None)[source]#

Bases: ConsDb

Query the ConsDB through pandas with a SQLAlchemy Postgres connection.

Parameters:

  • site (str, default: 'usdf') – Two options for site, to connect directly to the postgres servers, either ‘usdf’ or ‘summit’. Note that these postgres servers are not exposed outside of the USDF or Summit; you must use one of the other ConsDb query services in that case.

  • connection_string (str | None, default: None) – Optional kwarg to explicitly set the connection string instead of using the defaults for ‘usdf’ or ‘summit’.

Notes

Credentials must be available in ~/.lsst/postgres-credentials.txt

For access external to the USDF, base or summit, a different method must be used.

async async_query(query)[source]#

The ConsDbSql client uses sqlalchemy + pandas.read_sql. Async queries are unavailable.

Parameters:

query (str)

Return type:

DataFrame

query(query)[source]#

Execute a SQL query

Parameters:

query (str) – SQL query.

Returns:

results

Return type:

pd.DataFrame

class rubin_nights.consdb_query.ConsDbTap(api_base, token, query_timeout=600)[source]#

Bases: ConsDb

Query the ConsDB through the TAP service.

Parameters:

  • api_base (str) – Base API for services. e.g. https://usdf-rsp.slac.stanford.edu

  • token (str) – The token for authentication.

  • query_timeout (float, default: 600) – Seconds to wait for a query to complete, in remote service.

Notes

This class provides a pyvo.dal.TAPService connection to the Consdb, accessible at ConsDbTap.tap.

async async_query(query)[source]#

PyvoTapService handles async queries outside of asyncio. Use the tap attribute directly.

Parameters:

query (str)

Return type:

DataFrame

query(query)[source]#

Execute TAP ConsDB query.

Parameters:

query (str) – SQL query.

Returns:

results

Return type:

pd.DataFrame

class rubin_nights.influx_query.InfluxQueryClient(site='usdf', db_name='efd', repertoire_site=None, auth=None, id_tag=None, results_as_dataframe=True, query_timeout=300)[source]#

Bases: object

Query for InfluxDB data such as EFD.

Parameters:

  • site (str, default: 'usdf') – The site to use for the influx db, e.g. usdf, summit, base. Note that influxdb sites can be special: e.g. currently the EFD at USDF is only on usdf-rsp (usdf, not -dev) and the creds for the EFD at USDF are also served at summit.

  • db_name (str, default: 'efd') – The database to query. Default is “efd”.

  • repertoire_site (str | None, default: None) –

    The site to use for repertoire discovery for influx credentials.

    Does not necessarily have to be the same as ‘site’ for the influx db itself, but should match the auth token. If None, will match ‘site’.

  • auth (tuple | None, default: None) – The username and password for authentication to repertoire. Note that repertoire auth is site-specific, even though influx databases can be only available at one site (thus you may need cross-site authentication). Also only usdf-rsp currently works. If None, will fallback to segwarides.

  • id_tag (str | None, default: None) – Add the service name or user name as a comment to the query. This aids in tracking query sources in EFD logs. If None, will be set to username.

  • results_as_dataframe (bool, default: True) – If True, convert query results into a pandas DataFrame. If False, results are returned as a list of dictionaries.

  • query_timeout (float, default: 300) – Time (in seconds) to wait for query to return.

async async_get_fields(measurement)[source]#

Query the list of field names for a topic.

Parameters:

measurement (str) – Name of measurement/topic to query for field names.

Returns:

fields – DataFrame with fieldKey / fieldType columns.

Return type:

pd.DataFrame

async async_query(query)[source]#

Send and receive results from the InfluxDB API, with an asynchronous query.

Parameters:

query (str) – The query to send to the InfluxDb.

Returns:

result

Return type:

dict or pd.DataFrame

async async_select_time_series(topic_name, fields, t_start, t_end, index=None)[source]#

Async query to return data from topic_name between t_start and t_end.

Parameters:

  • topic_name (str) – The name of the topic or measurement to query.

  • fields (str | list[str]) – The field or fields to return from topic_name. The entry ‘*’ will return all fields.

  • t_start (Time) – The start of the time window for the query.

  • t_end (Time) – The end of the time window for the query.

  • index (int | None, default: None)

Returns:

query_results – The result of the query.

Return type:

pd.DataFrame or list [ dict ]

async async_select_top_n(topic_name, fields, num, time_cut=None, index=None)[source]#

Async query to return num records from topic_name.

Parameters:

  • topic_name (str) – The name of the topic or measurement to query.

  • fields (str | list[str]) – The field or fields to return from topic_name. The entry ‘*’ will return all fields.

  • num (int) – The number of records to return.

  • time_cut (Time, default: None) – If not None, looks for records prior to this time.

  • index (int | None, default: None)

Returns:

query_results – The result of the query.

Return type:

pd.DataFrame or list [ dict ]

static build_influxdb_query(measurement, fields='*', time_range=None, filters=None)[source]#

Build an influx DB query for fields from measurement (topic), usually over a time range.

Parameters:

  • measurement (str) – The name of the topic / measurement.

  • fields (list[str] | str, default: '*') – List of fields to return from the topic. Default * returns all fields.

  • time_range (tuple[Time, Time] | None, default: None) – The time window (in astropy.time.Time) to query.

  • filters (list[tuple[str, str]] | None, default: None) – The additional conditions to match for the query. e.g. (‘salIndex’, 1) would add salIndex=1 to the query.

Returns:

query

Return type:

str

static build_influxdb_top_n_query(measurement, fields='*', num=10, time_cut=None, filters=None)[source]#

Build an influx DB query for fields from measurement, restricted to num records.

Parameters:

  • measurement (str) – The name of the topic / measurement.

  • fields (list[str] | str, default: '*') – List of fields to return from the topic. Default * will return all fields.

  • num (int, default: 10) – The maximum number of records to return.

  • time_cut (Time | None, default: None) – Search for only records at or before this time.

  • filters (list[tuple[str, str]] | None, default: None) – The additional conditions to match for the query. e.g. (‘salIndex’, 1) would add salIndex=1 to the query.

Returns:

query

Return type:

str

get_fields(measurement)[source]#

Query the list of field names for a topic.

Parameters:

measurement (str) – Name of measurement/topic to query for field names.

Returns:

fields – DataFrame with fieldKey / fieldType columns.

Return type:

pd.DataFrame

get_topics()[source]#

Find all available topics.

Return type:

list[str]

query(query)[source]#

Send and receive results from the InfluxDB API, with a synchronous query.

Parameters:

query (str) – The query to send to the InfluxDb.

Returns:

result

Return type:

dict or pd.DataFrame

select_time_series(topic_name, fields, t_start, t_end, index=None)[source]#

Sync query to return data from topic_name between t_start and t_end.

Parameters:

  • topic_name (str) – The name of the topic or measurement to query.

  • fields (str | list[str]) – The field or fields to return from topic_name. The entry ‘*’ will return all fields.

  • t_start (Time) – The start of the time window for the query.

  • t_end (Time) – The end of the time window for the query.

  • index (int | None, default: None)

Returns:

query_results – The result of the query.

Return type:

pd.DataFrame or list [ dict ]

select_top_n(topic_name, fields, num, time_cut=None, index=None)[source]#

Sync query to return num records from topic_name.

Parameters:

  • topic_name (str) – The name of the topic or measurement to query.

  • fields (str | list[str]) – The field or fields to return from topic_name. The entry ‘*’ will return all fields.

  • num (int) – The number of records to return.

  • time_cut (Time, default: None) – If not None, looks for records prior to this time.

  • index (int | None, default: None)

Returns:

query_results – The result of the query.

Return type:

pd.DataFrame or list [ dict ]

rubin_nights.influx_query.convert_time_to_tz_datetime(time)[source]#

Convert astropy Time to datetime.datetime with UTC timezone.

Useful for when you want to select EFD entries based on a Time, as the EFD indexes are timezone-aware datetimes.

Parameters:

time (Time)

Return type:

datetime

rubin_nights.influx_query.day_obs_from_efd_index(x)[source]#

Use with pandas apply(efd_values, axis=1) to get dayobs.

Parameters:

x (Series)

Return type:

int

class rubin_nights.logging_query.ExposureLogClient(api_base, auth)[source]#

Bases: LoggingServiceClient

Query for the exposure log.

Parameters:
async async_query_log(t_start, t_end, user_params=None)[source]#

Get exposure log message entries over a specified timespan.

Parameters:

  • t_start (Time) – Time of start of exposure log query.

  • t_end (Time) – Time of end of exposure log query.

  • user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

messages – Exposure log messages.

Return type:

pd.DataFrame

query_log(t_start, t_end, user_params=None)[source]#

Get exposure log message entries over a specified timespan.

Parameters:

  • t_start (Time) – Time of start of exposure log query.

  • t_end (Time) – Time of end of exposure log query.

  • user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

messages – Exposure log messages.

Return type:

pd.DataFrame

static query_params(t_start, t_end, user_params=None)[source]#

Set query parameters for exposure log query.

Parameters:

  • t_start (Time) – Time of start of exposure log query. This is translated to min_day_obs for the query.

  • t_end (Time) – Time of end of exposure log query. This is translated to max_day_obs for the query.

  • user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

params

Return type:

dict

Notes

The exposure log contains the day_obs and seq_num for the exposure records, as well as date_added. It does not contain the time of the exposure itself directly. Thus t_start and t_end are translated to less specific min_day_obs_min and max_day_obs for the query.

class rubin_nights.logging_query.NarrativeLogClient(api_base, auth)[source]#

Bases: LoggingServiceClient

Query for the narrative log.

Parameters:
async async_query_log(t_start, t_end, user_params=None)[source]#

Async get narrative log entries over a specified timespan.

Parameters:

  • t_start (Time) – Time of start of narrative log query.

  • t_end (Time) – Time of end of narrative log query.

  • user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

messages – Narrative log messages.

Return type:

pd.DataFrame

Notes

Some modifications are made to the raw narrative logs. Extra space is stripped out and a simple “Log <component>” key is added to the dataframe (identifying Simonyi/Auxtel specific issues). The index is replaced by a time, in order to insert the narrative log values into other events at the telescope.

query_log(t_start, t_end, user_params=None)[source]#

Get narrative log entries over a specified timespan.

Parameters:

  • t_start (Time) – Time of start of narrative log query.

  • t_end (Time) – Time of end of narrative log query.

  • user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

messages – Narrative log messages.

Return type:

pd.DataFrame

Notes

Some modifications are made to the raw narrative logs. Extra space is stripped out and a simple “Log <component>” key is added to the dataframe (identifying Simonyi/Auxtel specific issues). The index is replaced by a time, in order to insert the narrative log values into other events at the telescope.

static query_params(t_start, t_end, user_params=None)[source]#

Set query parameters for narrative log query.

Parameters:

  • t_start (Time) – Time of start of narrative log query.

  • t_end (Time) – Time of end of narrative log query.

  • user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

params

Return type:

dict

class rubin_nights.logging_query.NightReportClient(api_base, auth)[source]#

Bases: LoggingServiceClient

Query for the night report log.

Parameters:
async async_query_night_report(day_obs, telescope=None, return_html=True)[source]#

Fetch the night report logs.

Parameters:

  • day_obs (str | int) – The day_obs of the night report. Format YYYY-MM-DD (str) or YYYYMMDD (int).

  • telescope (Optional[Literal['AuxTel', 'Simonyi']], default: None) – Format the night report logs for this telescope. Options: AuxTel, Simonyi or None (None will return both). The night_report now returns both telescope’s summary reports.

  • return_html (bool, default: True) – Send back an HTML formatted version of the first night report log, optionally for a given telescope only.

Return type:

tuple[list[dict], str]

Returns:

  • night_reports (list {dict}) – The night report logs, which are a list (often a single-element list, but can be multiple during the night) of dictionary key:value pairs describing the night report.

  • html (str (optional)) – If return_html is True, also return an HTML formatted version of the night report, potentially for a given telescope only.

query_night_report(day_obs, telescope=None, return_html=True)[source]#

Fetch the night report logs.

Parameters:

  • day_obs (str | int) – The day_obs of the night report. Format YYYY-MM-DD (str) or YYYYMMDD (int).

  • telescope (Optional[Literal['AuxTel', 'Simonyi']], default: None) – Format the night report logs for this telescope. Options: AuxTel, Simonyi or None (None will return both). The night_report now returns both telescope’s summary reports.

  • return_html (bool, default: True) – Send back an HTML formatted version of the first night report log, optionally for a given telescope only.

Return type:

tuple[list[dict], str]

Returns:

  • night_reports (list {dict}) – The night report logs, which are a list (often a single-element list, but can be multiple during the night) of dictionary key:value pairs describing the night report.

  • html (str (optional)) – If return_html is True, also return an HTML formatted version of the night report, potentially for a given telescope only.

static query_params(day_obs)[source]#

Set query parameters for night report query.

Parameters:

day_obs (str | int) – The day_obs for the night report.

Returns:

params

Return type:

dict