Query Clients#

The basic query clients, with some simple embellishments.

rubin_nights.connections.get_access_token(tokenfile=None)[source]#

Retrieve RSP access token.

Parameters:

tokenfile (str | None, default: None) –

Path to the RSP token file. See documentation on RSP tokens at https://rsp.lsst.io/guides/auth/creating-user-tokens.html The token will be read from the tokenfile if available. If tokenfile is None, then further attempts will be made to access the token value from:

lsst.rsp.get_access_token
the environment variable “ACCESS_TOKEN”
the environment variable “ACCESS_TOKEN_FILE”
the home directory + ‘.lsst’ + DEFAULT_TOKENFILE

If no RSP token is available, access to most services will not be available.

Returns:

token – Token value. A zero-length token will not be valid for use.

Return type:

str

Notes

RSP access tokens are unique to the different RSP sites, and the services which run on a particular site must receive tokens from the same site.

rubin_nights.connections.get_clients(tokenfile=None, site=None, auth_token=None)[source]#

Return a wide set of site-specific client connections.

Parameters:

tokenfile (str | None, default: None) – Path to the RSP tokenfile. See also get_access_token. Can be None if one of other methods to set token will be successful.
site (str | None, default: None) – Override site location to a preferred site. Most likely to be used to specify usdf-dev vs usdf.
auth_token (str | None, default: None) – The bare authentication token string. If not None, this will override any tokenfile argument. Useful in services running behind Gafaelfawr authentication.

Returns:

endpoints – Dictionary with efd, obsenv, sasquatch, narrative_log, exposure_log, night_log, and consdb connection information.

Return type:

dict

Note

The authentication token required to access the log services is an RSP token, and is RSP site-specific (including usdf vs usdf-dev). For users outside the RSP, a token can be created as described in https://rsp.lsst.io/v/usdfprod/guides/auth/creating-user-tokens.html

rubin_nights.connections.usdf_lfa(uri, bucket='s3://lfa@')[source]#

Convert LFA uri recorded in the EFD to a version accessible at USDF.

Parameters:

uri (str) – The URI written into the EFD from the summit.
bucket (str) – The bucket access at the USDF.

Returns:

uri – The LFA uri at USDF.

Return type:

str

class rubin_nights.consdb_query.ConsDbFastAPI(api_base, auth, query_timeout=600)[source]#

Bases: ConsDb

Query the ConsDB through the REST API / FastAPI interface.

Parameters:

api_base (str) – Base API for services. e.g. https://usdf-rsp.slac.stanford.edu
auth (tuple) – The username and password for authentication.
query_timeout (float, default: 600) – Seconds to wait for the query to return data.

async async_query(query)[source]#

Execute asynchronous FastAPI ConsDB query.

Parameters:: query (str) – SQL query.
Returns:: results
Return type:: pd.DataFrame

query(query)[source]#

Execute synchronous FastAPI ConsDB query.

Parameters:: query (str) – SQL query.
Returns:: results
Return type:: pd.DataFrame

class rubin_nights.consdb_query.ConsDbSql(site='usdf', connection_string=None)[source]#

Bases: ConsDb

Query the ConsDB through pandas with a SQLAlchemy Postgres connection.

Parameters:

site (str, default: 'usdf') – Two options for site, to connect directly to the postgres servers, either ‘usdf’ or ‘summit’. Note that these postgres servers are not exposed outside of the USDF or Summit; you must use one of the other ConsDb query services in that case.
connection_string (str | None, default: None) – Optional kwarg to explicitly set the connection string instead of using the defaults for ‘usdf’ or ‘summit’.

Notes

Credentials must be available in ~/.lsst/postgres-credentials.txt

For access external to the USDF, base or summit, a different method must be used.

async async_query(query)[source]#

The ConsDbSql client uses sqlalchemy + pandas.read_sql. Async queries are unavailable.

Parameters:: query (str)
Return type:: DataFrame

query(query)[source]#

Execute a SQL query

Parameters:: query (str) – SQL query.
Returns:: results
Return type:: pd.DataFrame

class rubin_nights.consdb_query.ConsDbTap(api_base, token, query_timeout=600)[source]#

Bases: ConsDb

Query the ConsDB through the TAP service.

Parameters:

api_base (str) – Base API for services. e.g. https://usdf-rsp.slac.stanford.edu
token (str) – The token for authentication.
query_timeout (float, default: 600) – Seconds to wait for a query to complete, in remote service.

Notes

This class provides a pyvo.dal.TAPService connection to the Consdb, accessible at ConsDbTap.tap.

async async_query(query)[source]#

PyvoTapService handles async queries outside of asyncio. Use the tap attribute directly.

Parameters:: query (str)
Return type:: DataFrame

query(query)[source]#

Execute TAP ConsDB query.

Parameters:: query (str) – SQL query.
Returns:: results
Return type:: pd.DataFrame

class rubin_nights.influx_query.InfluxQueryClient(site='usdf', db_name='efd', auth=None, id_tag=None, results_as_dataframe=True, query_timeout=300)[source]#

Bases: object

Query for InfluxDB data such as EFD.

Parameters:

site (str, default: 'usdf') – The site to use for the EFD, e.g. usdf, summit, base. Note that influxdb sites can be special: e.g. currently the EFD at USDF is only on usdf-rsp (usdf, not -dev) and Sasquatch (PP metrics) is only at usdf-rsp-dev (usdf-dev).
db_name (str, default: 'efd') – The database to query. Not used for credentials info, but will be used to help guide to the correct location to fetch the credentials (usdf-int efd =>usdf efd). Default is “efd”.
auth (tuple | None, default: None) – The username and password for authentication to repertoire. Note that repertoire auth is site-specific, even though influx databases can be only available at one site (thus you may need cross-site authentication). Also only usdf-rsp currently works. If None, will fallback to segwarides.
id_tag (str | None, default: None) – Add the service name or user name as a comment to the query. This aids in tracking query sources in EFD logs. If None, will be set to username.
results_as_dataframe (bool, default: True) – If True, convert query results into a pandas DataFrame. If False, results are returned as a list of dictionaries.
query_timeout (float, default: 300) – Time (in seconds) to wait for query to return.

async async_get_fields(measurement)[source]#

Query the list of field names for a topic.

Parameters:: measurement (str) – Name of measurement/topic to query for field names.
Returns:: fields – DataFrame with fieldKey / fieldType columns.
Return type:: pd.DataFrame

async async_query(query)[source]#

Send and receive results from the InfluxDB API, with an asynchronous query.

Parameters:: query (str) – The query to send to the InfluxDb.
Returns:: result
Return type:: dict or pd.DataFrame

async async_select_time_series(topic_name, fields, t_start, t_end, index=None)[source]#

Async query to return data from topic_name between t_start and t_end.

Parameters:

topic_name (str) – The name of the topic or measurement to query.
fields (str | list[str]) – The field or fields to return from topic_name. The entry ‘*’ will return all fields.
t_start (Time) – The start of the time window for the query.
t_end (Time) – The end of the time window for the query.
index (int | None, default: None)

Returns:

query_results – The result of the query.

Return type:

pd.DataFrame or list [ dict ]

async async_select_top_n(topic_name, fields, num, time_cut=None, index=None)[source]#

Async query to return num records from topic_name.

Parameters:

topic_name (str) – The name of the topic or measurement to query.
fields (str | list[str]) – The field or fields to return from topic_name. The entry ‘*’ will return all fields.
num (int) – The number of records to return.
time_cut (Time, default: None) – If not None, looks for records prior to this time.
index (int | None, default: None)

Returns:

query_results – The result of the query.

Return type:

pd.DataFrame or list [ dict ]

static build_influxdb_query(measurement, fields='*', time_range=None, filters=None)[source]#

Build an influx DB query for fields from measurement (topic), usually over a time range.

Parameters:

measurement (str) – The name of the topic / measurement.
fields (list[str] | str, default: '*') – List of fields to return from the topic. Default * returns all fields.
time_range (tuple[Time, Time] | None, default: None) – The time window (in astropy.time.Time) to query.
filters (list[tuple[str, str]] | None, default: None) – The additional conditions to match for the query. e.g. (‘salIndex’, 1) would add salIndex=1 to the query.

Returns:

query

Return type:

str

static build_influxdb_top_n_query(measurement, fields='*', num=10, time_cut=None, filters=None)[source]#

Build an influx DB query for fields from measurement, restricted to num records.

Parameters:

measurement (str) – The name of the topic / measurement.
fields (list[str] | str, default: '*') – List of fields to return from the topic. Default * will return all fields.
num (int, default: 10) – The maximum number of records to return.
time_cut (Time | None, default: None) – Search for only records at or before this time.
filters (list[tuple[str, str]] | None, default: None) – The additional conditions to match for the query. e.g. (‘salIndex’, 1) would add salIndex=1 to the query.

Returns:

query

Return type:

str

get_fields(measurement)[source]#

Query the list of field names for a topic.

Parameters:: measurement (str) – Name of measurement/topic to query for field names.
Returns:: fields – DataFrame with fieldKey / fieldType columns.
Return type:: pd.DataFrame

get_topics()[source]#

Find all available topics.

Return type:: list[str]

query(query)[source]#

Send and receive results from the InfluxDB API, with a synchronous query.

Parameters:: query (str) – The query to send to the InfluxDb.
Returns:: result
Return type:: dict or pd.DataFrame

select_time_series(topic_name, fields, t_start, t_end, index=None)[source]#

Sync query to return data from topic_name between t_start and t_end.

Parameters:

topic_name (str) – The name of the topic or measurement to query.
fields (str | list[str]) – The field or fields to return from topic_name. The entry ‘*’ will return all fields.
t_start (Time) – The start of the time window for the query.
t_end (Time) – The end of the time window for the query.
index (int | None, default: None)

Returns:

query_results – The result of the query.

Return type:

pd.DataFrame or list [ dict ]

select_top_n(topic_name, fields, num, time_cut=None, index=None)[source]#

Sync query to return num records from topic_name.

Parameters:

topic_name (str) – The name of the topic or measurement to query.
fields (str | list[str]) – The field or fields to return from topic_name. The entry ‘*’ will return all fields.
num (int) – The number of records to return.
time_cut (Time, default: None) – If not None, looks for records prior to this time.
index (int | None, default: None)

Returns:

query_results – The result of the query.

Return type:

pd.DataFrame or list [ dict ]

rubin_nights.influx_query.day_obs_from_efd_index(x)[source]#

Use with pandas apply(efd_values, axis=1) to get dayobs.

Parameters:: x (Series)
Return type:: int

class rubin_nights.logging_query.ExposureLogClient(api_base, auth)[source]#

Bases: LoggingServiceClient

Query for the exposure log.

Parameters:

api_base (str) – Base API for services. e.g. https://usdf-rsp.slac.stanford.edu
auth (tuple) – The username and password for authentication.

async async_query_log(t_start, t_end, user_params=None)[source]#

Get exposure log message entries over a specified timespan.

Parameters:

t_start (Time) – Time of start of exposure log query.
t_end (Time) – Time of end of exposure log query.
user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

messages – Exposure log messages.

Return type:

pd.DataFrame

query_log(t_start, t_end, user_params=None)[source]#

Get exposure log message entries over a specified timespan.

Parameters:

t_start (Time) – Time of start of exposure log query.
t_end (Time) – Time of end of exposure log query.
user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

messages – Exposure log messages.

Return type:

pd.DataFrame

static query_params(t_start, t_end, user_params=None)[source]#

Set query parameters for exposure log query.

Parameters:

t_start (Time) – Time of start of exposure log query.
t_end (Time) – Time of end of exposure log query.
user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

params

Return type:

dict

class rubin_nights.logging_query.NarrativeLogClient(api_base, auth)[source]#

Bases: LoggingServiceClient

Query for the narrative log.

Parameters:

api_base (str) – Base API for services. e.g. https://usdf-rsp.slac.stanford.edu
auth (tuple) – The username and password for authentication.

async async_query_log(t_start, t_end, user_params=None)[source]#

Async get narrative log entries over a specified timespan.

Parameters:

t_start (Time) – Time of start of narrative log query.
t_end (Time) – Time of end of narrative log query.
user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

messages – Narrative log messages.

Return type:

pd.DataFrame

Notes

Some modifications are made to the raw narrative logs. Extra space is stripped out and a simple “Log <component>” key is added to the dataframe (identifying Simonyi/Auxtel specific issues). The index is replaced by a time, in order to insert the narrative log values into other events at the telescope.

query_log(t_start, t_end, user_params=None)[source]#

Get narrative log entries over a specified timespan.

Parameters:

t_start (Time) – Time of start of narrative log query.
t_end (Time) – Time of end of narrative log query.
user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

messages – Narrative log messages.

Return type:

pd.DataFrame

Notes

Some modifications are made to the raw narrative logs. Extra space is stripped out and a simple “Log <component>” key is added to the dataframe (identifying Simonyi/Auxtel specific issues). The index is replaced by a time, in order to insert the narrative log values into other events at the telescope.

static query_params(t_start, t_end, user_params=None)[source]#

Set query parameters for narrative log query.

Parameters:

t_start (Time) – Time of start of narrative log query.
t_end (Time) – Time of end of narrative log query.
user_params (dict | None, default: None) – Additional parameters to add or override defaults. Passing {'limit': int} can override the default limit.

Returns:

params

Return type:

dict

class rubin_nights.logging_query.NightReportClient(api_base, auth)[source]#

Bases: LoggingServiceClient

Query for the night report log.

Parameters:

api_base (str) – Base API for services. e.g. https://usdf-rsp.slac.stanford.edu
auth (tuple) – The username and password for authentication.

async async_query_night_report(day_obs, telescope=None, return_html=True)[source]#

Fetch the night report logs.

Parameters:

day_obs (str | int) – The day_obs of the night report. Format YYYY-MM-DD (str) or YYYYMMDD (int).
telescope (Optional[Literal['AuxTel', 'Simonyi']], default: None) – Format the night report logs for this telescope. Options: AuxTel, Simonyi or None (None will return both). The night_report now returns both telescope’s summary reports.
return_html (bool, default: True) – Send back an HTML formatted version of the first night report log, optionally for a given telescope only.

Return type:

tuple[list[dict], str]

Returns:

night_reports (list {dict}) – The night report logs, which are a list (often a single-element list, but can be multiple during the night) of dictionary key:value pairs describing the night report.
html (str (optional)) – If return_html is True, also return an HTML formatted version of the night report, potentially for a given telescope only.

query_night_report(day_obs, telescope=None, return_html=True)[source]#

Fetch the night report logs.

Parameters:

day_obs (str | int) – The day_obs of the night report. Format YYYY-MM-DD (str) or YYYYMMDD (int).
telescope (Optional[Literal['AuxTel', 'Simonyi']], default: None) – Format the night report logs for this telescope. Options: AuxTel, Simonyi or None (None will return both). The night_report now returns both telescope’s summary reports.
return_html (bool, default: True) – Send back an HTML formatted version of the first night report log, optionally for a given telescope only.

Return type:

tuple[list[dict], str]

Returns:

night_reports (list {dict}) – The night report logs, which are a list (often a single-element list, but can be multiple during the night) of dictionary key:value pairs describing the night report.
html (str (optional)) – If return_html is True, also return an HTML formatted version of the night report, potentially for a given telescope only.

static query_params(day_obs)[source]#

Set query parameters for night report query.

Parameters:: day_obs (str | int) – The day_obs for the night report.
Returns:: params
Return type:: dict