Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,5 @@ scripts/download/data/*
# Version
_version.py

# agent research and plans
.agents/
180 changes: 67 additions & 113 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,42 +17,87 @@ Welcome to snowexsql
.. image:: https://codecov.io/gh/SnowEx/snowexsql/graph/badge.svg?token=B27OKGBOTR
:target: https://codecov.io/gh/SnowEx/snowexsql

.. image:: https://zenodo.org/badge/doi/10.5281/zenodo.7618101.svg
:target: https://doi.org/10.5281/zenodo.7618101
:alt: DOI

About
-----
Database access and tools for using the `SnowEx database`_. This tool is
simply a client for accessing the database using python

NASA SnowEx was a multi-year airborne and field campaign aimed at understanding
the seasonal snowpack across the western United States and Alaska. Each campaign
combined airborne remote sensing (lidar, radar, hyperspectral imagery) with
intensive ground truth measurements across a variety of different snow climates.
The goal was to improve snow water equivalent (SWE) retrieval algorithms
for future spaceborne missions.

The `SnowEx database`_ consolidates measurements from all campaigns into a
single queryable PostgreSQL/PostGIS database. It holds point measurements (snow
depths, Federal Sampler SWE) and snow pit information (density, temperature,
stratigraphy from snow pits). This software is a client for accessing the
database using Python.

.. _SnowEx database: https://www.github.com/SnowEx/snowex_db

WARNING - This is under active development in preparation for SnowEx Hackweek. Use at your own risk. Data will change as it is QA/QC'd and the end goal is for all data in this database to be pulled from NSIDC. The goal is for this to become a community database open to all.

Installing
----------

Features
--------
Install using pip:

* Database access for SnowEx Database
* Analysis tools
* Useful conversions to pandas and geopandas
* Lots of examples_
.. code-block::

.. _examples: https://snowexsql.readthedocs.io/en/latest/gallery/index.html
pip install snowexsql

Full Tutorial
-------------

Setup
-----
For a complete walkthrough of accessing and querying the SnowEx database,
including spatial queries, filtering by campaign or instrument, and working
with the returned data, see the Project Pythia Snow Observations Cookbook:

Installing
==========
Install using pip:
* `SnowEx Database Tutorial`_ — step-by-step guide to using the Lambda client
and the API classes

.. code-block::
.. _SnowEx Database Tutorial: https://projectpythia.org/snow-observations-cookbook/notebooks/snowexsql-database/

Accessing the Database
----------------------

There are two ways to access the SnowEx database:

**Public access via Lambda client (no credentials required)**
The recommended approach for most users. The
:class:`~snowexsql.lambda_client.SnowExLambdaClient` connects to a public
AWS Lambda Function URL that proxies queries to the database. No AWS
account or database credentials are needed.

.. code-block:: python

from snowexsql.lambda_client import SnowExLambdaClient

client = SnowExLambdaClient()
classes = client.get_measurement_classes()
PointMeasurements = classes['PointMeasurements']

df = PointMeasurements.from_filter(type='depth', limit=100)

**Direct database access (credentials required)**
For users with database credentials, the
:mod:`snowexsql.api` classes can be used directly without going through
Lambda. This path also supports raster queries.

.. code-block:: python

from snowexsql.api import PointMeasurements, LayerMeasurements

df = LayerMeasurements.from_filter(type='density', limit=100)

pip install snowexsql

Configuring the database connection
===================================
Using this library requires setting up the database connection credentials.
There are two options to do this:
-----------------------------------
For users wishing to have direct access to the database, there are two options
for setting up the credentials:

* Set database connection URL via ``SNOWEX_DB_CONNECTION`` environment variable
Example:
Expand Down Expand Up @@ -80,107 +125,16 @@ There are two options to do this:
"password": "password"
}


Accessing the SnowEx data
-----------------

There are two ways to access SnowEx data through this library:

1. **Direct Database Access** (requires database credentials)
2. **Lambda Client** (no credentials required - serverless access, recommended)

Direct Database Access
=======================
A programmatic API has been created for fast and standard
access to Point and Layer data. There are two examples_ covering the
features and usage of the api. See the specific api_ documentation for
detailed description.

.. _api: https://snowexsql.readthedocs.io/en/latest/api.html

.. code-block:: python

from snowexsql.api import PointMeasurements, LayerMeasurements
# The main functions we will use are `from_area` and `from_filter` like this
df = PointMeasurements.from_filter(
date=date(2020, 5, 28), instrument='camera'
)
print(df.head())

Lambda Client (Serverless Access)
==================================
For users who prefer serverless access or don't want to manage database
connections, we provide an AWS Lambda-based client with a public Function URL.

**No credentials required!** The Lambda function handles all database
credentials internally via AWS Secrets Manager.

**Requirements:**

* No AWS credentials needed - public HTTP endpoint
* No database credentials needed - handled by Lambda
* requests library installed (included with snowexsql)

**Usage:**

.. code-block:: python

from snowexsql.lambda_client import SnowExLambdaClient
from datetime import date

# Initialize client - no credentials needed!
client = SnowExLambdaClient()

# Get measurement classes
classes = client.get_measurement_classes()
PointMeasurements = classes['PointMeasurements']

# Query data (same API as direct access)
df = PointMeasurements.from_filter(
date=date(2020, 5, 28), instrument='camera'
)

See the `lambda_example notebook <https://snowexsql.readthedocs.io/en/latest/gallery/lambda_example.html>`_
for complete examples.

**How It Works:**

- Public Lambda Function URL allows anyone to query the database
- Database credentials stored securely in AWS Secrets Manager (never exposed)
- Database only accepts connections from Lambda (not public internet)
- All queries go through Lambda for security and monitoring


Getting help
------------
Jump over to `our discussion forum <https://github.com/SnowEx/snowexsql/discussions>`_
and get help from our community.


Documentation
-------------

There is a whole host of resources for users in the documentation. It has been
setup for you to preview in your browser.

In there you will find:

* Examples of database use
* Database structure
* API to the python package snowexsql
* Links to other resources
* Notes about the data uploaded
* And more!

To see the documentation in your browser:

**Warning**: To see the examples/gallery, the snowex db needs to be up. Otherwise they will be left with the
last image submitted to GitHub.

.. code-block:: bash

make docs

Our read the docs pages include `documentation of the API structure <https://snowexsql.readthedocs.io/en/latest/api.html>`_
and `provides a detailed description of the database schema <https://snowexsql.readthedocs.io/en/latest/database_structure.html>`_.

I want to contribute
---------------------
Expand Down
2 changes: 1 addition & 1 deletion docs/_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,10 @@ exclude_patterns: ['_build', 'Thumbs.db', '.DS_Store']
sphinx:
extra_extensions:
- 'sphinx.ext.autodoc'
- 'sphinxcontrib.apidoc'
- 'sphinx.ext.viewcode'
- 'sphinx_gallery.load_style'
- 'sphinx.ext.autosectionlabel'
- 'sphinxcontrib.mermaid'
config:
html_js_files:
- https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.4/require.min.js
19 changes: 0 additions & 19 deletions docs/_toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,8 @@
format: jb-book
root: readme
chapters:
- file: usage
- file: community/index
- file: database_structure
- file: data_notes
- file: gallery/index
sections:
- file: gallery/what_is_in_the_db_example
- file: gallery/raster_union_and_more_example
- file: gallery/plot_raster_example
- file: gallery/plot_pit_swe_example
- file: gallery/overview_example
- file: gallery/graupel_smp_example
- file: gallery/graupel_pits_example
- file: gallery/getting_started_example
- file: gallery/get_spiral_example
- file: gallery/compare_UAVSAR_to_depths_example
- file: gallery/compare_SSA_instruments_example
- file: gallery/camera_derive_snow_depths_example
- file: gallery/api_intro_example
- file: gallery/api_plot_pit_density_example
- file: cheat_sheet
- file: qgis
- file: api
- file: history
Loading
Loading